2024-06-10T00:04:53.543 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-10T00:04:53.543 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_teuthology@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/blogbench}} --archive /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320 --name teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320/orig.config.yaml 2024-06-10T00:04:53.626 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-10T00:04:53.711 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/blogbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7747320' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 7b7a5d375cf4e4afab96b9d4d229b699ae8304bb owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 178 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 130/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7b7a5d375cf4e4afab96b9d4d229b699ae8304bb syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcMKBZQ482S79Z/Xq9tqLcnxnOm4JiXmcmc/ORsLrBGGYkcs5K9nxA6SEhdv77qiktHKYAXY19nfHe4NTXKZzk= smithi135.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHCgpQhc8/9UqCYJiuVmAC7iHXDp115YUzYIrYubZgmfN3OwX1Yz8H2WEjx/vpeNGIU1TjmZVAaqO/VaB8wX718= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDaFgYiy6oxY3Wt7H/Gp/li5i2r4gkE2iaDuJ/C1ASESo+01syLJy4ORfvHZoePAfWr4gxI3Beb/5zZun/hZZg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - quiescer: null - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-09_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-10T00:04:53.711 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa; will attempt to use it 2024-06-10T00:04:53.712 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks 2024-06-10T00:04:53.712 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-10T00:04:53.713 INFO:teuthology.task.internal:Checking packages... 2024-06-10T00:04:53.734 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-10T00:04:53.734 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-10T00:04:53.734 INFO:teuthology.packaging:ref: None 2024-06-10T00:04:53.734 INFO:teuthology.packaging:tag: None 2024-06-10T00:04:53.734 INFO:teuthology.packaging:branch: squid 2024-06-10T00:04:53.734 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-10T00:04:53.734 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-10T00:04:53.891 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2970.g7b7a5d37 2024-06-10T00:04:53.892 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-10T00:04:53.951 INFO:teuthology.task.internal:no buildpackages task found 2024-06-10T00:04:53.951 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-10T00:04:53.959 INFO:teuthology.task.internal:Saving configuration 2024-06-10T00:04:53.979 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-10T00:04:53.989 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-10T00:04:54.011 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-09 23:59:53.794324', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOcMKBZQ482S79Z/Xq9tqLcnxnOm4JiXmcmc/ORsLrBGGYkcs5K9nxA6SEhdv77qiktHKYAXY19nfHe4NTXKZzk='} 2024-06-10T00:04:54.036 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-09 23:59:53.796271', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHCgpQhc8/9UqCYJiuVmAC7iHXDp115YUzYIrYubZgmfN3OwX1Yz8H2WEjx/vpeNGIU1TjmZVAaqO/VaB8wX718='} 2024-06-10T00:04:54.059 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-09 23:59:53.797967', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCDaFgYiy6oxY3Wt7H/Gp/li5i2r4gkE2iaDuJ/C1ASESo+01syLJy4ORfvHZoePAfWr4gxI3Beb/5zZun/hZZg='} 2024-06-10T00:04:54.059 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-10T00:04:54.101 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-06-10T00:04:54.101 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-06-10T00:04:54.101 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-06-10T00:04:54.101 INFO:teuthology.run_tasks:Running task console_log... 2024-06-10T00:04:54.195 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbd7a54bbe0>, signals=[15]) 2024-06-10T00:04:54.195 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-10T00:04:54.217 INFO:teuthology.task.internal:Opening connections... 2024-06-10T00:04:54.217 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-06-10T00:04:54.218 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-10T00:04:54.293 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi135.front.sepia.ceph.com 2024-06-10T00:04:54.293 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-10T00:04:54.367 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-06-10T00:04:54.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-10T00:04:54.442 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-10T00:04:54.599 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-06-10T00:04:54.617 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-06-10T00:04:54.618 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-06-10T00:04:54.674 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-06-10T00:04:54.674 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-06-10T00:04:54.674 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-06-10T00:04:54.674 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-06-10T00:04:54.675 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-06-10T00:04:54.675 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-06-10T00:04:54.675 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-10T00:04:54.675 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-06-10T00:04:54.675 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-06-10T00:04:54.675 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-10T00:04:54.675 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-06-10T00:04:54.675 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-10T00:04:54.675 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-10T00:04:54.675 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-10T00:04:54.676 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-06-10T00:04:54.705 DEBUG:teuthology.orchestra.run.smithi135:> uname -m 2024-06-10T00:04:54.722 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2024-06-10T00:04:54.723 DEBUG:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2024-06-10T00:04:54.779 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2024-06-10T00:04:54.779 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="9" 2024-06-10T00:04:54.779 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2024-06-10T00:04:54.779 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2024-06-10T00:04:54.779 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="9" 2024-06-10T00:04:54.780 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el9" 2024-06-10T00:04:54.780 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-10T00:04:54.780 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2024-06-10T00:04:54.780 INFO:teuthology.orchestra.run.smithi135.stdout:LOGO="fedora-logo-icon" 2024-06-10T00:04:54.780 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-10T00:04:54.780 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2024-06-10T00:04:54.780 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-10T00:04:54.780 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-10T00:04:54.780 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-10T00:04:54.780 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2024-06-10T00:04:54.822 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-06-10T00:04:54.839 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-06-10T00:04:54.839 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-06-10T00:04:54.896 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-06-10T00:04:54.896 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-06-10T00:04:54.896 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-06-10T00:04:54.896 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-06-10T00:04:54.896 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-06-10T00:04:54.896 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-06-10T00:04:54.896 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-10T00:04:54.896 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-06-10T00:04:54.896 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-06-10T00:04:54.896 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-10T00:04:54.896 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-06-10T00:04:54.897 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-10T00:04:54.897 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-10T00:04:54.897 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-10T00:04:54.897 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-06-10T00:04:54.922 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-10T00:04:55.089 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-10T00:04:55.142 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-10T00:04:55.142 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-06-10T00:04:55.144 DEBUG:teuthology.orchestra.run.smithi135:> test '!' -e /home/ubuntu/cephtest 2024-06-10T00:04:55.147 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-06-10T00:04:55.163 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-10T00:04:55.170 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-10T00:04:55.171 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-06-10T00:04:55.201 DEBUG:teuthology.orchestra.run.smithi135:> test -z $(ls -A /var/lib/ceph) 2024-06-10T00:04:55.204 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-06-10T00:04:55.333 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-10T00:04:55.483 INFO:teuthology.run_tasks:Running task kernel... 2024-06-10T00:04:55.520 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-06-10T00:04:55.520 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-06-10T00:04:55.520 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-06-10T00:04:55.521 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-10T00:04:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:04:55.537 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-06-10T00:04:55.594 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-437.el9.x86_64 2024-06-10T00:04:55.595 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-437.el9.x86_64 2024-06-10T00:04:55.595 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-06-10T00:04:58.959 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.1 MB 00:01 2024-06-10T00:05:06.566 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 19 MB 00:06 2024-06-10T00:05:11.691 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.2 MB 00:01 2024-06-10T00:05:13.407 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 17 kB 00:00 2024-06-10T00:05:14.277 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-06-10T00:05:19.465 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-06-10T00:05:21.455 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-10T00:05:21.455 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-10T00:05:21.554 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-06-10T00:05:21.566 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-06-10T00:05:21.567 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2024-06-10T00:05:21.568 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:05:21.568 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 80 M 2024-06-10T00:05:21.568 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 124 M 2024-06-10T00:05:21.568 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-06-10T00:05:21.921 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:05:21.926 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:05:21.926 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:05:22.874 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 866 kB/s | 893 kB 00:01 2024-06-10T00:05:23.458 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 9.8 MB/s | 16 MB 00:01 2024-06-10T00:05:24.133 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 15 MB/s | 35 MB 00:02 2024-06-10T00:05:25.641 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): kernel-modules-core-5.14.0-452.el9.x86_6 10 MB/s | 29 MB 00:02 2024-06-10T00:05:25.644 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-06-10T00:05:25.645 INFO:teuthology.orchestra.run.smithi121.stdout:Total 20 MB/s | 80 MB 00:04 2024-06-10T00:05:26.346 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-06-10T00:05:26.676 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-06-10T00:05:26.676 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-06-10T00:05:27.636 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-06-10T00:05:27.636 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-06-10T00:05:30.032 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-06-10T00:05:30.687 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-10T00:05:30.767 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-10T00:05:33.394 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-10T00:05:33.546 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-10T00:05:38.259 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-10T00:05:38.367 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-10T00:05:43.123 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-10T00:06:12.327 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-10T00:06:12.368 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-10T00:06:12.718 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-10T00:06:12.718 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-10T00:06:12.719 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-10T00:06:12.719 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-10T00:06:13.529 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-10T00:06:13.529 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:06:13.529 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-06-10T00:06:13.529 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-10T00:06:13.529 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-10T00:06:13.529 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-10T00:06:13.530 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-10T00:06:13.530 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:06:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-06-10T00:06:13.884 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-06-10T00:06:13.949 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-10T00:06:13.950 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-10T00:06:13.950 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-10T00:06:13.950 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-10T00:06:13.950 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-10T00:06:13.950 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-06-10T00:06:13.950 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-06-10T00:06:13.950 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-06-10T00:06:14.003 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-10T00:06:14.003 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-10T00:06:14.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-06-10T00:06:14.033 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-06-10T00:06:14.033 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-06-10T00:06:14.033 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.06 2024-06-10T00:06:14.033 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 68.el9 2024-06-10T00:06:14.033 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:Group : Unspecified 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 8264254 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : builder@centos.org 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-10T00:06:14.034 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-06-10T00:06:14.035 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-06-10T00:06:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:06:14.035 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-10T00:06:14.035 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-10T00:06:14.035 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-10T00:06:14.035 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-06-10T00:06:14.035 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:06:14.035 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-06-10T00:06:14.036 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-10T00:06:14.036 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 5.14.0-452.el9.x86_64 2024-06-10T00:06:14.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-10T00:06:14.703 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-06-10T00:06:16.106 INFO:teuthology.orchestra.run.smithi121.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-10T00:06:16.135 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-06-10T00:06:16.137 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-06-10T00:06:16.157 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.ThHixRQK6T 2024-06-10T00:06:16.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ThHixRQK6T 2024-06-10T00:06:16.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.ThHixRQK6T 2024-06-10T00:06:16.373 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.ThHixRQK6T is 6KB 2024-06-10T00:06:16.403 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.ThHixRQK6T 2024-06-10T00:06:16.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-06-10T00:06:16.482 INFO:teuthology.orchestra.run.smithi121.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-10T00:06:16.482 INFO:teuthology.orchestra.run.smithi121.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-10T00:06:16.482 INFO:teuthology.orchestra.run.smithi121.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-10T00:06:16.483 INFO:teuthology.orchestra.run.smithi121.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-10T00:06:16.484 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-10T00:06:16.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-06-10T00:06:46.637 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-06-10T00:06:46.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-06-10T00:06:46.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-10T00:07:46.642 DEBUG:teuthology.orchestra.remote:timed out 2024-06-10T00:07:55.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-06-10T00:07:55.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-10T00:07:56.281 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-06-10T00:07:56.716 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-06-10T00:07:56.716 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-452.el9.x86_64"... 2024-06-10T00:07:56.716 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-06-10T00:07:56.731 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-452.el9.x86_64 2024-06-10T00:07:56.731 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-10T00:07:56.731 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-10T00:07:56.731 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-10T00:07:57.732 DEBUG:teuthology.parallel:result is None 2024-06-10T00:07:57.733 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-10T00:07:57.739 INFO:teuthology.task.internal:Creating test directory... 2024-06-10T00:07:57.740 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-10T00:07:57.743 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-10T00:07:57.745 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-10T00:07:57.763 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-10T00:07:57.769 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-10T00:07:57.778 INFO:teuthology.task.internal:Creating archive directory... 2024-06-10T00:07:57.778 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-10T00:07:57.799 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-10T00:07:57.803 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-10T00:07:58.125 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-10T00:07:58.131 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-10T00:07:58.131 DEBUG:teuthology.orchestra.run.smithi135:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-10T00:07:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:07:58.148 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-10T00:07:58.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:07:58.165 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-10T00:07:58.168 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-10T00:07:58.191 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-10T00:07:58.222 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-10T00:07:58.244 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-10T00:07:58.260 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-10T00:07:58.263 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-10T00:07:58.263 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-10T00:07:58.337 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-10T00:07:58.339 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-10T00:07:58.347 INFO:teuthology.task.internal:Configuring sudo... 2024-06-10T00:07:58.347 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-10T00:07:58.349 DEBUG:teuthology.orchestra.run.smithi135:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-10T00:07:58.352 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-10T00:07:58.379 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-10T00:07:58.389 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-10T00:07:58.389 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-10T00:07:58.414 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-10T00:07:58.420 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-10T00:07:58.436 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-10T00:07:58.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-10T00:07:58.554 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-10T00:07:58.635 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-10T00:07:58.679 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:07:58.679 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-10T00:07:58.771 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-10T00:07:58.817 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-10T00:07:58.880 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-10T00:07:58.925 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-10T00:07:58.965 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-06-10T00:07:58.966 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-10T00:07:59.031 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-10T00:07:59.078 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-10T00:07:59.130 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-10T00:07:59.210 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-10T00:07:59.256 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-06-10T00:07:59.256 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-10T00:07:59.320 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-06-10T00:07:59.322 DEBUG:teuthology.orchestra.run.smithi135:> sudo service rsyslog restart 2024-06-10T00:07:59.325 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-06-10T00:07:59.362 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-10T00:07:59.363 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-10T00:07:59.393 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-10T00:07:59.919 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-10T00:07:59.925 INFO:teuthology.task.internal:Starting timer... 2024-06-10T00:07:59.926 INFO:teuthology.run_tasks:Running task pcp... 2024-06-10T00:07:59.935 INFO:teuthology.run_tasks:Running task selinux... 2024-06-10T00:07:59.944 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-10T00:07:59.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-06-10T00:08:00.015 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-06-10T00:08:00.017 DEBUG:teuthology.orchestra.run.smithi135:> sudo service auditd rotate 2024-06-10T00:08:00.090 INFO:teuthology.orchestra.run.smithi135.stdout:Rotating logs: 2024-06-10T00:08:00.092 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-06-10T00:08:00.157 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-06-10T00:08:00.159 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-10T00:08:00.159 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-06-10T00:08:00.177 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-06-10T00:08:00.177 DEBUG:teuthology.orchestra.run.smithi135:> /usr/sbin/getenforce 2024-06-10T00:08:00.201 INFO:teuthology.orchestra.run.smithi135.stdout:Permissive 2024-06-10T00:08:00.202 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-06-10T00:08:00.221 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-06-10T00:08:00.222 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi135.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-06-10T00:08:00.222 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-10T00:08:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:08:00.247 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-10T00:08:00.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:08:00.274 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-10T00:08:00.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:08:00.301 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-10T00:08:00.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-06-10T00:08:00.338 DEBUG:teuthology.orchestra.run.smithi135:> sudo /usr/sbin/setenforce permissive 2024-06-10T00:08:00.379 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-06-10T00:08:00.407 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-10T00:08:00.416 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-10T00:08:00.441 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-10T00:08:00.442 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi121.front.sepia.ceph.com,smithi135.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-10T00:12:40.535 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi135.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-06-10T00:12:40.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-06-10T00:12:40.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-10T00:12:40.616 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-06-10T00:12:40.698 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-06-10T00:12:40.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-06-10T00:12:40.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-10T00:12:40.779 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-06-10T00:12:40.853 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-06-10T00:12:40.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-06-10T00:12:40.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-10T00:12:40.931 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-06-10T00:12:40.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-06-10T00:12:40.970 INFO:teuthology.run_tasks:Running task clock... 2024-06-10T00:12:40.985 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-10T00:12:40.985 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-10T00:12:40.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-10T00:12:40.989 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-10T00:12:40.989 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-10T00:12:40.991 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-10T00:12:40.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-10T00:12:41.016 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-10T00:12:41.017 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-10T00:12:41.030 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-10T00:12:41.030 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-10T00:12:41.037 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-10T00:12:41.050 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-10T00:12:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-06-10T00:12:41.054 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: ntpd: command not found 2024-06-10T00:12:41.064 INFO:teuthology.orchestra.run.smithi135.stdout:506 Cannot talk to daemon 2024-06-10T00:12:41.064 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-06-10T00:12:41.072 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-06-10T00:12:41.078 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-10T00:12:41.078 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-10T00:12:41.082 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-06-10T00:12:41.090 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-10T00:12:41.092 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-10T00:12:41.096 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-10T00:12:41.108 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-10T00:12:41.146 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-06-10T00:12:41.149 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-10T00:12:41.149 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-06-10T00:12:41.149 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.149 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.150 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.150 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.151 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-06-10T00:12:41.154 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-10T00:12:41.154 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-06-10T00:12:41.154 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.155 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.155 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.155 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.163 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-06-10T00:12:41.167 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-10T00:12:41.167 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-06-10T00:12:41.167 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.167 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.167 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.167 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:12:41.168 INFO:teuthology.run_tasks:Running task install... 2024-06-10T00:12:41.180 DEBUG:teuthology.task.install:project ceph 2024-06-10T00:12:41.180 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}} 2024-06-10T00:12:41.180 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-10T00:12:41.180 INFO:teuthology.task.install:Using flavor: default 2024-06-10T00:12:41.186 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-10T00:12:41.186 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-06-10T00:12:41.186 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-10T00:12:41.187 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-10T00:12:41.188 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-10T00:12:41.189 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-10T00:12:41.190 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-10T00:12:41.190 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-10T00:12:41.407 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/ 2024-06-10T00:12:41.407 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2946.g3d89accf 2024-06-10T00:12:41.415 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/ 2024-06-10T00:12:41.416 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2946.g3d89accf 2024-06-10T00:12:41.420 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/ 2024-06-10T00:12:41.420 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2946.g3d89accf 2024-06-10T00:12:41.525 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-10T00:12:41.525 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:12:41.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-10T00:12:41.552 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-10T00:12:41.552 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d89accfe3201286f000e065a1faea2a88c7a6ba/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-10T00:12:41.592 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-10T00:12:41.592 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-06-10T00:12:41.593 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-10T00:12:41.606 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-10T00:12:41.606 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-06-10T00:12:41.606 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-10T00:12:41.617 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-10T00:12:41.618 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d89accfe3201286f000e065a1faea2a88c7a6ba/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-10T00:12:41.619 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-10T00:12:41.631 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-10T00:12:41.631 DEBUG:teuthology.orchestra.run.smithi135:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d89accfe3201286f000e065a1faea2a88c7a6ba/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-10T00:12:41.682 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-10T00:12:41.694 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-10T00:12:41.696 DEBUG:teuthology.orchestra.run.smithi135:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-10T00:12:41.754 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-10T00:12:41.760 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-06-10T00:12:41.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-06-10T00:12:41.769 DEBUG:teuthology.orchestra.run.smithi135:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-10T00:12:41.817 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-06-10T00:12:41.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-06-10T00:12:41.832 INFO:teuthology.orchestra.run.smithi135.stdout:check_obsoletes = 1 2024-06-10T00:12:41.834 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum clean all 2024-06-10T00:12:42.000 INFO:teuthology.orchestra.run.smithi121.stdout:45 files removed 2024-06-10T00:12:42.029 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-10T00:12:42.041 INFO:teuthology.orchestra.run.smithi193.stdout:45 files removed 2024-06-10T00:12:42.055 INFO:teuthology.orchestra.run.smithi135.stdout:45 files removed 2024-06-10T00:12:42.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-10T00:12:42.082 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-10T00:12:42.782 INFO:teuthology.orchestra.run.smithi135.stdout:ceph packages for x86_64 188 kB/s | 83 kB 00:00 2024-06-10T00:12:42.800 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 167 kB/s | 83 kB 00:00 2024-06-10T00:12:42.823 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 169 kB/s | 83 kB 00:00 2024-06-10T00:12:43.192 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 34 kB/s | 11 kB 00:00 2024-06-10T00:12:43.217 INFO:teuthology.orchestra.run.smithi135.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-06-10T00:12:43.273 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-06-10T00:12:43.601 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-06-10T00:12:43.609 INFO:teuthology.orchestra.run.smithi135.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-06-10T00:12:43.666 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-06-10T00:12:44.253 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.1 MB 00:00 2024-06-10T00:12:45.214 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.1 MB 00:01 2024-06-10T00:12:45.412 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.1 MB 00:01 2024-06-10T00:12:46.213 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-06-10T00:12:47.055 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-06-10T00:12:50.531 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.2 MB 00:00 2024-06-10T00:12:51.488 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.2 MB 00:00 2024-06-10T00:12:52.173 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 17 kB 00:00 2024-06-10T00:12:53.046 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 17 kB 00:00 2024-06-10T00:12:53.126 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-06-10T00:12:54.341 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-06-10T00:12:58.299 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-06-10T00:12:59.605 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-06-10T00:13:00.626 INFO:teuthology.orchestra.run.smithi135.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-10T00:13:00.626 INFO:teuthology.orchestra.run.smithi135.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-10T00:13:00.637 INFO:teuthology.orchestra.run.smithi135.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-10T00:13:00.638 INFO:teuthology.orchestra.run.smithi135.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-10T00:13:00.639 INFO:teuthology.orchestra.run.smithi135.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-10T00:13:00.640 INFO:teuthology.orchestra.run.smithi135.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-10T00:13:00.641 INFO:teuthology.orchestra.run.smithi135.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-10T00:13:00.644 INFO:teuthology.orchestra.run.smithi135.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-10T00:13:00.644 INFO:teuthology.orchestra.run.smithi135.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-10T00:13:00.707 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-06-10T00:13:00.724 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================= 2024-06-10T00:13:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-06-10T00:13:00.724 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================= 2024-06-10T00:13:00.724 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-06-10T00:13:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-10T00:13:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: ceph x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.6 k 2024-06-10T00:13:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-10T00:13:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 890 k 2024-06-10T00:13:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 145 k 2024-06-10T00:13:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-10T00:13:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 142 k 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 3.5 M 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 7.4 M 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 49 k 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 10 M 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 48 M 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 264 k 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 763 k 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 221 k 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-shell x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 27 k 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-top noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 18 k 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-10T00:13:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-10T00:13:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-10T00:13:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 32 k 2024-06-10T00:13:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 747 k 2024-06-10T00:13:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 127 k 2024-06-10T00:13:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-06-10T00:13:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-10T00:13:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-06-10T00:13:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-10T00:13:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-10T00:13:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-10T00:13:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 162 k 2024-06-10T00:13:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 321 k 2024-06-10T00:13:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 300 k 2024-06-10T00:13:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 k 2024-06-10T00:13:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 85 k 2024-06-10T00:13:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-10T00:13:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 171 k 2024-06-10T00:13:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-06-10T00:13:00.727 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrading: 2024-06-10T00:13:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager x86_64 1:1.47.91-1.el9 baseos 2.3 M 2024-06-10T00:13:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-libnm x86_64 1:1.47.91-1.el9 baseos 1.8 M 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-team x86_64 1:1.47.91-1.el9 baseos 41 k 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-tui x86_64 1:1.47.91-1.el9 baseos 248 k 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: librados2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.4 M 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-10T00:13:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-10T00:13:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-10T00:13:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 21 M 2024-06-10T00:13:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 32 k 2024-06-10T00:13:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 2.4 M 2024-06-10T00:13:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 249 k 2024-06-10T00:13:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 4.7 M 2024-06-10T00:13:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 17 M 2024-06-10T00:13:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 16 k 2024-06-10T00:13:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 25 k 2024-06-10T00:13:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-10T00:13:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-10T00:13:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-10T00:13:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-10T00:13:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-10T00:13:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-10T00:13:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-10T00:13:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-10T00:13:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 163 k 2024-06-10T00:13:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-10T00:13:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-10T00:13:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-10T00:13:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-10T00:13:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-10T00:13:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 501 k 2024-06-10T00:13:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-10T00:13:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.3 M 2024-06-10T00:13:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-10T00:13:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 135 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-10T00:13:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-10T00:13:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-10T00:13:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-10T00:13:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-10T00:13:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout:Installing weak dependencies: 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================= 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout:Install 148 Packages 2024-06-10T00:13:00.738 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrade 11 Packages 2024-06-10T00:13:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:00.739 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 225 M 2024-06-10T00:13:00.739 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-06-10T00:13:01.426 INFO:teuthology.orchestra.run.smithi135.stdout:(1/159): ceph-19.0.0-2946.g3d89accf.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-06-10T00:13:01.734 INFO:teuthology.orchestra.run.smithi135.stdout:(2/159): ceph-fuse-19.0.0-2946.g3d89accf.el9.x8 2.8 MB/s | 890 kB 00:00 2024-06-10T00:13:01.802 INFO:teuthology.orchestra.run.smithi135.stdout:(3/159): ceph-immutable-object-cache-19.0.0-294 2.1 MB/s | 145 kB 00:00 2024-06-10T00:13:01.881 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-10T00:13:01.882 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-10T00:13:01.892 INFO:teuthology.orchestra.run.smithi121.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-10T00:13:01.893 INFO:teuthology.orchestra.run.smithi121.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-10T00:13:01.894 INFO:teuthology.orchestra.run.smithi121.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-10T00:13:01.895 INFO:teuthology.orchestra.run.smithi121.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-10T00:13:01.896 INFO:teuthology.orchestra.run.smithi121.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-10T00:13:01.899 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-10T00:13:01.899 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-10T00:13:01.961 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-06-10T00:13:01.977 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.6 k 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 890 k 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 145 k 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 142 k 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 3.5 M 2024-06-10T00:13:01.978 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 7.4 M 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 49 k 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 10 M 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 48 M 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 264 k 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 763 k 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 221 k 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-shell x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 27 k 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 18 k 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 32 k 2024-06-10T00:13:01.979 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 747 k 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 127 k 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 162 k 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 321 k 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 300 k 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 k 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 85 k 2024-06-10T00:13:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 171 k 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager x86_64 1:1.47.91-1.el9 baseos 2.3 M 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-libnm x86_64 1:1.47.91-1.el9 baseos 1.8 M 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-team x86_64 1:1.47.91-1.el9 baseos 41 k 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-tui x86_64 1:1.47.91-1.el9 baseos 248 k 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.4 M 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-10T00:13:01.981 INFO:teuthology.orchestra.run.smithi121.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 21 M 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 32 k 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 2.4 M 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 249 k 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 4.7 M 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 17 M 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 16 k 2024-06-10T00:13:01.982 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 25 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 163 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-10T00:13:01.983 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 501 k 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.3 M 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-10T00:13:01.984 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 k 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 135 k 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-10T00:13:01.985 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-10T00:13:01.986 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-10T00:13:01.987 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-10T00:13:01.988 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-10T00:13:01.989 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-06-10T00:13:01.990 INFO:teuthology.orchestra.run.smithi121.stdout:Install 148 Packages 2024-06-10T00:13:01.991 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 11 Packages 2024-06-10T00:13:01.991 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:13:01.991 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 225 M 2024-06-10T00:13:01.991 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-06-10T00:13:02.160 INFO:teuthology.orchestra.run.smithi135.stdout:(4/159): ceph-mds-19.0.0-2946.g3d89accf.el9.x86 6.6 MB/s | 2.4 MB 00:00 2024-06-10T00:13:02.277 INFO:teuthology.orchestra.run.smithi135.stdout:(5/159): ceph-base-19.0.0-2946.g3d89accf.el9.x8 5.2 MB/s | 5.4 MB 00:01 2024-06-10T00:13:02.419 INFO:teuthology.orchestra.run.smithi135.stdout:(6/159): ceph-mgr-19.0.0-2946.g3d89accf.el9.x86 5.8 MB/s | 1.5 MB 00:00 2024-06-10T00:13:02.720 INFO:teuthology.orchestra.run.smithi135.stdout:(7/159): ceph-mon-19.0.0-2946.g3d89accf.el9.x86 11 MB/s | 4.7 MB 00:00 2024-06-10T00:13:03.356 INFO:teuthology.orchestra.run.smithi121.stdout:(1/159): ceph-19.0.0-2946.g3d89accf.el9.x86_64. 11 kB/s | 6.6 kB 00:00 2024-06-10T00:13:03.650 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:17 2024-06-10T00:13:03.653 INFO:teuthology.orchestra.run.smithi135.stdout:(8/159): ceph-radosgw-19.0.0-2946.g3d89accf.el9 11 MB/s | 10 MB 00:00 2024-06-10T00:13:03.729 INFO:teuthology.orchestra.run.smithi135.stdout:(9/159): ceph-selinux-19.0.0-2946.g3d89accf.el9 335 kB/s | 25 kB 00:00 2024-06-10T00:13:03.981 INFO:teuthology.orchestra.run.smithi121.stdout:(2/159): ceph-fuse-19.0.0-2946.g3d89accf.el9.x8 1.4 MB/s | 890 kB 00:00 2024-06-10T00:13:04.048 INFO:teuthology.orchestra.run.smithi121.stdout:(3/159): ceph-immutable-object-cache-19.0.0-294 2.1 MB/s | 145 kB 00:00 2024-06-10T00:13:04.054 INFO:teuthology.orchestra.run.smithi135.stdout:(10/159): ceph-osd-19.0.0-2946.g3d89accf.el9.x8 10 MB/s | 17 MB 00:01 2024-06-10T00:13:04.197 INFO:teuthology.orchestra.run.smithi135.stdout:(11/159): cephfs-mirror-19.0.0-2946.g3d89accf.e 1.5 MB/s | 221 kB 00:00 2024-06-10T00:13:04.271 INFO:teuthology.orchestra.run.smithi135.stdout:(12/159): cephfs-shell-19.0.0-2946.g3d89accf.el 360 kB/s | 27 kB 00:00 2024-06-10T00:13:04.347 INFO:teuthology.orchestra.run.smithi135.stdout:(13/159): libcephfs-devel-19.0.0-2946.g3d89accf 423 kB/s | 32 kB 00:00 2024-06-10T00:13:04.747 INFO:teuthology.orchestra.run.smithi135.stdout:(14/159): libcephfs2-19.0.0-2946.g3d89accf.el9. 1.8 MB/s | 747 kB 00:00 2024-06-10T00:13:04.848 INFO:teuthology.orchestra.run.smithi135.stdout:(15/159): libcephsqlite-19.0.0-2946.g3d89accf.e 1.6 MB/s | 163 kB 00:00 2024-06-10T00:13:04.882 INFO:teuthology.orchestra.run.smithi121.stdout:(4/159): ceph-mds-19.0.0-2946.g3d89accf.el9.x86 2.8 MB/s | 2.4 MB 00:00 2024-06-10T00:13:04.948 INFO:teuthology.orchestra.run.smithi135.stdout:(16/159): librados-devel-19.0.0-2946.g3d89accf. 1.2 MB/s | 127 kB 00:00 2024-06-10T00:13:05.190 INFO:teuthology.orchestra.run.smithi135.stdout:(17/159): libradosstriper1-19.0.0-2946.g3d89acc 2.0 MB/s | 501 kB 00:00 2024-06-10T00:13:05.332 INFO:teuthology.orchestra.run.smithi121.stdout:(5/159): ceph-mgr-19.0.0-2946.g3d89accf.el9.x86 3.3 MB/s | 1.5 MB 00:00 2024-06-10T00:13:06.332 INFO:teuthology.orchestra.run.smithi121.stdout:(6/159): ceph-mon-19.0.0-2946.g3d89accf.el9.x86 4.7 MB/s | 4.7 MB 00:00 2024-06-10T00:13:06.424 INFO:teuthology.orchestra.run.smithi135.stdout:(18/159): ceph-common-19.0.0-2946.g3d89accf.el9 4.1 MB/s | 21 MB 00:05 2024-06-10T00:13:06.516 INFO:teuthology.orchestra.run.smithi135.stdout:(19/159): python3-ceph-argparse-19.0.0-2946.g3d 492 kB/s | 45 kB 00:00 2024-06-10T00:13:06.608 INFO:teuthology.orchestra.run.smithi135.stdout:(20/159): python3-ceph-common-19.0.0-2946.g3d89 1.4 MB/s | 135 kB 00:00 2024-06-10T00:13:06.708 INFO:teuthology.orchestra.run.smithi135.stdout:(21/159): python3-cephfs-19.0.0-2946.g3d89accf. 1.6 MB/s | 162 kB 00:00 2024-06-10T00:13:06.867 INFO:teuthology.orchestra.run.smithi135.stdout:(22/159): python3-rados-19.0.0-2946.g3d89accf.e 2.0 MB/s | 321 kB 00:00 2024-06-10T00:13:06.975 INFO:teuthology.orchestra.run.smithi135.stdout:(23/159): python3-rbd-19.0.0-2946.g3d89accf.el9 2.7 MB/s | 300 kB 00:00 2024-06-10T00:13:07.051 INFO:teuthology.orchestra.run.smithi135.stdout:(24/159): librgw2-19.0.0-2946.g3d89accf.el9.x86 2.9 MB/s | 5.3 MB 00:01 2024-06-10T00:13:07.076 INFO:teuthology.orchestra.run.smithi135.stdout:(25/159): python3-rgw-19.0.0-2946.g3d89accf.el9 987 kB/s | 99 kB 00:00 2024-06-10T00:13:07.127 INFO:teuthology.orchestra.run.smithi135.stdout:(26/159): rbd-fuse-19.0.0-2946.g3d89accf.el9.x8 1.1 MB/s | 85 kB 00:00 2024-06-10T00:13:07.190 INFO:teuthology.orchestra.run.smithi121.stdout:(7/159): ceph-base-19.0.0-2946.g3d89accf.el9.x8 1.2 MB/s | 5.4 MB 00:04 2024-06-10T00:13:07.202 INFO:teuthology.orchestra.run.smithi135.stdout:(27/159): rbd-nbd-19.0.0-2946.g3d89accf.el9.x86 2.2 MB/s | 171 kB 00:00 2024-06-10T00:13:07.286 INFO:teuthology.orchestra.run.smithi135.stdout:(28/159): ceph-grafana-dashboards-19.0.0-2946.g 380 kB/s | 32 kB 00:00 2024-06-10T00:13:07.378 INFO:teuthology.orchestra.run.smithi135.stdout:(29/159): ceph-mgr-cephadm-19.0.0-2946.g3d89acc 1.5 MB/s | 142 kB 00:00 2024-06-10T00:13:07.912 INFO:teuthology.orchestra.run.smithi135.stdout:(30/159): rbd-mirror-19.0.0-2946.g3d89accf.el9. 3.7 MB/s | 3.1 MB 00:00 2024-06-10T00:13:08.404 INFO:teuthology.orchestra.run.smithi135.stdout:(31/159): ceph-mgr-dashboard-19.0.0-2946.g3d89a 3.4 MB/s | 3.5 MB 00:01 2024-06-10T00:13:08.821 INFO:teuthology.orchestra.run.smithi135.stdout:(32/159): ceph-test-19.0.0-2946.g3d89accf.el9.x 9.5 MB/s | 48 MB 00:05 2024-06-10T00:13:08.846 INFO:teuthology.orchestra.run.smithi135.stdout:(33/159): ceph-mgr-modules-core-19.0.0-2946.g3d 563 kB/s | 249 kB 00:00 2024-06-10T00:13:08.871 INFO:teuthology.orchestra.run.smithi135.stdout:(34/159): ceph-mgr-rook-19.0.0-2946.g3d89accf.e 987 kB/s | 49 kB 00:00 2024-06-10T00:13:08.897 INFO:teuthology.orchestra.run.smithi135.stdout:(35/159): ceph-prometheus-alerts-19.0.0-2946.g3 327 kB/s | 16 kB 00:00 2024-06-10T00:13:08.947 INFO:teuthology.orchestra.run.smithi135.stdout:(36/159): ceph-volume-19.0.0-2946.g3d89accf.el9 3.4 MB/s | 264 kB 00:00 2024-06-10T00:13:08.950 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.2 MB 00:01 2024-06-10T00:13:08.999 INFO:teuthology.orchestra.run.smithi121.stdout:(8/159): ceph-osd-19.0.0-2946.g3d89accf.el9.x86 6.3 MB/s | 17 MB 00:02 2024-06-10T00:13:09.006 INFO:teuthology.orchestra.run.smithi135.stdout:(37/159): cephfs-top-19.0.0-2946.g3d89accf.el9. 312 kB/s | 18 kB 00:00 2024-06-10T00:13:09.049 INFO:teuthology.orchestra.run.smithi121.stdout:(9/159): ceph-selinux-19.0.0-2946.g3d89accf.el9 508 kB/s | 25 kB 00:00 2024-06-10T00:13:09.064 INFO:teuthology.orchestra.run.smithi135.stdout:(38/159): cephadm-19.0.0-2946.g3d89accf.el9.noa 4.4 MB/s | 763 kB 00:00 2024-06-10T00:13:09.068 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:09.103 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:09.173 INFO:teuthology.orchestra.run.smithi135.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 481 kB/s | 80 kB 00:00 2024-06-10T00:13:09.174 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:09.257 INFO:teuthology.orchestra.run.smithi135.stdout:(40/159): ceph-mgr-diskprediction-local-19.0.0- 5.5 MB/s | 7.4 MB 00:01 2024-06-10T00:13:09.299 INFO:teuthology.orchestra.run.smithi135.stdout:(41/159): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 195 kB 00:00 2024-06-10T00:13:09.326 INFO:teuthology.orchestra.run.smithi135.stdout:(42/159): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-06-10T00:13:09.366 INFO:teuthology.orchestra.run.smithi135.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-06-10T00:13:09.400 INFO:teuthology.orchestra.run.smithi135.stdout:(44/159): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-06-10T00:13:09.425 INFO:teuthology.orchestra.run.smithi135.stdout:(45/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-06-10T00:13:09.451 INFO:teuthology.orchestra.run.smithi135.stdout:(46/159): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-06-10T00:13:09.476 INFO:teuthology.orchestra.run.smithi135.stdout:(47/159): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-06-10T00:13:09.477 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] automake-1.16.2-8.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/automake-1.16.2-8.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-10T00:13:09.477 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] bison-3.7.4-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/bison-3.7.4-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:09.501 INFO:teuthology.orchestra.run.smithi135.stdout:(48/159): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-06-10T00:13:09.502 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:09.544 INFO:teuthology.orchestra.run.smithi135.stdout:(49/159): automake-1.16.2-8.el9.noarch.rpm 7.3 MB/s | 689 kB 00:00 2024-06-10T00:13:09.569 INFO:teuthology.orchestra.run.smithi135.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-06-10T00:13:09.586 INFO:teuthology.orchestra.run.smithi135.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 517 kB/s | 22 kB 00:00 2024-06-10T00:13:09.620 INFO:teuthology.orchestra.run.smithi135.stdout:(52/159): bison-3.7.4-5.el9.x86_64.rpm 6.4 MB/s | 943 kB 00:00 2024-06-10T00:13:09.645 INFO:teuthology.orchestra.run.smithi135.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 556 kB/s | 33 kB 00:00 2024-06-10T00:13:09.649 INFO:teuthology.orchestra.run.smithi121.stdout:(10/159): ceph-common-19.0.0-2946.g3d89accf.el9 3.0 MB/s | 21 MB 00:06 2024-06-10T00:13:09.679 INFO:teuthology.orchestra.run.smithi135.stdout:(54/159): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-06-10T00:13:09.704 INFO:teuthology.orchestra.run.smithi135.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 310 kB/s | 18 kB 00:00 2024-06-10T00:13:09.730 INFO:teuthology.orchestra.run.smithi135.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-06-10T00:13:09.749 INFO:teuthology.orchestra.run.smithi121.stdout:(11/159): cephfs-mirror-19.0.0-2946.g3d89accf.e 2.2 MB/s | 221 kB 00:00 2024-06-10T00:13:09.755 INFO:teuthology.orchestra.run.smithi135.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-06-10T00:13:09.780 INFO:teuthology.orchestra.run.smithi135.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 899 kB/s | 45 kB 00:00 2024-06-10T00:13:09.808 INFO:teuthology.orchestra.run.smithi121.stdout:(12/159): cephfs-shell-19.0.0-2946.g3d89accf.el 466 kB/s | 27 kB 00:00 2024-06-10T00:13:09.814 INFO:teuthology.orchestra.run.smithi135.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-06-10T00:13:09.856 INFO:teuthology.orchestra.run.smithi135.stdout:(60/159): libtool-2.4.6-45.el9.x86_64.rpm 7.5 MB/s | 582 kB 00:00 2024-06-10T00:13:09.866 INFO:teuthology.orchestra.run.smithi121.stdout:(13/159): libcephfs-devel-19.0.0-2946.g3d89accf 544 kB/s | 32 kB 00:00 2024-06-10T00:13:09.881 INFO:teuthology.orchestra.run.smithi135.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-06-10T00:13:09.926 INFO:teuthology.orchestra.run.smithi135.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 687 kB/s | 47 kB 00:00 2024-06-10T00:13:09.957 INFO:teuthology.orchestra.run.smithi135.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-06-10T00:13:09.983 INFO:teuthology.orchestra.run.smithi135.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-06-10T00:13:10.008 INFO:teuthology.orchestra.run.smithi135.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 699 kB/s | 35 kB 00:00 2024-06-10T00:13:10.042 INFO:teuthology.orchestra.run.smithi121.stdout:(14/159): libcephfs2-19.0.0-2946.g3d89accf.el9. 4.2 MB/s | 747 kB 00:00 2024-06-10T00:13:10.075 INFO:teuthology.orchestra.run.smithi135.stdout:(66/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.6 MB/s | 3.0 MB 00:00 2024-06-10T00:13:10.142 INFO:teuthology.orchestra.run.smithi121.stdout:(15/159): libcephsqlite-19.0.0-2946.g3d89accf.e 1.6 MB/s | 163 kB 00:00 2024-06-10T00:13:10.159 INFO:teuthology.orchestra.run.smithi135.stdout:(67/159): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-06-10T00:13:10.193 INFO:teuthology.orchestra.run.smithi135.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-06-10T00:13:10.209 INFO:teuthology.orchestra.run.smithi121.stdout:(16/159): librados-devel-19.0.0-2946.g3d89accf. 1.9 MB/s | 127 kB 00:00 2024-06-10T00:13:10.251 INFO:teuthology.orchestra.run.smithi135.stdout:(69/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-06-10T00:13:10.277 INFO:teuthology.orchestra.run.smithi135.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-06-10T00:13:10.310 INFO:teuthology.orchestra.run.smithi135.stdout:(71/159): postgresql-contrib-13.14-1.el9.x86_64 5.9 MB/s | 911 kB 00:00 2024-06-10T00:13:10.344 INFO:teuthology.orchestra.run.smithi135.stdout:(72/159): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-06-10T00:13:10.351 INFO:teuthology.orchestra.run.smithi121.stdout:(17/159): libradosstriper1-19.0.0-2946.g3d89acc 3.5 MB/s | 501 kB 00:00 2024-06-10T00:13:10.369 INFO:teuthology.orchestra.run.smithi135.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-06-10T00:13:10.411 INFO:teuthology.orchestra.run.smithi135.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-06-10T00:13:10.495 INFO:teuthology.orchestra.run.smithi135.stdout:(75/159): postgresql-server-13.14-1.el9.x86_64. 24 MB/s | 5.8 MB 00:00 2024-06-10T00:13:10.542 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 17 kB 00:00 2024-06-10T00:13:10.571 INFO:teuthology.orchestra.run.smithi135.stdout:(76/159): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 6.0 MB 00:00 2024-06-10T00:13:10.596 INFO:teuthology.orchestra.run.smithi135.stdout:(77/159): python3-devel-3.9.18-3.el9.x86_64.rpm 1.3 MB/s | 244 kB 00:00 2024-06-10T00:13:10.621 INFO:teuthology.orchestra.run.smithi135.stdout:(78/159): python3-jinja2-2.11.3-6.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-06-10T00:13:10.646 INFO:teuthology.orchestra.run.smithi135.stdout:(79/159): python3-jsonpatch-1.21-16.el9.noarch. 339 kB/s | 26 kB 00:00 2024-06-10T00:13:10.672 INFO:teuthology.orchestra.run.smithi135.stdout:(80/159): python3-jsonpointer-2.0-4.el9.noarch. 253 kB/s | 19 kB 00:00 2024-06-10T00:13:10.697 INFO:teuthology.orchestra.run.smithi135.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 692 kB/s | 35 kB 00:00 2024-06-10T00:13:10.723 INFO:teuthology.orchestra.run.smithi135.stdout:(82/159): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-06-10T00:13:10.748 INFO:teuthology.orchestra.run.smithi135.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.8 MB/s | 250 kB 00:00 2024-06-10T00:13:10.773 INFO:teuthology.orchestra.run.smithi135.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-06-10T00:13:10.799 INFO:teuthology.orchestra.run.smithi135.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-06-10T00:13:10.809 INFO:teuthology.orchestra.run.smithi121.stdout:(18/159): ceph-radosgw-19.0.0-2946.g3d89accf.el 2.8 MB/s | 10 MB 00:03 2024-06-10T00:13:10.824 INFO:teuthology.orchestra.run.smithi135.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 840 kB/s | 42 kB 00:00 2024-06-10T00:13:10.849 INFO:teuthology.orchestra.run.smithi135.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-06-10T00:13:10.860 INFO:teuthology.orchestra.run.smithi121.stdout:(19/159): python3-ceph-argparse-19.0.0-2946.g3d 909 kB/s | 45 kB 00:00 2024-06-10T00:13:10.866 INFO:teuthology.orchestra.run.smithi135.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.7 MB/s | 159 kB 00:00 2024-06-10T00:13:10.892 INFO:teuthology.orchestra.run.smithi135.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 6.5 MB/s | 279 kB 00:00 2024-06-10T00:13:10.917 INFO:teuthology.orchestra.run.smithi135.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-06-10T00:13:10.927 INFO:teuthology.orchestra.run.smithi121.stdout:(20/159): python3-ceph-common-19.0.0-2946.g3d89 2.0 MB/s | 135 kB 00:00 2024-06-10T00:13:10.942 INFO:teuthology.orchestra.run.smithi135.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-06-10T00:13:10.985 INFO:teuthology.orchestra.run.smithi135.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 995 kB/s | 42 kB 00:00 2024-06-10T00:13:10.993 INFO:teuthology.orchestra.run.smithi121.stdout:(21/159): python3-cephfs-19.0.0-2946.g3d89accf. 2.4 MB/s | 162 kB 00:00 2024-06-10T00:13:11.060 INFO:teuthology.orchestra.run.smithi135.stdout:(93/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 5.1 MB 00:00 2024-06-10T00:13:11.094 INFO:teuthology.orchestra.run.smithi121.stdout:(22/159): python3-rados-19.0.0-2946.g3d89accf.e 3.1 MB/s | 321 kB 00:00 2024-06-10T00:13:11.095 INFO:teuthology.orchestra.run.smithi135.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 406 kB/s | 44 kB 00:00 2024-06-10T00:13:11.119 INFO:teuthology.orchestra.run.smithi135.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-06-10T00:13:11.145 INFO:teuthology.orchestra.run.smithi135.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 713 kB/s | 36 kB 00:00 2024-06-10T00:13:11.170 INFO:teuthology.orchestra.run.smithi135.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-06-10T00:13:11.186 INFO:teuthology.orchestra.run.smithi121.stdout:(23/159): python3-rbd-19.0.0-2946.g3d89accf.el9 3.2 MB/s | 300 kB 00:00 2024-06-10T00:13:11.195 INFO:teuthology.orchestra.run.smithi135.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-06-10T00:13:11.221 INFO:teuthology.orchestra.run.smithi135.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-06-10T00:13:11.245 INFO:teuthology.orchestra.run.smithi121.stdout:(24/159): python3-rgw-19.0.0-2946.g3d89accf.el9 1.7 MB/s | 99 kB 00:00 2024-06-10T00:13:11.246 INFO:teuthology.orchestra.run.smithi135.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-06-10T00:13:11.271 INFO:teuthology.orchestra.run.smithi135.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-06-10T00:13:11.297 INFO:teuthology.orchestra.run.smithi135.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 891 kB/s | 45 kB 00:00 2024-06-10T00:13:11.297 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:11.300 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:11.303 INFO:teuthology.orchestra.run.smithi121.stdout:(25/159): rbd-fuse-19.0.0-2946.g3d89accf.el9.x8 1.4 MB/s | 85 kB 00:00 2024-06-10T00:13:11.523 INFO:teuthology.orchestra.run.smithi135.stdout:(103/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-06-10T00:13:11.553 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-06-10T00:13:11.556 INFO:teuthology.orchestra.run.smithi135.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 16 MB/s | 548 kB 00:00 2024-06-10T00:13:11.562 INFO:teuthology.orchestra.run.smithi121.stdout:(26/159): librgw2-19.0.0-2946.g3d89accf.el9.x86 4.4 MB/s | 5.3 MB 00:01 2024-06-10T00:13:11.598 INFO:teuthology.orchestra.run.smithi135.stdout:(105/159): gperftools-libs-2.9.1-2.el9.x86_64.r 7.2 MB/s | 309 kB 00:00 2024-06-10T00:13:11.623 INFO:teuthology.orchestra.run.smithi135.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 783 kB/s | 19 kB 00:00 2024-06-10T00:13:11.637 INFO:teuthology.orchestra.run.smithi121.stdout:(27/159): rbd-nbd-19.0.0-2946.g3d89accf.el9.x86 2.2 MB/s | 171 kB 00:00 2024-06-10T00:13:11.713 INFO:teuthology.orchestra.run.smithi121.stdout:(28/159): ceph-grafana-dashboards-19.0.0-2946.g 422 kB/s | 32 kB 00:00 2024-06-10T00:13:11.732 INFO:teuthology.orchestra.run.smithi135.stdout:(107/159): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-06-10T00:13:11.780 INFO:teuthology.orchestra.run.smithi121.stdout:(29/159): ceph-mgr-cephadm-19.0.0-2946.g3d89acc 2.1 MB/s | 142 kB 00:00 2024-06-10T00:13:11.807 INFO:teuthology.orchestra.run.smithi135.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-06-10T00:13:11.833 INFO:teuthology.orchestra.run.smithi135.stdout:(109/159): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-10T00:13:11.858 INFO:teuthology.orchestra.run.smithi135.stdout:(110/159): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-06-10T00:13:11.884 INFO:teuthology.orchestra.run.smithi135.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 6.3 MB/s | 151 kB 00:00 2024-06-10T00:13:11.934 INFO:teuthology.orchestra.run.smithi135.stdout:(112/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 16 MB/s | 839 kB 00:00 2024-06-10T00:13:11.938 INFO:teuthology.orchestra.run.smithi121.stdout:(30/159): rbd-mirror-19.0.0-2946.g3d89accf.el9. 4.9 MB/s | 3.1 MB 00:00 2024-06-10T00:13:11.967 INFO:teuthology.orchestra.run.smithi135.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-06-10T00:13:11.993 INFO:teuthology.orchestra.run.smithi135.stdout:(114/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-06-10T00:13:12.018 INFO:teuthology.orchestra.run.smithi135.stdout:(115/159): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-06-10T00:13:12.043 INFO:teuthology.orchestra.run.smithi135.stdout:(116/159): python3-certifi-2021.10.8-2.el9.noar 594 kB/s | 15 kB 00:00 2024-06-10T00:13:12.069 INFO:teuthology.orchestra.run.smithi135.stdout:(117/159): python3-cheroot-8.6.0-4.el9.noarch.r 6.6 MB/s | 172 kB 00:00 2024-06-10T00:13:12.103 INFO:teuthology.orchestra.run.smithi135.stdout:(118/159): python3-cherrypy-18.6.1-2.el9.noarch 11 MB/s | 358 kB 00:00 2024-06-10T00:13:12.120 INFO:teuthology.orchestra.run.smithi135.stdout:(119/159): lua-devel-5.4.4-4.el9.x86_64.rpm 26 kB/s | 22 kB 00:00 2024-06-10T00:13:12.145 INFO:teuthology.orchestra.run.smithi135.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.2 MB/s | 303 kB 00:00 2024-06-10T00:13:12.179 INFO:teuthology.orchestra.run.smithi135.stdout:(121/159): python3-google-auth-2.29.0-2.el9.noa 3.4 MB/s | 205 kB 00:00 2024-06-10T00:13:12.213 INFO:teuthology.orchestra.run.smithi135.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-06-10T00:13:12.238 INFO:teuthology.orchestra.run.smithi135.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 426 kB/s | 11 kB 00:00 2024-06-10T00:13:12.263 INFO:teuthology.orchestra.run.smithi135.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 707 kB/s | 18 kB 00:00 2024-06-10T00:13:12.289 INFO:teuthology.orchestra.run.smithi135.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 928 kB/s | 23 kB 00:00 2024-06-10T00:13:12.314 INFO:teuthology.orchestra.run.smithi135.stdout:(126/159): python3-jaraco-functools-3.5.0-2.el9 780 kB/s | 19 kB 00:00 2024-06-10T00:13:12.364 INFO:teuthology.orchestra.run.smithi135.stdout:(127/159): python3-grpcio-1.46.7-10.el9.x86_64. 9.3 MB/s | 2.0 MB 00:00 2024-06-10T00:13:12.390 INFO:teuthology.orchestra.run.smithi135.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-06-10T00:13:12.415 INFO:teuthology.orchestra.run.smithi135.stdout:(129/159): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-06-10T00:13:12.449 INFO:teuthology.orchestra.run.smithi135.stdout:(130/159): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-06-10T00:13:12.463 INFO:teuthology.orchestra.run.smithi121.stdout:(31/159): ceph-mgr-dashboard-19.0.0-2946.g3d89a 5.1 MB/s | 3.5 MB 00:00 2024-06-10T00:13:12.474 INFO:teuthology.orchestra.run.smithi135.stdout:(131/159): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-06-10T00:13:12.508 INFO:teuthology.orchestra.run.smithi135.stdout:(132/159): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-06-10T00:13:12.533 INFO:teuthology.orchestra.run.smithi135.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 661 kB/s | 16 kB 00:00 2024-06-10T00:13:12.572 INFO:teuthology.orchestra.run.smithi121.stdout:(32/159): ceph-mgr-modules-core-19.0.0-2946.g3d 2.3 MB/s | 249 kB 00:00 2024-06-10T00:13:12.575 INFO:teuthology.orchestra.run.smithi135.stdout:(134/159): protobuf-compiler-3.14.0-13.el9.x86_ 676 kB/s | 863 kB 00:01 2024-06-10T00:13:12.634 INFO:teuthology.orchestra.run.smithi135.stdout:(135/159): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-06-10T00:13:12.639 INFO:teuthology.orchestra.run.smithi121.stdout:(33/159): ceph-mgr-rook-19.0.0-2946.g3d89accf.e 741 kB/s | 49 kB 00:00 2024-06-10T00:13:12.659 INFO:teuthology.orchestra.run.smithi135.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 716 kB/s | 90 kB 00:00 2024-06-10T00:13:12.684 INFO:teuthology.orchestra.run.smithi135.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 1.3 MB/s | 31 kB 00:00 2024-06-10T00:13:12.706 INFO:teuthology.orchestra.run.smithi121.stdout:(34/159): ceph-prometheus-alerts-19.0.0-2946.g3 247 kB/s | 16 kB 00:00 2024-06-10T00:13:12.710 INFO:teuthology.orchestra.run.smithi135.stdout:(138/159): python3-pyperclip-1.8.2-2.el9.noarch 210 kB/s | 28 kB 00:00 2024-06-10T00:13:12.735 INFO:teuthology.orchestra.run.smithi135.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-06-10T00:13:12.769 INFO:teuthology.orchestra.run.smithi135.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 608 kB/s | 36 kB 00:00 2024-06-10T00:13:12.795 INFO:teuthology.orchestra.run.smithi135.stdout:(141/159): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-06-10T00:13:12.820 INFO:teuthology.orchestra.run.smithi135.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 606 kB/s | 51 kB 00:00 2024-06-10T00:13:12.823 INFO:teuthology.orchestra.run.smithi121.stdout:(35/159): ceph-volume-19.0.0-2946.g3d89accf.el9 2.2 MB/s | 264 kB 00:00 2024-06-10T00:13:12.845 INFO:teuthology.orchestra.run.smithi135.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-06-10T00:13:12.870 INFO:teuthology.orchestra.run.smithi135.stdout:(144/159): python3-zc-lockfile-2.0-10.el9.noarc 797 kB/s | 20 kB 00:00 2024-06-10T00:13:12.896 INFO:teuthology.orchestra.run.smithi135.stdout:(145/159): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-06-10T00:13:12.921 INFO:teuthology.orchestra.run.smithi135.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-06-10T00:13:12.955 INFO:teuthology.orchestra.run.smithi135.stdout:(147/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-06-10T00:13:13.005 INFO:teuthology.orchestra.run.smithi135.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-06-10T00:13:13.006 INFO:teuthology.orchestra.run.smithi121.stdout:(36/159): cephadm-19.0.0-2946.g3d89accf.el9.noa 4.1 MB/s | 763 kB 00:00 2024-06-10T00:13:13.073 INFO:teuthology.orchestra.run.smithi121.stdout:(37/159): cephfs-top-19.0.0-2946.g3d89accf.el9. 274 kB/s | 18 kB 00:00 2024-06-10T00:13:13.131 INFO:teuthology.orchestra.run.smithi135.stdout:(149/159): NetworkManager-1.47.91-1.el9.x86_64. 19 MB/s | 2.3 MB 00:00 2024-06-10T00:13:13.198 INFO:teuthology.orchestra.run.smithi135.stdout:(150/159): NetworkManager-libnm-1.47.91-1.el9.x 27 MB/s | 1.8 MB 00:00 2024-06-10T00:13:13.207 INFO:teuthology.orchestra.run.smithi121.stdout:(38/159): ceph-mgr-diskprediction-local-19.0.0- 5.8 MB/s | 7.4 MB 00:01 2024-06-10T00:13:13.211 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:13.240 INFO:teuthology.orchestra.run.smithi135.stdout:(151/159): NetworkManager-team-1.47.91-1.el9.x8 993 kB/s | 41 kB 00:00 2024-06-10T00:13:13.247 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:13.282 INFO:teuthology.orchestra.run.smithi135.stdout:(152/159): NetworkManager-tui-1.47.91-1.el9.x86 5.8 MB/s | 248 kB 00:00 2024-06-10T00:13:13.315 INFO:teuthology.orchestra.run.smithi135.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 1.1 MB/s | 37 kB 00:00 2024-06-10T00:13:13.358 INFO:teuthology.orchestra.run.smithi135.stdout:(154/159): elfutils-libelf-0.191-4.el9.x86_64.r 4.9 MB/s | 207 kB 00:00 2024-06-10T00:13:13.400 INFO:teuthology.orchestra.run.smithi135.stdout:(155/159): elfutils-libs-0.191-4.el9.x86_64.rpm 6.1 MB/s | 260 kB 00:00 2024-06-10T00:13:13.450 INFO:teuthology.orchestra.run.smithi135.stdout:(156/159): librbd1-19.0.0-2946.g3d89accf.el9.x8 6.3 MB/s | 3.1 MB 00:00 2024-06-10T00:13:13.492 INFO:teuthology.orchestra.run.smithi135.stdout:(157/159): openssl-3.2.1-1.el9.x86_64.rpm 15 MB/s | 1.3 MB 00:00 2024-06-10T00:13:13.601 INFO:teuthology.orchestra.run.smithi135.stdout:(158/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-06-10T00:13:13.651 INFO:teuthology.orchestra.run.smithi135.stdout:(159/159): librados2-19.0.0-2946.g3d89accf.el9. 4.6 MB/s | 3.4 MB 00:00 2024-06-10T00:13:13.676 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-06-10T00:13:13.676 INFO:teuthology.orchestra.run.smithi135.stdout:Total 17 MB/s | 225 MB 00:12 2024-06-10T00:13:13.916 INFO:teuthology.orchestra.run.smithi121.stdout:(39/159): ceph-test-19.0.0-2946.g3d89accf.el9.x 9.9 MB/s | 48 MB 00:04 2024-06-10T00:13:13.941 INFO:teuthology.orchestra.run.smithi121.stdout:(40/159): fuse-2.9.9-15.el9.x86_64.rpm 93 kB/s | 80 kB 00:00 2024-06-10T00:13:13.942 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:13.942 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:14.216 INFO:teuthology.orchestra.run.smithi121.stdout:(41/159): libquadmath-11.4.1-3.el9.x86_64.rpm 651 kB/s | 195 kB 00:00 2024-06-10T00:13:14.242 INFO:teuthology.orchestra.run.smithi121.stdout:(42/159): libgfortran-11.4.1-3.el9.x86_64.rpm 779 kB/s | 803 kB 00:01 2024-06-10T00:13:14.350 INFO:teuthology.orchestra.run.smithi121.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 983 kB/s | 106 kB 00:00 2024-06-10T00:13:14.459 INFO:teuthology.orchestra.run.smithi121.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-06-10T00:13:14.534 INFO:teuthology.orchestra.run.smithi121.stdout:(45/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 427 kB/s | 253 kB 00:00 2024-06-10T00:13:14.568 INFO:teuthology.orchestra.run.smithi121.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-06-10T00:13:14.569 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] automake-1.16.2-8.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/automake-1.16.2-8.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-10T00:13:14.718 INFO:teuthology.orchestra.run.smithi121.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 1.1 MB/s | 215 kB 00:00 2024-06-10T00:13:14.752 INFO:teuthology.orchestra.run.smithi121.stdout:(48/159): python3-cryptography-36.0.1-4.el9.x86 2.3 MB/s | 1.2 MB 00:00 2024-06-10T00:13:14.752 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] bison-3.7.4-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/bison-3.7.4-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:14.753 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:14.835 INFO:teuthology.orchestra.run.smithi121.stdout:(49/159): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 689 kB 00:00 2024-06-10T00:13:14.861 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-06-10T00:13:14.877 INFO:teuthology.orchestra.run.smithi121.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 522 kB/s | 22 kB 00:00 2024-06-10T00:13:14.903 INFO:teuthology.orchestra.run.smithi121.stdout:(51/159): boost-program-options-1.75.0-8.el9.x8 709 kB/s | 107 kB 00:00 2024-06-10T00:13:14.945 INFO:teuthology.orchestra.run.smithi121.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 788 kB/s | 33 kB 00:00 2024-06-10T00:13:14.970 INFO:teuthology.orchestra.run.smithi121.stdout:(53/159): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-06-10T00:13:15.020 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-06-10T00:13:15.020 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-06-10T00:13:15.021 INFO:teuthology.orchestra.run.smithi121.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-06-10T00:13:15.062 INFO:teuthology.orchestra.run.smithi121.stdout:(55/159): bison-3.7.4-5.el9.x86_64.rpm 2.7 MB/s | 943 kB 00:00 2024-06-10T00:13:15.088 INFO:teuthology.orchestra.run.smithi121.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 166 kB/s | 11 kB 00:00 2024-06-10T00:13:15.113 INFO:teuthology.orchestra.run.smithi121.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-06-10T00:13:15.138 INFO:teuthology.orchestra.run.smithi121.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-06-10T00:13:15.288 INFO:teuthology.orchestra.run.smithi121.stdout:(59/159): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 582 kB 00:00 2024-06-10T00:13:15.322 INFO:teuthology.orchestra.run.smithi121.stdout:(60/159): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-06-10T00:13:15.347 INFO:teuthology.orchestra.run.smithi121.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 433 kB/s | 25 kB 00:00 2024-06-10T00:13:15.373 INFO:teuthology.orchestra.run.smithi121.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-06-10T00:13:15.431 INFO:teuthology.orchestra.run.smithi121.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-06-10T00:13:15.456 INFO:teuthology.orchestra.run.smithi121.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-06-10T00:13:15.482 INFO:teuthology.orchestra.run.smithi121.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 699 kB/s | 35 kB 00:00 2024-06-10T00:13:15.590 INFO:teuthology.orchestra.run.smithi121.stdout:(66/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.6 MB/s | 3.0 MB 00:00 2024-06-10T00:13:15.891 INFO:teuthology.orchestra.run.smithi121.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-06-10T00:13:16.074 INFO:teuthology.orchestra.run.smithi121.stdout:(68/159): postgresql-contrib-13.14-1.el9.x86_64 4.9 MB/s | 911 kB 00:00 2024-06-10T00:13:16.133 INFO:teuthology.orchestra.run.smithi121.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-06-10T00:13:16.308 INFO:teuthology.orchestra.run.smithi121.stdout:(70/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 5.4 MB/s | 4.4 MB 00:00 2024-06-10T00:13:16.375 INFO:teuthology.orchestra.run.smithi121.stdout:(71/159): openblas-openmp-0.3.21-2.el9.x86_64.r 5.2 MB/s | 4.7 MB 00:00 2024-06-10T00:13:16.434 INFO:teuthology.orchestra.run.smithi121.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-06-10T00:13:16.551 INFO:teuthology.orchestra.run.smithi121.stdout:(73/159): protobuf-3.14.0-13.el9.x86_64.rpm 4.2 MB/s | 1.0 MB 00:00 2024-06-10T00:13:16.609 INFO:teuthology.orchestra.run.smithi121.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.0 MB/s | 61 kB 00:00 2024-06-10T00:13:16.693 INFO:teuthology.orchestra.run.smithi121.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-06-10T00:13:16.776 INFO:teuthology.orchestra.run.smithi121.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-06-10T00:13:16.821 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-06-10T00:13:16.821 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-06-10T00:13:16.835 INFO:teuthology.orchestra.run.smithi121.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 440 kB/s | 26 kB 00:00 2024-06-10T00:13:16.894 INFO:teuthology.orchestra.run.smithi121.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 329 kB/s | 19 kB 00:00 2024-06-10T00:13:16.952 INFO:teuthology.orchestra.run.smithi121.stdout:(79/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-06-10T00:13:16.983 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 5.5 kB/s | 1.7 kB 00:00 2024-06-10T00:13:17.011 INFO:teuthology.orchestra.run.smithi121.stdout:(80/159): python3-markupsafe-1.1.1-12.el9.x86_6 596 kB/s | 35 kB 00:00 2024-06-10T00:13:17.145 INFO:teuthology.orchestra.run.smithi121.stdout:(81/159): postgresql-server-13.14-1.el9.x86_64. 5.7 MB/s | 5.8 MB 00:01 2024-06-10T00:13:17.212 INFO:teuthology.orchestra.run.smithi121.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-06-10T00:13:17.271 INFO:teuthology.orchestra.run.smithi121.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-06-10T00:13:17.329 INFO:teuthology.orchestra.run.smithi121.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-06-10T00:13:17.379 INFO:teuthology.orchestra.run.smithi121.stdout:(85/159): python3-prettytable-0.7.2-27.el9.noar 849 kB/s | 42 kB 00:00 2024-06-10T00:13:17.455 INFO:teuthology.orchestra.run.smithi121.stdout:(86/159): python3-babel-2.9.1-2.el9.noarch.rpm 5.8 MB/s | 6.0 MB 00:01 2024-06-10T00:13:17.505 INFO:teuthology.orchestra.run.smithi121.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-06-10T00:13:17.530 INFO:teuthology.orchestra.run.smithi121.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-06-10T00:13:17.581 INFO:teuthology.orchestra.run.smithi121.stdout:(89/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-06-10T00:13:17.614 INFO:teuthology.orchestra.run.smithi121.stdout:(90/159): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-06-10T00:13:17.639 INFO:teuthology.orchestra.run.smithi121.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 916 kB/s | 54 kB 00:00 2024-06-10T00:13:17.690 INFO:teuthology.orchestra.run.smithi121.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 834 kB/s | 42 kB 00:00 2024-06-10T00:13:17.831 INFO:teuthology.orchestra.run.smithi121.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 313 kB/s | 44 kB 00:00 2024-06-10T00:13:17.932 INFO:teuthology.orchestra.run.smithi121.stdout:(94/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.5 MB/s | 5.1 MB 00:00 2024-06-10T00:13:17.958 INFO:teuthology.orchestra.run.smithi121.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-06-10T00:13:17.983 INFO:teuthology.orchestra.run.smithi121.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 715 kB/s | 36 kB 00:00 2024-06-10T00:13:18.008 INFO:teuthology.orchestra.run.smithi121.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-06-10T00:13:18.067 INFO:teuthology.orchestra.run.smithi121.stdout:(98/159): uuid-1.6.2-55.el9.x86_64.rpm 992 kB/s | 58 kB 00:00 2024-06-10T00:13:18.092 INFO:teuthology.orchestra.run.smithi121.stdout:(99/159): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 305 kB 00:00 2024-06-10T00:13:18.126 INFO:teuthology.orchestra.run.smithi121.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-06-10T00:13:18.151 INFO:teuthology.orchestra.run.smithi121.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-06-10T00:13:18.152 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:18.184 INFO:teuthology.orchestra.run.smithi121.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 769 kB/s | 45 kB 00:00 2024-06-10T00:13:18.186 INFO:teuthology.orchestra.run.smithi121.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:18.243 INFO:teuthology.orchestra.run.smithi121.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-06-10T00:13:18.285 INFO:teuthology.orchestra.run.smithi121.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 13 MB/s | 548 kB 00:00 2024-06-10T00:13:18.319 INFO:teuthology.orchestra.run.smithi121.stdout:(105/159): gperftools-libs-2.9.1-2.el9.x86_64.r 9.1 MB/s | 309 kB 00:00 2024-06-10T00:13:18.344 INFO:teuthology.orchestra.run.smithi121.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-06-10T00:13:18.403 INFO:teuthology.orchestra.run.smithi121.stdout:(107/159): protobuf-compiler-3.14.0-13.el9.x86_ 3.9 MB/s | 863 kB 00:00 2024-06-10T00:13:18.470 INFO:teuthology.orchestra.run.smithi121.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 396 kB/s | 26 kB 00:00 2024-06-10T00:13:18.504 INFO:teuthology.orchestra.run.smithi121.stdout:(109/159): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-06-10T00:13:18.529 INFO:teuthology.orchestra.run.smithi121.stdout:(110/159): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-06-10T00:13:18.562 INFO:teuthology.orchestra.run.smithi121.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-06-10T00:13:18.630 INFO:teuthology.orchestra.run.smithi121.stdout:(112/159): libarrow-9.0.0-11.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-06-10T00:13:18.663 INFO:teuthology.orchestra.run.smithi121.stdout:(113/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-06-10T00:13:18.688 INFO:teuthology.orchestra.run.smithi121.stdout:(114/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-06-10T00:13:18.714 INFO:teuthology.orchestra.run.smithi121.stdout:(115/159): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-06-10T00:13:18.739 INFO:teuthology.orchestra.run.smithi121.stdout:(116/159): python3-asyncssh-2.13.2-5.el9.noarch 4.9 MB/s | 548 kB 00:00 2024-06-10T00:13:18.765 INFO:teuthology.orchestra.run.smithi121.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 297 kB/s | 15 kB 00:00 2024-06-10T00:13:18.789 INFO:teuthology.orchestra.run.smithi121.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-06-10T00:13:18.815 INFO:teuthology.orchestra.run.smithi121.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-06-10T00:13:18.840 INFO:teuthology.orchestra.run.smithi121.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-06-10T00:13:18.865 INFO:teuthology.orchestra.run.smithi121.stdout:(121/159): python3-google-auth-2.29.0-2.el9.noa 4.0 MB/s | 205 kB 00:00 2024-06-10T00:13:18.891 INFO:teuthology.orchestra.run.smithi121.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 5.7 MB/s | 144 kB 00:00 2024-06-10T00:13:18.908 INFO:teuthology.orchestra.run.smithi121.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 638 kB/s | 11 kB 00:00 2024-06-10T00:13:18.925 INFO:teuthology.orchestra.run.smithi121.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 1.0 MB/s | 18 kB 00:00 2024-06-10T00:13:18.950 INFO:teuthology.orchestra.run.smithi121.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 931 kB/s | 23 kB 00:00 2024-06-10T00:13:18.975 INFO:teuthology.orchestra.run.smithi121.stdout:(126/159): python3-jaraco-functools-3.5.0-2.el9 777 kB/s | 19 kB 00:00 2024-06-10T00:13:19.000 INFO:teuthology.orchestra.run.smithi121.stdout:(127/159): python3-jaraco-text-3.2.0-6.el9.noar 788 kB/s | 20 kB 00:00 2024-06-10T00:13:19.068 INFO:teuthology.orchestra.run.smithi121.stdout:(128/159): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-06-10T00:13:19.101 INFO:teuthology.orchestra.run.smithi121.stdout:(129/159): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-06-10T00:13:19.126 INFO:teuthology.orchestra.run.smithi121.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-06-10T00:13:19.143 INFO:teuthology.orchestra.run.smithi121.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-06-10T00:13:19.160 INFO:teuthology.orchestra.run.smithi121.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-06-10T00:13:19.177 INFO:teuthology.orchestra.run.smithi121.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 997 kB/s | 16 kB 00:00 2024-06-10T00:13:19.203 INFO:teuthology.orchestra.run.smithi121.stdout:(134/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.6 MB/s | 90 kB 00:00 2024-06-10T00:13:19.228 INFO:teuthology.orchestra.run.smithi121.stdout:(135/159): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-06-10T00:13:19.253 INFO:teuthology.orchestra.run.smithi121.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 557 kB/s | 28 kB 00:00 2024-06-10T00:13:19.269 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-10T00:13:19.269 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-10T00:13:19.279 INFO:teuthology.orchestra.run.smithi121.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 607 kB/s | 31 kB 00:00 2024-06-10T00:13:19.280 INFO:teuthology.orchestra.run.smithi193.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-10T00:13:19.281 INFO:teuthology.orchestra.run.smithi193.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-10T00:13:19.282 INFO:teuthology.orchestra.run.smithi193.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-10T00:13:19.283 INFO:teuthology.orchestra.run.smithi193.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-10T00:13:19.285 INFO:teuthology.orchestra.run.smithi193.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-10T00:13:19.287 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-10T00:13:19.288 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-10T00:13:19.304 INFO:teuthology.orchestra.run.smithi121.stdout:(138/159): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-06-10T00:13:19.329 INFO:teuthology.orchestra.run.smithi121.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-06-10T00:13:19.353 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-06-10T00:13:19.363 INFO:teuthology.orchestra.run.smithi121.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 611 kB/s | 36 kB 00:00 2024-06-10T00:13:19.374 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-06-10T00:13:19.374 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-06-10T00:13:19.374 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-06-10T00:13:19.374 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-06-10T00:13:19.374 INFO:teuthology.orchestra.run.smithi193.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-10T00:13:19.374 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.6 k 2024-06-10T00:13:19.374 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-10T00:13:19.374 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 890 k 2024-06-10T00:13:19.374 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 145 k 2024-06-10T00:13:19.374 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-10T00:13:19.374 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 142 k 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 3.5 M 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 7.4 M 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 49 k 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 10 M 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 48 M 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 264 k 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 763 k 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 221 k 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 27 k 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 18 k 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-10T00:13:19.375 INFO:teuthology.orchestra.run.smithi193.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 32 k 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 747 k 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 127 k 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 162 k 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 321 k 2024-06-10T00:13:19.376 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 300 k 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 k 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 85 k 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 171 k 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager x86_64 1:1.47.91-1.el9 baseos 2.3 M 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-libnm x86_64 1:1.47.91-1.el9 baseos 1.8 M 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-team x86_64 1:1.47.91-1.el9 baseos 41 k 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-tui x86_64 1:1.47.91-1.el9 baseos 248 k 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-10T00:13:19.377 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.4 M 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 21 M 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 32 k 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 2.4 M 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 249 k 2024-06-10T00:13:19.378 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 4.7 M 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 17 M 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 16 k 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 25 k 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 163 k 2024-06-10T00:13:19.379 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 501 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.3 M 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-10T00:13:19.380 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 k 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 135 k 2024-06-10T00:13:19.381 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-10T00:13:19.382 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-10T00:13:19.383 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-10T00:13:19.384 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-10T00:13:19.385 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-06-10T00:13:19.386 INFO:teuthology.orchestra.run.smithi193.stdout:Install 148 Packages 2024-06-10T00:13:19.387 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 11 Packages 2024-06-10T00:13:19.387 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:13:19.387 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 225 M 2024-06-10T00:13:19.387 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-06-10T00:13:19.388 INFO:teuthology.orchestra.run.smithi121.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 872 kB/s | 51 kB 00:00 2024-06-10T00:13:19.414 INFO:teuthology.orchestra.run.smithi121.stdout:(142/159): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-06-10T00:13:19.430 INFO:teuthology.orchestra.run.smithi121.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-06-10T00:13:19.456 INFO:teuthology.orchestra.run.smithi121.stdout:(144/159): python3-zc-lockfile-2.0-10.el9.noarc 798 kB/s | 20 kB 00:00 2024-06-10T00:13:19.481 INFO:teuthology.orchestra.run.smithi121.stdout:(145/159): re2-20211101-3.el9.x86_64.rpm 7.5 MB/s | 192 kB 00:00 2024-06-10T00:13:19.506 INFO:teuthology.orchestra.run.smithi121.stdout:(146/159): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-06-10T00:13:19.565 INFO:teuthology.orchestra.run.smithi121.stdout:(147/159): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-06-10T00:13:19.899 INFO:teuthology.orchestra.run.smithi121.stdout:(148/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.1 MB/s | 16 MB 00:02 2024-06-10T00:13:19.951 INFO:teuthology.orchestra.run.smithi193.stdout:(1/159): ceph-19.0.0-2946.g3d89accf.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-06-10T00:13:19.957 INFO:teuthology.orchestra.run.smithi121.stdout:(149/159): librados2-19.0.0-2946.g3d89accf.el9. 7.5 MB/s | 3.4 MB 00:00 2024-06-10T00:13:20.168 INFO:teuthology.orchestra.run.smithi193.stdout:(2/159): ceph-fuse-19.0.0-2946.g3d89accf.el9.x8 4.0 MB/s | 890 kB 00:00 2024-06-10T00:13:20.174 INFO:teuthology.orchestra.run.smithi121.stdout:(150/159): librbd1-19.0.0-2946.g3d89accf.el9.x8 5.1 MB/s | 3.1 MB 00:00 2024-06-10T00:13:20.218 INFO:teuthology.orchestra.run.smithi193.stdout:(3/159): ceph-immutable-object-cache-19.0.0-294 2.8 MB/s | 145 kB 00:00 2024-06-10T00:13:20.469 INFO:teuthology.orchestra.run.smithi193.stdout:(4/159): ceph-mds-19.0.0-2946.g3d89accf.el9.x86 9.4 MB/s | 2.4 MB 00:00 2024-06-10T00:13:20.516 INFO:teuthology.orchestra.run.smithi121.stdout:(151/159): NetworkManager-team-1.47.91-1.el9.x8 121 kB/s | 41 kB 00:00 2024-06-10T00:13:20.619 INFO:teuthology.orchestra.run.smithi193.stdout:(5/159): ceph-mgr-19.0.0-2946.g3d89accf.el9.x86 10 MB/s | 1.5 MB 00:00 2024-06-10T00:13:20.833 INFO:teuthology.orchestra.run.smithi121.stdout:(152/159): NetworkManager-1.47.91-1.el9.x86_64. 2.5 MB/s | 2.3 MB 00:00 2024-06-10T00:13:20.883 INFO:teuthology.orchestra.run.smithi121.stdout:(153/159): NetworkManager-libnm-1.47.91-1.el9.x 1.9 MB/s | 1.8 MB 00:00 2024-06-10T00:13:20.908 INFO:teuthology.orchestra.run.smithi121.stdout:(154/159): NetworkManager-tui-1.47.91-1.el9.x86 632 kB/s | 248 kB 00:00 2024-06-10T00:13:20.934 INFO:teuthology.orchestra.run.smithi121.stdout:(155/159): elfutils-debuginfod-client-0.191-4.e 365 kB/s | 37 kB 00:00 2024-06-10T00:13:20.992 INFO:teuthology.orchestra.run.smithi121.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.9 MB/s | 207 kB 00:00 2024-06-10T00:13:20.994 INFO:teuthology.orchestra.run.smithi193.stdout:(6/159): ceph-base-19.0.0-2946.g3d89accf.el9.x8 4.4 MB/s | 5.4 MB 00:01 2024-06-10T00:13:21.070 INFO:teuthology.orchestra.run.smithi193.stdout:(7/159): ceph-mon-19.0.0-2946.g3d89accf.el9.x86 10 MB/s | 4.7 MB 00:00 2024-06-10T00:13:21.092 INFO:teuthology.orchestra.run.smithi121.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 1.4 MB/s | 260 kB 00:00 2024-06-10T00:13:21.134 INFO:teuthology.orchestra.run.smithi121.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 6.7 MB/s | 1.3 MB 00:00 2024-06-10T00:13:21.301 INFO:teuthology.orchestra.run.smithi121.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 7.8 MB/s | 2.4 MB 00:00 2024-06-10T00:13:21.307 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-06-10T00:13:21.307 INFO:teuthology.orchestra.run.smithi121.stdout:Total 12 MB/s | 225 MB 00:19 2024-06-10T00:13:22.478 INFO:teuthology.orchestra.run.smithi193.stdout:(8/159): ceph-radosgw-19.0.0-2946.g3d89accf.el9 7.3 MB/s | 10 MB 00:01 2024-06-10T00:13:22.520 INFO:teuthology.orchestra.run.smithi193.stdout:(9/159): ceph-selinux-19.0.0-2946.g3d89accf.el9 603 kB/s | 25 kB 00:00 2024-06-10T00:13:22.548 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-06-10T00:13:22.702 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-06-10T00:13:22.703 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-06-10T00:13:22.787 INFO:teuthology.orchestra.run.smithi193.stdout:(10/159): ceph-common-19.0.0-2946.g3d89accf.el9 7.0 MB/s | 21 MB 00:03 2024-06-10T00:13:22.954 INFO:teuthology.orchestra.run.smithi193.stdout:(11/159): ceph-osd-19.0.0-2946.g3d89accf.el9.x8 8.6 MB/s | 17 MB 00:01 2024-06-10T00:13:22.979 INFO:teuthology.orchestra.run.smithi193.stdout:(12/159): cephfs-mirror-19.0.0-2946.g3d89accf.e 1.1 MB/s | 221 kB 00:00 2024-06-10T00:13:23.029 INFO:teuthology.orchestra.run.smithi193.stdout:(13/159): cephfs-shell-19.0.0-2946.g3d89accf.el 361 kB/s | 27 kB 00:00 2024-06-10T00:13:23.055 INFO:teuthology.orchestra.run.smithi193.stdout:(14/159): libcephfs-devel-19.0.0-2946.g3d89accf 421 kB/s | 32 kB 00:00 2024-06-10T00:13:23.100 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-06-10T00:13:23.136 INFO:teuthology.orchestra.run.smithi193.stdout:(15/159): libcephfs2-19.0.0-2946.g3d89accf.el9. 6.9 MB/s | 747 kB 00:00 2024-06-10T00:13:23.164 INFO:teuthology.orchestra.run.smithi193.stdout:(16/159): libcephsqlite-19.0.0-2946.g3d89accf.e 1.5 MB/s | 163 kB 00:00 2024-06-10T00:13:23.199 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-06-10T00:13:23.206 INFO:teuthology.orchestra.run.smithi193.stdout:(17/159): librados-devel-19.0.0-2946.g3d89accf. 1.8 MB/s | 127 kB 00:00 2024-06-10T00:13:23.237 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-06-10T00:13:23.247 INFO:teuthology.orchestra.run.smithi193.stdout:(18/159): libradosstriper1-19.0.0-2946.g3d89acc 5.9 MB/s | 501 kB 00:00 2024-06-10T00:13:23.290 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-06-10T00:13:23.325 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-06-10T00:13:23.331 INFO:teuthology.orchestra.run.smithi193.stdout:(19/159): python3-ceph-argparse-19.0.0-2946.g3d 542 kB/s | 45 kB 00:00 2024-06-10T00:13:23.359 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-06-10T00:13:23.401 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-06-10T00:13:23.431 INFO:teuthology.orchestra.run.smithi193.stdout:(20/159): python3-ceph-common-19.0.0-2946.g3d89 1.3 MB/s | 135 kB 00:00 2024-06-10T00:13:23.532 INFO:teuthology.orchestra.run.smithi193.stdout:(21/159): python3-cephfs-19.0.0-2946.g3d89accf. 1.6 MB/s | 162 kB 00:00 2024-06-10T00:13:23.599 INFO:teuthology.orchestra.run.smithi193.stdout:(22/159): librgw2-19.0.0-2946.g3d89accf.el9.x86 14 MB/s | 5.3 MB 00:00 2024-06-10T00:13:23.624 INFO:teuthology.orchestra.run.smithi193.stdout:(23/159): python3-rados-19.0.0-2946.g3d89accf.e 3.4 MB/s | 321 kB 00:00 2024-06-10T00:13:23.675 INFO:teuthology.orchestra.run.smithi193.stdout:(24/159): python3-rbd-19.0.0-2946.g3d89accf.el9 3.9 MB/s | 300 kB 00:00 2024-06-10T00:13:23.700 INFO:teuthology.orchestra.run.smithi193.stdout:(25/159): python3-rgw-19.0.0-2946.g3d89accf.el9 1.3 MB/s | 99 kB 00:00 2024-06-10T00:13:23.721 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-06-10T00:13:23.750 INFO:teuthology.orchestra.run.smithi193.stdout:(26/159): rbd-fuse-19.0.0-2946.g3d89accf.el9.x8 1.1 MB/s | 85 kB 00:00 2024-06-10T00:13:23.834 INFO:teuthology.orchestra.run.smithi193.stdout:(27/159): rbd-nbd-19.0.0-2946.g3d89accf.el9.x86 2.0 MB/s | 171 kB 00:00 2024-06-10T00:13:23.835 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-10T00:13:23.917 INFO:teuthology.orchestra.run.smithi193.stdout:(28/159): ceph-grafana-dashboards-19.0.0-2946.g 381 kB/s | 32 kB 00:00 2024-06-10T00:13:23.955 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-10T00:13:23.984 INFO:teuthology.orchestra.run.smithi193.stdout:(29/159): rbd-mirror-19.0.0-2946.g3d89accf.el9. 11 MB/s | 3.1 MB 00:00 2024-06-10T00:13:23.985 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-10T00:13:24.009 INFO:teuthology.orchestra.run.smithi193.stdout:(30/159): ceph-mgr-cephadm-19.0.0-2946.g3d89acc 1.5 MB/s | 142 kB 00:00 2024-06-10T00:13:24.077 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-10T00:13:24.107 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-10T00:13:24.139 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-06-10T00:13:24.326 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-06-10T00:13:24.368 INFO:teuthology.orchestra.run.smithi193.stdout:(31/159): ceph-mgr-dashboard-19.0.0-2946.g3d89a 9.2 MB/s | 3.5 MB 00:00 2024-06-10T00:13:24.379 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-06-10T00:13:24.407 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-10T00:13:24.443 INFO:teuthology.orchestra.run.smithi193.stdout:(32/159): ceph-mgr-modules-core-19.0.0-2946.g3d 3.3 MB/s | 249 kB 00:00 2024-06-10T00:13:24.511 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-10T00:13:24.533 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-10T00:13:24.535 INFO:teuthology.orchestra.run.smithi193.stdout:(33/159): ceph-mgr-rook-19.0.0-2946.g3d89accf.e 537 kB/s | 49 kB 00:00 2024-06-10T00:13:24.579 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-10T00:13:24.601 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-10T00:13:24.620 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-10T00:13:24.652 INFO:teuthology.orchestra.run.smithi193.stdout:(34/159): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-06-10T00:13:24.677 INFO:teuthology.orchestra.run.smithi193.stdout:(35/159): ceph-prometheus-alerts-19.0.0-2946.g3 116 kB/s | 16 kB 00:00 2024-06-10T00:13:24.727 INFO:teuthology.orchestra.run.smithi193.stdout:(36/159): ceph-volume-19.0.0-2946.g3d89accf.el9 3.4 MB/s | 264 kB 00:00 2024-06-10T00:13:24.768 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-10T00:13:24.803 INFO:teuthology.orchestra.run.smithi193.stdout:(37/159): cephadm-19.0.0-2946.g3d89accf.el9.noa 5.9 MB/s | 763 kB 00:00 2024-06-10T00:13:24.817 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-06-10T00:13:24.818 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-06-10T00:13:24.837 INFO:teuthology.orchestra.run.smithi193.stdout:(38/159): cephfs-top-19.0.0-2946.g3d89accf.el9. 167 kB/s | 18 kB 00:00 2024-06-10T00:13:24.891 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:24.892 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-10T00:13:24.897 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:25.048 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-10T00:13:25.092 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-10T00:13:25.183 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-10T00:13:25.277 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-10T00:13:25.362 INFO:teuthology.orchestra.run.smithi193.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 144 kB/s | 80 kB 00:00 2024-06-10T00:13:25.363 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:25.491 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-10T00:13:25.595 INFO:teuthology.orchestra.run.smithi193.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 838 kB/s | 195 kB 00:00 2024-06-10T00:13:25.599 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-06-10T00:13:25.662 INFO:teuthology.orchestra.run.smithi193.stdout:(41/159): libgfortran-11.4.1-3.el9.x86_64.rpm 973 kB/s | 803 kB 00:00 2024-06-10T00:13:25.695 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-06-10T00:13:25.724 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-06-10T00:13:25.763 INFO:teuthology.orchestra.run.smithi193.stdout:(42/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-06-10T00:13:25.767 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-06-10T00:13:25.832 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-06-10T00:13:25.867 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-06-10T00:13:25.910 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-06-10T00:13:25.938 INFO:teuthology.orchestra.run.smithi193.stdout:(43/159): python3-cryptography-36.0.1-4.el9.x86 4.5 MB/s | 1.2 MB 00:00 2024-06-10T00:13:25.943 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-06-10T00:13:25.963 INFO:teuthology.orchestra.run.smithi193.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 533 kB/s | 106 kB 00:00 2024-06-10T00:13:25.972 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-06-10T00:13:26.039 INFO:teuthology.orchestra.run.smithi193.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 135 kB 00:00 2024-06-10T00:13:26.040 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-06-10T00:13:26.064 INFO:teuthology.orchestra.run.smithi193.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-06-10T00:13:26.065 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] automake-1.16.2-8.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/automake-1.16.2-8.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-10T00:13:26.103 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-06-10T00:13:26.142 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-06-10T00:13:26.156 INFO:teuthology.orchestra.run.smithi193.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-06-10T00:13:26.156 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] bison-3.7.4-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/bison-3.7.4-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:26.178 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-06-10T00:13:26.253 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-06-10T00:13:26.288 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 34/170 2024-06-10T00:13:26.331 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 35/170 2024-06-10T00:13:26.360 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 36/170 2024-06-10T00:13:26.431 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-06-10T00:13:26.475 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-06-10T00:13:26.549 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-06-10T00:13:26.581 INFO:teuthology.orchestra.run.smithi193.stdout:(48/159): ceph-test-19.0.0-2946.g3d89accf.el9.x 12 MB/s | 48 MB 00:04 2024-06-10T00:13:26.582 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:26.623 INFO:teuthology.orchestra.run.smithi193.stdout:(49/159): automake-1.16.2-8.el9.noarch.rpm 1.2 MB/s | 689 kB 00:00 2024-06-10T00:13:26.665 INFO:teuthology.orchestra.run.smithi193.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 521 kB/s | 22 kB 00:00 2024-06-10T00:13:26.691 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-06-10T00:13:26.707 INFO:teuthology.orchestra.run.smithi193.stdout:(51/159): boost-program-options-1.75.0-8.el9.x8 854 kB/s | 107 kB 00:00 2024-06-10T00:13:26.732 INFO:teuthology.orchestra.run.smithi193.stdout:(52/159): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 313 kB 00:00 2024-06-10T00:13:26.752 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-06-10T00:13:26.757 INFO:teuthology.orchestra.run.smithi193.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 653 kB/s | 33 kB 00:00 2024-06-10T00:13:26.791 INFO:teuthology.orchestra.run.smithi193.stdout:(54/159): bison-3.7.4-5.el9.x86_64.rpm 1.5 MB/s | 943 kB 00:00 2024-06-10T00:13:26.808 INFO:teuthology.orchestra.run.smithi193.stdout:(55/159): flexiblas-openblas-openmp-3.0.4-8.el9 362 kB/s | 18 kB 00:00 2024-06-10T00:13:26.825 INFO:teuthology.orchestra.run.smithi193.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 332 kB/s | 11 kB 00:00 2024-06-10T00:13:26.858 INFO:teuthology.orchestra.run.smithi193.stdout:(57/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-06-10T00:13:26.884 INFO:teuthology.orchestra.run.smithi193.stdout:(58/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-06-10T00:13:26.968 INFO:teuthology.orchestra.run.smithi193.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-06-10T00:13:27.001 INFO:teuthology.orchestra.run.smithi193.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 770 kB/s | 25 kB 00:00 2024-06-10T00:13:27.043 INFO:teuthology.orchestra.run.smithi193.stdout:(61/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-06-10T00:13:27.077 INFO:teuthology.orchestra.run.smithi193.stdout:(62/159): libtool-2.4.6-45.el9.x86_64.rpm 3.0 MB/s | 582 kB 00:00 2024-06-10T00:13:27.152 INFO:teuthology.orchestra.run.smithi193.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-06-10T00:13:27.169 INFO:teuthology.orchestra.run.smithi193.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-06-10T00:13:27.194 INFO:teuthology.orchestra.run.smithi193.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 841 kB/s | 35 kB 00:00 2024-06-10T00:13:27.280 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-06-10T00:13:27.365 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-06-10T00:13:27.394 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-06-10T00:13:27.430 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-06-10T00:13:27.479 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-06-10T00:13:27.517 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 47/170 2024-06-10T00:13:27.558 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-06-10T00:13:27.586 INFO:teuthology.orchestra.run.smithi193.stdout:(66/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.5 MB/s | 3.0 MB 00:00 2024-06-10T00:13:27.721 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-06-10T00:13:27.769 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-06-10T00:13:27.824 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-06-10T00:13:27.898 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-06-10T00:13:27.928 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-06-10T00:13:27.965 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-06-10T00:13:28.012 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-06-10T00:13:28.047 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-06-10T00:13:28.070 INFO:teuthology.orchestra.run.smithi193.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 3.3 MB/s | 1.6 MB 00:00 2024-06-10T00:13:28.091 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-06-10T00:13:28.133 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-06-10T00:13:28.269 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-06-10T00:13:28.309 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-06-10T00:13:28.345 INFO:teuthology.orchestra.run.smithi193.stdout:(68/159): postgresql-contrib-13.14-1.el9.x86_64 3.2 MB/s | 911 kB 00:00 2024-06-10T00:13:28.351 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-06-10T00:13:28.420 INFO:teuthology.orchestra.run.smithi193.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-06-10T00:13:28.445 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-06-10T00:13:28.512 INFO:teuthology.orchestra.run.smithi193.stdout:(70/159): openblas-openmp-0.3.21-2.el9.x86_64.r 3.5 MB/s | 4.7 MB 00:01 2024-06-10T00:13:28.579 INFO:teuthology.orchestra.run.smithi193.stdout:(71/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.2 MB/s | 4.4 MB 00:01 2024-06-10T00:13:28.621 INFO:teuthology.orchestra.run.smithi193.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-06-10T00:13:28.686 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 63/170 2024-06-10T00:13:28.819 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-06-10T00:13:28.850 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-06-10T00:13:28.875 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-06-10T00:13:28.888 INFO:teuthology.orchestra.run.smithi193.stdout:(73/159): protobuf-3.14.0-13.el9.x86_64.rpm 2.7 MB/s | 1.0 MB 00:00 2024-06-10T00:13:28.930 INFO:teuthology.orchestra.run.smithi193.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-06-10T00:13:28.958 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-06-10T00:13:29.022 INFO:teuthology.orchestra.run.smithi193.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.6 MB/s | 244 kB 00:00 2024-06-10T00:13:29.088 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-06-10T00:13:29.122 INFO:teuthology.orchestra.run.smithi193.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-06-10T00:13:29.157 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-06-10T00:13:29.164 INFO:teuthology.orchestra.run.smithi193.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 616 kB/s | 26 kB 00:00 2024-06-10T00:13:29.185 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-06-10T00:13:29.206 INFO:teuthology.orchestra.run.smithi193.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 462 kB/s | 19 kB 00:00 2024-06-10T00:13:29.227 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-06-10T00:13:29.267 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-06-10T00:13:29.281 INFO:teuthology.orchestra.run.smithi193.stdout:(79/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-06-10T00:13:29.298 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-06-10T00:13:29.328 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-06-10T00:13:29.340 INFO:teuthology.orchestra.run.smithi193.stdout:(80/159): python3-markupsafe-1.1.1-12.el9.x86_6 597 kB/s | 35 kB 00:00 2024-06-10T00:13:29.362 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-06-10T00:13:29.395 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-06-10T00:13:29.437 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-06-10T00:13:29.472 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-06-10T00:13:29.559 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-06-10T00:13:29.602 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-06-10T00:13:29.633 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-06-10T00:13:29.681 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-06-10T00:13:29.779 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-06-10T00:13:29.826 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-06-10T00:13:30.290 INFO:teuthology.orchestra.run.smithi193.stdout:(81/159): postgresql-server-13.14-1.el9.x86_64. 3.1 MB/s | 5.8 MB 00:01 2024-06-10T00:13:30.407 INFO:teuthology.orchestra.run.smithi193.stdout:(82/159): python3-babel-2.9.1-2.el9.noarch.rpm 3.3 MB/s | 6.0 MB 00:01 2024-06-10T00:13:30.433 INFO:teuthology.orchestra.run.smithi193.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 250 kB 00:00 2024-06-10T00:13:30.491 INFO:teuthology.orchestra.run.smithi193.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-06-10T00:13:30.516 INFO:teuthology.orchestra.run.smithi193.stdout:(85/159): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-06-10T00:13:30.542 INFO:teuthology.orchestra.run.smithi193.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 842 kB/s | 42 kB 00:00 2024-06-10T00:13:30.554 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-06-10T00:13:30.598 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-06-10T00:13:30.632 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-06-10T00:13:30.633 INFO:teuthology.orchestra.run.smithi193.stdout:(87/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-06-10T00:13:30.659 INFO:teuthology.orchestra.run.smithi193.stdout:(88/159): python3-protobuf-3.14.0-13.el9.noarch 1.9 MB/s | 269 kB 00:00 2024-06-10T00:13:30.726 INFO:teuthology.orchestra.run.smithi193.stdout:(89/159): python3-pytz-2021.1-5.el9.noarch.rpm 765 kB/s | 51 kB 00:00 2024-06-10T00:13:30.733 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-06-10T00:13:30.767 INFO:teuthology.orchestra.run.smithi193.stdout:(90/159): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-06-10T00:13:30.787 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-06-10T00:13:30.793 INFO:teuthology.orchestra.run.smithi193.stdout:(91/159): python3-pyasn1-modules-0.4.8-6.el9.no 1.7 MB/s | 279 kB 00:00 2024-06-10T00:13:30.832 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-06-10T00:13:30.835 INFO:teuthology.orchestra.run.smithi193.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-06-10T00:13:30.877 INFO:teuthology.orchestra.run.smithi193.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-06-10T00:13:30.920 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-06-10T00:13:30.960 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-06-10T00:13:30.969 INFO:teuthology.orchestra.run.smithi193.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 2.4 MB/s | 221 kB 00:00 2024-06-10T00:13:31.036 INFO:teuthology.orchestra.run.smithi193.stdout:(95/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.0 MB/s | 5.1 MB 00:01 2024-06-10T00:13:31.070 INFO:teuthology.orchestra.run.smithi193.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 356 kB/s | 36 kB 00:00 2024-06-10T00:13:31.095 INFO:teuthology.orchestra.run.smithi193.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 806 kB/s | 47 kB 00:00 2024-06-10T00:13:31.136 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-06-10T00:13:31.137 INFO:teuthology.orchestra.run.smithi193.stdout:(98/159): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-06-10T00:13:31.174 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-06-10T00:13:31.193 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-06-10T00:13:31.195 INFO:teuthology.orchestra.run.smithi193.stdout:(99/159): socat-1.7.4.1-5.el9.x86_64.rpm 2.4 MB/s | 305 kB 00:00 2024-06-10T00:13:31.213 INFO:teuthology.orchestra.run.smithi193.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-06-10T00:13:31.252 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-06-10T00:13:31.254 INFO:teuthology.orchestra.run.smithi193.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-06-10T00:13:31.272 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-06-10T00:13:31.280 INFO:teuthology.orchestra.run.smithi193.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 670 kB/s | 45 kB 00:00 2024-06-10T00:13:31.280 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:31.281 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-10T00:13:31.316 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-06-10T00:13:31.377 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-06-10T00:13:31.396 INFO:teuthology.orchestra.run.smithi193.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-06-10T00:13:31.414 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-06-10T00:13:31.430 INFO:teuthology.orchestra.run.smithi193.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 16 MB/s | 548 kB 00:00 2024-06-10T00:13:31.439 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-06-10T00:13:31.472 INFO:teuthology.orchestra.run.smithi193.stdout:(105/159): gperftools-libs-2.9.1-2.el9.x86_64.r 7.3 MB/s | 309 kB 00:00 2024-06-10T00:13:31.492 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-06-10T00:13:31.497 INFO:teuthology.orchestra.run.smithi193.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 780 kB/s | 19 kB 00:00 2024-06-10T00:13:31.606 INFO:teuthology.orchestra.run.smithi193.stdout:(107/159): libarrow-9.0.0-11.el9.x86_64.rpm 41 MB/s | 4.4 MB 00:00 2024-06-10T00:13:31.642 INFO:teuthology.orchestra.run.smithi193.stdout:(108/159): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-06-10T00:13:31.665 INFO:teuthology.orchestra.run.smithi193.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-06-10T00:13:31.715 INFO:teuthology.orchestra.run.smithi193.stdout:(110/159): libunwind-1.6.2-1.el9.x86_64.rpm 2.8 MB/s | 67 kB 00:00 2024-06-10T00:13:31.749 INFO:teuthology.orchestra.run.smithi193.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 4.8 MB/s | 151 kB 00:00 2024-06-10T00:13:31.774 INFO:teuthology.orchestra.run.smithi193.stdout:(112/159): liboath-2.6.7-2.el9.x86_64.rpm 362 kB/s | 49 kB 00:00 2024-06-10T00:13:31.821 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-06-10T00:13:31.824 INFO:teuthology.orchestra.run.smithi193.stdout:(113/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-06-10T00:13:31.849 INFO:teuthology.orchestra.run.smithi193.stdout:(114/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-06-10T00:13:31.875 INFO:teuthology.orchestra.run.smithi193.stdout:(115/159): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-06-10T00:13:31.900 INFO:teuthology.orchestra.run.smithi193.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-06-10T00:13:31.904 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-06-10T00:13:31.924 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-10T00:13:31.925 INFO:teuthology.orchestra.run.smithi193.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-06-10T00:13:31.951 INFO:teuthology.orchestra.run.smithi193.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-06-10T00:13:31.976 INFO:teuthology.orchestra.run.smithi193.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-06-10T00:13:32.001 INFO:teuthology.orchestra.run.smithi193.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-06-10T00:13:32.013 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-06-10T00:13:32.027 INFO:teuthology.orchestra.run.smithi193.stdout:(121/159): python3-google-auth-2.29.0-2.el9.noa 4.0 MB/s | 205 kB 00:00 2024-06-10T00:13:32.055 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-10T00:13:32.060 INFO:teuthology.orchestra.run.smithi193.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-06-10T00:13:32.084 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-10T00:13:32.094 INFO:teuthology.orchestra.run.smithi193.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 323 kB/s | 11 kB 00:00 2024-06-10T00:13:32.119 INFO:teuthology.orchestra.run.smithi193.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 705 kB/s | 18 kB 00:00 2024-06-10T00:13:32.153 INFO:teuthology.orchestra.run.smithi193.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 694 kB/s | 23 kB 00:00 2024-06-10T00:13:32.161 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-10T00:13:32.178 INFO:teuthology.orchestra.run.smithi193.stdout:(126/159): python3-jaraco-functools-3.5.0-2.el9 777 kB/s | 19 kB 00:00 2024-06-10T00:13:32.188 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-10T00:13:32.204 INFO:teuthology.orchestra.run.smithi193.stdout:(127/159): python3-jaraco-text-3.2.0-6.el9.noar 795 kB/s | 20 kB 00:00 2024-06-10T00:13:32.229 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-06-10T00:13:32.271 INFO:teuthology.orchestra.run.smithi193.stdout:(128/159): python3-grpcio-1.46.7-10.el9.x86_64. 7.6 MB/s | 2.0 MB 00:00 2024-06-10T00:13:32.296 INFO:teuthology.orchestra.run.smithi193.stdout:(129/159): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-06-10T00:13:32.321 INFO:teuthology.orchestra.run.smithi193.stdout:(130/159): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-06-10T00:13:32.355 INFO:teuthology.orchestra.run.smithi193.stdout:(131/159): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-06-10T00:13:32.372 INFO:teuthology.orchestra.run.smithi193.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-06-10T00:13:32.388 INFO:teuthology.orchestra.run.smithi193.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 997 kB/s | 16 kB 00:00 2024-06-10T00:13:32.408 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-06-10T00:13:32.414 INFO:teuthology.orchestra.run.smithi193.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-06-10T00:13:32.431 INFO:teuthology.orchestra.run.smithi193.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-06-10T00:13:32.448 INFO:teuthology.orchestra.run.smithi193.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 1.7 MB/s | 28 kB 00:00 2024-06-10T00:13:32.470 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-06-10T00:13:32.473 INFO:teuthology.orchestra.run.smithi193.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 737 kB/s | 31 kB 00:00 2024-06-10T00:13:32.498 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-10T00:13:32.498 INFO:teuthology.orchestra.run.smithi193.stdout:(138/159): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-06-10T00:13:32.524 INFO:teuthology.orchestra.run.smithi193.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-06-10T00:13:32.549 INFO:teuthology.orchestra.run.smithi193.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 716 kB/s | 36 kB 00:00 2024-06-10T00:13:32.574 INFO:teuthology.orchestra.run.smithi193.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 2.0 MB/s | 51 kB 00:00 2024-06-10T00:13:32.601 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-10T00:13:32.607 INFO:teuthology.orchestra.run.smithi193.stdout:(142/159): python3-websocket-client-1.2.3-2.el9 2.7 MB/s | 90 kB 00:00 2024-06-10T00:13:32.623 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-10T00:13:32.650 INFO:teuthology.orchestra.run.smithi193.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-06-10T00:13:32.675 INFO:teuthology.orchestra.run.smithi193.stdout:(144/159): python3-zc-lockfile-2.0-10.el9.noarc 811 kB/s | 20 kB 00:00 2024-06-10T00:13:32.686 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-10T00:13:32.700 INFO:teuthology.orchestra.run.smithi193.stdout:(145/159): re2-20211101-3.el9.x86_64.rpm 7.5 MB/s | 192 kB 00:00 2024-06-10T00:13:32.707 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-10T00:13:32.725 INFO:teuthology.orchestra.run.smithi193.stdout:(146/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-06-10T00:13:32.726 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-10T00:13:32.776 INFO:teuthology.orchestra.run.smithi193.stdout:(147/159): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-06-10T00:13:32.918 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-10T00:13:33.034 INFO:teuthology.orchestra.run.smithi193.stdout:(148/159): librados2-19.0.0-2946.g3d89accf.el9. 11 MB/s | 3.4 MB 00:00 2024-06-10T00:13:33.110 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-10T00:13:33.193 INFO:teuthology.orchestra.run.smithi193.stdout:(149/159): librbd1-19.0.0-2946.g3d89accf.el9.x8 7.5 MB/s | 3.1 MB 00:00 2024-06-10T00:13:33.235 INFO:teuthology.orchestra.run.smithi193.stdout:(150/159): NetworkManager-1.47.91-1.el9.x86_64. 12 MB/s | 2.3 MB 00:00 2024-06-10T00:13:33.244 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-06-10T00:13:33.280 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-10T00:13:33.327 INFO:teuthology.orchestra.run.smithi193.stdout:(151/159): NetworkManager-libnm-1.47.91-1.el9.x 14 MB/s | 1.8 MB 00:00 2024-06-10T00:13:33.327 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-06-10T00:13:33.340 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-10T00:13:33.352 INFO:teuthology.orchestra.run.smithi193.stdout:(152/159): NetworkManager-team-1.47.91-1.el9.x8 354 kB/s | 41 kB 00:00 2024-06-10T00:13:33.355 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-06-10T00:13:33.388 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-06-10T00:13:33.427 INFO:teuthology.orchestra.run.smithi193.stdout:(153/159): NetworkManager-tui-1.47.91-1.el9.x86 2.4 MB/s | 248 kB 00:00 2024-06-10T00:13:33.452 INFO:teuthology.orchestra.run.smithi193.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 366 kB/s | 37 kB 00:00 2024-06-10T00:13:33.456 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-10T00:13:33.525 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-10T00:13:33.536 INFO:teuthology.orchestra.run.smithi193.stdout:(155/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.9 MB/s | 207 kB 00:00 2024-06-10T00:13:33.561 INFO:teuthology.orchestra.run.smithi193.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 2.3 MB/s | 260 kB 00:00 2024-06-10T00:13:33.662 INFO:teuthology.orchestra.run.smithi193.stdout:(157/159): openssl-3.2.1-1.el9.x86_64.rpm 11 MB/s | 1.3 MB 00:00 2024-06-10T00:13:33.701 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-06-10T00:13:33.712 INFO:teuthology.orchestra.run.smithi193.stdout:(158/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-06-10T00:13:33.742 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-10T00:13:33.742 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-10T00:13:33.833 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-10T00:13:33.852 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-06-10T00:13:33.864 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 104/170 2024-06-10T00:13:33.900 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-06-10T00:13:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-06-10T00:13:33.990 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-06-10T00:13:34.024 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-06-10T00:13:34.089 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-06-10T00:13:34.124 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-06-10T00:13:34.167 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-06-10T00:13:34.208 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-06-10T00:13:34.236 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-06-10T00:13:34.305 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-06-10T00:13:34.358 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-06-10T00:13:34.369 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-06-10T00:13:34.389 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-06-10T00:13:34.404 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-10T00:13:34.433 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-06-10T00:13:34.474 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-10T00:13:34.497 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 108/170 2024-06-10T00:13:34.508 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-06-10T00:13:34.543 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 34/170 2024-06-10T00:13:34.587 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 35/170 2024-06-10T00:13:34.615 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 36/170 2024-06-10T00:13:34.677 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-06-10T00:13:34.730 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-06-10T00:13:34.804 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-06-10T00:13:34.936 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-06-10T00:13:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-06-10T00:13:35.505 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-06-10T00:13:35.604 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-06-10T00:13:35.632 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-06-10T00:13:35.685 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-06-10T00:13:35.726 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-06-10T00:13:35.764 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 47/170 2024-06-10T00:13:35.805 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-06-10T00:13:35.950 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-06-10T00:13:36.017 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-06-10T00:13:36.071 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-06-10T00:13:36.137 INFO:teuthology.orchestra.run.smithi193.stdout:(159/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.0 MB/s | 16 MB 00:05 2024-06-10T00:13:36.144 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-06-10T00:13:36.144 INFO:teuthology.orchestra.run.smithi193.stdout:Total 13 MB/s | 225 MB 00:16 2024-06-10T00:13:36.144 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-06-10T00:13:36.176 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-06-10T00:13:36.220 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-06-10T00:13:36.267 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-06-10T00:13:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-06-10T00:13:36.346 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-06-10T00:13:36.388 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-06-10T00:13:36.522 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-06-10T00:13:36.566 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-06-10T00:13:36.606 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-06-10T00:13:36.699 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-06-10T00:13:36.779 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-10T00:13:36.824 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 63/170 2024-06-10T00:13:36.827 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-10T00:13:36.866 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-06-10T00:13:36.897 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-06-10T00:13:36.922 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-06-10T00:13:37.004 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-06-10T00:13:37.132 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-06-10T00:13:37.171 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-06-10T00:13:37.199 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-06-10T00:13:37.239 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-06-10T00:13:37.281 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-06-10T00:13:37.312 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-06-10T00:13:37.342 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-06-10T00:13:37.359 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-06-10T00:13:37.364 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-10T00:13:37.368 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-06-10T00:13:37.397 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-10T00:13:37.402 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-06-10T00:13:37.443 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-06-10T00:13:37.478 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-06-10T00:13:37.486 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-10T00:13:37.486 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-10T00:13:37.486 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-10T00:13:37.486 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:37.514 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-10T00:13:37.516 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-06-10T00:13:37.516 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-06-10T00:13:37.564 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-06-10T00:13:37.609 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-06-10T00:13:37.639 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-06-10T00:13:37.686 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-06-10T00:13:37.792 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-06-10T00:13:37.851 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-06-10T00:13:38.564 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-06-10T00:13:38.606 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-06-10T00:13:38.640 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-06-10T00:13:38.739 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-06-10T00:13:38.787 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-06-10T00:13:38.832 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-06-10T00:13:38.919 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-06-10T00:13:38.951 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-06-10T00:13:39.128 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-06-10T00:13:39.158 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-06-10T00:13:39.239 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-06-10T00:13:39.319 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-06-10T00:13:39.319 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-06-10T00:13:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-06-10T00:13:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-06-10T00:13:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-06-10T00:13:41.336 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-06-10T00:13:41.364 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-06-10T00:13:41.397 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-06-10T00:13:41.709 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-06-10T00:13:41.751 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-10T00:13:41.843 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-10T00:13:41.873 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 104/170 2024-06-10T00:13:41.909 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-06-10T00:13:42.379 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-06-10T00:13:42.405 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-10T00:13:42.459 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-10T00:13:42.498 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 108/170 2024-06-10T00:13:44.774 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-10T00:13:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-10T00:13:45.349 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-10T00:13:45.389 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-10T00:13:45.465 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-10T00:13:45.465 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-10T00:13:45.465 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-10T00:13:45.465 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:13:45.490 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-10T00:13:45.600 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-06-10T00:13:45.691 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-06-10T00:13:45.729 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-06-10T00:13:45.782 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-06-10T00:13:45.816 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-06-10T00:13:45.876 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-06-10T00:13:45.918 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-06-10T00:13:46.240 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-06-10T00:13:46.344 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-10T00:13:46.464 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-10T00:13:46.494 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-10T00:13:46.586 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-10T00:13:46.615 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-10T00:13:46.647 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-06-10T00:13:46.839 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-06-10T00:13:46.888 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-06-10T00:13:46.916 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-10T00:13:47.020 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-10T00:13:47.042 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-10T00:13:47.105 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 15/170 2024-06-10T00:13:47.127 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-10T00:13:47.145 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-06-10T00:13:47.335 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-10T00:13:47.465 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-06-10T00:13:47.623 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-10T00:13:47.675 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 18/170 2024-06-10T00:13:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-10T00:13:47.852 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-10T00:13:48.044 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 19/170 2024-06-10T00:13:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-06-10T00:13:48.244 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-06-10T00:13:48.291 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-06-10T00:13:48.325 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-06-10T00:13:48.389 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-06-10T00:13:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-06-10T00:13:48.468 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-06-10T00:13:48.501 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-06-10T00:13:48.529 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-06-10T00:13:48.597 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-06-10T00:13:48.653 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-06-10T00:13:48.683 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-06-10T00:13:48.719 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-06-10T00:13:48.795 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-06-10T00:13:48.830 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 34/170 2024-06-10T00:13:48.873 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 35/170 2024-06-10T00:13:48.901 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 36/170 2024-06-10T00:13:48.972 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-06-10T00:13:49.025 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-06-10T00:13:49.099 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-06-10T00:13:49.231 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-06-10T00:13:49.294 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-06-10T00:13:49.382 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-10T00:13:49.382 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /sys 2024-06-10T00:13:49.383 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /proc 2024-06-10T00:13:49.383 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /mnt 2024-06-10T00:13:49.383 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /var/tmp 2024-06-10T00:13:49.383 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /home 2024-06-10T00:13:49.383 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /root 2024-06-10T00:13:49.383 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /tmp 2024-06-10T00:13:49.383 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:49.552 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-10T00:13:49.567 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-10T00:13:49.568 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:49.568 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-10T00:13:49.568 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:49.809 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-06-10T00:13:49.889 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-06-10T00:13:49.919 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-06-10T00:13:49.955 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-06-10T00:13:49.995 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-06-10T00:13:50.034 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 47/170 2024-06-10T00:13:50.076 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-06-10T00:13:50.230 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-06-10T00:13:50.287 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-06-10T00:13:50.342 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-06-10T00:13:50.414 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-06-10T00:13:50.446 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-06-10T00:13:50.482 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-06-10T00:13:50.530 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-06-10T00:13:50.565 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-06-10T00:13:50.608 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-06-10T00:13:50.651 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-06-10T00:13:50.785 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-06-10T00:13:50.827 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-06-10T00:13:50.868 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-06-10T00:13:50.963 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-06-10T00:13:50.994 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 63/170 2024-06-10T00:13:51.019 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-10T00:13:51.026 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-06-10T00:13:51.044 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-10T00:13:51.045 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:51.045 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-10T00:13:51.045 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-10T00:13:51.045 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-10T00:13:51.045 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:51.051 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-06-10T00:13:51.076 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-06-10T00:13:51.151 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-06-10T00:13:51.277 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-06-10T00:13:51.278 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-10T00:13:51.301 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-10T00:13:51.301 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:51.301 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-10T00:13:51.301 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-10T00:13:51.301 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-10T00:13:51.301 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:51.317 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-06-10T00:13:51.344 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-06-10T00:13:51.387 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-06-10T00:13:51.426 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-06-10T00:13:51.458 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-06-10T00:13:51.584 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-06-10T00:13:51.737 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-10T00:13:51.740 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-06-10T00:13:51.759 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-10T00:13:51.759 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:51.759 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-10T00:13:51.760 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-10T00:13:51.760 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-10T00:13:51.760 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:51.773 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-06-10T00:13:51.815 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-06-10T00:13:51.849 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-06-10T00:13:51.867 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-06-10T00:13:51.915 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-06-10T00:13:51.939 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-06-10T00:13:51.971 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-06-10T00:13:51.988 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-06-10T00:13:52.019 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-06-10T00:13:52.067 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-06-10T00:13:52.165 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-06-10T00:13:52.172 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-06-10T00:13:52.221 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-06-10T00:13:52.237 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-06-10T00:13:52.315 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-06-10T00:13:52.403 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-06-10T00:13:52.542 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-06-10T00:13:52.604 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-06-10T00:13:52.645 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-06-10T00:13:52.672 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-06-10T00:13:52.962 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-06-10T00:13:53.009 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-06-10T00:13:53.042 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-06-10T00:13:53.091 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-10T00:13:53.120 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-10T00:13:53.142 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-06-10T00:13:53.152 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 128/170 2024-06-10T00:13:53.197 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-06-10T00:13:53.214 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 129/170 2024-06-10T00:13:53.243 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-06-10T00:13:53.252 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-10T00:13:53.329 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-06-10T00:13:53.362 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-06-10T00:13:53.563 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-06-10T00:13:53.585 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-06-10T00:13:53.661 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-06-10T00:13:54.280 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-10T00:13:54.304 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-06-10T00:13:54.320 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-10T00:13:54.424 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-06-10T00:13:55.281 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-10T00:13:55.321 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-10T00:13:55.422 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-10T00:13:55.584 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 133/170 2024-06-10T00:13:55.614 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-10T00:13:55.637 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-10T00:13:55.637 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:55.637 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-10T00:13:55.637 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-10T00:13:55.638 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-10T00:13:55.638 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:55.646 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-06-10T00:13:55.681 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-10T00:13:55.693 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-10T00:13:55.790 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 136/170 2024-06-10T00:13:55.838 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 137/170 2024-06-10T00:13:55.979 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-06-10T00:13:56.008 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-06-10T00:13:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-06-10T00:13:56.353 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-06-10T00:13:56.399 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-10T00:13:56.487 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 103/170 2024-06-10T00:13:56.517 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 104/170 2024-06-10T00:13:56.553 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-06-10T00:13:56.730 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-06-10T00:13:56.767 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-10T00:13:56.790 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-10T00:13:56.790 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:56.790 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-10T00:13:56.790 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-10T00:13:56.790 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-10T00:13:56.790 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:56.791 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-10T00:13:56.791 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-06-10T00:13:56.792 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-06-10T00:13:56.792 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-06-10T00:13:56.792 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-06-10T00:13:56.792 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-06-10T00:13:56.792 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-06-10T00:13:56.792 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-06-10T00:13:56.792 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:13:56.835 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-10T00:13:56.856 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-10T00:13:56.857 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:56.857 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-10T00:13:56.857 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:56.902 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-10T00:13:56.924 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-10T00:13:56.924 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:56.924 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-10T00:13:56.924 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-10T00:13:56.924 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-10T00:13:56.925 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:56.955 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-10T00:13:56.971 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-10T00:13:56.971 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:56.971 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-10T00:13:56.971 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:13:57.017 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-06-10T00:13:57.049 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-10T00:13:57.103 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 107/170 2024-06-10T00:13:57.126 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 108/170 2024-06-10T00:13:57.227 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-10T00:13:57.250 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-10T00:13:57.250 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:57.250 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-10T00:13:57.250 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-10T00:13:57.251 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-10T00:13:57.251 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:13:58.457 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-10T00:13:58.484 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-10T00:13:58.484 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:58.484 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-10T00:13:58.484 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-10T00:13:58.484 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-10T00:13:58.484 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:13:58.708 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-10T00:13:58.731 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-10T00:13:58.731 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:58.732 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-10T00:13:58.732 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-10T00:13:58.732 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-10T00:13:58.732 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:13:59.184 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-10T00:13:59.207 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-10T00:13:59.208 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:13:59.208 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-10T00:13:59.208 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-10T00:13:59.208 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-10T00:13:59.208 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:13:59.314 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-06-10T00:13:59.371 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-06-10T00:13:59.427 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-06-10T00:13:59.428 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-10T00:13:59.464 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-10T00:13:59.612 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-06-10T00:13:59.660 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-06-10T00:13:59.738 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-06-10T00:13:59.827 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-06-10T00:13:59.966 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-06-10T00:14:00.002 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 109/170 2024-06-10T00:14:00.009 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-06-10T00:14:00.041 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-06-10T00:14:00.042 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-10T00:14:00.078 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-06-10T00:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 110/170 2024-06-10T00:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-10T00:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-10T00:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-10T00:14:00.523 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-10T00:14:00.551 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-10T00:14:00.583 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 128/170 2024-06-10T00:14:00.644 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 129/170 2024-06-10T00:14:00.674 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-10T00:14:01.703 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-10T00:14:01.743 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-10T00:14:01.757 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 143/170 2024-06-10T00:14:01.797 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 144/170 2024-06-10T00:14:01.838 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 145/170 2024-06-10T00:14:01.870 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-06-10T00:14:01.898 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-06-10T00:14:02.084 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 148/170 2024-06-10T00:14:02.127 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-06-10T00:14:02.160 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 150/170 2024-06-10T00:14:02.192 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-06-10T00:14:02.230 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : NetworkManager-team-1:1.47.91-1.el9.x86_64 152/170 2024-06-10T00:14:02.276 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : NetworkManager-tui-1:1.47.91-1.el9.x86_64 153/170 2024-06-10T00:14:02.704 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-10T00:14:02.737 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-10T00:14:02.839 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-10T00:14:02.849 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-10T00:14:02.890 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-10T00:14:02.910 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-10T00:14:03.000 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 133/170 2024-06-10T00:14:03.031 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-10T00:14:03.055 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-10T00:14:03.055 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:03.055 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-10T00:14:03.055 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-10T00:14:03.055 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-10T00:14:03.056 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:14:03.107 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-10T00:14:03.118 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-10T00:14:03.216 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 136/170 2024-06-10T00:14:03.273 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 137/170 2024-06-10T00:14:03.738 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 155/170 2024-06-10T00:14:03.837 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-06-10T00:14:03.886 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-06-10T00:14:03.976 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-06-10T00:14:04.041 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-06-10T00:14:04.065 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-10T00:14:04.090 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-06-10T00:14:04.117 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-06-10T00:14:04.117 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-10T00:14:04.166 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-06-10T00:14:04.192 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-10T00:14:04.193 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-10T00:14:04.216 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-10T00:14:04.216 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:04.216 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-10T00:14:04.216 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-10T00:14:04.216 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-10T00:14:04.216 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:14:04.217 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-06-10T00:14:04.218 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-10T00:14:04.268 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-10T00:14:04.285 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-10T00:14:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-10T00:14:04.291 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:04.291 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-10T00:14:04.291 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:14:04.310 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-06-10T00:14:04.315 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-10T00:14:04.315 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-10T00:14:04.336 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-10T00:14:04.359 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-10T00:14:04.359 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:04.360 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-10T00:14:04.360 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-10T00:14:04.360 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-10T00:14:04.360 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:14:04.496 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-10T00:14:04.496 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-10T00:14:04.580 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-10T00:14:04.605 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-06-10T00:14:04.661 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-10T00:14:04.670 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-10T00:14:04.684 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-10T00:14:04.684 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:04.684 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-10T00:14:04.684 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-10T00:14:04.685 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-10T00:14:04.685 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:14:09.214 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 143/170 2024-06-10T00:14:09.256 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 144/170 2024-06-10T00:14:09.296 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 145/170 2024-06-10T00:14:09.328 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-06-10T00:14:09.356 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-06-10T00:14:09.565 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 148/170 2024-06-10T00:14:09.611 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-06-10T00:14:09.644 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 150/170 2024-06-10T00:14:09.683 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-06-10T00:14:09.713 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-team-1:1.47.91-1.el9.x86_64 152/170 2024-06-10T00:14:09.759 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-tui-1:1.47.91-1.el9.x86_64 153/170 2024-06-10T00:14:10.359 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-10T00:14:10.396 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-10T00:14:10.396 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 1/170 2024-06-10T00:14:10.396 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2/170 2024-06-10T00:14:10.396 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 3/170 2024-06-10T00:14:10.396 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 4/170 2024-06-10T00:14:10.396 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 5/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 6/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 7/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 11/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 12/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 13/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 15/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 16/170 2024-06-10T00:14:10.397 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 17/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 18/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 19/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 20/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 21/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 22/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 23/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 24/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 25/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 26/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 27/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 28/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 29/170 2024-06-10T00:14:10.398 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 30/170 2024-06-10T00:14:10.407 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-10T00:14:10.412 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 31/170 2024-06-10T00:14:10.412 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 32/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 33/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 34/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 35/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 36/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 37/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 38/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 39/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-06-10T00:14:10.413 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-06-10T00:14:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-06-10T00:14:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-06-10T00:14:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-06-10T00:14:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-06-10T00:14:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-06-10T00:14:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-06-10T00:14:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-06-10T00:14:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-06-10T00:14:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-06-10T00:14:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-06-10T00:14:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-06-10T00:14:10.415 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-06-10T00:14:10.415 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-06-10T00:14:10.415 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-06-10T00:14:10.415 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-06-10T00:14:10.415 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-06-10T00:14:10.415 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-06-10T00:14:10.415 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-06-10T00:14:10.415 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-06-10T00:14:10.415 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-06-10T00:14:10.415 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-06-10T00:14:10.416 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-06-10T00:14:10.417 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-06-10T00:14:10.418 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-06-10T00:14:10.419 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 122/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-06-10T00:14:10.420 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-06-10T00:14:10.421 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 149/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 151/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-1:1.47.91-1.el9.x86_64 153/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 155/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-team-1:1.47.91-1.el9.x86_64 157/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-tui-1:1.47.91-1.el9.x86_64 159/170 2024-06-10T00:14:10.422 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-10T00:14:10.423 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-06-10T00:14:10.423 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-06-10T00:14:10.423 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-06-10T00:14:10.423 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-06-10T00:14:10.423 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-06-10T00:14:10.423 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-06-10T00:14:10.423 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-06-10T00:14:10.423 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-06-10T00:14:10.423 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-06-10T00:14:10.424 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-10T00:14:11.243 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 155/170 2024-06-10T00:14:11.321 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-06-10T00:14:11.362 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-06-10T00:14:11.449 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-06-10T00:14:11.483 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-06-10T00:14:11.508 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-10T00:14:11.533 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-06-10T00:14:11.559 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-06-10T00:14:11.559 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-10T00:14:11.644 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-10T00:14:11.685 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-06-10T00:14:11.686 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-10T00:14:11.721 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 111/170 2024-06-10T00:14:11.721 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-06-10T00:14:11.721 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-06-10T00:14:11.721 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-06-10T00:14:11.721 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-06-10T00:14:11.721 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-06-10T00:14:11.721 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /root 2024-06-10T00:14:11.721 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-06-10T00:14:11.721 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:11.770 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-10T00:14:11.795 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-06-10T00:14:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-10T00:14:11.805 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-10T00:14:11.882 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-10T00:14:11.899 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 112/170 2024-06-10T00:14:11.899 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:11.899 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-10T00:14:11.899 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:11.981 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-10T00:14:11.981 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-10T00:14:12.065 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-10T00:14:12.089 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-06-10T00:14:12.154 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-10T00:14:12.926 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout:Upgraded: 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-team-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: NetworkManager-tui-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.927 INFO:teuthology.orchestra.run.smithi135.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: automake-1.16.2-8.el9.noarch 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: bison-3.7.4-5.el9.x86_64 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.928 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.929 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: flex-2.6.4-9.el9.x86_64 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-10T00:14:12.930 INFO:teuthology.orchestra.run.smithi135.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.931 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: libtool-2.4.6-45.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: lua-5.4.4-4.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-10T00:14:12.932 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-13.14-1.el9.x86_64 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.933 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-10T00:14:12.934 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-10T00:14:12.935 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ply-3.11-14.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-10T00:14:12.936 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-10T00:14:12.937 INFO:teuthology.orchestra.run.smithi135.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: re2-1:20211101-3.el9.x86_64 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-10T00:14:12.938 INFO:teuthology.orchestra.run.smithi135.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-10T00:14:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-06-10T00:14:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-10T00:14:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-10T00:14:12.939 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-06-10T00:14:12.939 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-06-10T00:14:13.385 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-10T00:14:13.411 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 113/170 2024-06-10T00:14:13.411 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:13.411 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-10T00:14:13.411 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-10T00:14:13.411 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-10T00:14:13.411 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:13.418 DEBUG:teuthology.parallel:result is None 2024-06-10T00:14:13.636 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-10T00:14:13.659 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 114/170 2024-06-10T00:14:13.659 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:13.659 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-10T00:14:13.659 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-10T00:14:13.659 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-10T00:14:13.659 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:14.096 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-10T00:14:14.118 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 115/170 2024-06-10T00:14:14.118 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:14.118 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-10T00:14:14.118 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-10T00:14:14.118 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-10T00:14:14.118 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:14.217 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-06-10T00:14:14.266 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-06-10T00:14:14.322 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-06-10T00:14:14.507 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-06-10T00:14:14.555 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-06-10T00:14:14.633 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-06-10T00:14:14.721 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-06-10T00:14:14.860 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-06-10T00:14:14.913 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-06-10T00:14:14.945 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-06-10T00:14:14.974 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-06-10T00:14:15.418 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-10T00:14:15.447 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 127/170 2024-06-10T00:14:15.487 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 128/170 2024-06-10T00:14:15.558 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 129/170 2024-06-10T00:14:15.595 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-10T00:14:16.627 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 130/170 2024-06-10T00:14:16.664 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-10T00:14:17.628 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 131/170 2024-06-10T00:14:17.656 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-10T00:14:17.719 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-10T00:14:17.719 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 1/170 2024-06-10T00:14:17.719 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 3/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 4/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 5/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 6/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 7/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 11/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 12/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 13/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-10T00:14:17.720 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 15/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 16/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 17/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 18/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 19/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 20/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 21/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 22/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 23/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 24/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 25/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 26/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 27/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 28/170 2024-06-10T00:14:17.721 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 29/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 30/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 31/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 32/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 33/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 34/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 35/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 36/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 37/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 38/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 39/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-06-10T00:14:17.722 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-06-10T00:14:17.723 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-06-10T00:14:17.724 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-06-10T00:14:17.725 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-06-10T00:14:17.726 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-06-10T00:14:17.727 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-06-10T00:14:17.728 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 122/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-06-10T00:14:17.729 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-06-10T00:14:17.730 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 149/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 151/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-1:1.47.91-1.el9.x86_64 153/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 155/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-team-1:1.47.91-1.el9.x86_64 157/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-tui-1:1.47.91-1.el9.x86_64 159/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-06-10T00:14:17.731 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-06-10T00:14:17.732 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-06-10T00:14:17.732 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-06-10T00:14:17.732 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-06-10T00:14:17.732 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-06-10T00:14:17.732 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-06-10T00:14:17.732 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-06-10T00:14:17.757 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 132/170 2024-06-10T00:14:17.919 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 133/170 2024-06-10T00:14:17.950 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-10T00:14:17.972 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 134/170 2024-06-10T00:14:17.972 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:17.973 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-10T00:14:17.973 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-10T00:14:17.973 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-10T00:14:17.973 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:18.017 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-10T00:14:18.028 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 135/170 2024-06-10T00:14:18.126 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 136/170 2024-06-10T00:14:18.182 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 137/170 2024-06-10T00:14:19.087 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-06-10T00:14:19.120 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-10T00:14:19.143 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 139/170 2024-06-10T00:14:19.143 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:19.143 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-10T00:14:19.143 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-10T00:14:19.143 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-10T00:14:19.143 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:19.187 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-10T00:14:19.209 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 140/170 2024-06-10T00:14:19.210 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:19.210 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-10T00:14:19.210 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:19.263 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-10T00:14:19.286 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 141/170 2024-06-10T00:14:19.286 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:19.286 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-10T00:14:19.286 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-10T00:14:19.286 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-10T00:14:19.286 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:19.589 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-10T00:14:19.612 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/170 2024-06-10T00:14:19.612 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-10T00:14:19.612 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-10T00:14:19.612 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-10T00:14:19.612 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-10T00:14:19.612 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:20.523 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-06-10T00:14:20.523 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:14:20.524 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-06-10T00:14:20.524 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:20.524 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:20.524 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-team-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:20.524 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-tui-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:20.524 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-10T00:14:20.524 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-10T00:14:20.524 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-10T00:14:20.524 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.524 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.525 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-06-10T00:14:20.525 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-06-10T00:14:20.525 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-06-10T00:14:20.525 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-10T00:14:20.525 INFO:teuthology.orchestra.run.smithi121.stdout: automake-1.16.2-8.el9.noarch 2024-06-10T00:14:20.525 INFO:teuthology.orchestra.run.smithi121.stdout: bison-3.7.4-5.el9.x86_64 2024-06-10T00:14:20.525 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-10T00:14:20.525 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.526 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.526 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.526 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.526 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:20.526 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.526 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.526 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.526 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:20.527 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:20.527 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:20.527 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:20.527 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:20.527 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.527 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.527 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:20.527 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.527 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.527 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.528 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:20.528 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:20.528 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.528 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.528 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:20.528 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-10T00:14:20.528 INFO:teuthology.orchestra.run.smithi121.stdout: flex-2.6.4-9.el9.x86_64 2024-06-10T00:14:20.528 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-10T00:14:20.528 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-10T00:14:20.529 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-10T00:14:20.529 INFO:teuthology.orchestra.run.smithi121.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-10T00:14:20.529 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-10T00:14:20.529 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-10T00:14:20.529 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-10T00:14:20.529 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-10T00:14:20.529 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-10T00:14:20.529 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.529 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.529 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.530 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-10T00:14:20.530 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-10T00:14:20.530 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-10T00:14:20.530 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-10T00:14:20.530 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-10T00:14:20.530 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.530 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.530 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-10T00:14:20.530 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.530 INFO:teuthology.orchestra.run.smithi121.stdout: libtool-2.4.6-45.el9.x86_64 2024-06-10T00:14:20.531 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-10T00:14:20.531 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-10T00:14:20.531 INFO:teuthology.orchestra.run.smithi121.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-10T00:14:20.531 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-10T00:14:20.531 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2024-06-10T00:14:20.531 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-10T00:14:20.531 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-10T00:14:20.532 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-10T00:14:20.532 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-10T00:14:20.532 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-06-10T00:14:20.532 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-10T00:14:20.532 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-13.14-1.el9.x86_64 2024-06-10T00:14:20.532 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-10T00:14:20.532 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-10T00:14:20.532 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-10T00:14:20.532 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-10T00:14:20.532 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-10T00:14:20.532 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-10T00:14:20.533 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-10T00:14:20.533 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-10T00:14:20.533 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-10T00:14:20.533 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-10T00:14:20.533 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.533 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.533 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.533 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-10T00:14:20.533 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-10T00:14:20.533 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-10T00:14:20.534 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-10T00:14:20.534 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-10T00:14:20.534 INFO:teuthology.orchestra.run.smithi121.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-10T00:14:20.534 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-10T00:14:20.534 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-10T00:14:20.534 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-10T00:14:20.534 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-10T00:14:20.534 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-10T00:14:20.534 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-10T00:14:20.534 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-10T00:14:20.534 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-10T00:14:20.535 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-10T00:14:20.535 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-10T00:14:20.535 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-10T00:14:20.535 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-10T00:14:20.535 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-10T00:14:20.535 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-10T00:14:20.535 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-10T00:14:20.535 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-10T00:14:20.535 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-10T00:14:20.536 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-10T00:14:20.536 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-10T00:14:20.536 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-10T00:14:20.536 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-10T00:14:20.536 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-10T00:14:20.537 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-10T00:14:20.537 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-10T00:14:20.537 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-06-10T00:14:20.537 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-10T00:14:20.537 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-10T00:14:20.537 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-10T00:14:20.537 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-10T00:14:20.537 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-10T00:14:20.537 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-10T00:14:20.537 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-10T00:14:20.538 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-10T00:14:20.538 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-10T00:14:20.538 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.538 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.538 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-10T00:14:20.538 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-10T00:14:20.539 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-10T00:14:20.539 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.539 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-10T00:14:20.539 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-10T00:14:20.539 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-10T00:14:20.539 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-10T00:14:20.539 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-10T00:14:20.539 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-10T00:14:20.540 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-10T00:14:20.541 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-10T00:14:20.541 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-10T00:14:20.541 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-10T00:14:20.541 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-10T00:14:20.541 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-10T00:14:20.541 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-10T00:14:20.542 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-10T00:14:20.542 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-10T00:14:20.542 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.542 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.542 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:20.542 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-3.el9.x86_64 2024-06-10T00:14:20.542 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-10T00:14:20.542 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-10T00:14:20.542 INFO:teuthology.orchestra.run.smithi121.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-10T00:14:20.542 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-06-10T00:14:20.542 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-10T00:14:20.543 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-10T00:14:20.543 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:14:20.543 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-06-10T00:14:20.949 DEBUG:teuthology.parallel:result is None 2024-06-10T00:14:24.125 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 143/170 2024-06-10T00:14:24.166 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 144/170 2024-06-10T00:14:24.216 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 145/170 2024-06-10T00:14:24.248 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-06-10T00:14:24.276 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-06-10T00:14:24.455 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 148/170 2024-06-10T00:14:24.505 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-06-10T00:14:24.538 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 150/170 2024-06-10T00:14:24.578 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-06-10T00:14:24.608 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-team-1:1.47.91-1.el9.x86_64 152/170 2024-06-10T00:14:24.646 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-tui-1:1.47.91-1.el9.x86_64 153/170 2024-06-10T00:14:25.242 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-10T00:14:25.286 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-10T00:14:25.305 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-06-10T00:14:26.135 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 155/170 2024-06-10T00:14:26.208 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-06-10T00:14:26.249 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-06-10T00:14:26.331 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-06-10T00:14:26.370 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-06-10T00:14:26.395 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-10T00:14:26.421 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-06-10T00:14:26.455 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-06-10T00:14:26.456 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-10T00:14:26.540 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-06-10T00:14:26.573 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-06-10T00:14:26.573 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-10T00:14:26.658 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-06-10T00:14:26.683 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-06-10T00:14:26.688 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-10T00:14:26.688 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-10T00:14:26.868 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-06-10T00:14:26.868 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-10T00:14:26.944 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-06-10T00:14:26.977 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-06-10T00:14:27.049 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-10T00:14:32.671 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-06-10T00:14:32.671 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 1/170 2024-06-10T00:14:32.671 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 3/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 4/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 5/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 6/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 7/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 8/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 9/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 10/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 11/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 12/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 13/170 2024-06-10T00:14:32.672 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 14/170 2024-06-10T00:14:32.673 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 15/170 2024-06-10T00:14:32.673 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 16/170 2024-06-10T00:14:32.673 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 17/170 2024-06-10T00:14:32.673 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 18/170 2024-06-10T00:14:32.673 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 19/170 2024-06-10T00:14:32.673 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 20/170 2024-06-10T00:14:32.673 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 21/170 2024-06-10T00:14:32.674 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 22/170 2024-06-10T00:14:32.674 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 23/170 2024-06-10T00:14:32.674 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 24/170 2024-06-10T00:14:32.674 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 25/170 2024-06-10T00:14:32.674 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 26/170 2024-06-10T00:14:32.674 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 27/170 2024-06-10T00:14:32.674 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 28/170 2024-06-10T00:14:32.674 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 29/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 30/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 31/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 32/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 33/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 34/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 35/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 36/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 37/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 38/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 39/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-06-10T00:14:32.688 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-06-10T00:14:32.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-06-10T00:14:32.690 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-06-10T00:14:32.690 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-06-10T00:14:32.690 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-06-10T00:14:32.690 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-06-10T00:14:32.690 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-06-10T00:14:32.690 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-06-10T00:14:32.690 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-06-10T00:14:32.690 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-06-10T00:14:32.690 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-06-10T00:14:32.690 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-06-10T00:14:32.691 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-06-10T00:14:32.691 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-06-10T00:14:32.691 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-06-10T00:14:32.691 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-06-10T00:14:32.691 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-06-10T00:14:32.691 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-06-10T00:14:32.691 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-06-10T00:14:32.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-06-10T00:14:32.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-06-10T00:14:32.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-06-10T00:14:32.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-06-10T00:14:32.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-06-10T00:14:32.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-06-10T00:14:32.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-06-10T00:14:32.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-06-10T00:14:32.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-06-10T00:14:32.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-06-10T00:14:32.692 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-06-10T00:14:32.693 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-06-10T00:14:32.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-06-10T00:14:32.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-06-10T00:14:32.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-06-10T00:14:32.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-06-10T00:14:32.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-06-10T00:14:32.694 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-06-10T00:14:32.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-06-10T00:14:32.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-06-10T00:14:32.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-06-10T00:14:32.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-06-10T00:14:32.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-06-10T00:14:32.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-06-10T00:14:32.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-06-10T00:14:32.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-06-10T00:14:32.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-06-10T00:14:32.695 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-06-10T00:14:32.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-06-10T00:14:32.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-06-10T00:14:32.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-06-10T00:14:32.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-06-10T00:14:32.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-06-10T00:14:32.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-06-10T00:14:32.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-06-10T00:14:32.696 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-06-10T00:14:32.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-06-10T00:14:32.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-06-10T00:14:32.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 122/170 2024-06-10T00:14:32.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-06-10T00:14:32.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-06-10T00:14:32.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-06-10T00:14:32.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-06-10T00:14:32.697 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-06-10T00:14:32.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-06-10T00:14:32.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-06-10T00:14:32.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-06-10T00:14:32.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-06-10T00:14:32.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-06-10T00:14:32.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-06-10T00:14:32.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-06-10T00:14:32.698 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-06-10T00:14:32.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-06-10T00:14:32.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-06-10T00:14:32.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-06-10T00:14:32.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-06-10T00:14:32.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-06-10T00:14:32.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-06-10T00:14:32.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-06-10T00:14:32.699 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-06-10T00:14:32.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-06-10T00:14:32.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-06-10T00:14:32.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-06-10T00:14:32.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-06-10T00:14:32.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-06-10T00:14:32.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 149/170 2024-06-10T00:14:32.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-06-10T00:14:32.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 151/170 2024-06-10T00:14:32.700 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-06-10T00:14:32.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-1:1.47.91-1.el9.x86_64 153/170 2024-06-10T00:14:32.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-06-10T00:14:32.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 155/170 2024-06-10T00:14:32.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-06-10T00:14:32.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-team-1:1.47.91-1.el9.x86_64 157/170 2024-06-10T00:14:32.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-06-10T00:14:32.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-tui-1:1.47.91-1.el9.x86_64 159/170 2024-06-10T00:14:32.701 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-06-10T00:14:32.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-06-10T00:14:32.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-06-10T00:14:32.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-06-10T00:14:32.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-06-10T00:14:32.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-06-10T00:14:32.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-06-10T00:14:32.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-06-10T00:14:32.702 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-06-10T00:14:32.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-06-10T00:14:35.278 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-team-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-tui-1:1.47.91-1.el9.x86_64 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.279 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: automake-1.16.2-8.el9.noarch 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: bison-3.7.4-5.el9.x86_64 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.280 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:35.281 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: flex-2.6.4-9.el9.x86_64 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-10T00:14:35.282 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-10T00:14:35.283 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: libtool-2.4.6-45.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: lua-5.4.4-4.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-13.14-1.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-10T00:14:35.284 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-10T00:14:35.285 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-10T00:14:35.286 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-10T00:14:35.287 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply-3.11-14.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-10T00:14:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-10T00:14:35.289 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: re2-1:20211101-3.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-10T00:14:35.290 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:14:35.291 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-06-10T00:14:35.697 DEBUG:teuthology.parallel:result is None 2024-06-10T00:14:35.698 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-10T00:14:35.923 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-10T00:14:35.950 INFO:teuthology.orchestra.run.smithi121.stdout:19.0.0-2946.g3d89accf.el9 2024-06-10T00:14:35.950 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946.g3d89accf.el9 2024-06-10T00:14:35.950 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946.g3d89accf is installed. 2024-06-10T00:14:35.952 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-10T00:14:36.095 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-10T00:14:36.121 INFO:teuthology.orchestra.run.smithi135.stdout:19.0.0-2946.g3d89accf.el9 2024-06-10T00:14:36.121 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946.g3d89accf.el9 2024-06-10T00:14:36.121 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946.g3d89accf is installed. 2024-06-10T00:14:36.123 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-10T00:14:36.342 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-10T00:14:36.364 INFO:teuthology.orchestra.run.smithi193.stdout:19.0.0-2946.g3d89accf.el9 2024-06-10T00:14:36.365 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946.g3d89accf.el9 2024-06-10T00:14:36.365 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946.g3d89accf is installed. 2024-06-10T00:14:36.366 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-10T00:14:36.367 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:14:36.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-10T00:14:36.400 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-06-10T00:14:36.400 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-10T00:14:36.431 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-06-10T00:14:36.431 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-10T00:14:36.459 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-10T00:14:36.459 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:14:36.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-06-10T00:14:36.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-10T00:14:36.561 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-06-10T00:14:36.561 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/daemon-helper 2024-06-10T00:14:36.590 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-10T00:14:36.657 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-06-10T00:14:36.657 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-06-10T00:14:36.683 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-10T00:14:36.747 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-10T00:14:36.747 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:14:36.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-10T00:14:36.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-10T00:14:36.837 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-06-10T00:14:36.837 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-10T00:14:36.865 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-10T00:14:36.930 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-06-10T00:14:36.931 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-10T00:14:36.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-10T00:14:37.019 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-10T00:14:37.020 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:14:37.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-06-10T00:14:37.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-10T00:14:37.109 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-06-10T00:14:37.109 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/stdin-killer 2024-06-10T00:14:37.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-10T00:14:37.196 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-06-10T00:14:37.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-06-10T00:14:37.220 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-10T00:14:37.284 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-10T00:14:37.374 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'subvols': {'create': 2}} 2024-06-10T00:14:37.375 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-10T00:14:37.375 INFO:tasks.cephadm:Cluster fsid is 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:14:37.375 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-10T00:14:37.375 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.121', 'mon.b': '172.21.15.135', 'mon.c': '172.21.15.193'} 2024-06-10T00:14:37.376 INFO:tasks.cephadm:First mon is mon.a on smithi121 2024-06-10T00:14:37.376 INFO:tasks.cephadm:First mgr is x 2024-06-10T00:14:37.376 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-10T00:14:37.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname $(hostname -s) 2024-06-10T00:14:37.401 DEBUG:teuthology.orchestra.run.smithi135:> sudo hostname $(hostname -s) 2024-06-10T00:14:37.425 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2024-06-10T00:14:37.449 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-06-10T00:14:37.450 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-10T00:14:37.595 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55273/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2946-g3d89accf', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2946.g3d89accf'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-30 17:04:17.570799', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-06-10T00:14:37.803 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 3d89accfe3201286f000e065a1faea2a88c7a6ba from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-10T00:14:37.805 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/x86_64/flavors/default/cephadm 2024-06-10T00:14:37.805 DEBUG:teuthology.orchestra.run.smithi121:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-10T00:14:38.315 INFO:teuthology.orchestra.run.smithi121.stdout:-rw-r--r--. 1 ubuntu ubuntu 781907 Jun 10 00:14 /home/ubuntu/cephtest/cephadm 2024-06-10T00:14:38.316 DEBUG:teuthology.orchestra.run.smithi135:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-10T00:14:38.864 INFO:teuthology.orchestra.run.smithi135.stdout:-rw-r--r--. 1 ubuntu ubuntu 781907 Jun 10 00:14 /home/ubuntu/cephtest/cephadm 2024-06-10T00:14:38.865 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-10T00:14:39.398 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r--. 1 ubuntu ubuntu 781907 Jun 10 00:14 /home/ubuntu/cephtest/cephadm 2024-06-10T00:14:39.398 DEBUG:teuthology.orchestra.run.smithi121:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-10T00:14:39.418 DEBUG:teuthology.orchestra.run.smithi135:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-10T00:14:39.437 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-10T00:14:39.473 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba on all hosts... 2024-06-10T00:14:39.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba pull 2024-06-10T00:14:39.476 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba pull 2024-06-10T00:14:39.480 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba pull 2024-06-10T00:14:40.336 INFO:teuthology.orchestra.run.smithi193.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba... 2024-06-10T00:14:40.345 INFO:teuthology.orchestra.run.smithi135.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba... 2024-06-10T00:14:40.372 INFO:teuthology.orchestra.run.smithi121.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba... 2024-06-10T00:15:18.558 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-06-10T00:15:18.558 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_version": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-10T00:15:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "image_id": "b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1", 2024-06-10T00:15:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "repo_digests": [ 2024-06-10T00:15:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9" 2024-06-10T00:15:18.559 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-06-10T00:15:18.559 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-06-10T00:15:26.692 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:15:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-10T00:15:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "image_id": "b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1", 2024-06-10T00:15:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "repo_digests": [ 2024-06-10T00:15:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9" 2024-06-10T00:15:26.693 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:15:26.693 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:15:33.653 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-06-10T00:15:33.653 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_version": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-10T00:15:33.653 INFO:teuthology.orchestra.run.smithi135.stdout: "image_id": "b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1", 2024-06-10T00:15:33.653 INFO:teuthology.orchestra.run.smithi135.stdout: "repo_digests": [ 2024-06-10T00:15:33.653 INFO:teuthology.orchestra.run.smithi135.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9" 2024-06-10T00:15:33.653 INFO:teuthology.orchestra.run.smithi135.stdout: ] 2024-06-10T00:15:33.653 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-06-10T00:15:33.677 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph 2024-06-10T00:15:33.709 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph 2024-06-10T00:15:33.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2024-06-10T00:15:33.766 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /etc/ceph 2024-06-10T00:15:33.795 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 777 /etc/ceph 2024-06-10T00:15:33.817 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2024-06-10T00:15:33.844 INFO:tasks.cephadm:Writing seed config... 2024-06-10T00:15:33.845 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-10T00:15:33.845 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-10T00:15:33.845 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-10T00:15:33.846 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-10T00:15:33.847 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-06-10T00:15:33.848 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-06-10T00:15:33.849 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:15:33.849 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-10T00:15:33.867 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6c432244-26be-11ef-bc9b-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-10T00:15:33.867 DEBUG:teuthology.orchestra.run.smithi121:mon.a> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.a.service 2024-06-10T00:15:33.910 DEBUG:teuthology.orchestra.run.smithi121:mgr.x> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.x.service 2024-06-10T00:15:33.953 INFO:tasks.cephadm:Bootstrapping... 2024-06-10T00:15:33.953 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba -v bootstrap --fsid 6c432244-26be-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.121 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-10T00:15:34.146 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-06-10T00:15:34.146 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba', '-v', 'bootstrap', '--fsid', '6c432244-26be-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.121', '--skip-admin-label'] 2024-06-10T00:15:34.146 INFO:teuthology.orchestra.run.smithi121.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-10T00:15:34.147 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying podman|docker is present... 2024-06-10T00:15:34.172 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stdout 5.0.2 2024-06-10T00:15:34.172 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying lvm2 is present... 2024-06-10T00:15:34.172 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying time synchronization is in place... 2024-06-10T00:15:34.179 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-10T00:15:34.179 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-10T00:15:34.186 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-10T00:15:34.186 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-06-10T00:15:34.193 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout enabled 2024-06-10T00:15:34.199 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout active 2024-06-10T00:15:34.199 INFO:teuthology.orchestra.run.smithi121.stdout:Unit chronyd.service is enabled and running 2024-06-10T00:15:34.199 INFO:teuthology.orchestra.run.smithi121.stdout:Repeating the final host check... 2024-06-10T00:15:34.222 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stdout 5.0.2 2024-06-10T00:15:34.222 INFO:teuthology.orchestra.run.smithi121.stdout:podman (/bin/podman) version 5.0.2 is present 2024-06-10T00:15:34.222 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl is present 2024-06-10T00:15:34.222 INFO:teuthology.orchestra.run.smithi121.stdout:lvcreate is present 2024-06-10T00:15:34.228 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-10T00:15:34.229 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-10T00:15:34.235 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-10T00:15:34.235 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-06-10T00:15:34.241 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout enabled 2024-06-10T00:15:34.248 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout active 2024-06-10T00:15:34.248 INFO:teuthology.orchestra.run.smithi121.stdout:Unit chronyd.service is enabled and running 2024-06-10T00:15:34.248 INFO:teuthology.orchestra.run.smithi121.stdout:Host looks OK 2024-06-10T00:15:34.248 INFO:teuthology.orchestra.run.smithi121.stdout:Cluster fsid: 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:15:34.248 INFO:teuthology.orchestra.run.smithi121.stdout:Acquiring lock 140098227135872 on /run/cephadm/6c432244-26be-11ef-bc9b-c7b262605968.lock 2024-06-10T00:15:34.248 INFO:teuthology.orchestra.run.smithi121.stdout:Lock 140098227135872 acquired on /run/cephadm/6c432244-26be-11ef-bc9b-c7b262605968.lock 2024-06-10T00:15:34.249 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying IP 172.21.15.121 port 3300 ... 2024-06-10T00:15:34.249 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying IP 172.21.15.121 port 6789 ... 2024-06-10T00:15:34.249 INFO:teuthology.orchestra.run.smithi121.stdout:Base mon IP(s) is [172.21.15.121:3300, 172.21.15.121:6789], mon addrv is [v2:172.21.15.121:3300,v1:172.21.15.121:6789] 2024-06-10T00:15:34.252 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.121 metric 100 2024-06-10T00:15:34.252 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.121 metric 100 2024-06-10T00:15:34.254 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-10T00:15:34.254 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-10T00:15:34.255 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1789sec hoplimit 64 pref medium 2024-06-10T00:15:34.257 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-10T00:15:34.257 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-10T00:15:34.257 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-10T00:15:34.257 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-10T00:15:34.257 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2024-06-10T00:15:34.257 INFO:teuthology.orchestra.run.smithi121.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-10T00:15:34.258 INFO:teuthology.orchestra.run.smithi121.stdout:Mon IP `172.21.15.121` is in CIDR network `172.21.0.0/20` 2024-06-10T00:15:34.258 INFO:teuthology.orchestra.run.smithi121.stdout:Mon IP `172.21.15.121` is in CIDR network `172.21.0.0/20` 2024-06-10T00:15:34.258 INFO:teuthology.orchestra.run.smithi121.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-10T00:15:34.259 INFO:teuthology.orchestra.run.smithi121.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-10T00:15:34.259 INFO:teuthology.orchestra.run.smithi121.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba... 2024-06-10T00:15:34.693 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stdout b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 2024-06-10T00:15:34.693 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba... 2024-06-10T00:15:34.694 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Getting image source signatures 2024-06-10T00:15:34.694 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Copying blob sha256:53b2d4191269e9d20972bfa652565296fad6514cf6a3d99f19db644875f67876 2024-06-10T00:15:34.694 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Copying blob sha256:3ca969c7b6d79946c6f10490fffc00028babe7df67f50ed53c7bc209325e0612 2024-06-10T00:15:34.694 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Copying config sha256:b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 2024-06-10T00:15:34.694 INFO:teuthology.orchestra.run.smithi121.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-10T00:15:35.640 INFO:teuthology.orchestra.run.smithi121.stdout:ceph: stdout ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc) 2024-06-10T00:15:35.640 INFO:teuthology.orchestra.run.smithi121.stdout:Ceph version: ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc) 2024-06-10T00:15:35.640 INFO:teuthology.orchestra.run.smithi121.stdout:Extracting ceph user uid/gid from container image... 2024-06-10T00:15:36.554 INFO:teuthology.orchestra.run.smithi121.stdout:stat: stdout 167 167 2024-06-10T00:15:36.554 INFO:teuthology.orchestra.run.smithi121.stdout:Creating initial keys... 2024-06-10T00:15:37.517 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph-authtool: stdout AQCoRWZmBNX5NBAAkI94Fx3EPvcEiWWgWsLORw== 2024-06-10T00:15:38.474 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph-authtool: stdout AQCpRWZmdSTiMRAAU6u5t+hb3+EQnCtZCiP4Fw== 2024-06-10T00:15:39.312 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph-authtool: stdout AQCqRWZmGYVGMBAAwhdG++WeOlwjmXf+OybkNA== 2024-06-10T00:15:39.313 INFO:teuthology.orchestra.run.smithi121.stdout:Creating initial monmap... 2024-06-10T00:15:40.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-10T00:15:40.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-06-10T00:15:40.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:15:40.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-10T00:15:40.203 INFO:teuthology.orchestra.run.smithi121.stdout:monmaptool for a [v2:172.21.15.121:3300,v1:172.21.15.121:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-10T00:15:40.203 INFO:teuthology.orchestra.run.smithi121.stdout:setting min_mon_release = quincy 2024-06-10T00:15:40.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: set fsid to 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:15:40.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-10T00:15:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:15:40.203 INFO:teuthology.orchestra.run.smithi121.stdout:Creating mon... 2024-06-10T00:15:41.141 INFO:teuthology.orchestra.run.smithi121.stdout:create mon.a on 2024-06-10T00:15:41.351 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-10T00:15:41.493 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-10T00:15:41.652 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6c432244-26be-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-6c432244-26be-11ef-bc9b-c7b262605968.target. 2024-06-10T00:15:41.652 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-6c432244-26be-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-6c432244-26be-11ef-bc9b-c7b262605968.target. 2024-06-10T00:15:41.858 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.a 2024-06-10T00:15:41.859 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.a.service: Unit ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.a.service not loaded. 2024-06-10T00:15:42.058 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6c432244-26be-11ef-bc9b-c7b262605968.target.wants/ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.a.service → /etc/systemd/system/ceph-6c432244-26be-11ef-bc9b-c7b262605968@.service. 2024-06-10T00:15:42.617 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-10T00:15:42.617 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-06-10T00:15:42.624 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-10T00:15:42.624 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-06-10T00:15:42.625 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-06-10T00:15:42.625 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to enable service . firewalld.service is not available 2024-06-10T00:15:42.625 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mon to start... 2024-06-10T00:15:42.625 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mon... 2024-06-10T00:15:42.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:42 smithi121 systemd[1]: Started Ceph mon.a for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:15:43.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-10T00:15:43.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: monmap epoch 1 2024-06-10T00:15:43.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:15:43.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: last_changed 2024-06-10T00:15:39.680682+0000 2024-06-10T00:15:43.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: created 2024-06-10T00:15:39.680682+0000 2024-06-10T00:15:43.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: min_mon_release 19 (squid) 2024-06-10T00:15:43.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: election_strategy: 1 2024-06-10T00:15:43.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-06-10T00:15:43.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: fsmap 2024-06-10T00:15:43.676 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: osdmap e1: 0 total, 0 up, 0 in 2024-06-10T00:15:43.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: mgrmap e1: no daemons active 2024-06-10T00:15:43.677 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:43 smithi121 ceph-mon[32001]: from='client.? 172.21.15.121:0/4061781355' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout cluster: 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout id: 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout services: 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.570958s) 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout data: 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-10T00:15:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout pgs: 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.039+0000 7fadbae68700 1 Processor -- start 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.039+0000 7fadbae68700 1 -- start start 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.039+0000 7fadbae68700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb410ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.039+0000 7fadbae68700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fadb407c640 con 0x7fadb4108a30 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.040+0000 7fadb8c04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb410ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.040+0000 7fadb8c04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb410ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53424/0 (socket says 172.21.15.121:53424) 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.040+0000 7fadb8c04700 1 -- 172.21.15.121:0/2466879276 learned_addr learned my addr 172.21.15.121:0/2466879276 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.040+0000 7fadb8c04700 1 -- 172.21.15.121:0/2466879276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb410b360 con 0x7fadb4108a30 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.040+0000 7fadb8c04700 1 --2- 172.21.15.121:0/2466879276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb410ae20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fada80098a0 tx=0x7fada8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915f4298449faa3e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.040+0000 7fadb37fe700 1 -- 172.21.15.121:0/2466879276 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fada8004160 con 0x7fadb4108a30 2024-06-10T00:15:43.685 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.040+0000 7fadb37fe700 1 -- 172.21.15.121:0/2466879276 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada8004300 con 0x7fadb4108a30 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.040+0000 7fadb37fe700 1 -- 172.21.15.121:0/2466879276 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fada80036a0 con 0x7fadb4108a30 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.041+0000 7fadbae68700 1 -- 172.21.15.121:0/2466879276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 msgr2=0x7fadb410ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.041+0000 7fadbae68700 1 --2- 172.21.15.121:0/2466879276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb410ae20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fada80098a0 tx=0x7fada8009bb0 comp rx=0 tx=0).stop 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.041+0000 7fadbae68700 1 -- 172.21.15.121:0/2466879276 shutdown_connections 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.041+0000 7fadbae68700 1 --2- 172.21.15.121:0/2466879276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb410ae20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.041+0000 7fadbae68700 1 -- 172.21.15.121:0/2466879276 >> 172.21.15.121:0/2466879276 conn(0x7fadb4102460 msgr2=0x7fadb4104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.041+0000 7fadbae68700 1 -- 172.21.15.121:0/2466879276 shutdown_connections 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.041+0000 7fadbae68700 1 -- 172.21.15.121:0/2466879276 wait complete. 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.042+0000 7fadbae68700 1 Processor -- start 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.042+0000 7fadbae68700 1 -- start start 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.042+0000 7fadbae68700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb419b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:15:43.686 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.042+0000 7fadbae68700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fadb410b710 con 0x7fadb4108a30 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.042+0000 7fadb8c04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb419b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.042+0000 7fadb8c04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb419b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53428/0 (socket says 172.21.15.121:53428) 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.042+0000 7fadb8c04700 1 -- 172.21.15.121:0/4061781355 learned_addr learned my addr 172.21.15.121:0/4061781355 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.043+0000 7fadb8c04700 1 -- 172.21.15.121:0/4061781355 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb419baf0 con 0x7fadb4108a30 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.043+0000 7fadb8c04700 1 --2- 172.21.15.121:0/4061781355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb419b5b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fada8009fb0 tx=0x7fada8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.043+0000 7fadb1ffb700 1 -- 172.21.15.121:0/4061781355 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fada8003de0 con 0x7fadb4108a30 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.043+0000 7fadb1ffb700 1 -- 172.21.15.121:0/4061781355 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fada8011630 con 0x7fadb4108a30 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.043+0000 7fadbae68700 1 -- 172.21.15.121:0/4061781355 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadb419bd80 con 0x7fadb4108a30 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.043+0000 7fadb1ffb700 1 -- 172.21.15.121:0/4061781355 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fada8011930 con 0x7fadb4108a30 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.043+0000 7fadbae68700 1 -- 172.21.15.121:0/4061781355 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb419ea70 con 0x7fadb4108a30 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.043+0000 7fadb1ffb700 1 -- 172.21.15.121:0/4061781355 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fada8011ad0 con 0x7fadb4108a30 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.043+0000 7fadb1ffb700 1 -- 172.21.15.121:0/4061781355 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fada801b450 con 0x7fadb4108a30 2024-06-10T00:15:43.687 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.044+0000 7fadbae68700 1 -- 172.21.15.121:0/4061781355 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadb4063af0 con 0x7fadb4108a30 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.046+0000 7fadb1ffb700 1 -- 172.21.15.121:0/4061781355 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fada8004260 con 0x7fadb4108a30 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.117+0000 7fadbae68700 1 -- 172.21.15.121:0/4061781355 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fadb402ddf0 con 0x7fadb4108a30 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.118+0000 7fadb1ffb700 1 -- 172.21.15.121:0/4061781355 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fada8025480 con 0x7fadb4108a30 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.119+0000 7fadbae68700 1 -- 172.21.15.121:0/4061781355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 msgr2=0x7fadb419b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.119+0000 7fadbae68700 1 --2- 172.21.15.121:0/4061781355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb419b5b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fada8009fb0 tx=0x7fada8004750 comp rx=0 tx=0).stop 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.119+0000 7fadbae68700 1 -- 172.21.15.121:0/4061781355 shutdown_connections 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.119+0000 7fadbae68700 1 --2- 172.21.15.121:0/4061781355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fadb4108a30 0x7fadb419b5b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.119+0000 7fadbae68700 1 -- 172.21.15.121:0/4061781355 >> 172.21.15.121:0/4061781355 conn(0x7fadb4102460 msgr2=0x7fadb4104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.119+0000 7fadbae68700 1 -- 172.21.15.121:0/4061781355 shutdown_connections 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:43.119+0000 7fadbae68700 1 -- 172.21.15.121:0/4061781355 wait complete. 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:mon is available 2024-06-10T00:15:43.688 INFO:teuthology.orchestra.run.smithi121.stdout:Assimilating anything we can from ceph.conf... 2024-06-10T00:15:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:44 smithi121 ceph-mon[32001]: from='client.? 172.21.15.121:0/3012792788' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-10T00:15:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:44 smithi121 ceph-mon[32001]: from='client.? 172.21.15.121:0/3012792788' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-10T00:15:44.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:15:44.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [global] 2024-06-10T00:15:44.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout fsid = 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:15:44.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.121:3300,v1:172.21.15.121:6789] 2024-06-10T00:15:44.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-10T00:15:44.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-10T00:15:44.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-10T00:15:44.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-10T00:15:44.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:15:44.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [osd] 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.126+0000 7fc52d175700 1 Processor -- start 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.126+0000 7fc52d175700 1 -- start start 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.127+0000 7fc52d175700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52810adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.127+0000 7fc52d175700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc52807be40 con 0x7fc528108a00 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.127+0000 7fc526d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52810adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:15:44.785 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.127+0000 7fc526d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52810adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53440/0 (socket says 172.21.15.121:53440) 2024-06-10T00:15:44.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.127+0000 7fc526d9d700 1 -- 172.21.15.121:0/892942284 learned_addr learned my addr 172.21.15.121:0/892942284 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:15:44.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.127+0000 7fc526d9d700 1 -- 172.21.15.121:0/892942284 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc52810b330 con 0x7fc528108a00 2024-06-10T00:15:44.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.127+0000 7fc526d9d700 1 --2- 172.21.15.121:0/892942284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52810adf0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc5100098a0 tx=0x7fc510009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa98d6cf861d02a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:15:44.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.127+0000 7fc525d9b700 1 -- 172.21.15.121:0/892942284 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc510004160 con 0x7fc528108a00 2024-06-10T00:15:44.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.128+0000 7fc525d9b700 1 -- 172.21.15.121:0/892942284 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc510004300 con 0x7fc528108a00 2024-06-10T00:15:44.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.128+0000 7fc525d9b700 1 -- 172.21.15.121:0/892942284 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc5100036a0 con 0x7fc528108a00 2024-06-10T00:15:44.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.128+0000 7fc52d175700 1 -- 172.21.15.121:0/892942284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 msgr2=0x7fc52810adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:15:44.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.128+0000 7fc52d175700 1 --2- 172.21.15.121:0/892942284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52810adf0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fc5100098a0 tx=0x7fc510009bb0 comp rx=0 tx=0).stop 2024-06-10T00:15:44.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.128+0000 7fc52d175700 1 -- 172.21.15.121:0/892942284 shutdown_connections 2024-06-10T00:15:44.786 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.128+0000 7fc52d175700 1 --2- 172.21.15.121:0/892942284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52810adf0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:15:44.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.128+0000 7fc52d175700 1 -- 172.21.15.121:0/892942284 >> 172.21.15.121:0/892942284 conn(0x7fc528102410 msgr2=0x7fc528104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:15:44.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.128+0000 7fc52d175700 1 -- 172.21.15.121:0/892942284 shutdown_connections 2024-06-10T00:15:44.787 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.128+0000 7fc52d175700 1 -- 172.21.15.121:0/892942284 wait complete. 2024-06-10T00:15:44.790 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.129+0000 7fc52d175700 1 Processor -- start 2024-06-10T00:15:44.790 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.129+0000 7fc52d175700 1 -- start start 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.129+0000 7fc52d175700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52819b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.129+0000 7fc52d175700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc52810b7a0 con 0x7fc528108a00 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.129+0000 7fc526d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52819b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.129+0000 7fc526d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52819b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53442/0 (socket says 172.21.15.121:53442) 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.129+0000 7fc526d9d700 1 -- 172.21.15.121:0/3012792788 learned_addr learned my addr 172.21.15.121:0/3012792788 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.130+0000 7fc526d9d700 1 -- 172.21.15.121:0/3012792788 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc52819bb70 con 0x7fc528108a00 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.130+0000 7fc526d9d700 1 --2- 172.21.15.121:0/3012792788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52819b630 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc510000c00 tx=0x7fc510004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.130+0000 7fc51ffff700 1 -- 172.21.15.121:0/3012792788 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc510003db0 con 0x7fc528108a00 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.130+0000 7fc51ffff700 1 -- 172.21.15.121:0/3012792788 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc510011630 con 0x7fc528108a00 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.130+0000 7fc52d175700 1 -- 172.21.15.121:0/3012792788 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc52819be00 con 0x7fc528108a00 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.130+0000 7fc51ffff700 1 -- 172.21.15.121:0/3012792788 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc510011950 con 0x7fc528108a00 2024-06-10T00:15:44.791 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.130+0000 7fc52d175700 1 -- 172.21.15.121:0/3012792788 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc52819eaf0 con 0x7fc528108a00 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.130+0000 7fc51ffff700 1 -- 172.21.15.121:0/3012792788 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc510011af0 con 0x7fc528108a00 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.130+0000 7fc51ffff700 1 -- 172.21.15.121:0/3012792788 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc51001c450 con 0x7fc528108a00 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.131+0000 7fc52d175700 1 -- 172.21.15.121:0/3012792788 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc528063af0 con 0x7fc528108a00 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.133+0000 7fc51ffff700 1 -- 172.21.15.121:0/3012792788 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc510004260 con 0x7fc528108a00 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.201+0000 7fc52d175700 1 -- 172.21.15.121:0/3012792788 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fc52802de20 con 0x7fc528108a00 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.204+0000 7fc51ffff700 1 -- 172.21.15.121:0/3012792788 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7fc51001b450 con 0x7fc528108a00 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.205+0000 7fc52d175700 1 -- 172.21.15.121:0/3012792788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 msgr2=0x7fc52819b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.205+0000 7fc51ffff700 1 -- 172.21.15.121:0/3012792788 <== mon.0 v2:172.21.15.121:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc51001b630 con 0x7fc528108a00 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.205+0000 7fc52d175700 1 --2- 172.21.15.121:0/3012792788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52819b630 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fc510000c00 tx=0x7fc510004750 comp rx=0 tx=0).stop 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.205+0000 7fc52d175700 1 -- 172.21.15.121:0/3012792788 shutdown_connections 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.205+0000 7fc52d175700 1 --2- 172.21.15.121:0/3012792788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc528108a00 0x7fc52819b630 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.205+0000 7fc52d175700 1 -- 172.21.15.121:0/3012792788 >> 172.21.15.121:0/3012792788 conn(0x7fc528102410 msgr2=0x7fc528104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.205+0000 7fc52d175700 1 -- 172.21.15.121:0/3012792788 shutdown_connections 2024-06-10T00:15:44.793 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:44.205+0000 7fc52d175700 1 -- 172.21.15.121:0/3012792788 wait complete. 2024-06-10T00:15:44.793 INFO:teuthology.orchestra.run.smithi121.stdout:Generating new minimal ceph.conf... 2024-06-10T00:15:45.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:45 smithi121 ceph-mon[32001]: from='client.? 172.21.15.121:0/3990945955' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.192+0000 7fe2f6a6d700 1 Processor -- start 2024-06-10T00:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.192+0000 7fe2f6a6d700 1 -- start start 2024-06-10T00:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.192+0000 7fe2f6a6d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f0106c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.192+0000 7fe2f6a6d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f007be00 con 0x7fe2f0106830 2024-06-10T00:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.193+0000 7fe2f4809700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f0106c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.193+0000 7fe2f4809700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f0106c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53444/0 (socket says 172.21.15.121:53444) 2024-06-10T00:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.193+0000 7fe2f4809700 1 -- 172.21.15.121:0/1345352667 learned_addr learned my addr 172.21.15.121:0/1345352667 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.193+0000 7fe2f4809700 1 -- 172.21.15.121:0/1345352667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f0107190 con 0x7fe2f0106830 2024-06-10T00:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.193+0000 7fe2f4809700 1 --2- 172.21.15.121:0/1345352667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f0106c50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009e80 tx=0x7fe2e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ad7ee8821b75d2 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.193+0000 7fe2ef7fe700 1 -- 172.21.15.121:0/1345352667 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2e0017070 con 0x7fe2f0106830 2024-06-10T00:15:46.781 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.193+0000 7fe2ef7fe700 1 -- 172.21.15.121:0/1345352667 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe2e000b8c0 con 0x7fe2f0106830 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.193+0000 7fe2ef7fe700 1 -- 172.21.15.121:0/1345352667 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2e00045c0 con 0x7fe2f0106830 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.194+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/1345352667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 msgr2=0x7fe2f0106c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.194+0000 7fe2f6a6d700 1 --2- 172.21.15.121:0/1345352667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f0106c50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009e80 tx=0x7fe2e000b330 comp rx=0 tx=0).stop 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.194+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/1345352667 shutdown_connections 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.194+0000 7fe2f6a6d700 1 --2- 172.21.15.121:0/1345352667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f0106c50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.194+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/1345352667 >> 172.21.15.121:0/1345352667 conn(0x7fe2f0102410 msgr2=0x7fe2f0104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.194+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/1345352667 shutdown_connections 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.194+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/1345352667 wait complete. 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2f6a6d700 1 Processor -- start 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2f6a6d700 1 -- start start 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2f6a6d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f019e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2f6a6d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f0108220 con 0x7fe2f0106830 2024-06-10T00:15:46.782 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2f4809700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f019e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2f4809700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f019e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53448/0 (socket says 172.21.15.121:53448) 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2f4809700 1 -- 172.21.15.121:0/3990945955 learned_addr learned my addr 172.21.15.121:0/3990945955 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2f4809700 1 -- 172.21.15.121:0/3990945955 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f019e610 con 0x7fe2f0106830 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2f4809700 1 --2- 172.21.15.121:0/3990945955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f019e0d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00072a0 tx=0x7fe2e000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2edffb700 1 -- 172.21.15.121:0/3990945955 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2e0020020 con 0x7fe2f0106830 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2edffb700 1 -- 172.21.15.121:0/3990945955 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe2e0003b50 con 0x7fe2f0106830 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/3990945955 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f019e8a0 con 0x7fe2f0106830 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.195+0000 7fe2edffb700 1 -- 172.21.15.121:0/3990945955 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2e0017040 con 0x7fe2f0106830 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.196+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/3990945955 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f01a1590 con 0x7fe2f0106830 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.196+0000 7fe2edffb700 1 -- 172.21.15.121:0/3990945955 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe2e0023020 con 0x7fe2f0106830 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.196+0000 7fe2edffb700 1 -- 172.21.15.121:0/3990945955 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe2e000f050 con 0x7fe2f0106830 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.197+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/3990945955 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f0063af0 con 0x7fe2f0106830 2024-06-10T00:15:46.783 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.199+0000 7fe2edffb700 1 -- 172.21.15.121:0/3990945955 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe2e002a020 con 0x7fe2f0106830 2024-06-10T00:15:46.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.264+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/3990945955 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fe2f002de20 con 0x7fe2f0106830 2024-06-10T00:15:46.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.265+0000 7fe2edffb700 1 -- 172.21.15.121:0/3990945955 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7fe2e000b860 con 0x7fe2f0106830 2024-06-10T00:15:46.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.265+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/3990945955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 msgr2=0x7fe2f019e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:15:46.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.265+0000 7fe2f6a6d700 1 --2- 172.21.15.121:0/3990945955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f019e0d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00072a0 tx=0x7fe2e000bd50 comp rx=0 tx=0).stop 2024-06-10T00:15:46.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.266+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/3990945955 shutdown_connections 2024-06-10T00:15:46.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.266+0000 7fe2f6a6d700 1 --2- 172.21.15.121:0/3990945955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f0106830 0x7fe2f019e0d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:15:46.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.266+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/3990945955 >> 172.21.15.121:0/3990945955 conn(0x7fe2f0102410 msgr2=0x7fe2f0194d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:15:46.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.266+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/3990945955 shutdown_connections 2024-06-10T00:15:46.784 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:45.266+0000 7fe2f6a6d700 1 -- 172.21.15.121:0/3990945955 wait complete. 2024-06-10T00:15:46.784 INFO:teuthology.orchestra.run.smithi121.stdout:Restarting the monitor... 2024-06-10T00:15:47.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:46 smithi121 systemd[1]: Stopping Ceph mon.a for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:15:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:47 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a[31989]: 2024-06-10T00:15:47.977+0000 7f23c5229700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:15:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:47 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a[31989]: 2024-06-10T00:15:47.977+0000 7f23c5229700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-10T00:15:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:48 smithi121 podman[32249]: 2024-06-10 00:15:48.030147751 +0000 UTC m=+1.083550873 container died cccce6463ff1480219a106a60e5c9a94ae6da8f02260e1ef491342508a981710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-06-10T00:15:51.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:51 smithi121 podman[32249]: 2024-06-10 00:15:51.344140027 +0000 UTC m=+4.397543149 container cleanup cccce6463ff1480219a106a60e5c9a94ae6da8f02260e1ef491342508a981710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=) 2024-06-10T00:15:51.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:51 smithi121 bash[32249]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a 2024-06-10T00:15:53.845 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:53 smithi121 podman[32261]: 2024-06-10 00:15:53.498673767 +0000 UTC m=+5.459932047 container remove cccce6463ff1480219a106a60e5c9a94ae6da8f02260e1ef491342508a981710 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-10T00:15:53.845 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:53 smithi121 systemd[1]: ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-10T00:15:53.845 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:53 smithi121 systemd[1]: Stopped Ceph mon.a for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:15:53.845 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:53 smithi121 systemd[1]: Starting Ceph mon.a for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:15:54.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:53 smithi121 podman[32359]: 2024-06-10 00:15:53.748231585 +0000 UTC m=+0.016856759 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-10T00:15:54.925 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 podman[32359]: 2024-06-10 00:15:54.654472638 +0000 UTC m=+0.923097784 container create abafb9418971b452a9073e897aa81dc2000522fb57ae1e29f9c0b38e8a76495a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-06-10T00:15:54.926 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 podman[32359]: 2024-06-10 00:15:54.898362868 +0000 UTC m=+1.166988022 container init abafb9418971b452a9073e897aa81dc2000522fb57ae1e29f9c0b38e8a76495a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-10T00:15:54.926 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 podman[32359]: 2024-06-10 00:15:54.901606259 +0000 UTC m=+1.170231413 container start abafb9418971b452a9073e897aa81dc2000522fb57ae1e29f9c0b38e8a76495a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:15:54.985 INFO:teuthology.orchestra.run.smithi121.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-10T00:15:55.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: set uid:gid to 167:167 (ceph:ceph) 2024-06-10T00:15:55.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc), process ceph-mon, pid 2 2024-06-10T00:15:55.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: pidfile_write: ignore empty --pid-file 2024-06-10T00:15:55.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: load: jerasure load: lrc 2024-06-10T00:15:55.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: RocksDB version: 7.9.2 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Git sha 0 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Compile date 2024-05-30 15:42:47 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: DB SUMMARY 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: DB Session ID: ZNAT5Q2JOT3F5KU175PU 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: CURRENT file: CURRENT 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: IDENTITY file: IDENTITY 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105336 ; 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.error_if_exists: 0 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.create_if_missing: 0 2024-06-10T00:15:55.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.paranoid_checks: 1 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.env: 0x558333f2cc60 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.fs: PosixFileSystem 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.info_log: 0x558335311b20 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_file_opening_threads: 16 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.statistics: (nil) 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.use_fsync: 0 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_log_file_size: 0 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-10T00:15:55.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-10T00:15:55.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.keep_log_file_num: 1000 2024-06-10T00:15:55.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.recycle_log_file_num: 0 2024-06-10T00:15:55.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.allow_fallocate: 1 2024-06-10T00:15:55.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.allow_mmap_reads: 0 2024-06-10T00:15:55.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.allow_mmap_writes: 0 2024-06-10T00:15:55.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.use_direct_reads: 0 2024-06-10T00:15:55.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-10T00:15:55.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.create_missing_column_families: 0 2024-06-10T00:15:55.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.db_log_dir: 2024-06-10T00:15:55.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.wal_dir: 2024-06-10T00:15:55.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-10T00:15:55.205 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-10T00:15:55.205 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-10T00:15:55.205 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-10T00:15:55.205 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-10T00:15:55.205 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-10T00:15:55.205 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.advise_random_on_open: 1 2024-06-10T00:15:55.205 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.db_write_buffer_size: 0 2024-06-10T00:15:55.205 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.write_buffer_manager: 0x5583352e21e0 2024-06-10T00:15:55.205 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-10T00:15:55.205 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-10T00:15:55.206 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-10T00:15:55.206 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.rate_limiter: (nil) 2024-06-10T00:15:55.206 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-10T00:15:55.206 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.wal_recovery_mode: 2 2024-06-10T00:15:55.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.enable_thread_tracking: 0 2024-06-10T00:15:55.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.enable_pipelined_write: 0 2024-06-10T00:15:55.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.unordered_write: 0 2024-06-10T00:15:55.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-10T00:15:55.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-10T00:15:55.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-10T00:15:55.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-10T00:15:55.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.row_cache: None 2024-06-10T00:15:55.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.wal_filter: None 2024-06-10T00:15:55.209 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.allow_ingest_behind: 0 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.two_write_queues: 0 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.manual_wal_flush: 0 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.wal_compression: 0 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.atomic_flush: 0 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.log_readahead_size: 0 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.best_efforts_recovery: 0 2024-06-10T00:15:55.210 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.allow_data_in_errors: 0 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.db_host_id: __hostname__ 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_background_jobs: 2 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_background_compactions: -1 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_subcompactions: 1 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_total_wal_size: 0 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-10T00:15:55.211 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_open_files: -1 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bytes_per_sync: 0 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_readahead_size: 0 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_background_flushes: -1 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Compression algorithms supported: 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: kZSTD supported: 0 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: kXpressCompression supported: 0 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: kLZ4HCCompression supported: 1 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: kZlibCompression supported: 1 2024-06-10T00:15:55.212 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: kSnappyCompression supported: 1 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: kLZ4Compression supported: 1 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: kBZip2Compression supported: 0 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.merge_operator: 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_filter: None 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_filter_factory: None 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.sst_partitioner_factory: None 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558335311c40) 2024-06-10T00:15:55.213 INFO:journalctl@ceph.mon.a.smithi121.stdout: cache_index_and_filter_blocks: 1 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: pin_top_level_index_and_filter: 1 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: index_type: 0 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: data_block_index_type: 0 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: index_shortening: 1 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: checksum: 4 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: no_block_cache: 0 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_cache: 0x558335313090 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_cache_name: BinnedLRUCache 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_cache_options: 2024-06-10T00:15:55.214 INFO:journalctl@ceph.mon.a.smithi121.stdout: capacity : 536870912 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: num_shard_bits : 4 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: strict_capacity_limit : 0 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: high_pri_pool_ratio: 0.000 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_cache_compressed: (nil) 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: persistent_cache: (nil) 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_size: 4096 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_size_deviation: 10 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_restart_interval: 16 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: index_block_restart_interval: 1 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: metadata_block_size: 4096 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: partition_filters: 0 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: use_delta_encoding: 1 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: filter_policy: bloomfilter 2024-06-10T00:15:55.215 INFO:journalctl@ceph.mon.a.smithi121.stdout: whole_key_filtering: 1 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout: verify_compression: 0 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout: read_amp_bytes_per_bit: 0 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout: format_version: 5 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout: enable_index_compression: 1 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout: block_align: 0 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout: max_auto_readahead_size: 262144 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout: prepopulate_block_cache: 0 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout: initial_auto_readahead_size: 8192 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout: num_file_reads_for_auto_readahead: 2 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.write_buffer_size: 33554432 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_write_buffer_number: 2 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compression: NoCompression 2024-06-10T00:15:55.216 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bottommost_compression: Disabled 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.prefix_extractor: nullptr 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.num_levels: 7 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-10T00:15:55.217 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-10T00:15:55.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-10T00:15:55.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-10T00:15:55.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-10T00:15:55.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compression_opts.level: 32767 2024-06-10T00:15:55.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compression_opts.strategy: 0 2024-06-10T00:15:55.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-10T00:15:55.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-10T00:15:55.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-10T00:15:55.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-10T00:15:55.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compression_opts.enabled: false 2024-06-10T00:15:55.218 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.target_file_size_base: 67108864 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-10T00:15:55.219 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.arena_block_size: 1048576 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.disable_auto_compactions: 0 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-10T00:15:55.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.table_properties_collectors: 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.inplace_update_support: 0 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.bloom_locality: 0 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.max_successive_merges: 0 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-10T00:15:55.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.paranoid_file_checks: 0 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.force_consistency_checks: 1 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.report_bg_io_stats: 0 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.ttl: 2592000 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.enable_blob_files: false 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.min_blob_size: 0 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.blob_file_size: 268435456 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-10T00:15:55.222 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.blob_file_starting_level: 0 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 82fe3fb9-03a3-4a3f-89ef-4c058c2888da 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717978554932964, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717978554933928, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100365, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98285, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14298, "raw_average_key_size": 49, "raw_value_size": 90553, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717978554, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "82fe3fb9-03a3-4a3f-89ef-4c058c2888da", "db_session_id": "ZNAT5Q2JOT3F5KU175PU", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717978554934013, "job": 1, "event": "recovery_finished"} 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558335402000 2024-06-10T00:15:55.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: DB pointer 0x5583353ec000 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: ** DB Stats ** 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: ** Compaction Stats [default] ** 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-10T00:15:55.224 INFO:journalctl@ceph.mon.a.smithi121.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: L0 2/0 99.85 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: Sum 2/0 99.85 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: ** Compaction Stats [default] ** 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 164.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-10T00:15:55.225 INFO:journalctl@ceph.mon.a.smithi121.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout: Cumulative compaction: 0.00 GB write, 30.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout: Interval compaction: 0.00 GB write, 30.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout: Block cache BinnedLRUCache@0x558335313090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: starting mon.a rank 0 at public addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] at bind addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: mon.a@-1(???) e1 preinit fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:15:55.226 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: mon.a@-1(???).mds e1 new map 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: mon.a@-1(???).mds e1 print_map 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout: e1 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout: legacy client fscid: -1 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout: 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout: No filesystems configured 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-10T00:15:55.227 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 bash[32359]: abafb9418971b452a9073e897aa81dc2000522fb57ae1e29f9c0b38e8a76495a 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 systemd[1]: Started Ceph mon.a for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: monmap epoch 1 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: last_changed 2024-06-10T00:15:39.680682+0000 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: created 2024-06-10T00:15:39.680682+0000 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: min_mon_release 19 (squid) 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: election_strategy: 1 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: fsmap 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: osdmap e1: 0 total, 0 up, 0 in 2024-06-10T00:15:55.228 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:54 smithi121 ceph-mon[32378]: mgrmap e1: no daemons active 2024-06-10T00:15:55.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.404+0000 7fcdb28df700 1 Processor -- start 2024-06-10T00:15:55.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.404+0000 7fcdb28df700 1 -- start start 2024-06-10T00:15:55.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.405+0000 7fcdb28df700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac108390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:15:55.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.405+0000 7fcdb28df700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcdac1088d0 con 0x7fcdac107f50 2024-06-10T00:15:55.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.405+0000 7fcdabfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:15:55.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.405+0000 7fcdabfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac108390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43254/0 (socket says 172.21.15.121:43254) 2024-06-10T00:15:55.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.405+0000 7fcdabfff700 1 -- 172.21.15.121:0/704535573 learned_addr learned my addr 172.21.15.121:0/704535573 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:15:55.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.405+0000 7fcdabfff700 1 -- 172.21.15.121:0/704535573 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdac1090e0 con 0x7fcdac107f50 2024-06-10T00:15:55.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.405+0000 7fcdabfff700 1 --2- 172.21.15.121:0/704535573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac108390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0098a0 tx=0x7fcd9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2acf598e7afb1d1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:15:55.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.406+0000 7fcdaaffd700 1 -- 172.21.15.121:0/704535573 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd9c004160 con 0x7fcdac107f50 2024-06-10T00:15:55.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.406+0000 7fcdaaffd700 1 -- 172.21.15.121:0/704535573 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcd9c004300 con 0x7fcdac107f50 2024-06-10T00:15:55.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.406+0000 7fcdaaffd700 1 -- 172.21.15.121:0/704535573 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd9c011b50 con 0x7fcdac107f50 2024-06-10T00:15:55.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.406+0000 7fcdb28df700 1 -- 172.21.15.121:0/704535573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 msgr2=0x7fcdac108390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:15:55.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.406+0000 7fcdb28df700 1 --2- 172.21.15.121:0/704535573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac108390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0098a0 tx=0x7fcd9c009bb0 comp rx=0 tx=0).stop 2024-06-10T00:15:55.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.406+0000 7fcdb28df700 1 -- 172.21.15.121:0/704535573 shutdown_connections 2024-06-10T00:15:55.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.406+0000 7fcdb28df700 1 --2- 172.21.15.121:0/704535573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac108390 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:15:55.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.406+0000 7fcdb28df700 1 -- 172.21.15.121:0/704535573 >> 172.21.15.121:0/704535573 conn(0x7fcdac1035a0 msgr2=0x7fcdac1059c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:15:55.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.406+0000 7fcdb28df700 1 -- 172.21.15.121:0/704535573 shutdown_connections 2024-06-10T00:15:55.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.407+0000 7fcdb28df700 1 -- 172.21.15.121:0/704535573 wait complete. 2024-06-10T00:15:55.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.407+0000 7fcdb28df700 1 Processor -- start 2024-06-10T00:15:55.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.407+0000 7fcdb28df700 1 -- start start 2024-06-10T00:15:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.407+0000 7fcdb28df700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac19bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:15:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.407+0000 7fcdb28df700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcdac10c3a0 con 0x7fcdac107f50 2024-06-10T00:15:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.408+0000 7fcdabfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac19bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:15:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.408+0000 7fcdabfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac19bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43260/0 (socket says 172.21.15.121:43260) 2024-06-10T00:15:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.408+0000 7fcdabfff700 1 -- 172.21.15.121:0/2792689789 learned_addr learned my addr 172.21.15.121:0/2792689789 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:15:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.408+0000 7fcdabfff700 1 -- 172.21.15.121:0/2792689789 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdac19c480 con 0x7fcdac107f50 2024-06-10T00:15:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.408+0000 7fcdabfff700 1 --2- 172.21.15.121:0/2792689789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac19bf40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c000c00 tx=0x7fcd9c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:15:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.408+0000 7fcda97fa700 1 -- 172.21.15.121:0/2792689789 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd9c003b20 con 0x7fcdac107f50 2024-06-10T00:15:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.408+0000 7fcdb28df700 1 -- 172.21.15.121:0/2792689789 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdac19c710 con 0x7fcdac107f50 2024-06-10T00:15:55.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.408+0000 7fcda97fa700 1 -- 172.21.15.121:0/2792689789 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcd9c003cc0 con 0x7fcdac107f50 2024-06-10T00:15:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.408+0000 7fcda97fa700 1 -- 172.21.15.121:0/2792689789 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd9c024910 con 0x7fcdac107f50 2024-06-10T00:15:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.408+0000 7fcdb28df700 1 -- 172.21.15.121:0/2792689789 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdac19f400 con 0x7fcdac107f50 2024-06-10T00:15:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.408+0000 7fcda97fa700 1 -- 172.21.15.121:0/2792689789 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcd9c022020 con 0x7fcdac107f50 2024-06-10T00:15:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.409+0000 7fcda97fa700 1 -- 172.21.15.121:0/2792689789 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcd9c011e00 con 0x7fcdac107f50 2024-06-10T00:15:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.409+0000 7fcdb28df700 1 -- 172.21.15.121:0/2792689789 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdac10c520 con 0x7fcdac107f50 2024-06-10T00:15:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.414+0000 7fcda97fa700 1 -- 172.21.15.121:0/2792689789 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fcd9c02ddf0 con 0x7fcdac107f50 2024-06-10T00:15:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.483+0000 7fcdb28df700 1 -- 172.21.15.121:0/2792689789 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fcdac063af0 con 0x7fcdac107f50 2024-06-10T00:15:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.487+0000 7fcda97fa700 1 -- 172.21.15.121:0/2792689789 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fcd9c032030 con 0x7fcdac107f50 2024-06-10T00:15:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.488+0000 7fcdb28df700 1 -- 172.21.15.121:0/2792689789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 msgr2=0x7fcdac19bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:15:55.981 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.488+0000 7fcdb28df700 1 --2- 172.21.15.121:0/2792689789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac19bf40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c000c00 tx=0x7fcd9c011c10 comp rx=0 tx=0).stop 2024-06-10T00:15:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.488+0000 7fcdb28df700 1 -- 172.21.15.121:0/2792689789 shutdown_connections 2024-06-10T00:15:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.488+0000 7fcdb28df700 1 --2- 172.21.15.121:0/2792689789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdac107f50 0x7fcdac19bf40 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:15:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.488+0000 7fcdb28df700 1 -- 172.21.15.121:0/2792689789 >> 172.21.15.121:0/2792689789 conn(0x7fcdac1035a0 msgr2=0x7fcdac104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:15:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.488+0000 7fcdb28df700 1 -- 172.21.15.121:0/2792689789 shutdown_connections 2024-06-10T00:15:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:55.489+0000 7fcdb28df700 1 -- 172.21.15.121:0/2792689789 wait complete. 2024-06-10T00:15:55.982 INFO:teuthology.orchestra.run.smithi121.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-10T00:15:55.995 INFO:teuthology.orchestra.run.smithi121.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-10T00:15:55.995 INFO:teuthology.orchestra.run.smithi121.stdout:Creating mgr... 2024-06-10T00:15:55.995 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying port 0.0.0.0:9283 ... 2024-06-10T00:15:55.996 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying port 0.0.0.0:8765 ... 2024-06-10T00:15:56.241 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.x 2024-06-10T00:15:56.241 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.x.service: Unit ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.x.service not loaded. 2024-06-10T00:15:56.389 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6c432244-26be-11ef-bc9b-c7b262605968.target.wants/ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-6c432244-26be-11ef-bc9b-c7b262605968@.service. 2024-06-10T00:15:56.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:56 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2792689789' entity='client.admin' 2024-06-10T00:15:56.700 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:56 smithi121 podman[32603]: 2024-06-10 00:15:56.648753953 +0000 UTC m=+0.129537565 container create a57b33200d7e35e61755f9a2fb650e53d6eef3559516af6db996e36dcce0e23c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-10T00:15:56.914 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-10T00:15:56.914 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-06-10T00:15:56.922 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-10T00:15:56.922 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-06-10T00:15:56.922 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-06-10T00:15:56.922 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to enable service . firewalld.service is not available 2024-06-10T00:15:56.930 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-10T00:15:56.930 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-06-10T00:15:56.938 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-10T00:15:56.938 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-06-10T00:15:56.938 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-06-10T00:15:56.938 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-06-10T00:15:56.938 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr to start... 2024-06-10T00:15:56.938 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr... 2024-06-10T00:15:56.988 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:56 smithi121 podman[32603]: 2024-06-10 00:15:56.826381277 +0000 UTC m=+0.307164920 container init a57b33200d7e35e61755f9a2fb650e53d6eef3559516af6db996e36dcce0e23c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-10T00:15:56.988 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:56 smithi121 podman[32603]: 2024-06-10 00:15:56.829496269 +0000 UTC m=+0.310279897 container start a57b33200d7e35e61755f9a2fb650e53d6eef3559516af6db996e36dcce0e23c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-10T00:15:56.988 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:56 smithi121 bash[32603]: a57b33200d7e35e61755f9a2fb650e53d6eef3559516af6db996e36dcce0e23c 2024-06-10T00:15:56.988 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:56 smithi121 systemd[1]: Started Ceph mgr.x for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:15:57.281 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:56 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:56.987+0000 7f00cfe98200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-10T00:15:57.281 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:57.176+0000 7f00cfe98200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:15:57.281 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:57.280+0000 7f00cfe98200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-10T00:15:57.606 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:15:57 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/756692214' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-10T00:15:57.949 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:57.605+0000 7f00cfe98200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-10T00:15:57.949 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:57.695+0000 7f00cfe98200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-10T00:15:58.044 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:15:58.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-06-10T00:15:58.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:15:58.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "health": { 2024-06-10T00:15:58.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-10T00:15:58.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-10T00:15:58.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-10T00:15:58.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:15:58.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-10T00:15:58.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-10T00:15:58.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 0 2024-06-10T00:15:58.045 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "a" 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-10T00:15:58.046 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-10T00:15:58.047 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-10T00:15:58.055 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-10T00:15:58.056 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "restful" 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-06-10T00:15:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modified": "2024-06-10T00:15:42.549333+0000", 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.387+0000 7f4058ea7700 1 Processor -- start 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.387+0000 7f4058ea7700 1 -- start start 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.387+0000 7f4058ea7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f4054106620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:15:58.058 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.387+0000 7f4058ea7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4054106bf0 con 0x7f4054106200 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.388+0000 7f405259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f4054106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.388+0000 7f405259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f4054106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43286/0 (socket says 172.21.15.121:43286) 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.388+0000 7f405259c700 1 -- 172.21.15.121:0/3242416535 learned_addr learned my addr 172.21.15.121:0/3242416535 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.388+0000 7f405259c700 1 -- 172.21.15.121:0/3242416535 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4054106d70 con 0x7f4054106200 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.388+0000 7f405259c700 1 --2- 172.21.15.121:0/3242416535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f4054106620 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f40440098a0 tx=0x7f4044009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90bb17c393404903 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.388+0000 7f405159a700 1 -- 172.21.15.121:0/3242416535 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4044004160 con 0x7f4054106200 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.388+0000 7f405159a700 1 -- 172.21.15.121:0/3242416535 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4044004300 con 0x7f4054106200 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.388+0000 7f405159a700 1 -- 172.21.15.121:0/3242416535 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4044011b50 con 0x7f4054106200 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.389+0000 7f4058ea7700 1 -- 172.21.15.121:0/3242416535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 msgr2=0x7f4054106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.389+0000 7f4058ea7700 1 --2- 172.21.15.121:0/3242416535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f4054106620 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f40440098a0 tx=0x7f4044009bb0 comp rx=0 tx=0).stop 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.389+0000 7f4058ea7700 1 -- 172.21.15.121:0/3242416535 shutdown_connections 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.389+0000 7f4058ea7700 1 --2- 172.21.15.121:0/3242416535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f4054106620 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:15:58.059 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.389+0000 7f4058ea7700 1 -- 172.21.15.121:0/3242416535 >> 172.21.15.121:0/3242416535 conn(0x7f4054101b40 msgr2=0x7f4054103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.389+0000 7f4058ea7700 1 -- 172.21.15.121:0/3242416535 shutdown_connections 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.389+0000 7f4058ea7700 1 -- 172.21.15.121:0/3242416535 wait complete. 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.390+0000 7f4058ea7700 1 Processor -- start 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.390+0000 7f4058ea7700 1 -- start start 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.390+0000 7f4058ea7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f405419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.390+0000 7f4058ea7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f405410a0c0 con 0x7f4054106200 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.390+0000 7f405259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f405419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.390+0000 7f405259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f405419bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43298/0 (socket says 172.21.15.121:43298) 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.390+0000 7f405259c700 1 -- 172.21.15.121:0/756692214 learned_addr learned my addr 172.21.15.121:0/756692214 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.391+0000 7f405259c700 1 -- 172.21.15.121:0/756692214 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405419c3e0 con 0x7f4054106200 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.391+0000 7f405259c700 1 --2- 172.21.15.121:0/756692214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f405419bea0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4044011670 tx=0x7f4044011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.391+0000 7f40437fe700 1 -- 172.21.15.121:0/756692214 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f40440039c0 con 0x7f4054106200 2024-06-10T00:15:58.060 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.391+0000 7f4058ea7700 1 -- 172.21.15.121:0/756692214 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f405419c6d0 con 0x7f4054106200 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.391+0000 7f40437fe700 1 -- 172.21.15.121:0/756692214 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4044011e50 con 0x7f4054106200 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.391+0000 7f40437fe700 1 -- 172.21.15.121:0/756692214 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f404401a5c0 con 0x7f4054106200 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.391+0000 7f4058ea7700 1 -- 172.21.15.121:0/756692214 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f405419f360 con 0x7f4054106200 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.391+0000 7f40437fe700 1 -- 172.21.15.121:0/756692214 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f404401ac50 con 0x7f4054106200 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.391+0000 7f40437fe700 1 -- 172.21.15.121:0/756692214 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4044019700 con 0x7f4054106200 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.392+0000 7f4058ea7700 1 -- 172.21.15.121:0/756692214 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4054063af0 con 0x7f4054106200 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.394+0000 7f40437fe700 1 -- 172.21.15.121:0/756692214 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4044032080 con 0x7f4054106200 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.466+0000 7f4058ea7700 1 -- 172.21.15.121:0/756692214 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f405419f530 con 0x7f4054106200 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.467+0000 7f40437fe700 1 -- 172.21.15.121:0/756692214 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4044003b60 con 0x7f4054106200 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.468+0000 7f4058ea7700 1 -- 172.21.15.121:0/756692214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 msgr2=0x7f405419bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.468+0000 7f4058ea7700 1 --2- 172.21.15.121:0/756692214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f405419bea0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4044011670 tx=0x7f4044011b60 comp rx=0 tx=0).stop 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.468+0000 7f4058ea7700 1 -- 172.21.15.121:0/756692214 shutdown_connections 2024-06-10T00:15:58.061 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.468+0000 7f4058ea7700 1 --2- 172.21.15.121:0/756692214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4054106200 0x7f405419bea0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:15:58.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.468+0000 7f4058ea7700 1 -- 172.21.15.121:0/756692214 >> 172.21.15.121:0/756692214 conn(0x7f4054101b40 msgr2=0x7f4054102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:15:58.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.468+0000 7f4058ea7700 1 -- 172.21.15.121:0/756692214 shutdown_connections 2024-06-10T00:15:58.062 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:15:57.468+0000 7f4058ea7700 1 -- 172.21.15.121:0/756692214 wait complete. 2024-06-10T00:15:58.062 INFO:teuthology.orchestra.run.smithi121.stdout:mgr not available, waiting (1/15)... 2024-06-10T00:15:59.199 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:58 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:58.885+0000 7f00cfe98200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-10T00:15:59.199 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:59.051+0000 7f00cfe98200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-10T00:15:59.498 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:59.218+0000 7f00cfe98200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-10T00:15:59.499 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:59.499+0000 7f00cfe98200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-10T00:15:59.786 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:59.588+0000 7f00cfe98200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-10T00:15:59.786 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:59.786+0000 7f00cfe98200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-10T00:16:00.176 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:15:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:15:59.975+0000 7f00cfe98200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:16:00.949 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:00.583+0000 7f00cfe98200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-10T00:16:01.307 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:01 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:01.009+0000 7f00cfe98200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-10T00:16:01.307 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:01 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:01.107+0000 7f00cfe98200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-10T00:16:01.307 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:01 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:01.307+0000 7f00cfe98200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-10T00:16:02.168 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:01 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:01.828+0000 7f00cfe98200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-10T00:16:02.169 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:01 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:01.919+0000 7f00cfe98200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-10T00:16:02.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:02 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:02.169+0000 7f00cfe98200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-10T00:16:02.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:02 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:02.301+0000 7f00cfe98200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-10T00:16:02.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:02 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:02.397+0000 7f00cfe98200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-10T00:16:02.765 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:02 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:02.515+0000 7f00cfe98200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-10T00:16:03.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:02 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1473760872' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-10T00:16:03.100 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:02 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:02.824+0000 7f00cfe98200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-10T00:16:03.327 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:16:03.330 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-06-10T00:16:03.330 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:16:03.330 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "health": { 2024-06-10T00:16:03.330 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-10T00:16:03.330 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-10T00:16:03.330 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 0 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "a" 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-10T00:16:03.331 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:03.332 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-10T00:16:03.332 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:16:03.332 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-10T00:16:03.332 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-10T00:16:03.332 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-10T00:16:03.339 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-10T00:16:03.339 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-10T00:16:03.339 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-10T00:16:03.339 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:03.339 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-10T00:16:03.339 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-10T00:16:03.339 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-10T00:16:03.340 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "restful" 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modified": "2024-06-10T00:15:42.549333+0000", 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-10T00:16:03.341 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.674+0000 7ff5a7b04700 1 Processor -- start 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.674+0000 7ff5a7b04700 1 -- start start 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.675+0000 7ff5a7b04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a0108390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.675+0000 7ff5a7b04700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a01088d0 con 0x7ff5a0107f50 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.675+0000 7ff5a58a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a0108390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.675+0000 7ff5a58a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a0108390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53506/0 (socket says 172.21.15.121:53506) 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.675+0000 7ff5a58a0700 1 -- 172.21.15.121:0/3671452289 learned_addr learned my addr 172.21.15.121:0/3671452289 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.675+0000 7ff5a58a0700 1 -- 172.21.15.121:0/3671452289 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a01090e0 con 0x7ff5a0107f50 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.675+0000 7ff5a58a0700 1 --2- 172.21.15.121:0/3671452289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a0108390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff59c0098a0 tx=0x7ff59c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddcb29b71316698a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.675+0000 7ff5a489e700 1 -- 172.21.15.121:0/3671452289 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff59c004160 con 0x7ff5a0107f50 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.676+0000 7ff5a489e700 1 -- 172.21.15.121:0/3671452289 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff59c004300 con 0x7ff5a0107f50 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.676+0000 7ff5a489e700 1 -- 172.21.15.121:0/3671452289 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff59c011b50 con 0x7ff5a0107f50 2024-06-10T00:16:03.342 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.676+0000 7ff5a7b04700 1 -- 172.21.15.121:0/3671452289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 msgr2=0x7ff5a0108390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.676+0000 7ff5a7b04700 1 --2- 172.21.15.121:0/3671452289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a0108390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff59c0098a0 tx=0x7ff59c009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.676+0000 7ff5a7b04700 1 -- 172.21.15.121:0/3671452289 shutdown_connections 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.676+0000 7ff5a7b04700 1 --2- 172.21.15.121:0/3671452289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a0108390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.676+0000 7ff5a7b04700 1 -- 172.21.15.121:0/3671452289 >> 172.21.15.121:0/3671452289 conn(0x7ff5a01035a0 msgr2=0x7ff5a01059c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.676+0000 7ff5a7b04700 1 -- 172.21.15.121:0/3671452289 shutdown_connections 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.676+0000 7ff5a7b04700 1 -- 172.21.15.121:0/3671452289 wait complete. 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.677+0000 7ff5a7b04700 1 Processor -- start 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.677+0000 7ff5a7b04700 1 -- start start 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.677+0000 7ff5a7b04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.677+0000 7ff5a7b04700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a010c3a0 con 0x7ff5a0107f50 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.677+0000 7ff5a58a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.677+0000 7ff5a58a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a019bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53508/0 (socket says 172.21.15.121:53508) 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.677+0000 7ff5a58a0700 1 -- 172.21.15.121:0/1473760872 learned_addr learned my addr 172.21.15.121:0/1473760872 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:03.343 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.678+0000 7ff5a58a0700 1 -- 172.21.15.121:0/1473760872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a019c480 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.678+0000 7ff5a58a0700 1 --2- 172.21.15.121:0/1473760872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a019bf40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff59c000c00 tx=0x7ff59c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.678+0000 7ff596ffd700 1 -- 172.21.15.121:0/1473760872 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff59c003b20 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.678+0000 7ff596ffd700 1 -- 172.21.15.121:0/1473760872 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff59c003cc0 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.678+0000 7ff596ffd700 1 -- 172.21.15.121:0/1473760872 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff59c024910 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.678+0000 7ff5a7b04700 1 -- 172.21.15.121:0/1473760872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a019c710 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.678+0000 7ff5a7b04700 1 -- 172.21.15.121:0/1473760872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a019f400 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.678+0000 7ff596ffd700 1 -- 172.21.15.121:0/1473760872 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff59c022020 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.678+0000 7ff596ffd700 1 -- 172.21.15.121:0/1473760872 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff59c011e00 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.679+0000 7ff5a7b04700 1 -- 172.21.15.121:0/1473760872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a010c520 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.681+0000 7ff596ffd700 1 -- 172.21.15.121:0/1473760872 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff59c01acd0 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.754+0000 7ff5a7b04700 1 -- 172.21.15.121:0/1473760872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff5a0063af0 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.754+0000 7ff596ffd700 1 -- 172.21.15.121:0/1473760872 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff59c032030 con 0x7ff5a0107f50 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.755+0000 7ff5a7b04700 1 -- 172.21.15.121:0/1473760872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 msgr2=0x7ff5a019bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:03.344 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.755+0000 7ff5a7b04700 1 --2- 172.21.15.121:0/1473760872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a019bf40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff59c000c00 tx=0x7ff59c011c10 comp rx=0 tx=0).stop 2024-06-10T00:16:03.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.755+0000 7ff5a7b04700 1 -- 172.21.15.121:0/1473760872 shutdown_connections 2024-06-10T00:16:03.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.755+0000 7ff5a7b04700 1 --2- 172.21.15.121:0/1473760872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5a0107f50 0x7ff5a019bf40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:03.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.755+0000 7ff5a7b04700 1 -- 172.21.15.121:0/1473760872 >> 172.21.15.121:0/1473760872 conn(0x7ff5a01035a0 msgr2=0x7ff5a0104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:03.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.755+0000 7ff5a7b04700 1 -- 172.21.15.121:0/1473760872 shutdown_connections 2024-06-10T00:16:03.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:02.755+0000 7ff5a7b04700 1 -- 172.21.15.121:0/1473760872 wait complete. 2024-06-10T00:16:03.345 INFO:teuthology.orchestra.run.smithi121.stdout:mgr not available, waiting (2/15)... 2024-06-10T00:16:03.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:03.423+0000 7f00cfe98200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-10T00:16:03.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:03.633+0000 7f00cfe98200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-10T00:16:04.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: Activating manager daemon x 2024-06-10T00:16:04.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: mgrmap e2: x(active, starting, since 0.00306094s) 2024-06-10T00:16:04.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: from='mgr.14100 172.21.15.121:0/1813427849' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-10T00:16:04.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: from='mgr.14100 172.21.15.121:0/1813427849' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-10T00:16:04.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: from='mgr.14100 172.21.15.121:0/1813427849' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-10T00:16:04.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: from='mgr.14100 172.21.15.121:0/1813427849' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:16:04.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: from='mgr.14100 172.21.15.121:0/1813427849' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-10T00:16:04.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: Manager daemon x is now available 2024-06-10T00:16:04.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: from='mgr.14100 172.21.15.121:0/1813427849' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-10T00:16:04.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: from='mgr.14100 172.21.15.121:0/1813427849' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-10T00:16:04.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: from='mgr.14100 172.21.15.121:0/1813427849' entity='mgr.x' 2024-06-10T00:16:04.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: from='mgr.14100 172.21.15.121:0/1813427849' entity='mgr.x' 2024-06-10T00:16:04.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:03 smithi121 ceph-mon[32378]: from='mgr.14100 172.21.15.121:0/1813427849' entity='mgr.x' 2024-06-10T00:16:05.692 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:05 smithi121 ceph-mon[32378]: mgrmap e3: x(active, since 1.00711s) 2024-06-10T00:16:06.735 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:16:06.735 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "health": { 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 0 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "a" 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-06-10T00:16:06.736 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-06-10T00:16:06.737 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-10T00:16:06.737 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:16:06.737 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-06-10T00:16:06.737 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-10T00:16:06.737 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:06.737 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-10T00:16:06.737 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:16:06.737 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-10T00:16:06.737 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-10T00:16:06.737 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-10T00:16:06.745 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-10T00:16:06.745 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-10T00:16:06.745 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-10T00:16:06.745 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:06.745 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-10T00:16:06.745 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-10T00:16:06.745 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:06.746 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "restful" 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ], 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "modified": "2024-06-10T00:15:42.549333+0000", 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-10T00:16:06.747 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout }, 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.691+0000 7f924c714700 1 Processor -- start 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.691+0000 7f924c714700 1 -- start start 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.691+0000 7f924c714700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244107270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.691+0000 7f924c714700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f92441077b0 con 0x7f9244106e50 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.692+0000 7f924a4b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244107270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.692+0000 7f924a4b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244107270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53578/0 (socket says 172.21.15.121:53578) 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.692+0000 7f924a4b0700 1 -- 172.21.15.121:0/1617019862 learned_addr learned my addr 172.21.15.121:0/1617019862 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.692+0000 7f924a4b0700 1 -- 172.21.15.121:0/1617019862 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9244107fb0 con 0x7f9244106e50 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.692+0000 7f924a4b0700 1 --2- 172.21.15.121:0/1617019862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244107270 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f923c009e80 tx=0x7f923c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc18b3f7a3783d8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.692+0000 7f92494ae700 1 -- 172.21.15.121:0/1617019862 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f923c017070 con 0x7f9244106e50 2024-06-10T00:16:06.748 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.692+0000 7f92494ae700 1 -- 172.21.15.121:0/1617019862 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f923c00b8c0 con 0x7f9244106e50 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.692+0000 7f92494ae700 1 -- 172.21.15.121:0/1617019862 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f923c0045c0 con 0x7f9244106e50 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.693+0000 7f924c714700 1 -- 172.21.15.121:0/1617019862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 msgr2=0x7f9244107270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.693+0000 7f924c714700 1 --2- 172.21.15.121:0/1617019862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244107270 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f923c009e80 tx=0x7f923c00b330 comp rx=0 tx=0).stop 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.693+0000 7f924c714700 1 -- 172.21.15.121:0/1617019862 shutdown_connections 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.693+0000 7f924c714700 1 --2- 172.21.15.121:0/1617019862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244107270 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.693+0000 7f924c714700 1 -- 172.21.15.121:0/1617019862 >> 172.21.15.121:0/1617019862 conn(0x7f9244102410 msgr2=0x7f9244104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.693+0000 7f924c714700 1 -- 172.21.15.121:0/1617019862 shutdown_connections 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.693+0000 7f924c714700 1 -- 172.21.15.121:0/1617019862 wait complete. 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.694+0000 7f924c714700 1 Processor -- start 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.694+0000 7f924c714700 1 -- start start 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.694+0000 7f924c714700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244191ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.694+0000 7f924c714700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f924410b3f0 con 0x7f9244106e50 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.694+0000 7f924a4b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244191ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:06.749 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.694+0000 7f924a4b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244191ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53590/0 (socket says 172.21.15.121:53590) 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.694+0000 7f924a4b0700 1 -- 172.21.15.121:0/3989825986 learned_addr learned my addr 172.21.15.121:0/3989825986 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.694+0000 7f924a4b0700 1 -- 172.21.15.121:0/3989825986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9244192530 con 0x7f9244106e50 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.694+0000 7f924a4b0700 1 --2- 172.21.15.121:0/3989825986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244191ff0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f923c004350 tx=0x7f923c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.695+0000 7f92437fe700 1 -- 172.21.15.121:0/3989825986 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f923c020020 con 0x7f9244106e50 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.695+0000 7f92437fe700 1 -- 172.21.15.121:0/3989825986 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f923c003c10 con 0x7f9244106e50 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.695+0000 7f924c714700 1 -- 172.21.15.121:0/3989825986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9244190720 con 0x7f9244106e50 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.695+0000 7f92437fe700 1 -- 172.21.15.121:0/3989825986 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f923c017040 con 0x7f9244106e50 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.695+0000 7f924c714700 1 -- 172.21.15.121:0/3989825986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9244190c60 con 0x7f9244106e50 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.695+0000 7f92437fe700 1 -- 172.21.15.121:0/3989825986 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f923c023020 con 0x7f9244106e50 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.696+0000 7f92437fe700 1 --2- 172.21.15.121:0/3989825986 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f922c039050 0x7f922c03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.696+0000 7f924c714700 1 -- 172.21.15.121:0/3989825986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9244063af0 con 0x7f9244106e50 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.696+0000 7f92437fe700 1 -- 172.21.15.121:0/3989825986 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f923c00f050 con 0x7f9244106e50 2024-06-10T00:16:06.750 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.696+0000 7f9249caf700 1 --2- 172.21.15.121:0/3989825986 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f922c039050 0x7f922c03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.696+0000 7f9249caf700 1 --2- 172.21.15.121:0/3989825986 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f922c039050 0x7f922c03b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9238006fd0 tx=0x7f9238006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.701+0000 7f92437fe700 1 -- 172.21.15.121:0/3989825986 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f923c02a020 con 0x7f9244106e50 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.983+0000 7f924c714700 1 -- 172.21.15.121:0/3989825986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9244190e30 con 0x7f9244106e50 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.983+0000 7f92437fe700 1 -- 172.21.15.121:0/3989825986 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f923c02b260 con 0x7f9244106e50 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.987+0000 7f924c714700 1 -- 172.21.15.121:0/3989825986 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f922c039050 msgr2=0x7f922c03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.987+0000 7f924c714700 1 --2- 172.21.15.121:0/3989825986 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f922c039050 0x7f922c03b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9238006fd0 tx=0x7f9238006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.987+0000 7f924c714700 1 -- 172.21.15.121:0/3989825986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 msgr2=0x7f9244191ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.987+0000 7f924c714700 1 --2- 172.21.15.121:0/3989825986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244191ff0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f923c004350 tx=0x7f923c00bd50 comp rx=0 tx=0).stop 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.987+0000 7f924c714700 1 -- 172.21.15.121:0/3989825986 shutdown_connections 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.987+0000 7f924c714700 1 --2- 172.21.15.121:0/3989825986 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f922c039050 0x7f922c03b510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.987+0000 7f924c714700 1 --2- 172.21.15.121:0/3989825986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9244106e50 0x7f9244191ff0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.987+0000 7f924c714700 1 -- 172.21.15.121:0/3989825986 >> 172.21.15.121:0/3989825986 conn(0x7f9244102410 msgr2=0x7f9244102eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.987+0000 7f924c714700 1 -- 172.21.15.121:0/3989825986 shutdown_connections 2024-06-10T00:16:06.751 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:05.987+0000 7f924c714700 1 -- 172.21.15.121:0/3989825986 wait complete. 2024-06-10T00:16:06.752 INFO:teuthology.orchestra.run.smithi121.stdout:mgr is available 2024-06-10T00:16:06.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:06 smithi121 ceph-mon[32378]: mgrmap e4: x(active, since 2s) 2024-06-10T00:16:06.831 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:06 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3989825986' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-10T00:16:07.853 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:07 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/56212802' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-10T00:16:07.862 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:16:07.862 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [global] 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout fsid = 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout [osd] 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-10T00:16:07.863 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.106+0000 7fd4937b9700 1 Processor -- start 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.106+0000 7fd4937b9700 1 -- start start 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.106+0000 7fd4937b9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.106+0000 7fd4937b9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c108e80 con 0x7fd48c108490 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.106+0000 7fd491555700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.106+0000 7fd491555700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53598/0 (socket says 172.21.15.121:53598) 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.106+0000 7fd491555700 1 -- 172.21.15.121:0/2492919961 learned_addr learned my addr 172.21.15.121:0/2492919961 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.107+0000 7fd491555700 1 -- 172.21.15.121:0/2492919961 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48c1070f0 con 0x7fd48c108490 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.107+0000 7fd491555700 1 --2- 172.21.15.121:0/2492919961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1088b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd4880098a0 tx=0x7fd488009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e9d7b3c8fbdc60 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.107+0000 7fd483fff700 1 -- 172.21.15.121:0/2492919961 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd488004160 con 0x7fd48c108490 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.107+0000 7fd483fff700 1 -- 172.21.15.121:0/2492919961 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd488004300 con 0x7fd48c108490 2024-06-10T00:16:07.864 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.107+0000 7fd483fff700 1 -- 172.21.15.121:0/2492919961 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd488011b50 con 0x7fd48c108490 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.107+0000 7fd4937b9700 1 -- 172.21.15.121:0/2492919961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 msgr2=0x7fd48c1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.107+0000 7fd4937b9700 1 --2- 172.21.15.121:0/2492919961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1088b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd4880098a0 tx=0x7fd488009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.107+0000 7fd4937b9700 1 -- 172.21.15.121:0/2492919961 shutdown_connections 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.107+0000 7fd4937b9700 1 --2- 172.21.15.121:0/2492919961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1088b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.107+0000 7fd4937b9700 1 -- 172.21.15.121:0/2492919961 >> 172.21.15.121:0/2492919961 conn(0x7fd48c101c80 msgr2=0x7fd48c1040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.108+0000 7fd4937b9700 1 -- 172.21.15.121:0/2492919961 shutdown_connections 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.108+0000 7fd4937b9700 1 -- 172.21.15.121:0/2492919961 wait complete. 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.108+0000 7fd4937b9700 1 Processor -- start 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.108+0000 7fd4937b9700 1 -- start start 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.108+0000 7fd4937b9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1920b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.108+0000 7fd4937b9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd48c10b4b0 con 0x7fd48c108490 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.108+0000 7fd491555700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1920b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:07.865 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.108+0000 7fd491555700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1920b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53612/0 (socket says 172.21.15.121:53612) 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.108+0000 7fd491555700 1 -- 172.21.15.121:0/56212802 learned_addr learned my addr 172.21.15.121:0/56212802 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.109+0000 7fd491555700 1 -- 172.21.15.121:0/56212802 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48c1925f0 con 0x7fd48c108490 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.109+0000 7fd491555700 1 --2- 172.21.15.121:0/56212802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1920b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd488011670 tx=0x7fd488011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.109+0000 7fd4827fc700 1 -- 172.21.15.121:0/56212802 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4880039c0 con 0x7fd48c108490 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.109+0000 7fd4937b9700 1 -- 172.21.15.121:0/56212802 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd48c1907e0 con 0x7fd48c108490 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.109+0000 7fd4827fc700 1 -- 172.21.15.121:0/56212802 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd488011e50 con 0x7fd48c108490 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.109+0000 7fd4827fc700 1 -- 172.21.15.121:0/56212802 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd48801a5c0 con 0x7fd48c108490 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.109+0000 7fd4937b9700 1 -- 172.21.15.121:0/56212802 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd48c190d20 con 0x7fd48c108490 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.110+0000 7fd4827fc700 1 -- 172.21.15.121:0/56212802 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fd48801a8c0 con 0x7fd48c108490 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.110+0000 7fd4827fc700 1 --2- 172.21.15.121:0/56212802 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7fd478038fb0 0x7fd47803b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.110+0000 7fd4937b9700 1 -- 172.21.15.121:0/56212802 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd48c063af0 con 0x7fd48c108490 2024-06-10T00:16:07.866 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.110+0000 7fd4827fc700 1 -- 172.21.15.121:0/56212802 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd48804d420 con 0x7fd48c108490 2024-06-10T00:16:07.867 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.110+0000 7fd490d54700 1 --2- 172.21.15.121:0/56212802 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7fd478038fb0 0x7fd47803b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:07.867 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.110+0000 7fd490d54700 1 --2- 172.21.15.121:0/56212802 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7fd478038fb0 0x7fd47803b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd47c006fd0 tx=0x7fd47c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:07.867 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.115+0000 7fd4827fc700 1 -- 172.21.15.121:0/56212802 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd488019c70 con 0x7fd48c108490 2024-06-10T00:16:07.869 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.306+0000 7fd4937b9700 1 -- 172.21.15.121:0/56212802 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fd48c1959c0 con 0x7fd48c108490 2024-06-10T00:16:07.869 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.307+0000 7fd4827fc700 1 -- 172.21.15.121:0/56212802 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fd48802cc00 con 0x7fd48c108490 2024-06-10T00:16:07.869 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.310+0000 7fd4937b9700 1 -- 172.21.15.121:0/56212802 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7fd478038fb0 msgr2=0x7fd47803b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:07.869 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.310+0000 7fd4937b9700 1 --2- 172.21.15.121:0/56212802 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7fd478038fb0 0x7fd47803b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd47c006fd0 tx=0x7fd47c006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:07.869 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.310+0000 7fd4937b9700 1 -- 172.21.15.121:0/56212802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 msgr2=0x7fd48c1920b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:07.870 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.310+0000 7fd4937b9700 1 --2- 172.21.15.121:0/56212802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1920b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd488011670 tx=0x7fd488011b60 comp rx=0 tx=0).stop 2024-06-10T00:16:07.870 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.310+0000 7fd4937b9700 1 -- 172.21.15.121:0/56212802 shutdown_connections 2024-06-10T00:16:07.870 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.310+0000 7fd4937b9700 1 --2- 172.21.15.121:0/56212802 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7fd478038fb0 0x7fd47803b470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:07.870 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.310+0000 7fd4937b9700 1 --2- 172.21.15.121:0/56212802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd48c108490 0x7fd48c1920b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:07.870 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.310+0000 7fd4937b9700 1 -- 172.21.15.121:0/56212802 >> 172.21.15.121:0/56212802 conn(0x7fd48c101c80 msgr2=0x7fd48c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:07.870 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.310+0000 7fd4937b9700 1 -- 172.21.15.121:0/56212802 shutdown_connections 2024-06-10T00:16:07.870 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:07.310+0000 7fd4937b9700 1 -- 172.21.15.121:0/56212802 wait complete. 2024-06-10T00:16:07.870 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling cephadm module... 2024-06-10T00:16:08.943 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:08 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1644827668' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-10T00:16:08.943 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:08 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: ignoring --setuser ceph since I am not root 2024-06-10T00:16:08.943 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:08 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: ignoring --setgroup ceph since I am not root 2024-06-10T00:16:08.943 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:08 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:08.812+0000 7f02dcd41200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-10T00:16:09.193 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:09 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:08.999+0000 7f02dcd41200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:16:09.193 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:09 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:09.102+0000 7f02dcd41200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-10T00:16:09.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.300+0000 7f2a005d4700 1 Processor -- start 2024-06-10T00:16:09.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.300+0000 7f2a005d4700 1 -- start start 2024-06-10T00:16:09.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.300+0000 7f2a005d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f81083e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:09.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.300+0000 7f2a005d4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f29f8108920 con 0x7f29f8107fa0 2024-06-10T00:16:09.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.300+0000 7f29fe370700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f81083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:09.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.300+0000 7f29fe370700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f81083e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53626/0 (socket says 172.21.15.121:53626) 2024-06-10T00:16:09.200 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.300+0000 7f29fe370700 1 -- 172.21.15.121:0/4138780878 learned_addr learned my addr 172.21.15.121:0/4138780878 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.300+0000 7f29fe370700 1 -- 172.21.15.121:0/4138780878 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f81090e0 con 0x7f29f8107fa0 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.301+0000 7f29fe370700 1 --2- 172.21.15.121:0/4138780878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f81083e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f29f40098a0 tx=0x7f29f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a8ec02499b9232 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.301+0000 7f29fd36e700 1 -- 172.21.15.121:0/4138780878 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f4004160 con 0x7f29f8107fa0 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.301+0000 7f29fd36e700 1 -- 172.21.15.121:0/4138780878 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f29f4004300 con 0x7f29f8107fa0 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.301+0000 7f29fd36e700 1 -- 172.21.15.121:0/4138780878 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f4011b50 con 0x7f29f8107fa0 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.301+0000 7f2a005d4700 1 -- 172.21.15.121:0/4138780878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 msgr2=0x7f29f81083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.301+0000 7f2a005d4700 1 --2- 172.21.15.121:0/4138780878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f81083e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f29f40098a0 tx=0x7f29f4009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.301+0000 7f2a005d4700 1 -- 172.21.15.121:0/4138780878 shutdown_connections 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.301+0000 7f2a005d4700 1 --2- 172.21.15.121:0/4138780878 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f81083e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.301+0000 7f2a005d4700 1 -- 172.21.15.121:0/4138780878 >> 172.21.15.121:0/4138780878 conn(0x7f29f8103570 msgr2=0x7f29f81059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.301+0000 7f2a005d4700 1 -- 172.21.15.121:0/4138780878 shutdown_connections 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.301+0000 7f2a005d4700 1 -- 172.21.15.121:0/4138780878 wait complete. 2024-06-10T00:16:09.201 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.302+0000 7f2a005d4700 1 Processor -- start 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.302+0000 7f2a005d4700 1 -- start start 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.302+0000 7f2a005d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.302+0000 7f2a005d4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f29f810c3a0 con 0x7f29f8107fa0 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.302+0000 7f29fe370700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.302+0000 7f29fe370700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f819bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53628/0 (socket says 172.21.15.121:53628) 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.302+0000 7f29fe370700 1 -- 172.21.15.121:0/1644827668 learned_addr learned my addr 172.21.15.121:0/1644827668 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.303+0000 7f29fe370700 1 -- 172.21.15.121:0/1644827668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f819c480 con 0x7f29f8107fa0 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.303+0000 7f29fe370700 1 --2- 172.21.15.121:0/1644827668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f819bf40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f29f4011670 tx=0x7f29f4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.303+0000 7f29ef7fe700 1 -- 172.21.15.121:0/1644827668 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f40039c0 con 0x7f29f8107fa0 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.303+0000 7f29ef7fe700 1 -- 172.21.15.121:0/1644827668 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f29f4011e50 con 0x7f29f8107fa0 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.303+0000 7f2a005d4700 1 -- 172.21.15.121:0/1644827668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f819c710 con 0x7f29f8107fa0 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.303+0000 7f2a005d4700 1 -- 172.21.15.121:0/1644827668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f819f400 con 0x7f29f8107fa0 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.303+0000 7f29ef7fe700 1 -- 172.21.15.121:0/1644827668 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29f401a5c0 con 0x7f29f8107fa0 2024-06-10T00:16:09.202 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.304+0000 7f29ef7fe700 1 -- 172.21.15.121:0/1644827668 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f29f401a890 con 0x7f29f8107fa0 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.304+0000 7f29ef7fe700 1 --2- 172.21.15.121:0/1644827668 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f29e4039000 0x7f29e403b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.304+0000 7f29ef7fe700 1 -- 172.21.15.121:0/1644827668 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f29f404e250 con 0x7f29f8107fa0 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.304+0000 7f29fdb6f700 1 --2- 172.21.15.121:0/1644827668 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f29e4039000 0x7f29e403b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.304+0000 7f2a005d4700 1 -- 172.21.15.121:0/1644827668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f810c520 con 0x7f29f8107fa0 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.304+0000 7f29fdb6f700 1 --2- 172.21.15.121:0/1644827668 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f29e4039000 0x7f29e403b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29e8006fd0 tx=0x7f29e8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.309+0000 7f29ef7fe700 1 -- 172.21.15.121:0/1644827668 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f29f4019c70 con 0x7f29f8107fa0 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.562+0000 7f2a005d4700 1 -- 172.21.15.121:0/1644827668 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f29f819f5d0 con 0x7f29f8107fa0 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.646+0000 7f29ef7fe700 1 -- 172.21.15.121:0/1644827668 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f29f401ad60 con 0x7f29f8107fa0 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.648+0000 7f29ef7fe700 1 -- 172.21.15.121:0/1644827668 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f29f403f020 con 0x7f29f8107fa0 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.649+0000 7f2a005d4700 1 -- 172.21.15.121:0/1644827668 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f29e4039000 msgr2=0x7f29e403b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.649+0000 7f2a005d4700 1 --2- 172.21.15.121:0/1644827668 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f29e4039000 0x7f29e403b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29e8006fd0 tx=0x7f29e8006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.649+0000 7f2a005d4700 1 -- 172.21.15.121:0/1644827668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 msgr2=0x7f29f819bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.649+0000 7f2a005d4700 1 --2- 172.21.15.121:0/1644827668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f819bf40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f29f4011670 tx=0x7f29f4011b60 comp rx=0 tx=0).stop 2024-06-10T00:16:09.203 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.649+0000 7f2a005d4700 1 -- 172.21.15.121:0/1644827668 shutdown_connections 2024-06-10T00:16:09.204 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.649+0000 7f2a005d4700 1 --2- 172.21.15.121:0/1644827668 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f29e4039000 0x7f29e403b4c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:09.204 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.649+0000 7f2a005d4700 1 --2- 172.21.15.121:0/1644827668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29f8107fa0 0x7f29f819bf40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:09.204 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.649+0000 7f2a005d4700 1 -- 172.21.15.121:0/1644827668 >> 172.21.15.121:0/1644827668 conn(0x7f29f8103570 msgr2=0x7f29f8104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:09.204 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.649+0000 7f2a005d4700 1 -- 172.21.15.121:0/1644827668 shutdown_connections 2024-06-10T00:16:09.204 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:08.649+0000 7f2a005d4700 1 -- 172.21.15.121:0/1644827668 wait complete. 2024-06-10T00:16:09.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:09 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:09.419+0000 7f02dcd41200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-10T00:16:09.853 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:09 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1644827668' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-10T00:16:09.854 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:09 smithi121 ceph-mon[32378]: mgrmap e5: x(active, since 5s) 2024-06-10T00:16:09.854 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:09 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:09.511+0000 7f02dcd41200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-10T00:16:10.499 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.646+0000 7f57a0450700 1 Processor -- start 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.646+0000 7f57a0450700 1 -- start start 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.646+0000 7f57a0450700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f5798107280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.646+0000 7f57a0450700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f57981077c0 con 0x7f5798106e60 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.646+0000 7f579e1ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f5798107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.646+0000 7f579e1ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f5798107280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51944/0 (socket says 172.21.15.121:51944) 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.646+0000 7f579e1ec700 1 -- 172.21.15.121:0/3103169090 learned_addr learned my addr 172.21.15.121:0/3103169090 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:10.500 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.647+0000 7f579e1ec700 1 -- 172.21.15.121:0/3103169090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5798107f70 con 0x7f5798106e60 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.647+0000 7f579e1ec700 1 --2- 172.21.15.121:0/3103169090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f5798107280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f57940098a0 tx=0x7f5794009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa04229e6316fe4 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.649+0000 7f579d1ea700 1 -- 172.21.15.121:0/3103169090 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5794004160 con 0x7f5798106e60 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.649+0000 7f579d1ea700 1 -- 172.21.15.121:0/3103169090 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5794004300 con 0x7f5798106e60 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.649+0000 7f579d1ea700 1 -- 172.21.15.121:0/3103169090 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5794011b50 con 0x7f5798106e60 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.650+0000 7f57a0450700 1 -- 172.21.15.121:0/3103169090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 msgr2=0x7f5798107280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.650+0000 7f57a0450700 1 --2- 172.21.15.121:0/3103169090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f5798107280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f57940098a0 tx=0x7f5794009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.650+0000 7f57a0450700 1 -- 172.21.15.121:0/3103169090 shutdown_connections 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.650+0000 7f57a0450700 1 --2- 172.21.15.121:0/3103169090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f5798107280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.650+0000 7f57a0450700 1 -- 172.21.15.121:0/3103169090 >> 172.21.15.121:0/3103169090 conn(0x7f5798102460 msgr2=0x7f5798104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.650+0000 7f57a0450700 1 -- 172.21.15.121:0/3103169090 shutdown_connections 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.650+0000 7f57a0450700 1 -- 172.21.15.121:0/3103169090 wait complete. 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.651+0000 7f57a0450700 1 Processor -- start 2024-06-10T00:16:10.501 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.651+0000 7f57a0450700 1 -- start start 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.651+0000 7f57a0450700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f579807c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.651+0000 7f57a0450700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f579810b3b0 con 0x7f5798106e60 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.651+0000 7f579e1ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f579807c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.651+0000 7f579e1ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f579807c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51958/0 (socket says 172.21.15.121:51958) 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.651+0000 7f579e1ec700 1 -- 172.21.15.121:0/1639516839 learned_addr learned my addr 172.21.15.121:0/1639516839 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.651+0000 7f579e1ec700 1 -- 172.21.15.121:0/1639516839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f579819ec20 con 0x7f5798106e60 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.651+0000 7f579e1ec700 1 --2- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f579807c2d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5794011b60 tx=0x7f5794011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.652+0000 7f578f7fe700 1 -- 172.21.15.121:0/1639516839 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5794003b20 con 0x7f5798106e60 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.652+0000 7f578f7fe700 1 -- 172.21.15.121:0/1639516839 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5794003cc0 con 0x7f5798106e60 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.652+0000 7f57a0450700 1 -- 172.21.15.121:0/1639516839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f579819edf0 con 0x7f5798106e60 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.652+0000 7f578f7fe700 1 -- 172.21.15.121:0/1639516839 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5794024840 con 0x7f5798106e60 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.652+0000 7f57a0450700 1 -- 172.21.15.121:0/1639516839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f579819f210 con 0x7f5798106e60 2024-06-10T00:16:10.502 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.653+0000 7f578f7fe700 1 -- 172.21.15.121:0/1639516839 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f5794022020 con 0x7f5798106e60 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.653+0000 7f57a0450700 1 -- 172.21.15.121:0/1639516839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5798063af0 con 0x7f5798106e60 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.653+0000 7f578f7fe700 1 --2- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f5784039050 0x7f578403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.653+0000 7f578f7fe700 1 -- 172.21.15.121:0/1639516839 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5794011e00 con 0x7f5798106e60 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.653+0000 7f579d9eb700 1 -- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f5784039050 msgr2=0x7f578403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3588432900 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.653+0000 7f579d9eb700 1 --2- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f5784039050 0x7f578403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.658+0000 7f578f7fe700 1 -- 172.21.15.121:0/1639516839 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5794019c70 con 0x7f5798106e60 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.854+0000 7f579d9eb700 1 -- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f5784039050 msgr2=0x7f578403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3588432900 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.854+0000 7f579d9eb700 1 --2- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f5784039050 0x7f578403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.941+0000 7f57a0450700 1 -- 172.21.15.121:0/1639516839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f579819fa50 con 0x7f5798106e60 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.942+0000 7f578f7fe700 1 -- 172.21.15.121:0/1639516839 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f57940172b0 con 0x7f5798106e60 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.945+0000 7f57a0450700 1 -- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f5784039050 msgr2=0x7f578403b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.945+0000 7f57a0450700 1 --2- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f5784039050 0x7f578403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:10.503 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.945+0000 7f57a0450700 1 -- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 msgr2=0x7f579807c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.945+0000 7f57a0450700 1 --2- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f579807c2d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5794011b60 tx=0x7f5794011c40 comp rx=0 tx=0).stop 2024-06-10T00:16:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.945+0000 7f57a0450700 1 -- 172.21.15.121:0/1639516839 shutdown_connections 2024-06-10T00:16:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.945+0000 7f57a0450700 1 --2- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f5784039050 0x7f578403b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.945+0000 7f57a0450700 1 --2- 172.21.15.121:0/1639516839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5798106e60 0x7f579807c2d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.946+0000 7f57a0450700 1 -- 172.21.15.121:0/1639516839 >> 172.21.15.121:0/1639516839 conn(0x7f5798102460 msgr2=0x7f5798103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.946+0000 7f57a0450700 1 -- 172.21.15.121:0/1639516839 shutdown_connections 2024-06-10T00:16:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:09.946+0000 7f57a0450700 1 -- 172.21.15.121:0/1639516839 wait complete. 2024-06-10T00:16:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for the mgr to restart... 2024-06-10T00:16:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr epoch 5... 2024-06-10T00:16:10.742 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:10 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1639516839' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-10T00:16:10.742 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:10 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:10.715+0000 7f02dcd41200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-10T00:16:11.063 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:10 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:10.892+0000 7f02dcd41200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-10T00:16:11.064 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:11.063+0000 7f02dcd41200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-10T00:16:11.644 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:11.353+0000 7f02dcd41200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-10T00:16:11.645 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:11.444+0000 7f02dcd41200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-10T00:16:11.645 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:11.644+0000 7f02dcd41200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-10T00:16:11.949 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:11.836+0000 7f02dcd41200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:16:12.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:12 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:12.433+0000 7f02dcd41200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-10T00:16:13.147 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:12 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:12.851+0000 7f02dcd41200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-10T00:16:13.147 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:12 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:12.948+0000 7f02dcd41200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-10T00:16:13.148 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:13 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:13.147+0000 7f02dcd41200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-10T00:16:13.949 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:13 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:13.662+0000 7f02dcd41200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-10T00:16:13.949 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:13 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:13.753+0000 7f02dcd41200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-10T00:16:14.219 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:13 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:13.994+0000 7f02dcd41200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-10T00:16:14.220 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:14 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:14.125+0000 7f02dcd41200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-10T00:16:14.220 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:14 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:14.219+0000 7f02dcd41200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-10T00:16:14.629 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:14 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:14.333+0000 7f02dcd41200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-10T00:16:14.629 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:14 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:14.629+0000 7f02dcd41200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-10T00:16:15.493 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:15.223+0000 7f02dcd41200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-10T00:16:15.493 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:15.440+0000 7f02dcd41200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-10T00:16:15.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: Active manager daemon x restarted 2024-06-10T00:16:15.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: Activating manager daemon x 2024-06-10T00:16:15.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: osdmap e2: 0 total, 0 up, 0 in 2024-06-10T00:16:15.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: mgrmap e6: x(active, starting, since 0.00369152s) 2024-06-10T00:16:15.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:16:15.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-10T00:16:15.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-10T00:16:15.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-10T00:16:15.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-10T00:16:15.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: Manager daemon x is now available 2024-06-10T00:16:15.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:15.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:15.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:16:15.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:16:15.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:15 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-10T00:16:16.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:16 smithi121 ceph-mon[32378]: Found migration_current of "None". Setting to last migration. 2024-06-10T00:16:16.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:16 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-10T00:16:16.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:16 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:16.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:16 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:16.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:16 smithi121 ceph-mon[32378]: mgrmap e7: x(active, since 1.00602s) 2024-06-10T00:16:17.022 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-06-10T00:16:17.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-10T00:16:17.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-10T00:16:17.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-06-10T00:16:17.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.947+0000 7f671dcf9700 1 Processor -- start 2024-06-10T00:16:17.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.947+0000 7f671dcf9700 1 -- start start 2024-06-10T00:16:17.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.947+0000 7f671dcf9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671810aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:17.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.947+0000 7f671dcf9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f671807c630 con 0x7f6718108ac0 2024-06-10T00:16:17.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.947+0000 7f67177fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671810aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:17.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.948+0000 7f67177fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671810aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51962/0 (socket says 172.21.15.121:51962) 2024-06-10T00:16:17.023 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.948+0000 7f67177fe700 1 -- 172.21.15.121:0/1412591580 learned_addr learned my addr 172.21.15.121:0/1412591580 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.948+0000 7f67177fe700 1 -- 172.21.15.121:0/1412591580 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671810b3f0 con 0x7f6718108ac0 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.948+0000 7f67177fe700 1 --2- 172.21.15.121:0/1412591580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671810aeb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f670c0098a0 tx=0x7f670c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40bd054570571df2 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.948+0000 7f67167fc700 1 -- 172.21.15.121:0/1412591580 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f670c004160 con 0x7f6718108ac0 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.948+0000 7f67167fc700 1 -- 172.21.15.121:0/1412591580 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f670c004300 con 0x7f6718108ac0 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.948+0000 7f67167fc700 1 -- 172.21.15.121:0/1412591580 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f670c011b50 con 0x7f6718108ac0 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.949+0000 7f671dcf9700 1 -- 172.21.15.121:0/1412591580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 msgr2=0x7f671810aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.949+0000 7f671dcf9700 1 --2- 172.21.15.121:0/1412591580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671810aeb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f670c0098a0 tx=0x7f670c009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.949+0000 7f671dcf9700 1 -- 172.21.15.121:0/1412591580 shutdown_connections 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.949+0000 7f671dcf9700 1 --2- 172.21.15.121:0/1412591580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671810aeb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.949+0000 7f671dcf9700 1 -- 172.21.15.121:0/1412591580 >> 172.21.15.121:0/1412591580 conn(0x7f6718102430 msgr2=0x7f6718104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.949+0000 7f671dcf9700 1 -- 172.21.15.121:0/1412591580 shutdown_connections 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.949+0000 7f671dcf9700 1 -- 172.21.15.121:0/1412591580 wait complete. 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.950+0000 7f671dcf9700 1 Processor -- start 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.950+0000 7f671dcf9700 1 -- start start 2024-06-10T00:16:17.024 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.950+0000 7f671dcf9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671819bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.950+0000 7f671dcf9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f671810c350 con 0x7f6718108ac0 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.950+0000 7f67177fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671819bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.950+0000 7f67177fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671819bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51964/0 (socket says 172.21.15.121:51964) 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.950+0000 7f67177fe700 1 -- 172.21.15.121:0/1294410286 learned_addr learned my addr 172.21.15.121:0/1294410286 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.950+0000 7f67177fe700 1 -- 172.21.15.121:0/1294410286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671819c490 con 0x7f6718108ac0 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.951+0000 7f67177fe700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671819bf50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f670c009fb0 tx=0x7f670c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.951+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f670c0038b0 con 0x7f6718108ac0 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.951+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f670c011e50 con 0x7f6718108ac0 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.951+0000 7f671dcf9700 1 -- 172.21.15.121:0/1294410286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f671819c720 con 0x7f6718108ac0 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.951+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f670c019420 con 0x7f6718108ac0 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.951+0000 7f671dcf9700 1 -- 172.21.15.121:0/1294410286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f671819f410 con 0x7f6718108ac0 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.952+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f670c019730 con 0x7f6718108ac0 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.952+0000 7f6714ff9700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 0x7f670003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.952+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 --> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f670003bc20 con 0x7f6700039050 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.952+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f670c04d360 con 0x7f6718108ac0 2024-06-10T00:16:17.025 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.952+0000 7f6716ffd700 1 -- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 msgr2=0x7f670003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3588432900 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:10.952+0000 7f6716ffd700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 0x7f670003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:11.153+0000 7f6716ffd700 1 -- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 msgr2=0x7f670003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3588432900 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:11.153+0000 7f6716ffd700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 0x7f670003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:11.554+0000 7f6716ffd700 1 -- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 msgr2=0x7f670003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3588432900 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:11.554+0000 7f6716ffd700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 0x7f670003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:12.355+0000 7f6716ffd700 1 -- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 msgr2=0x7f670003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3588432900 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:12.355+0000 7f6716ffd700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 0x7f670003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:13.956+0000 7f6716ffd700 1 -- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 msgr2=0x7f670003b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3588432900 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:13.956+0000 7f6716ffd700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 0x7f670003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:15.444+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f670c019a70 con 0x7f6718108ac0 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:15.444+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 msgr2=0x7f670003b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:16:17.026 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:15.444+0000 7f6714ff9700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 0x7f670003b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:17.027 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.445+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f670c003a50 con 0x7f6718108ac0 2024-06-10T00:16:17.027 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.446+0000 7f6714ff9700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f670003c980 0x7f670003ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:17.027 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.446+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 --> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f670c012ea0 con 0x7f670003c980 2024-06-10T00:16:17.027 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.446+0000 7f6716ffd700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f670003c980 0x7f670003ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:17.027 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.447+0000 7f6716ffd700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f670003c980 0x7f670003ed70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6708003a10 tx=0x7f67080092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:17.027 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.447+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 <== mgr.14118 v2:172.21.15.121:6800/704666019 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f670c012ea0 con 0x7f670003c980 2024-06-10T00:16:17.027 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 -- 172.21.15.121:0/1294410286 --> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f67180695c0 con 0x7f670003c980 2024-06-10T00:16:17.027 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f6714ff9700 1 -- 172.21.15.121:0/1294410286 <== mgr.14118 v2:172.21.15.121:6800/704666019 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f67180695c0 con 0x7f670003c980 2024-06-10T00:16:17.028 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 -- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f670003c980 msgr2=0x7f670003ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:17.028 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f670003c980 0x7f670003ed70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6708003a10 tx=0x7f67080092b0 comp rx=0 tx=0).stop 2024-06-10T00:16:17.028 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 -- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 msgr2=0x7f671819bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:17.028 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671819bf50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f670c009fb0 tx=0x7f670c011bf0 comp rx=0 tx=0).stop 2024-06-10T00:16:17.028 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 -- 172.21.15.121:0/1294410286 shutdown_connections 2024-06-10T00:16:17.028 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f670003c980 0x7f670003ed70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:17.028 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6718108ac0 0x7f671819bf50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:17.029 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 --2- 172.21.15.121:0/1294410286 >> [v2:172.21.15.121:6800/3588432900,v1:172.21.15.121:6801/3588432900] conn(0x7f6700039050 0x7f670003b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:17.029 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 -- 172.21.15.121:0/1294410286 >> 172.21.15.121:0/1294410286 conn(0x7f6718102430 msgr2=0x7f6718103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:17.029 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 -- 172.21.15.121:0/1294410286 shutdown_connections 2024-06-10T00:16:17.029 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:16.454+0000 7f671dcf9700 1 -- 172.21.15.121:0/1294410286 wait complete. 2024-06-10T00:16:17.029 INFO:teuthology.orchestra.run.smithi121.stdout:mgr epoch 5 is available 2024-06-10T00:16:17.029 INFO:teuthology.orchestra.run.smithi121.stdout:Setting orchestrator backend to cephadm... 2024-06-10T00:16:17.657 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:17 smithi121 ceph-mon[32378]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-10T00:16:17.657 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:17 smithi121 ceph-mon[32378]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-10T00:16:17.657 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:17 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:16:18.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.434+0000 7fdb34392700 1 Processor -- start 2024-06-10T00:16:18.211 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.434+0000 7fdb34392700 1 -- start start 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.435+0000 7fdb34392700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c108430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.435+0000 7fdb34392700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c108a00 con 0x7fdb2c108010 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.435+0000 7fdb3212e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c108430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.435+0000 7fdb3212e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c108430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52016/0 (socket says 172.21.15.121:52016) 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.435+0000 7fdb3212e700 1 -- 172.21.15.121:0/2149530494 learned_addr learned my addr 172.21.15.121:0/2149530494 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.435+0000 7fdb3212e700 1 -- 172.21.15.121:0/2149530494 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2c1091c0 con 0x7fdb2c108010 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.435+0000 7fdb3212e700 1 --2- 172.21.15.121:0/2149530494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c108430 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdb280098a0 tx=0x7fdb28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4208bf1a9053177e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.435+0000 7fdb3112c700 1 -- 172.21.15.121:0/2149530494 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb28004160 con 0x7fdb2c108010 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.435+0000 7fdb3112c700 1 -- 172.21.15.121:0/2149530494 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdb28004300 con 0x7fdb2c108010 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.435+0000 7fdb3112c700 1 -- 172.21.15.121:0/2149530494 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb28011b50 con 0x7fdb2c108010 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.436+0000 7fdb34392700 1 -- 172.21.15.121:0/2149530494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 msgr2=0x7fdb2c108430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.436+0000 7fdb34392700 1 --2- 172.21.15.121:0/2149530494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c108430 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdb280098a0 tx=0x7fdb28009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:18.212 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.436+0000 7fdb34392700 1 -- 172.21.15.121:0/2149530494 shutdown_connections 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.436+0000 7fdb34392700 1 --2- 172.21.15.121:0/2149530494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c108430 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.436+0000 7fdb34392700 1 -- 172.21.15.121:0/2149530494 >> 172.21.15.121:0/2149530494 conn(0x7fdb2c103610 msgr2=0x7fdb2c105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.436+0000 7fdb34392700 1 -- 172.21.15.121:0/2149530494 shutdown_connections 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.436+0000 7fdb34392700 1 -- 172.21.15.121:0/2149530494 wait complete. 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb34392700 1 Processor -- start 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb34392700 1 -- start start 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb34392700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c19bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb34392700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2c10c570 con 0x7fdb2c108010 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb3212e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c19bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb3212e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c19bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52026/0 (socket says 172.21.15.121:52026) 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb3212e700 1 -- 172.21.15.121:0/2620151184 learned_addr learned my addr 172.21.15.121:0/2620151184 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb3212e700 1 -- 172.21.15.121:0/2620151184 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2c19c4b0 con 0x7fdb2c108010 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb3212e700 1 --2- 172.21.15.121:0/2620151184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c19bf70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdb28000c00 tx=0x7fdb28011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:18.213 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb1f7fe700 1 -- 172.21.15.121:0/2620151184 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb28003b20 con 0x7fdb2c108010 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb1f7fe700 1 -- 172.21.15.121:0/2620151184 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdb28003cc0 con 0x7fdb2c108010 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb1f7fe700 1 -- 172.21.15.121:0/2620151184 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb28024910 con 0x7fdb2c108010 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.437+0000 7fdb34392700 1 -- 172.21.15.121:0/2620151184 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb2c19c740 con 0x7fdb2c108010 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.438+0000 7fdb34392700 1 -- 172.21.15.121:0/2620151184 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb2c19f430 con 0x7fdb2c108010 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.438+0000 7fdb1f7fe700 1 -- 172.21.15.121:0/2620151184 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fdb28022020 con 0x7fdb2c108010 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.438+0000 7fdb34392700 1 -- 172.21.15.121:0/2620151184 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb2c063af0 con 0x7fdb2c108010 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.439+0000 7fdb1f7fe700 1 --2- 172.21.15.121:0/2620151184 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fdb18038f80 0x7fdb1803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.439+0000 7fdb1f7fe700 1 -- 172.21.15.121:0/2620151184 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdb28011e00 con 0x7fdb2c108010 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.439+0000 7fdb3192d700 1 --2- 172.21.15.121:0/2620151184 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fdb18038f80 0x7fdb1803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.439+0000 7fdb3192d700 1 --2- 172.21.15.121:0/2620151184 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fdb18038f80 0x7fdb1803b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdb20006fd0 tx=0x7fdb20006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.444+0000 7fdb1f7fe700 1 -- 172.21.15.121:0/2620151184 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdb28017ca0 con 0x7fdb2c108010 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.497+0000 7fdb1f7fe700 1 -- 172.21.15.121:0/2620151184 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fdb2801a430 con 0x7fdb2c108010 2024-06-10T00:16:18.214 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.657+0000 7fdb34392700 1 -- 172.21.15.121:0/2620151184 --> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fdb2c1048d0 con 0x7fdb18038f80 2024-06-10T00:16:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.663+0000 7fdb1f7fe700 1 -- 172.21.15.121:0/2620151184 <== mgr.14118 v2:172.21.15.121:6800/704666019 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdb2c1048d0 con 0x7fdb18038f80 2024-06-10T00:16:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.666+0000 7fdb34392700 1 -- 172.21.15.121:0/2620151184 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fdb18038f80 msgr2=0x7fdb1803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.666+0000 7fdb34392700 1 --2- 172.21.15.121:0/2620151184 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fdb18038f80 0x7fdb1803b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdb20006fd0 tx=0x7fdb20006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.666+0000 7fdb34392700 1 -- 172.21.15.121:0/2620151184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 msgr2=0x7fdb2c19bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.666+0000 7fdb34392700 1 --2- 172.21.15.121:0/2620151184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c19bf70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdb28000c00 tx=0x7fdb28011c10 comp rx=0 tx=0).stop 2024-06-10T00:16:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.666+0000 7fdb34392700 1 -- 172.21.15.121:0/2620151184 shutdown_connections 2024-06-10T00:16:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.666+0000 7fdb34392700 1 --2- 172.21.15.121:0/2620151184 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fdb18038f80 0x7fdb1803b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.666+0000 7fdb34392700 1 --2- 172.21.15.121:0/2620151184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb2c108010 0x7fdb2c19bf70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.666+0000 7fdb34392700 1 -- 172.21.15.121:0/2620151184 >> 172.21.15.121:0/2620151184 conn(0x7fdb2c103610 msgr2=0x7fdb2c1041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.666+0000 7fdb34392700 1 -- 172.21.15.121:0/2620151184 shutdown_connections 2024-06-10T00:16:18.215 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:17.666+0000 7fdb34392700 1 -- 172.21.15.121:0/2620151184 wait complete. 2024-06-10T00:16:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:18 smithi121 ceph-mon[32378]: [10/Jun/2024:00:16:16] ENGINE Bus STARTING 2024-06-10T00:16:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:18 smithi121 ceph-mon[32378]: [10/Jun/2024:00:16:16] ENGINE Serving on https://172.21.15.121:7150 2024-06-10T00:16:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:18 smithi121 ceph-mon[32378]: [10/Jun/2024:00:16:17] ENGINE Serving on http://172.21.15.121:8765 2024-06-10T00:16:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:18 smithi121 ceph-mon[32378]: [10/Jun/2024:00:16:17] ENGINE Bus STARTED 2024-06-10T00:16:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:18 smithi121 ceph-mon[32378]: mgrmap e8: x(active, since 2s) 2024-06-10T00:16:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:18 smithi121 ceph-mon[32378]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:16:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:18 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:18.581 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:18 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:16:19.334 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-10T00:16:19.334 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.615+0000 7fcb385b5700 1 Processor -- start 2024-06-10T00:16:19.334 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.615+0000 7fcb385b5700 1 -- start start 2024-06-10T00:16:19.334 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.615+0000 7fcb385b5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb301083e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:19.334 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.615+0000 7fcb385b5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcb30108920 con 0x7fcb30107fa0 2024-06-10T00:16:19.334 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.616+0000 7fcb36351700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb301083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:19.334 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.616+0000 7fcb36351700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb301083e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52042/0 (socket says 172.21.15.121:52042) 2024-06-10T00:16:19.334 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.616+0000 7fcb36351700 1 -- 172.21.15.121:0/1297762636 learned_addr learned my addr 172.21.15.121:0/1297762636 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:19.334 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.616+0000 7fcb36351700 1 -- 172.21.15.121:0/1297762636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb301090e0 con 0x7fcb30107fa0 2024-06-10T00:16:19.334 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.616+0000 7fcb36351700 1 --2- 172.21.15.121:0/1297762636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb301083e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009e80 tx=0x7fcb2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624ac6ac2d491ec9 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.616+0000 7fcb3534f700 1 -- 172.21.15.121:0/1297762636 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb2c017070 con 0x7fcb30107fa0 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.616+0000 7fcb3534f700 1 -- 172.21.15.121:0/1297762636 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcb2c00b8c0 con 0x7fcb30107fa0 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.616+0000 7fcb3534f700 1 -- 172.21.15.121:0/1297762636 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb2c0045c0 con 0x7fcb30107fa0 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.617+0000 7fcb385b5700 1 -- 172.21.15.121:0/1297762636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 msgr2=0x7fcb301083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.617+0000 7fcb385b5700 1 --2- 172.21.15.121:0/1297762636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb301083e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009e80 tx=0x7fcb2c00b330 comp rx=0 tx=0).stop 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.617+0000 7fcb385b5700 1 -- 172.21.15.121:0/1297762636 shutdown_connections 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.617+0000 7fcb385b5700 1 --2- 172.21.15.121:0/1297762636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb301083e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.617+0000 7fcb385b5700 1 -- 172.21.15.121:0/1297762636 >> 172.21.15.121:0/1297762636 conn(0x7fcb301035b0 msgr2=0x7fcb301059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.617+0000 7fcb385b5700 1 -- 172.21.15.121:0/1297762636 shutdown_connections 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.617+0000 7fcb385b5700 1 -- 172.21.15.121:0/1297762636 wait complete. 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.617+0000 7fcb385b5700 1 Processor -- start 2024-06-10T00:16:19.335 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb385b5700 1 -- start start 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb385b5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb3019beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb385b5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3010c3a0 con 0x7fcb30107fa0 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb36351700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb3019beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb36351700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb3019beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52058/0 (socket says 172.21.15.121:52058) 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb36351700 1 -- 172.21.15.121:0/3045600772 learned_addr learned my addr 172.21.15.121:0/3045600772 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb36351700 1 -- 172.21.15.121:0/3045600772 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb3019c3f0 con 0x7fcb30107fa0 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb36351700 1 --2- 172.21.15.121:0/3045600772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb3019beb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c0072a0 tx=0x7fcb2c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb277fe700 1 -- 172.21.15.121:0/3045600772 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb2c020020 con 0x7fcb30107fa0 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb385b5700 1 -- 172.21.15.121:0/3045600772 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb3019c680 con 0x7fcb30107fa0 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb277fe700 1 -- 172.21.15.121:0/3045600772 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcb2c003b50 con 0x7fcb30107fa0 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb277fe700 1 -- 172.21.15.121:0/3045600772 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcb2c017040 con 0x7fcb30107fa0 2024-06-10T00:16:19.336 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.618+0000 7fcb385b5700 1 -- 172.21.15.121:0/3045600772 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb3019f370 con 0x7fcb30107fa0 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.619+0000 7fcb385b5700 1 -- 172.21.15.121:0/3045600772 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb3010c520 con 0x7fcb30107fa0 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.619+0000 7fcb277fe700 1 -- 172.21.15.121:0/3045600772 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fcb2c023020 con 0x7fcb30107fa0 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.620+0000 7fcb277fe700 1 --2- 172.21.15.121:0/3045600772 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcb1c039000 0x7fcb1c03b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.620+0000 7fcb277fe700 1 -- 172.21.15.121:0/3045600772 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcb2c00f050 con 0x7fcb30107fa0 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.620+0000 7fcb35b50700 1 --2- 172.21.15.121:0/3045600772 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcb1c039000 0x7fcb1c03b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.620+0000 7fcb35b50700 1 --2- 172.21.15.121:0/3045600772 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcb1c039000 0x7fcb1c03b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb20006fd0 tx=0x7fcb20006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.624+0000 7fcb277fe700 1 -- 172.21.15.121:0/3045600772 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcb2c02a020 con 0x7fcb30107fa0 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.821+0000 7fcb385b5700 1 -- 172.21.15.121:0/3045600772 --> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fcb301047b0 con 0x7fcb1c039000 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.821+0000 7fcb277fe700 1 -- 172.21.15.121:0/3045600772 <== mgr.14118 v2:172.21.15.121:6800/704666019 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fcb301047b0 con 0x7fcb1c039000 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.824+0000 7fcb385b5700 1 -- 172.21.15.121:0/3045600772 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcb1c039000 msgr2=0x7fcb1c03b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.824+0000 7fcb385b5700 1 --2- 172.21.15.121:0/3045600772 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcb1c039000 0x7fcb1c03b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb20006fd0 tx=0x7fcb20006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:19.337 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.824+0000 7fcb385b5700 1 -- 172.21.15.121:0/3045600772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 msgr2=0x7fcb3019beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:19.338 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.824+0000 7fcb385b5700 1 --2- 172.21.15.121:0/3045600772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb3019beb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c0072a0 tx=0x7fcb2c00bd50 comp rx=0 tx=0).stop 2024-06-10T00:16:19.338 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.824+0000 7fcb385b5700 1 -- 172.21.15.121:0/3045600772 shutdown_connections 2024-06-10T00:16:19.338 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.824+0000 7fcb385b5700 1 --2- 172.21.15.121:0/3045600772 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcb1c039000 0x7fcb1c03b4c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:19.338 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.824+0000 7fcb385b5700 1 --2- 172.21.15.121:0/3045600772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb30107fa0 0x7fcb3019beb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:19.338 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.824+0000 7fcb385b5700 1 -- 172.21.15.121:0/3045600772 >> 172.21.15.121:0/3045600772 conn(0x7fcb301035b0 msgr2=0x7fcb301040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:19.338 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.824+0000 7fcb385b5700 1 -- 172.21.15.121:0/3045600772 shutdown_connections 2024-06-10T00:16:19.338 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:18.824+0000 7fcb385b5700 1 -- 172.21.15.121:0/3045600772 wait complete. 2024-06-10T00:16:19.338 INFO:teuthology.orchestra.run.smithi121.stdout:Generating ssh key... 2024-06-10T00:16:20.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:19 smithi121 ceph-mon[32378]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:16:20.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: Generating public/private rsa key pair. 2024-06-10T00:16:20.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: Your identification has been saved in /tmp/tmpzi02x4dv/key. 2024-06-10T00:16:20.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: Your public key has been saved in /tmp/tmpzi02x4dv/key.pub. 2024-06-10T00:16:20.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: The key fingerprint is: 2024-06-10T00:16:20.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: SHA256:z/MFBWXUGWmDVvpvlsXWUeGAoXhyZm0SzDPzgSaXDoA ceph-6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:16:20.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: The key's randomart image is: 2024-06-10T00:16:20.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: +---[RSA 3072]----+ 2024-06-10T00:16:20.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: | ... o.o.++*=*| 2024-06-10T00:16:20.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: | E o.@+o =+=o| 2024-06-10T00:16:20.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: | o*B=oo.oo.| 2024-06-10T00:16:20.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: | *.o. ...o| 2024-06-10T00:16:20.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: | S . .=| 2024-06-10T00:16:20.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: | o . .+| 2024-06-10T00:16:20.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: | + ..+| 2024-06-10T00:16:20.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: | o . o | 2024-06-10T00:16:20.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: | . | 2024-06-10T00:16:20.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: +----[SHA256]-----+ 2024-06-10T00:16:20.722 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.745+0000 7fccc07c4700 1 Processor -- start 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.745+0000 7fccc07c4700 1 -- start start 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.746+0000 7fccc07c4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb8106c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.746+0000 7fccc07c4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fccb807c640 con 0x7fccb8106860 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.746+0000 7fccbe560700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb8106c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.746+0000 7fccbe560700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb8106c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33646/0 (socket says 172.21.15.121:33646) 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.746+0000 7fccbe560700 1 -- 172.21.15.121:0/889973957 learned_addr learned my addr 172.21.15.121:0/889973957 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.746+0000 7fccbe560700 1 -- 172.21.15.121:0/889973957 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb81071c0 con 0x7fccb8106860 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.746+0000 7fccbe560700 1 --2- 172.21.15.121:0/889973957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb8106c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009e80 tx=0x7fccb000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9312bb5f194e28dd server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.746+0000 7fccbd55e700 1 -- 172.21.15.121:0/889973957 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb0017070 con 0x7fccb8106860 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.746+0000 7fccbd55e700 1 -- 172.21.15.121:0/889973957 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fccb000b8c0 con 0x7fccb8106860 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.746+0000 7fccbd55e700 1 -- 172.21.15.121:0/889973957 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb00045c0 con 0x7fccb8106860 2024-06-10T00:16:20.723 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.747+0000 7fccc07c4700 1 -- 172.21.15.121:0/889973957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 msgr2=0x7fccb8106c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.747+0000 7fccc07c4700 1 --2- 172.21.15.121:0/889973957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb8106c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fccb0009e80 tx=0x7fccb000b330 comp rx=0 tx=0).stop 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.747+0000 7fccc07c4700 1 -- 172.21.15.121:0/889973957 shutdown_connections 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.747+0000 7fccc07c4700 1 --2- 172.21.15.121:0/889973957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb8106c80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.747+0000 7fccc07c4700 1 -- 172.21.15.121:0/889973957 >> 172.21.15.121:0/889973957 conn(0x7fccb81024c0 msgr2=0x7fccb81048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.747+0000 7fccc07c4700 1 -- 172.21.15.121:0/889973957 shutdown_connections 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.747+0000 7fccc07c4700 1 -- 172.21.15.121:0/889973957 wait complete. 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.748+0000 7fccc07c4700 1 Processor -- start 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.748+0000 7fccc07c4700 1 -- start start 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.748+0000 7fccc07c4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb819e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.748+0000 7fccc07c4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fccb81080d0 con 0x7fccb8106860 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.748+0000 7fccbe560700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb819e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.748+0000 7fccbe560700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb819e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33660/0 (socket says 172.21.15.121:33660) 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.748+0000 7fccbe560700 1 -- 172.21.15.121:0/2652005717 learned_addr learned my addr 172.21.15.121:0/2652005717 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:20.724 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.748+0000 7fccbe560700 1 -- 172.21.15.121:0/2652005717 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb819e5b0 con 0x7fccb8106860 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.748+0000 7fccbe560700 1 --2- 172.21.15.121:0/2652005717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb819e070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fccb0004350 tx=0x7fccb000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.749+0000 7fccb77fe700 1 -- 172.21.15.121:0/2652005717 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb0020020 con 0x7fccb8106860 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.749+0000 7fccb77fe700 1 -- 172.21.15.121:0/2652005717 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fccb0003c10 con 0x7fccb8106860 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.749+0000 7fccc07c4700 1 -- 172.21.15.121:0/2652005717 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb819e840 con 0x7fccb8106860 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.749+0000 7fccb77fe700 1 -- 172.21.15.121:0/2652005717 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccb0017040 con 0x7fccb8106860 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.749+0000 7fccc07c4700 1 -- 172.21.15.121:0/2652005717 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb81a1530 con 0x7fccb8106860 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.749+0000 7fccb77fe700 1 -- 172.21.15.121:0/2652005717 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fccb0023020 con 0x7fccb8106860 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.750+0000 7fccc07c4700 1 -- 172.21.15.121:0/2652005717 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccb8063af0 con 0x7fccb8106860 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.750+0000 7fccb77fe700 1 --2- 172.21.15.121:0/2652005717 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcca0039050 0x7fcca003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.750+0000 7fccb77fe700 1 -- 172.21.15.121:0/2652005717 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fccb000f050 con 0x7fccb8106860 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.750+0000 7fccbdd5f700 1 --2- 172.21.15.121:0/2652005717 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcca0039050 0x7fcca003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.750+0000 7fccbdd5f700 1 --2- 172.21.15.121:0/2652005717 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcca0039050 0x7fcca003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fccac006fd0 tx=0x7fccac006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.755+0000 7fccb77fe700 1 -- 172.21.15.121:0/2652005717 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fccb002a020 con 0x7fccb8106860 2024-06-10T00:16:20.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:19.953+0000 7fccc07c4700 1 -- 172.21.15.121:0/2652005717 --> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fccb802e080 con 0x7fcca0039050 2024-06-10T00:16:20.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:20.177+0000 7fccb77fe700 1 -- 172.21.15.121:0/2652005717 <== mgr.14118 v2:172.21.15.121:6800/704666019 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fccb802e080 con 0x7fcca0039050 2024-06-10T00:16:20.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:20.179+0000 7fccc07c4700 1 -- 172.21.15.121:0/2652005717 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcca0039050 msgr2=0x7fcca003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:20.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:20.179+0000 7fccc07c4700 1 --2- 172.21.15.121:0/2652005717 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcca0039050 0x7fcca003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fccac006fd0 tx=0x7fccac006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:20.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:20.179+0000 7fccc07c4700 1 -- 172.21.15.121:0/2652005717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 msgr2=0x7fccb819e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:20.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:20.179+0000 7fccc07c4700 1 --2- 172.21.15.121:0/2652005717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb819e070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fccb0004350 tx=0x7fccb000bd50 comp rx=0 tx=0).stop 2024-06-10T00:16:20.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:20.179+0000 7fccc07c4700 1 -- 172.21.15.121:0/2652005717 shutdown_connections 2024-06-10T00:16:20.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:20.179+0000 7fccc07c4700 1 --2- 172.21.15.121:0/2652005717 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fcca0039050 0x7fcca003b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:20.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:20.179+0000 7fccc07c4700 1 --2- 172.21.15.121:0/2652005717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb8106860 0x7fccb819e070 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:20.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:20.179+0000 7fccc07c4700 1 -- 172.21.15.121:0/2652005717 >> 172.21.15.121:0/2652005717 conn(0x7fccb81024c0 msgr2=0x7fccb8194cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:20.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:20.179+0000 7fccc07c4700 1 -- 172.21.15.121:0/2652005717 shutdown_connections 2024-06-10T00:16:20.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:20.179+0000 7fccc07c4700 1 -- 172.21.15.121:0/2652005717 wait complete. 2024-06-10T00:16:21.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:21 smithi121 ceph-mon[32378]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:16:21.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:21 smithi121 ceph-mon[32378]: Generating ssh key... 2024-06-10T00:16:21.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:21 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:21.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:21 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:21.937 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXqsDLrmdVr/Qt6Pb3RHwIgCkU0CP3j8ySx92StlGDGyCcmdKDt1783xfFYWzKniBtxNe+5TebyJlP1227J/1x7dzojY2n3vS9dEIME3PI4oBZHjL3EDznLavz6GyjTPH2Y8HyL9h93qRWyo5IkVBNMNmungpF9SV+fvtKkZ+2+8673pcm481c7IfAD6Oh+LXZ0/2vlqfYKZCQMOMYli/8xK0wEBrivauzV/QvIAAyo5yi8mNnRP1kwfTbv//VeEhfGcZlkm0o/7ZV7fih4WH+N6Uyv13mH+9W6zKYybXC0WpxoDKBu+072dy1UM0kzNFw+rV/gYeTPJy+fxukYN3IfriFKMNjj9FwOmY7YlJ/JbUU7/khU+OCEumSQ1MKGndu2MkV3yVOAgKQ05pI5LOOOim4dxs6mncEoPDwThMS4eDf/cF1KJFa/Gq7otAKvlc+uMbvTgvZVm0SC1MX/mZYeFdbwDJ4w6426ykIOe0BoPfN4151rDAAf6q3oVIAfAM= ceph-6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.115+0000 7f870428f700 1 Processor -- start 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.115+0000 7f870428f700 1 -- start start 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.115+0000 7f870428f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc107280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.115+0000 7f870428f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc1077c0 con 0x7f86fc106e60 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.116+0000 7f870202b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc107280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.116+0000 7f870202b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc107280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33672/0 (socket says 172.21.15.121:33672) 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.116+0000 7f870202b700 1 -- 172.21.15.121:0/4123978331 learned_addr learned my addr 172.21.15.121:0/4123978331 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.116+0000 7f870202b700 1 -- 172.21.15.121:0/4123978331 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86fc107f70 con 0x7f86fc106e60 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.116+0000 7f870202b700 1 --2- 172.21.15.121:0/4123978331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc107280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f86f400d180 tx=0x7f86f400d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c79845c21db9cd4d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.116+0000 7f8701029700 1 -- 172.21.15.121:0/4123978331 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86f4019070 con 0x7f86fc106e60 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.116+0000 7f8701029700 1 -- 172.21.15.121:0/4123978331 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f86f400de20 con 0x7f86fc106e60 2024-06-10T00:16:21.938 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.116+0000 7f8701029700 1 -- 172.21.15.121:0/4123978331 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86f4009b90 con 0x7f86fc106e60 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.117+0000 7f870428f700 1 -- 172.21.15.121:0/4123978331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 msgr2=0x7f86fc107280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.117+0000 7f870428f700 1 --2- 172.21.15.121:0/4123978331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc107280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f86f400d180 tx=0x7f86f400d490 comp rx=0 tx=0).stop 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.117+0000 7f870428f700 1 -- 172.21.15.121:0/4123978331 shutdown_connections 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.117+0000 7f870428f700 1 --2- 172.21.15.121:0/4123978331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc107280 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.117+0000 7f870428f700 1 -- 172.21.15.121:0/4123978331 >> 172.21.15.121:0/4123978331 conn(0x7f86fc102460 msgr2=0x7f86fc104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.117+0000 7f870428f700 1 -- 172.21.15.121:0/4123978331 shutdown_connections 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.117+0000 7f870428f700 1 -- 172.21.15.121:0/4123978331 wait complete. 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.118+0000 7f870428f700 1 Processor -- start 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.118+0000 7f870428f700 1 -- start start 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.118+0000 7f870428f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.118+0000 7f870428f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc07b830 con 0x7f86fc106e60 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.118+0000 7f870202b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.118+0000 7f870202b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33688/0 (socket says 172.21.15.121:33688) 2024-06-10T00:16:21.939 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.118+0000 7f870202b700 1 -- 172.21.15.121:0/400528366 learned_addr learned my addr 172.21.15.121:0/400528366 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.118+0000 7f870202b700 1 -- 172.21.15.121:0/400528366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86fc19a1c0 con 0x7f86fc106e60 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.118+0000 7f870202b700 1 --2- 172.21.15.121:0/400528366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc199c80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009740 tx=0x7f86f4003ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.118+0000 7f86fb7fe700 1 -- 172.21.15.121:0/400528366 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86f4019050 con 0x7f86fc106e60 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.118+0000 7f86fb7fe700 1 -- 172.21.15.121:0/400528366 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f86f401a040 con 0x7f86fc106e60 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.119+0000 7f870428f700 1 -- 172.21.15.121:0/400528366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86fc19a450 con 0x7f86fc106e60 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.119+0000 7f86fb7fe700 1 -- 172.21.15.121:0/400528366 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86f4004560 con 0x7f86fc106e60 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.119+0000 7f870428f700 1 -- 172.21.15.121:0/400528366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86fc19d140 con 0x7f86fc106e60 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.119+0000 7f86fb7fe700 1 -- 172.21.15.121:0/400528366 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f86f4027020 con 0x7f86fc106e60 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.119+0000 7f870428f700 1 -- 172.21.15.121:0/400528366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86fc063af0 con 0x7f86fc106e60 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.120+0000 7f86fb7fe700 1 --2- 172.21.15.121:0/400528366 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f86e40390a0 0x7f86e403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.120+0000 7f86fb7fe700 1 -- 172.21.15.121:0/400528366 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f86f4025020 con 0x7f86fc106e60 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.120+0000 7f870182a700 1 --2- 172.21.15.121:0/400528366 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f86e40390a0 0x7f86e403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.120+0000 7f870182a700 1 --2- 172.21.15.121:0/400528366 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f86e40390a0 0x7f86e403b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86f0006fd0 tx=0x7f86f0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:21.940 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.125+0000 7f86fb7fe700 1 -- 172.21.15.121:0/400528366 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f86f401e400 con 0x7f86fc106e60 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.321+0000 7f870428f700 1 -- 172.21.15.121:0/400528366 --> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f86fc0628d0 con 0x7f86e40390a0 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.322+0000 7f86fb7fe700 1 -- 172.21.15.121:0/400528366 <== mgr.14118 v2:172.21.15.121:6800/704666019 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f86fc0628d0 con 0x7f86e40390a0 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.324+0000 7f870428f700 1 -- 172.21.15.121:0/400528366 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f86e40390a0 msgr2=0x7f86e403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.324+0000 7f870428f700 1 --2- 172.21.15.121:0/400528366 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f86e40390a0 0x7f86e403b560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86f0006fd0 tx=0x7f86f0006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.324+0000 7f870428f700 1 -- 172.21.15.121:0/400528366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 msgr2=0x7f86fc199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.324+0000 7f870428f700 1 --2- 172.21.15.121:0/400528366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc199c80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009740 tx=0x7f86f4003ad0 comp rx=0 tx=0).stop 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.324+0000 7f870428f700 1 -- 172.21.15.121:0/400528366 shutdown_connections 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.324+0000 7f870428f700 1 --2- 172.21.15.121:0/400528366 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f86e40390a0 0x7f86e403b560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.324+0000 7f870428f700 1 --2- 172.21.15.121:0/400528366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86fc106e60 0x7f86fc199c80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.324+0000 7f870428f700 1 -- 172.21.15.121:0/400528366 >> 172.21.15.121:0/400528366 conn(0x7f86fc102460 msgr2=0x7f86fc190220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.324+0000 7f870428f700 1 -- 172.21.15.121:0/400528366 shutdown_connections 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:21.324+0000 7f870428f700 1 -- 172.21.15.121:0/400528366 wait complete. 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-10T00:16:21.941 INFO:teuthology.orchestra.run.smithi121.stdout:Adding key to root@localhost authorized_keys... 2024-06-10T00:16:21.942 INFO:teuthology.orchestra.run.smithi121.stdout:Adding host smithi121... 2024-06-10T00:16:22.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:22 smithi121 ceph-mon[32378]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:16:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:23 smithi121 ceph-mon[32378]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi121", "addr": "172.21.15.121", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:16:24.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:24 smithi121 ceph-mon[32378]: Deploying cephadm binary to smithi121 2024-06-10T00:16:27.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Added host 'smithi121' with addr '172.21.15.121' 2024-06-10T00:16:27.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.385+0000 7feab0b49700 1 Processor -- start 2024-06-10T00:16:27.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.385+0000 7feab0b49700 1 -- start start 2024-06-10T00:16:27.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.385+0000 7feab0b49700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac07c550 0x7feaac10aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:27.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.385+0000 7feab0b49700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feaac10b090 con 0x7feaac07c550 2024-06-10T00:16:27.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.386+0000 7feaaa59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac07c550 0x7feaac10aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:27.345 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.386+0000 7feaaa59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac07c550 0x7feaac10aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33696/0 (socket says 172.21.15.121:33696) 2024-06-10T00:16:27.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.386+0000 7feaaa59c700 1 -- 172.21.15.121:0/696459770 learned_addr learned my addr 172.21.15.121:0/696459770 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:27.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.386+0000 7feaaa59c700 1 -- 172.21.15.121:0/696459770 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac10b210 con 0x7feaac07c550 2024-06-10T00:16:27.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.386+0000 7feaaa59c700 1 --2- 172.21.15.121:0/696459770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac07c550 0x7feaac10aac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0098a0 tx=0x7fea9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f05adf49faa725c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:27.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.386+0000 7feaa9d9b700 1 -- 172.21.15.121:0/696459770 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea9c004160 con 0x7feaac07c550 2024-06-10T00:16:27.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.386+0000 7feaa9d9b700 1 -- 172.21.15.121:0/696459770 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fea9c004300 con 0x7feaac07c550 2024-06-10T00:16:27.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.386+0000 7feaa9d9b700 1 -- 172.21.15.121:0/696459770 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea9c011b50 con 0x7feaac07c550 2024-06-10T00:16:27.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.387+0000 7feab0b49700 1 -- 172.21.15.121:0/696459770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac07c550 msgr2=0x7feaac10aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:27.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.387+0000 7feab0b49700 1 --2- 172.21.15.121:0/696459770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac07c550 0x7feaac10aac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0098a0 tx=0x7fea9c009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:27.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.387+0000 7feab0b49700 1 -- 172.21.15.121:0/696459770 shutdown_connections 2024-06-10T00:16:27.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.387+0000 7feab0b49700 1 --2- 172.21.15.121:0/696459770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac07c550 0x7feaac10aac0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:27.346 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.387+0000 7feab0b49700 1 -- 172.21.15.121:0/696459770 >> 172.21.15.121:0/696459770 conn(0x7feaac102460 msgr2=0x7feaac104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.387+0000 7feab0b49700 1 -- 172.21.15.121:0/696459770 shutdown_connections 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.387+0000 7feab0b49700 1 -- 172.21.15.121:0/696459770 wait complete. 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.387+0000 7feab0b49700 1 Processor -- start 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.388+0000 7feab0b49700 1 -- start start 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.388+0000 7feab0b49700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac19beb0 0x7feaac19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.388+0000 7feab0b49700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feaac10c2f0 con 0x7feaac19beb0 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.388+0000 7feaaa59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac19beb0 0x7feaac19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.388+0000 7feaaa59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac19beb0 0x7feaac19c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33702/0 (socket says 172.21.15.121:33702) 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.388+0000 7feaaa59c700 1 -- 172.21.15.121:0/1374202211 learned_addr learned my addr 172.21.15.121:0/1374202211 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.388+0000 7feaaa59c700 1 -- 172.21.15.121:0/1374202211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac19c810 con 0x7feaac19beb0 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.388+0000 7feaaa59c700 1 --2- 172.21.15.121:0/1374202211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac19beb0 0x7feaac19c2d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fea9c011670 tx=0x7fea9c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:27.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.388+0000 7feaa3fff700 1 -- 172.21.15.121:0/1374202211 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea9c0039c0 con 0x7feaac19beb0 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.389+0000 7feaa3fff700 1 -- 172.21.15.121:0/1374202211 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fea9c011e50 con 0x7feaac19beb0 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.389+0000 7feab0b49700 1 -- 172.21.15.121:0/1374202211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaac19caa0 con 0x7feaac19beb0 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.389+0000 7feaa3fff700 1 -- 172.21.15.121:0/1374202211 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea9c01a5c0 con 0x7feaac19beb0 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.389+0000 7feab0b49700 1 -- 172.21.15.121:0/1374202211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac19f580 con 0x7feaac19beb0 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.389+0000 7feaa3fff700 1 -- 172.21.15.121:0/1374202211 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fea9c01a8c0 con 0x7feaac19beb0 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.390+0000 7feaa3fff700 1 --2- 172.21.15.121:0/1374202211 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fea98039000 0x7fea9803b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.390+0000 7feaa3fff700 1 -- 172.21.15.121:0/1374202211 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fea9c04e460 con 0x7feaac19beb0 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.390+0000 7feaa1d9b700 1 --2- 172.21.15.121:0/1374202211 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fea98039000 0x7fea9803b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.390+0000 7feab0b49700 1 -- 172.21.15.121:0/1374202211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaac063af0 con 0x7feaac19beb0 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.390+0000 7feaa1d9b700 1 --2- 172.21.15.121:0/1374202211 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fea98039000 0x7fea9803b4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fea94006fd0 tx=0x7fea94006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.395+0000 7feaa3fff700 1 -- 172.21.15.121:0/1374202211 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fea9c019c70 con 0x7feaac19beb0 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:22.592+0000 7feab0b49700 1 -- 172.21.15.121:0/1374202211 --> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi121", "addr": "172.21.15.121", "target": ["mon-mgr", ""]}) -- 0x7feaac103750 con 0x7fea98039000 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:26.771+0000 7feaa3fff700 1 -- 172.21.15.121:0/1374202211 <== mgr.14118 v2:172.21.15.121:6800/704666019 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7feaac103750 con 0x7fea98039000 2024-06-10T00:16:27.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:26.774+0000 7feab0b49700 1 -- 172.21.15.121:0/1374202211 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fea98039000 msgr2=0x7fea9803b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:26.774+0000 7feab0b49700 1 --2- 172.21.15.121:0/1374202211 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fea98039000 0x7fea9803b4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fea94006fd0 tx=0x7fea94006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:26.774+0000 7feab0b49700 1 -- 172.21.15.121:0/1374202211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac19beb0 msgr2=0x7feaac19c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:26.774+0000 7feab0b49700 1 --2- 172.21.15.121:0/1374202211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac19beb0 0x7feaac19c2d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fea9c011670 tx=0x7fea9c011b60 comp rx=0 tx=0).stop 2024-06-10T00:16:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:26.774+0000 7feab0b49700 1 -- 172.21.15.121:0/1374202211 shutdown_connections 2024-06-10T00:16:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:26.774+0000 7feab0b49700 1 --2- 172.21.15.121:0/1374202211 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fea98039000 0x7fea9803b4c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:26.774+0000 7feab0b49700 1 --2- 172.21.15.121:0/1374202211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaac19beb0 0x7feaac19c2d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:26.774+0000 7feab0b49700 1 -- 172.21.15.121:0/1374202211 >> 172.21.15.121:0/1374202211 conn(0x7feaac102460 msgr2=0x7feaac103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:26.774+0000 7feab0b49700 1 -- 172.21.15.121:0/1374202211 shutdown_connections 2024-06-10T00:16:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:26.774+0000 7feab0b49700 1 -- 172.21.15.121:0/1374202211 wait complete. 2024-06-10T00:16:27.349 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying unmanaged mon service... 2024-06-10T00:16:27.944 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:27 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:27.945 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:27 smithi121 ceph-mon[32378]: Added host smithi121 2024-06-10T00:16:27.945 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:27 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:16:28.976 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-10T00:16:28.976 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.797+0000 7fce2ae33700 1 Processor -- start 2024-06-10T00:16:28.976 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.797+0000 7fce2ae33700 1 -- start start 2024-06-10T00:16:28.976 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.797+0000 7fce2ae33700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2410ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:28.976 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.797+0000 7fce2ae33700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fce2407c6a0 con 0x7fce24108a10 2024-06-10T00:16:28.976 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.797+0000 7fce28bcf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2410ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:28.976 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.797+0000 7fce28bcf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2410ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33714/0 (socket says 172.21.15.121:33714) 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.797+0000 7fce28bcf700 1 -- 172.21.15.121:0/3010119857 learned_addr learned my addr 172.21.15.121:0/3010119857 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.797+0000 7fce28bcf700 1 -- 172.21.15.121:0/3010119857 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce2410b340 con 0x7fce24108a10 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.798+0000 7fce28bcf700 1 --2- 172.21.15.121:0/3010119857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2410ae00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fce180098a0 tx=0x7fce18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6779e28ec82c1b13 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.798+0000 7fce237fe700 1 -- 172.21.15.121:0/3010119857 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce18004160 con 0x7fce24108a10 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.798+0000 7fce237fe700 1 -- 172.21.15.121:0/3010119857 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fce18004300 con 0x7fce24108a10 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.798+0000 7fce237fe700 1 -- 172.21.15.121:0/3010119857 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce18011b50 con 0x7fce24108a10 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.798+0000 7fce2ae33700 1 -- 172.21.15.121:0/3010119857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 msgr2=0x7fce2410ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.798+0000 7fce2ae33700 1 --2- 172.21.15.121:0/3010119857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2410ae00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fce180098a0 tx=0x7fce18009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.798+0000 7fce2ae33700 1 -- 172.21.15.121:0/3010119857 shutdown_connections 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.798+0000 7fce2ae33700 1 --2- 172.21.15.121:0/3010119857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2410ae00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.798+0000 7fce2ae33700 1 -- 172.21.15.121:0/3010119857 >> 172.21.15.121:0/3010119857 conn(0x7fce24102460 msgr2=0x7fce24104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.798+0000 7fce2ae33700 1 -- 172.21.15.121:0/3010119857 shutdown_connections 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.798+0000 7fce2ae33700 1 -- 172.21.15.121:0/3010119857 wait complete. 2024-06-10T00:16:28.977 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.799+0000 7fce2ae33700 1 Processor -- start 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.799+0000 7fce2ae33700 1 -- start start 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.799+0000 7fce2ae33700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.799+0000 7fce2ae33700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fce2410c300 con 0x7fce24108a10 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.799+0000 7fce28bcf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.799+0000 7fce28bcf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2419bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33728/0 (socket says 172.21.15.121:33728) 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.799+0000 7fce28bcf700 1 -- 172.21.15.121:0/1751654815 learned_addr learned my addr 172.21.15.121:0/1751654815 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.799+0000 7fce28bcf700 1 -- 172.21.15.121:0/1751654815 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce2419c3e0 con 0x7fce24108a10 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.800+0000 7fce28bcf700 1 --2- 172.21.15.121:0/1751654815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2419bea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fce18011b60 tx=0x7fce18011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.800+0000 7fce21ffb700 1 -- 172.21.15.121:0/1751654815 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce18003b20 con 0x7fce24108a10 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.800+0000 7fce2ae33700 1 -- 172.21.15.121:0/1751654815 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce2419c670 con 0x7fce24108a10 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.800+0000 7fce21ffb700 1 -- 172.21.15.121:0/1751654815 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fce18003cc0 con 0x7fce24108a10 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.800+0000 7fce21ffb700 1 -- 172.21.15.121:0/1751654815 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fce18024840 con 0x7fce24108a10 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.800+0000 7fce2ae33700 1 -- 172.21.15.121:0/1751654815 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce2419f360 con 0x7fce24108a10 2024-06-10T00:16:28.978 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.801+0000 7fce21ffb700 1 -- 172.21.15.121:0/1751654815 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fce18022020 con 0x7fce24108a10 2024-06-10T00:16:28.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.801+0000 7fce2ae33700 1 -- 172.21.15.121:0/1751654815 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce24063af0 con 0x7fce24108a10 2024-06-10T00:16:28.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.801+0000 7fce21ffb700 1 --2- 172.21.15.121:0/1751654815 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fce14039050 0x7fce1403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:28.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.801+0000 7fce21ffb700 1 -- 172.21.15.121:0/1751654815 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fce18011e00 con 0x7fce24108a10 2024-06-10T00:16:28.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.801+0000 7fce23fff700 1 --2- 172.21.15.121:0/1751654815 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fce14039050 0x7fce1403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:28.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.801+0000 7fce23fff700 1 --2- 172.21.15.121:0/1751654815 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fce14039050 0x7fce1403b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fce10006fd0 tx=0x7fce10006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:28.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:27.806+0000 7fce21ffb700 1 -- 172.21.15.121:0/1751654815 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fce1800ba30 con 0x7fce24108a10 2024-06-10T00:16:28.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.025+0000 7fce2ae33700 1 -- 172.21.15.121:0/1751654815 --> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fce241036e0 con 0x7fce14039050 2024-06-10T00:16:28.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.030+0000 7fce21ffb700 1 -- 172.21.15.121:0/1751654815 <== mgr.14118 v2:172.21.15.121:6800/704666019 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fce241036e0 con 0x7fce14039050 2024-06-10T00:16:28.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.032+0000 7fce2ae33700 1 -- 172.21.15.121:0/1751654815 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fce14039050 msgr2=0x7fce1403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:28.979 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.032+0000 7fce2ae33700 1 --2- 172.21.15.121:0/1751654815 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fce14039050 0x7fce1403b510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fce10006fd0 tx=0x7fce10006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:28.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.032+0000 7fce2ae33700 1 -- 172.21.15.121:0/1751654815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 msgr2=0x7fce2419bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:28.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.032+0000 7fce2ae33700 1 --2- 172.21.15.121:0/1751654815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2419bea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fce18011b60 tx=0x7fce18011c40 comp rx=0 tx=0).stop 2024-06-10T00:16:28.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.032+0000 7fce2ae33700 1 -- 172.21.15.121:0/1751654815 shutdown_connections 2024-06-10T00:16:28.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.032+0000 7fce2ae33700 1 --2- 172.21.15.121:0/1751654815 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fce14039050 0x7fce1403b510 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:28.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.032+0000 7fce2ae33700 1 --2- 172.21.15.121:0/1751654815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce24108a10 0x7fce2419bea0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:28.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.032+0000 7fce2ae33700 1 -- 172.21.15.121:0/1751654815 >> 172.21.15.121:0/1751654815 conn(0x7fce24102460 msgr2=0x7fce24102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:28.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.032+0000 7fce2ae33700 1 -- 172.21.15.121:0/1751654815 shutdown_connections 2024-06-10T00:16:28.980 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:28.032+0000 7fce2ae33700 1 -- 172.21.15.121:0/1751654815 wait complete. 2024-06-10T00:16:28.980 INFO:teuthology.orchestra.run.smithi121.stdout:Deploying unmanaged mgr service... 2024-06-10T00:16:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:29 smithi121 ceph-mon[32378]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:16:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:29 smithi121 ceph-mon[32378]: Saving service mon spec with placement count:5 2024-06-10T00:16:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:29 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:30.652 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:30 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:30.652 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:30 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:30.652 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:30 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:30.659 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-10T00:16:30.659 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.826+0000 7f6f08f5f700 1 Processor -- start 2024-06-10T00:16:30.659 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.826+0000 7f6f08f5f700 1 -- start start 2024-06-10T00:16:30.659 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.826+0000 7f6f08f5f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f04107270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.826+0000 7f6f08f5f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6f041077b0 con 0x7f6f04106e50 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.826+0000 7f6f0259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f04107270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.826+0000 7f6f0259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f04107270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40084/0 (socket says 172.21.15.121:40084) 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.826+0000 7f6f0259c700 1 -- 172.21.15.121:0/2827626977 learned_addr learned my addr 172.21.15.121:0/2827626977 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.827+0000 7f6f0259c700 1 -- 172.21.15.121:0/2827626977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f04107fb0 con 0x7f6f04106e50 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.827+0000 7f6f0259c700 1 --2- 172.21.15.121:0/2827626977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f04107270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40098a0 tx=0x7f6ef4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59626dad34d1a5a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.827+0000 7f6f0159a700 1 -- 172.21.15.121:0/2827626977 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ef4004160 con 0x7f6f04106e50 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.827+0000 7f6f0159a700 1 -- 172.21.15.121:0/2827626977 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef4004300 con 0x7f6f04106e50 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.827+0000 7f6f0159a700 1 -- 172.21.15.121:0/2827626977 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ef4011b20 con 0x7f6f04106e50 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.828+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2827626977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 msgr2=0x7f6f04107270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.828+0000 7f6f08f5f700 1 --2- 172.21.15.121:0/2827626977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f04107270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40098a0 tx=0x7f6ef4009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.828+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2827626977 shutdown_connections 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.828+0000 7f6f08f5f700 1 --2- 172.21.15.121:0/2827626977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f04107270 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:30.660 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.828+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2827626977 >> 172.21.15.121:0/2827626977 conn(0x7f6f04102410 msgr2=0x7f6f04104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.828+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2827626977 shutdown_connections 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.828+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2827626977 wait complete. 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.829+0000 7f6f08f5f700 1 Processor -- start 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.829+0000 7f6f08f5f700 1 -- start start 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.829+0000 7f6f08f5f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f0419beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.829+0000 7f6f08f5f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0410b3f0 con 0x7f6f04106e50 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.829+0000 7f6f0259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f0419beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.829+0000 7f6f0259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f0419beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40098/0 (socket says 172.21.15.121:40098) 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.829+0000 7f6f0259c700 1 -- 172.21.15.121:0/2731745151 learned_addr learned my addr 172.21.15.121:0/2731745151 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.829+0000 7f6f0259c700 1 -- 172.21.15.121:0/2731745151 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0419c3f0 con 0x7f6f04106e50 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.830+0000 7f6f0259c700 1 --2- 172.21.15.121:0/2731745151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f0419beb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40117e0 tx=0x7f6ef4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.830+0000 7f6ef37fe700 1 -- 172.21.15.121:0/2731745151 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ef4003bf0 con 0x7f6f04106e50 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.830+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2731745151 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0419c680 con 0x7f6f04106e50 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.830+0000 7f6ef37fe700 1 -- 172.21.15.121:0/2731745151 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef401a910 con 0x7f6f04106e50 2024-06-10T00:16:30.661 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.830+0000 7f6ef37fe700 1 -- 172.21.15.121:0/2731745151 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ef4024970 con 0x7f6f04106e50 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.830+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2731745151 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f0419cae0 con 0x7f6f04106e50 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.831+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2731745151 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f04063af0 con 0x7f6f04106e50 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.831+0000 7f6ef37fe700 1 -- 172.21.15.121:0/2731745151 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f6ef4022020 con 0x7f6f04106e50 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.831+0000 7f6ef37fe700 1 --2- 172.21.15.121:0/2731745151 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f6eec039050 0x7f6eec03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.831+0000 7f6ef37fe700 1 -- 172.21.15.121:0/2731745151 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6ef404d3a0 con 0x7f6f04106e50 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.831+0000 7f6f01d9b700 1 --2- 172.21.15.121:0/2731745151 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f6eec039050 0x7f6eec03b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.832+0000 7f6f01d9b700 1 --2- 172.21.15.121:0/2731745151 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f6eec039050 0x7f6eec03b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8006fd0 tx=0x7f6ef8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:29.836+0000 7f6ef37fe700 1 -- 172.21.15.121:0/2731745151 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6ef4011e20 con 0x7f6f04106e50 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.055+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2731745151 --> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f6f040628d0 con 0x7f6eec039050 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.060+0000 7f6ef37fe700 1 -- 172.21.15.121:0/2731745151 <== mgr.14118 v2:172.21.15.121:6800/704666019 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6f040628d0 con 0x7f6eec039050 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.062+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2731745151 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f6eec039050 msgr2=0x7f6eec03b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.062+0000 7f6f08f5f700 1 --2- 172.21.15.121:0/2731745151 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f6eec039050 0x7f6eec03b510 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8006fd0 tx=0x7f6ef8006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:30.662 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.062+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2731745151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 msgr2=0x7f6f0419beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:30.663 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.062+0000 7f6f08f5f700 1 --2- 172.21.15.121:0/2731745151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f0419beb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40117e0 tx=0x7f6ef4011c10 comp rx=0 tx=0).stop 2024-06-10T00:16:30.663 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.062+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2731745151 shutdown_connections 2024-06-10T00:16:30.663 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.062+0000 7f6f08f5f700 1 --2- 172.21.15.121:0/2731745151 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f6eec039050 0x7f6eec03b510 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:30.663 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.062+0000 7f6f08f5f700 1 --2- 172.21.15.121:0/2731745151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04106e50 0x7f6f0419beb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:30.663 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.062+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2731745151 >> 172.21.15.121:0/2731745151 conn(0x7f6f04102410 msgr2=0x7f6f041929d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:30.663 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.063+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2731745151 shutdown_connections 2024-06-10T00:16:30.663 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:30.063+0000 7f6f08f5f700 1 -- 172.21.15.121:0/2731745151 wait complete. 2024-06-10T00:16:31.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:31 smithi121 ceph-mon[32378]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:16:31.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:31 smithi121 ceph-mon[32378]: Saving service mgr spec with placement count:2 2024-06-10T00:16:31.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:31 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1527721557' entity='client.admin' 2024-06-10T00:16:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.082+0000 7f3316c07700 1 Processor -- start 2024-06-10T00:16:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.082+0000 7f3316c07700 1 -- start start 2024-06-10T00:16:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.082+0000 7f3316c07700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331010ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.082+0000 7f3316c07700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f331007c660 con 0x7f3310108a50 2024-06-10T00:16:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.082+0000 7f33149a3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331010ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.082+0000 7f33149a3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331010ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40106/0 (socket says 172.21.15.121:40106) 2024-06-10T00:16:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.082+0000 7f33149a3700 1 -- 172.21.15.121:0/1025234780 learned_addr learned my addr 172.21.15.121:0/1025234780 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.083+0000 7f33149a3700 1 -- 172.21.15.121:0/1025234780 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f331010b380 con 0x7f3310108a50 2024-06-10T00:16:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.083+0000 7f33149a3700 1 --2- 172.21.15.121:0/1025234780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331010ae40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f33040098a0 tx=0x7f3304009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c34a10e95a3a71c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:32.461 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.083+0000 7f330f7fe700 1 -- 172.21.15.121:0/1025234780 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3304004160 con 0x7f3310108a50 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.083+0000 7f330f7fe700 1 -- 172.21.15.121:0/1025234780 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3304004300 con 0x7f3310108a50 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.083+0000 7f330f7fe700 1 -- 172.21.15.121:0/1025234780 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3304011b20 con 0x7f3310108a50 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.083+0000 7f3316c07700 1 -- 172.21.15.121:0/1025234780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 msgr2=0x7f331010ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.083+0000 7f3316c07700 1 --2- 172.21.15.121:0/1025234780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331010ae40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f33040098a0 tx=0x7f3304009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f3316c07700 1 -- 172.21.15.121:0/1025234780 shutdown_connections 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f3316c07700 1 --2- 172.21.15.121:0/1025234780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331010ae40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f3316c07700 1 -- 172.21.15.121:0/1025234780 >> 172.21.15.121:0/1025234780 conn(0x7f3310102460 msgr2=0x7f3310104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f3316c07700 1 -- 172.21.15.121:0/1025234780 shutdown_connections 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f3316c07700 1 -- 172.21.15.121:0/1025234780 wait complete. 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f3316c07700 1 Processor -- start 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f3316c07700 1 -- start start 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f3316c07700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331019be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:32.462 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f3316c07700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f331010c380 con 0x7f3310108a50 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f33149a3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331019be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f33149a3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331019be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40108/0 (socket says 172.21.15.121:40108) 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.084+0000 7f33149a3700 1 -- 172.21.15.121:0/1527721557 learned_addr learned my addr 172.21.15.121:0/1527721557 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.085+0000 7f33149a3700 1 -- 172.21.15.121:0/1527721557 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f331019c3a0 con 0x7f3310108a50 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.085+0000 7f33149a3700 1 --2- 172.21.15.121:0/1527721557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331019be60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f33040117e0 tx=0x7f3304011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.085+0000 7f330dffb700 1 -- 172.21.15.121:0/1527721557 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3304003bf0 con 0x7f3310108a50 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.085+0000 7f3316c07700 1 -- 172.21.15.121:0/1527721557 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f331019c630 con 0x7f3310108a50 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.085+0000 7f330dffb700 1 -- 172.21.15.121:0/1527721557 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f330401a910 con 0x7f3310108a50 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.085+0000 7f330dffb700 1 -- 172.21.15.121:0/1527721557 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3304024970 con 0x7f3310108a50 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.085+0000 7f3316c07700 1 -- 172.21.15.121:0/1527721557 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f331019ca90 con 0x7f3310108a50 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.086+0000 7f330dffb700 1 -- 172.21.15.121:0/1527721557 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f3304022020 con 0x7f3310108a50 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.086+0000 7f3316c07700 1 -- 172.21.15.121:0/1527721557 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3310063af0 con 0x7f3310108a50 2024-06-10T00:16:32.463 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.086+0000 7f330dffb700 1 --2- 172.21.15.121:0/1527721557 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f3300039050 0x7f330003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.086+0000 7f330dffb700 1 -- 172.21.15.121:0/1527721557 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f330404d390 con 0x7f3310108a50 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.086+0000 7f330ffff700 1 --2- 172.21.15.121:0/1527721557 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f3300039050 0x7f330003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.087+0000 7f330ffff700 1 --2- 172.21.15.121:0/1527721557 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f3300039050 0x7f330003b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f32fc006fd0 tx=0x7f32fc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.091+0000 7f330dffb700 1 -- 172.21.15.121:0/1527721557 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3304011e20 con 0x7f3310108a50 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.303+0000 7f3316c07700 1 -- 172.21.15.121:0/1527721557 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f331019f320 con 0x7f3310108a50 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.306+0000 7f330dffb700 1 -- 172.21.15.121:0/1527721557 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f33040172b0 con 0x7f3310108a50 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.308+0000 7f3316c07700 1 -- 172.21.15.121:0/1527721557 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f3300039050 msgr2=0x7f330003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.308+0000 7f3316c07700 1 --2- 172.21.15.121:0/1527721557 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f3300039050 0x7f330003b510 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f32fc006fd0 tx=0x7f32fc006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.308+0000 7f3316c07700 1 -- 172.21.15.121:0/1527721557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 msgr2=0x7f331019be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.308+0000 7f3316c07700 1 --2- 172.21.15.121:0/1527721557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331019be60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f33040117e0 tx=0x7f3304011c10 comp rx=0 tx=0).stop 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.308+0000 7f3316c07700 1 -- 172.21.15.121:0/1527721557 shutdown_connections 2024-06-10T00:16:32.464 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.308+0000 7f3316c07700 1 --2- 172.21.15.121:0/1527721557 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f3300039050 0x7f330003b510 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:32.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.308+0000 7f3316c07700 1 --2- 172.21.15.121:0/1527721557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3310108a50 0x7f331019be60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:32.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.308+0000 7f3316c07700 1 -- 172.21.15.121:0/1527721557 >> 172.21.15.121:0/1527721557 conn(0x7f3310102460 msgr2=0x7f3310102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:32.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.308+0000 7f3316c07700 1 -- 172.21.15.121:0/1527721557 shutdown_connections 2024-06-10T00:16:32.465 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:31.308+0000 7f3316c07700 1 -- 172.21.15.121:0/1527721557 wait complete. 2024-06-10T00:16:33.129 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:32 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:33.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.911+0000 7f34d0a0a700 1 Processor -- start 2024-06-10T00:16:33.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.911+0000 7f34d0a0a700 1 -- start start 2024-06-10T00:16:33.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.911+0000 7f34d0a0a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc108820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:33.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.911+0000 7f34d0a0a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc108df0 con 0x7f34cc108400 2024-06-10T00:16:33.725 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.911+0000 7f34ca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc108820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.911+0000 7f34ca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc108820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40118/0 (socket says 172.21.15.121:40118) 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.911+0000 7f34ca59c700 1 -- 172.21.15.121:0/577054333 learned_addr learned my addr 172.21.15.121:0/577054333 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.912+0000 7f34ca59c700 1 -- 172.21.15.121:0/577054333 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc107060 con 0x7f34cc108400 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.912+0000 7f34ca59c700 1 --2- 172.21.15.121:0/577054333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc108820 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0098a0 tx=0x7f34bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439ecc07ce501ca0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.912+0000 7f34c959a700 1 -- 172.21.15.121:0/577054333 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34bc004160 con 0x7f34cc108400 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.912+0000 7f34c959a700 1 -- 172.21.15.121:0/577054333 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc004300 con 0x7f34cc108400 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.912+0000 7f34c959a700 1 -- 172.21.15.121:0/577054333 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34bc011b20 con 0x7f34cc108400 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.913+0000 7f34d0a0a700 1 -- 172.21.15.121:0/577054333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 msgr2=0x7f34cc108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.913+0000 7f34d0a0a700 1 --2- 172.21.15.121:0/577054333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc108820 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0098a0 tx=0x7f34bc009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.913+0000 7f34d0a0a700 1 -- 172.21.15.121:0/577054333 shutdown_connections 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.913+0000 7f34d0a0a700 1 --2- 172.21.15.121:0/577054333 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc108820 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.913+0000 7f34d0a0a700 1 -- 172.21.15.121:0/577054333 >> 172.21.15.121:0/577054333 conn(0x7f34cc101b90 msgr2=0x7f34cc103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:33.726 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.913+0000 7f34d0a0a700 1 -- 172.21.15.121:0/577054333 shutdown_connections 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.913+0000 7f34d0a0a700 1 -- 172.21.15.121:0/577054333 wait complete. 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.914+0000 7f34d0a0a700 1 Processor -- start 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.914+0000 7f34d0a0a700 1 -- start start 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.914+0000 7f34d0a0a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc19ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.914+0000 7f34d0a0a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc10b420 con 0x7f34cc108400 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.914+0000 7f34ca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc19ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.914+0000 7f34ca59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc19ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40130/0 (socket says 172.21.15.121:40130) 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.914+0000 7f34ca59c700 1 -- 172.21.15.121:0/880590913 learned_addr learned my addr 172.21.15.121:0/880590913 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.914+0000 7f34ca59c700 1 -- 172.21.15.121:0/880590913 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc19b2d0 con 0x7f34cc108400 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.914+0000 7f34ca59c700 1 --2- 172.21.15.121:0/880590913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc19ad90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0117e0 tx=0x7f34bc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.915+0000 7f34bb7fe700 1 -- 172.21.15.121:0/880590913 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34bc003ba0 con 0x7f34cc108400 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.915+0000 7f34bb7fe700 1 -- 172.21.15.121:0/880590913 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34bc01a910 con 0x7f34cc108400 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.915+0000 7f34d0a0a700 1 -- 172.21.15.121:0/880590913 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34cc19b560 con 0x7f34cc108400 2024-06-10T00:16:33.727 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.915+0000 7f34bb7fe700 1 -- 172.21.15.121:0/880590913 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34bc0248f0 con 0x7f34cc108400 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.915+0000 7f34d0a0a700 1 -- 172.21.15.121:0/880590913 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34cc19e210 con 0x7f34cc108400 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.916+0000 7f34bb7fe700 1 -- 172.21.15.121:0/880590913 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f34bc022020 con 0x7f34cc108400 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.916+0000 7f34d0a0a700 1 -- 172.21.15.121:0/880590913 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34cc063af0 con 0x7f34cc108400 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.916+0000 7f34bb7fe700 1 --2- 172.21.15.121:0/880590913 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f34b40390a0 0x7f34b403b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.916+0000 7f34bb7fe700 1 -- 172.21.15.121:0/880590913 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f34bc04d460 con 0x7f34cc108400 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.916+0000 7f34c9d9b700 1 --2- 172.21.15.121:0/880590913 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f34b40390a0 0x7f34b403b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.916+0000 7f34c9d9b700 1 --2- 172.21.15.121:0/880590913 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f34b40390a0 0x7f34b403b560 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f34c0006fd0 tx=0x7f34c0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:32.921+0000 7f34bb7fe700 1 -- 172.21.15.121:0/880590913 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f34bc02cdf0 con 0x7f34cc108400 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.130+0000 7f34d0a0a700 1 -- 172.21.15.121:0/880590913 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f34cc19b9c0 con 0x7f34cc108400 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.133+0000 7f34bb7fe700 1 -- 172.21.15.121:0/880590913 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f34bc020c10 con 0x7f34cc108400 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.135+0000 7f34d0a0a700 1 -- 172.21.15.121:0/880590913 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f34b40390a0 msgr2=0x7f34b403b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.135+0000 7f34d0a0a700 1 --2- 172.21.15.121:0/880590913 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f34b40390a0 0x7f34b403b560 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f34c0006fd0 tx=0x7f34c0006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.135+0000 7f34d0a0a700 1 -- 172.21.15.121:0/880590913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 msgr2=0x7f34cc19ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:33.728 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.135+0000 7f34d0a0a700 1 --2- 172.21.15.121:0/880590913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc19ad90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f34bc0117e0 tx=0x7f34bc011ba0 comp rx=0 tx=0).stop 2024-06-10T00:16:33.729 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.135+0000 7f34d0a0a700 1 -- 172.21.15.121:0/880590913 shutdown_connections 2024-06-10T00:16:33.729 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.135+0000 7f34d0a0a700 1 --2- 172.21.15.121:0/880590913 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f34b40390a0 0x7f34b403b560 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:33.729 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.135+0000 7f34d0a0a700 1 --2- 172.21.15.121:0/880590913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34cc108400 0x7f34cc19ad90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:33.729 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.135+0000 7f34d0a0a700 1 -- 172.21.15.121:0/880590913 >> 172.21.15.121:0/880590913 conn(0x7f34cc101b90 msgr2=0x7f34cc102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:33.729 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.136+0000 7f34d0a0a700 1 -- 172.21.15.121:0/880590913 shutdown_connections 2024-06-10T00:16:33.729 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:33.136+0000 7f34d0a0a700 1 -- 172.21.15.121:0/880590913 wait complete. 2024-06-10T00:16:33.729 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling the dashboard module... 2024-06-10T00:16:34.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:34 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/880590913' entity='client.admin' 2024-06-10T00:16:34.340 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:34 smithi121 ceph-mon[32378]: from='mgr.14118 172.21.15.121:0/1323672259' entity='mgr.x' 2024-06-10T00:16:35.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:35 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/833989299' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-10T00:16:35.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:35 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: ignoring --setuser ceph since I am not root 2024-06-10T00:16:35.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:35 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: ignoring --setgroup ceph since I am not root 2024-06-10T00:16:35.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:35 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:35.314+0000 7f6b57cd7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-10T00:16:35.830 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:35 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:35.503+0000 7f6b57cd7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:16:35.831 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:35 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:35.605+0000 7f6b57cd7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-10T00:16:35.838 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.140+0000 7fc3e7eec700 1 Processor -- start 2024-06-10T00:16:35.838 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.140+0000 7fc3e7eec700 1 -- start start 2024-06-10T00:16:35.838 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.141+0000 7fc3e7eec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e0108780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:35.838 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.141+0000 7fc3e7eec700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e0108cc0 con 0x7fc3e0108360 2024-06-10T00:16:35.838 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.141+0000 7fc3e5c88700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e0108780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:35.838 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.141+0000 7fc3e5c88700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e0108780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40134/0 (socket says 172.21.15.121:40134) 2024-06-10T00:16:35.838 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.141+0000 7fc3e5c88700 1 -- 172.21.15.121:0/881555064 learned_addr learned my addr 172.21.15.121:0/881555064 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:35.838 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.141+0000 7fc3e5c88700 1 -- 172.21.15.121:0/881555064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e0106fc0 con 0x7fc3e0108360 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.141+0000 7fc3e5c88700 1 --2- 172.21.15.121:0/881555064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e0108780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00098a0 tx=0x7fc3d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e98c9330722f84 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.141+0000 7fc3e4c86700 1 -- 172.21.15.121:0/881555064 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d0004160 con 0x7fc3e0108360 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.141+0000 7fc3e4c86700 1 -- 172.21.15.121:0/881555064 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3d0004300 con 0x7fc3e0108360 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.141+0000 7fc3e4c86700 1 -- 172.21.15.121:0/881555064 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d0011b20 con 0x7fc3e0108360 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.142+0000 7fc3e7eec700 1 -- 172.21.15.121:0/881555064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 msgr2=0x7fc3e0108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.142+0000 7fc3e7eec700 1 --2- 172.21.15.121:0/881555064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e0108780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00098a0 tx=0x7fc3d0009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.142+0000 7fc3e7eec700 1 -- 172.21.15.121:0/881555064 shutdown_connections 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.142+0000 7fc3e7eec700 1 --2- 172.21.15.121:0/881555064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e0108780 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.142+0000 7fc3e7eec700 1 -- 172.21.15.121:0/881555064 >> 172.21.15.121:0/881555064 conn(0x7fc3e0101b90 msgr2=0x7fc3e0103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.142+0000 7fc3e7eec700 1 -- 172.21.15.121:0/881555064 shutdown_connections 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.142+0000 7fc3e7eec700 1 -- 172.21.15.121:0/881555064 wait complete. 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.142+0000 7fc3e7eec700 1 Processor -- start 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3e7eec700 1 -- start start 2024-06-10T00:16:35.839 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3e7eec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e019adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3e7eec700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e010b290 con 0x7fc3e0108360 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3e5c88700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e019adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3e5c88700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e019adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40150/0 (socket says 172.21.15.121:40150) 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3e5c88700 1 -- 172.21.15.121:0/833989299 learned_addr learned my addr 172.21.15.121:0/833989299 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3e5c88700 1 -- 172.21.15.121:0/833989299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e019b2f0 con 0x7fc3e0108360 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3e5c88700 1 --2- 172.21.15.121:0/833989299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e019adb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00117e0 tx=0x7fc3d0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3d6ffd700 1 -- 172.21.15.121:0/833989299 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d0003bf0 con 0x7fc3e0108360 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3d6ffd700 1 -- 172.21.15.121:0/833989299 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3d001a910 con 0x7fc3e0108360 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3e7eec700 1 -- 172.21.15.121:0/833989299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e019b580 con 0x7fc3e0108360 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.143+0000 7fc3d6ffd700 1 -- 172.21.15.121:0/833989299 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc3d0024970 con 0x7fc3e0108360 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.144+0000 7fc3e7eec700 1 -- 172.21.15.121:0/833989299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e019e230 con 0x7fc3e0108360 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.145+0000 7fc3d6ffd700 1 -- 172.21.15.121:0/833989299 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fc3d0022020 con 0x7fc3e0108360 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.145+0000 7fc3e7eec700 1 -- 172.21.15.121:0/833989299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3e0063af0 con 0x7fc3e0108360 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.145+0000 7fc3d6ffd700 1 --2- 172.21.15.121:0/833989299 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fc3cc0390a0 0x7fc3cc03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:35.840 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.145+0000 7fc3d6ffd700 1 -- 172.21.15.121:0/833989299 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc3d004d3c0 con 0x7fc3e0108360 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.145+0000 7fc3e5487700 1 --2- 172.21.15.121:0/833989299 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fc3cc0390a0 0x7fc3cc03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.145+0000 7fc3e5487700 1 --2- 172.21.15.121:0/833989299 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fc3cc0390a0 0x7fc3cc03b560 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc006fd0 tx=0x7fc3dc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.150+0000 7fc3d6ffd700 1 -- 172.21.15.121:0/833989299 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc3d00212c0 con 0x7fc3e0108360 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:34.412+0000 7fc3e7eec700 1 -- 172.21.15.121:0/833989299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fc3e019b9e0 con 0x7fc3e0108360 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.135+0000 7fc3d6ffd700 1 -- 172.21.15.121:0/833989299 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fc3d001ab70 con 0x7fc3e0108360 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.137+0000 7fc3d6ffd700 1 -- 172.21.15.121:0/833989299 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fc3d002d3f0 con 0x7fc3e0108360 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.139+0000 7fc3e7eec700 1 -- 172.21.15.121:0/833989299 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fc3cc0390a0 msgr2=0x7fc3cc03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.139+0000 7fc3e7eec700 1 --2- 172.21.15.121:0/833989299 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fc3cc0390a0 0x7fc3cc03b560 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc006fd0 tx=0x7fc3dc006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.139+0000 7fc3e7eec700 1 -- 172.21.15.121:0/833989299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 msgr2=0x7fc3e019adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.139+0000 7fc3e7eec700 1 --2- 172.21.15.121:0/833989299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e019adb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00117e0 tx=0x7fc3d0011ba0 comp rx=0 tx=0).stop 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.139+0000 7fc3e7eec700 1 -- 172.21.15.121:0/833989299 shutdown_connections 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.139+0000 7fc3e7eec700 1 --2- 172.21.15.121:0/833989299 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7fc3cc0390a0 0x7fc3cc03b560 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.139+0000 7fc3e7eec700 1 --2- 172.21.15.121:0/833989299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e0108360 0x7fc3e019adb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:35.841 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.139+0000 7fc3e7eec700 1 -- 172.21.15.121:0/833989299 >> 172.21.15.121:0/833989299 conn(0x7fc3e0101b90 msgr2=0x7fc3e0102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:35.842 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.139+0000 7fc3e7eec700 1 -- 172.21.15.121:0/833989299 shutdown_connections 2024-06-10T00:16:35.842 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:35.139+0000 7fc3e7eec700 1 -- 172.21.15.121:0/833989299 wait complete. 2024-06-10T00:16:36.133 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:35 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:35.917+0000 7f6b57cd7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-10T00:16:36.134 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:36 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:36.003+0000 7f6b57cd7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-10T00:16:36.442 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:36 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/833989299' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-10T00:16:36.443 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:36 smithi121 ceph-mon[32378]: mgrmap e9: x(active, since 19s) 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.237+0000 7f22007b4700 1 Processor -- start 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.237+0000 7f22007b4700 1 -- start start 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.237+0000 7f22007b4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f810ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.237+0000 7f22007b4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f21f807c660 con 0x7f21f8108a30 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.237+0000 7f21fe550700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f810ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:37.079 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.237+0000 7f21fe550700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f810ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40168/0 (socket says 172.21.15.121:40168) 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.237+0000 7f21fe550700 1 -- 172.21.15.121:0/3755366334 learned_addr learned my addr 172.21.15.121:0/3755366334 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.238+0000 7f21fe550700 1 -- 172.21.15.121:0/3755366334 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f810b360 con 0x7f21f8108a30 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.238+0000 7f21fe550700 1 --2- 172.21.15.121:0/3755366334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f810ae20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f21f40098a0 tx=0x7f21f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca581b1d938f5dc server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.238+0000 7f21fd54e700 1 -- 172.21.15.121:0/3755366334 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f4004160 con 0x7f21f8108a30 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.238+0000 7f21fd54e700 1 -- 172.21.15.121:0/3755366334 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21f4004300 con 0x7f21f8108a30 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.238+0000 7f21fd54e700 1 -- 172.21.15.121:0/3755366334 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f4011b20 con 0x7f21f8108a30 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.239+0000 7f22007b4700 1 -- 172.21.15.121:0/3755366334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 msgr2=0x7f21f810ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.239+0000 7f22007b4700 1 --2- 172.21.15.121:0/3755366334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f810ae20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f21f40098a0 tx=0x7f21f4009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.239+0000 7f22007b4700 1 -- 172.21.15.121:0/3755366334 shutdown_connections 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.239+0000 7f22007b4700 1 --2- 172.21.15.121:0/3755366334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f810ae20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.239+0000 7f22007b4700 1 -- 172.21.15.121:0/3755366334 >> 172.21.15.121:0/3755366334 conn(0x7f21f8102460 msgr2=0x7f21f8104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.239+0000 7f22007b4700 1 -- 172.21.15.121:0/3755366334 shutdown_connections 2024-06-10T00:16:37.080 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.239+0000 7f22007b4700 1 -- 172.21.15.121:0/3755366334 wait complete. 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.239+0000 7f22007b4700 1 Processor -- start 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.239+0000 7f22007b4700 1 -- start start 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.240+0000 7f22007b4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f81082b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.240+0000 7f22007b4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f21f810c360 con 0x7f21f8108a30 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.240+0000 7f21fe550700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f81082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.240+0000 7f21fe550700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f81082b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40170/0 (socket says 172.21.15.121:40170) 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.240+0000 7f21fe550700 1 -- 172.21.15.121:0/1950287099 learned_addr learned my addr 172.21.15.121:0/1950287099 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.240+0000 7f21fe550700 1 -- 172.21.15.121:0/1950287099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f81087f0 con 0x7f21f8108a30 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.240+0000 7f21fe550700 1 --2- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f81082b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f21f40117e0 tx=0x7f21f4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.240+0000 7f21ef7fe700 1 -- 172.21.15.121:0/1950287099 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f4003bf0 con 0x7f21f8108a30 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.240+0000 7f21ef7fe700 1 -- 172.21.15.121:0/1950287099 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21f401a910 con 0x7f21f8108a30 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.240+0000 7f22007b4700 1 -- 172.21.15.121:0/1950287099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f81069e0 con 0x7f21f8108a30 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.241+0000 7f21ef7fe700 1 -- 172.21.15.121:0/1950287099 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f4024970 con 0x7f21f8108a30 2024-06-10T00:16:37.081 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.241+0000 7f22007b4700 1 -- 172.21.15.121:0/1950287099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f8106ea0 con 0x7f21f8108a30 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.242+0000 7f22007b4700 1 -- 172.21.15.121:0/1950287099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f8063af0 con 0x7f21f8108a30 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.242+0000 7f21ef7fe700 1 -- 172.21.15.121:0/1950287099 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f21f4022020 con 0x7f21f8108a30 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.242+0000 7f21ef7fe700 1 --2- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f21e4039050 0x7f21e403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.242+0000 7f21ef7fe700 1 -- 172.21.15.121:0/1950287099 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f21f404d3e0 con 0x7f21f8108a30 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.242+0000 7f21fdd4f700 1 -- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f21e4039050 msgr2=0x7f21e403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/704666019 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.242+0000 7f21fdd4f700 1 --2- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f21e4039050 0x7f21e403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.247+0000 7f21ef7fe700 1 -- 172.21.15.121:0/1950287099 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f21f4011e20 con 0x7f21f8108a30 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.443+0000 7f21fdd4f700 1 -- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f21e4039050 msgr2=0x7f21e403b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/704666019 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.443+0000 7f21fdd4f700 1 --2- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f21e4039050 0x7f21e403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.540+0000 7f22007b4700 1 -- 172.21.15.121:0/1950287099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f21f81071e0 con 0x7f21f8108a30 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.540+0000 7f21ef7fe700 1 -- 172.21.15.121:0/1950287099 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f21f81071e0 con 0x7f21f8108a30 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.543+0000 7f22007b4700 1 -- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f21e4039050 msgr2=0x7f21e403b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:16:37.082 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.543+0000 7f22007b4700 1 --2- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f21e4039050 0x7f21e403b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:37.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.543+0000 7f22007b4700 1 -- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 msgr2=0x7f21f81082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:37.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.543+0000 7f22007b4700 1 --2- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f81082b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f21f40117e0 tx=0x7f21f4011c10 comp rx=0 tx=0).stop 2024-06-10T00:16:37.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.544+0000 7f22007b4700 1 -- 172.21.15.121:0/1950287099 shutdown_connections 2024-06-10T00:16:37.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.544+0000 7f22007b4700 1 --2- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f21e4039050 0x7f21e403b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:37.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.544+0000 7f22007b4700 1 --2- 172.21.15.121:0/1950287099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21f8108a30 0x7f21f81082b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:37.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.544+0000 7f22007b4700 1 -- 172.21.15.121:0/1950287099 >> 172.21.15.121:0/1950287099 conn(0x7f21f8102460 msgr2=0x7f21f8102f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:37.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.544+0000 7f22007b4700 1 -- 172.21.15.121:0/1950287099 shutdown_connections 2024-06-10T00:16:37.083 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:36.544+0000 7f22007b4700 1 -- 172.21.15.121:0/1950287099 wait complete. 2024-06-10T00:16:37.083 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for the mgr to restart... 2024-06-10T00:16:37.083 INFO:teuthology.orchestra.run.smithi121.stdout:Waiting for mgr epoch 9... 2024-06-10T00:16:37.432 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:37 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1950287099' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-10T00:16:37.433 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:37 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:37.255+0000 7f6b57cd7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-10T00:16:37.433 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:37 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:37.432+0000 7f6b57cd7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-10T00:16:37.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:37 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:37.627+0000 7f6b57cd7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-10T00:16:38.199 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:37 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:37.917+0000 7f6b57cd7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-10T00:16:38.199 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:38 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:38.010+0000 7f6b57cd7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-10T00:16:38.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:38 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:38.213+0000 7f6b57cd7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-10T00:16:38.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:38 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:38.404+0000 7f6b57cd7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:16:39.199 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:38 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:38.997+0000 7f6b57cd7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-10T00:16:39.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:39 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:39.416+0000 7f6b57cd7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-10T00:16:39.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:39 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:39.512+0000 7f6b57cd7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-10T00:16:40.199 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:39 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:39.712+0000 7f6b57cd7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-10T00:16:40.563 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:40 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:40.231+0000 7f6b57cd7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-10T00:16:40.563 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:40 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:40.322+0000 7f6b57cd7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-10T00:16:40.905 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:40 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:40.563+0000 7f6b57cd7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-10T00:16:40.905 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:40 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:40.695+0000 7f6b57cd7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-10T00:16:40.906 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:40 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:40.790+0000 7f6b57cd7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-10T00:16:40.906 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:40 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:40.905+0000 7f6b57cd7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-10T00:16:41.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:41 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:41.201+0000 7f6b57cd7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-10T00:16:42.061 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:41 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:41.793+0000 7f6b57cd7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-10T00:16:42.062 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:16:42.009+0000 7f6b57cd7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-10T00:16:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-mon[32378]: Active manager daemon x restarted 2024-06-10T00:16:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-mon[32378]: Activating manager daemon x 2024-06-10T00:16:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-mon[32378]: osdmap e3: 0 total, 0 up, 0 in 2024-06-10T00:16:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-mon[32378]: mgrmap e10: x(active, starting, since 0.00352247s) 2024-06-10T00:16:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:16:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-10T00:16:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-10T00:16:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-10T00:16:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-10T00:16:42.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-mon[32378]: Manager daemon x is now available 2024-06-10T00:16:42.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:16:43.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-10T00:16:43.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-10T00:16:43.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:43 smithi121 ceph-mon[32378]: mgrmap e11: x(active, since 1.006s) 2024-06-10T00:16:43.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout { 2024-06-10T00:16:43.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-10T00:16:43.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-10T00:16:43.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout } 2024-06-10T00:16:43.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.459+0000 7f2e6191c700 1 Processor -- start 2024-06-10T00:16:43.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.459+0000 7f2e6191c700 1 -- start start 2024-06-10T00:16:43.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.459+0000 7f2e6191c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c1088a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:43.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.459+0000 7f2e6191c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5c108e70 con 0x7f2e5c108480 2024-06-10T00:16:43.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.459+0000 7f2e5affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c1088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:43.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.459+0000 7f2e5affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c1088a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40178/0 (socket says 172.21.15.121:40178) 2024-06-10T00:16:43.922 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.459+0000 7f2e5affd700 1 -- 172.21.15.121:0/1080391717 learned_addr learned my addr 172.21.15.121:0/1080391717 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.460+0000 7f2e5affd700 1 -- 172.21.15.121:0/1080391717 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5c108ff0 con 0x7f2e5c108480 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.460+0000 7f2e5affd700 1 --2- 172.21.15.121:0/1080391717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c1088a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2e500098a0 tx=0x7f2e50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61dc78629b56e099 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.460+0000 7f2e59ffb700 1 -- 172.21.15.121:0/1080391717 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e50004160 con 0x7f2e5c108480 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.460+0000 7f2e59ffb700 1 -- 172.21.15.121:0/1080391717 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e50004300 con 0x7f2e5c108480 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.460+0000 7f2e59ffb700 1 -- 172.21.15.121:0/1080391717 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e50011b20 con 0x7f2e5c108480 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.461+0000 7f2e6191c700 1 -- 172.21.15.121:0/1080391717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 msgr2=0x7f2e5c1088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.461+0000 7f2e6191c700 1 --2- 172.21.15.121:0/1080391717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c1088a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2e500098a0 tx=0x7f2e50009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.461+0000 7f2e6191c700 1 -- 172.21.15.121:0/1080391717 shutdown_connections 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.461+0000 7f2e6191c700 1 --2- 172.21.15.121:0/1080391717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c1088a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.461+0000 7f2e6191c700 1 -- 172.21.15.121:0/1080391717 >> 172.21.15.121:0/1080391717 conn(0x7f2e5c101b70 msgr2=0x7f2e5c103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.461+0000 7f2e6191c700 1 -- 172.21.15.121:0/1080391717 shutdown_connections 2024-06-10T00:16:43.923 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.461+0000 7f2e6191c700 1 -- 172.21.15.121:0/1080391717 wait complete. 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.461+0000 7f2e6191c700 1 Processor -- start 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.462+0000 7f2e6191c700 1 -- start start 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.462+0000 7f2e6191c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c19e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.462+0000 7f2e6191c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5c106ea0 con 0x7f2e5c108480 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.462+0000 7f2e5affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c19e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.462+0000 7f2e5affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c19e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40188/0 (socket says 172.21.15.121:40188) 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.462+0000 7f2e5affd700 1 -- 172.21.15.121:0/3527464269 learned_addr learned my addr 172.21.15.121:0/3527464269 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.462+0000 7f2e5affd700 1 -- 172.21.15.121:0/3527464269 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5c19e5a0 con 0x7f2e5c108480 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.462+0000 7f2e5affd700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c19e060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e50000c00 tx=0x7f2e50011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.462+0000 7f2e6091a700 1 -- 172.21.15.121:0/3527464269 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e50003b60 con 0x7f2e5c108480 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.463+0000 7f2e6191c700 1 -- 172.21.15.121:0/3527464269 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5c19e890 con 0x7f2e5c108480 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.463+0000 7f2e6091a700 1 -- 172.21.15.121:0/3527464269 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e5001a910 con 0x7f2e5c108480 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.463+0000 7f2e6091a700 1 -- 172.21.15.121:0/3527464269 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2e50024930 con 0x7f2e5c108480 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.463+0000 7f2e6191c700 1 -- 172.21.15.121:0/3527464269 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e5c1a1520 con 0x7f2e5c108480 2024-06-10T00:16:43.924 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.463+0000 7f2e6091a700 1 -- 172.21.15.121:0/3527464269 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f2e50022020 con 0x7f2e5c108480 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.464+0000 7f2e6091a700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 0x7f2e4403b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.464+0000 7f2e6191c700 1 -- 172.21.15.121:0/3527464269 --> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2e5c0695c0 con 0x7f2e44039100 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.464+0000 7f2e6091a700 1 -- 172.21.15.121:0/3527464269 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2e5004e3f0 con 0x7f2e5c108480 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.464+0000 7f2e5a7fc700 1 -- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 msgr2=0x7f2e4403b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/704666019 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.464+0000 7f2e5a7fc700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 0x7f2e4403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.665+0000 7f2e5a7fc700 1 -- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 msgr2=0x7f2e4403b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/704666019 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:37.665+0000 7f2e5a7fc700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 0x7f2e4403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:38.066+0000 7f2e5a7fc700 1 -- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 msgr2=0x7f2e4403b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/704666019 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:38.066+0000 7f2e5a7fc700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 0x7f2e4403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:38.867+0000 7f2e5a7fc700 1 -- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 msgr2=0x7f2e4403b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/704666019 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:38.867+0000 7f2e5a7fc700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 0x7f2e4403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:40.468+0000 7f2e5a7fc700 1 -- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 msgr2=0x7f2e4403b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/704666019 2024-06-10T00:16:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:40.468+0000 7f2e5a7fc700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 0x7f2e4403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:42.012+0000 7f2e6091a700 1 -- 172.21.15.121:0/3527464269 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f2e5001aac0 con 0x7f2e5c108480 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:42.012+0000 7f2e6091a700 1 -- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 msgr2=0x7f2e4403b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:42.012+0000 7f2e6091a700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 0x7f2e4403b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.013+0000 7f2e6091a700 1 -- 172.21.15.121:0/3527464269 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f2e5002dc40 con 0x7f2e5c108480 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.014+0000 7f2e6091a700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e4403c9d0 0x7f2e4403edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.014+0000 7f2e6091a700 1 -- 172.21.15.121:0/3527464269 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2e5c0695c0 con 0x7f2e4403c9d0 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.014+0000 7f2e5a7fc700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e4403c9d0 0x7f2e4403edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.015+0000 7f2e5a7fc700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e4403c9d0 0x7f2e4403edc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c003a10 tx=0x7f2e4c009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.015+0000 7f2e6091a700 1 -- 172.21.15.121:0/3527464269 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f2e5c0695c0 con 0x7f2e4403c9d0 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.022+0000 7f2e6191c700 1 -- 172.21.15.121:0/3527464269 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f2e5c195e20 con 0x7f2e4403c9d0 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.022+0000 7f2e6091a700 1 -- 172.21.15.121:0/3527464269 <== mgr.14150 v2:172.21.15.121:6800/3506007074 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f2e5c195e20 con 0x7f2e4403c9d0 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.023+0000 7f2e6191c700 1 -- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e4403c9d0 msgr2=0x7f2e4403edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.023+0000 7f2e6191c700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e4403c9d0 0x7f2e4403edc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c003a10 tx=0x7f2e4c009270 comp rx=0 tx=0).stop 2024-06-10T00:16:43.926 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.023+0000 7f2e6191c700 1 -- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 msgr2=0x7f2e5c19e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:43.927 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.023+0000 7f2e6191c700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c19e060 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e50000c00 tx=0x7f2e50011c10 comp rx=0 tx=0).stop 2024-06-10T00:16:43.927 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.023+0000 7f2e6191c700 1 -- 172.21.15.121:0/3527464269 shutdown_connections 2024-06-10T00:16:43.927 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.023+0000 7f2e6191c700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e4403c9d0 0x7f2e4403edc0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:43.927 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.023+0000 7f2e6191c700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e5c108480 0x7f2e5c19e060 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:43.927 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.023+0000 7f2e6191c700 1 --2- 172.21.15.121:0/3527464269 >> [v2:172.21.15.121:6800/704666019,v1:172.21.15.121:6801/704666019] conn(0x7f2e44039100 0x7f2e4403b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:43.927 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.023+0000 7f2e6191c700 1 -- 172.21.15.121:0/3527464269 >> 172.21.15.121:0/3527464269 conn(0x7f2e5c101b70 msgr2=0x7f2e5c1946a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:43.927 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.023+0000 7f2e6191c700 1 -- 172.21.15.121:0/3527464269 shutdown_connections 2024-06-10T00:16:43.927 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:43.023+0000 7f2e6191c700 1 -- 172.21.15.121:0/3527464269 wait complete. 2024-06-10T00:16:43.927 INFO:teuthology.orchestra.run.smithi121.stdout:mgr epoch 9 is available 2024-06-10T00:16:43.927 INFO:teuthology.orchestra.run.smithi121.stdout:Generating a dashboard self-signed certificate... 2024-06-10T00:16:44.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:44 smithi121 ceph-mon[32378]: [10/Jun/2024:00:16:42] ENGINE Bus STARTING 2024-06-10T00:16:44.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:44 smithi121 ceph-mon[32378]: [10/Jun/2024:00:16:42] ENGINE Serving on http://172.21.15.121:8765 2024-06-10T00:16:44.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:44 smithi121 ceph-mon[32378]: [10/Jun/2024:00:16:42] ENGINE Serving on https://172.21.15.121:7150 2024-06-10T00:16:44.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:44 smithi121 ceph-mon[32378]: [10/Jun/2024:00:16:42] ENGINE Bus STARTED 2024-06-10T00:16:44.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:44 smithi121 ceph-mon[32378]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-10T00:16:44.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:44 smithi121 ceph-mon[32378]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-10T00:16:44.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:45.286 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-10T00:16:45.286 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.335+0000 7f23f2538700 1 Processor -- start 2024-06-10T00:16:45.286 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.335+0000 7f23f2538700 1 -- start start 2024-06-10T00:16:45.286 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.336+0000 7f23f2538700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec108800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:45.286 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.336+0000 7f23f2538700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec108dd0 con 0x7f23ec1083e0 2024-06-10T00:16:45.286 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.336+0000 7f23ebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec108800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:45.286 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.336+0000 7f23ebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec108800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60658/0 (socket says 172.21.15.121:60658) 2024-06-10T00:16:45.286 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.336+0000 7f23ebfff700 1 -- 172.21.15.121:0/1145742516 learned_addr learned my addr 172.21.15.121:0/1145742516 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.336+0000 7f23ebfff700 1 -- 172.21.15.121:0/1145742516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ec108f50 con 0x7f23ec1083e0 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.336+0000 7f23ebfff700 1 --2- 172.21.15.121:0/1145742516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec108800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f23dc0098a0 tx=0x7f23dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c335d7ecfb151e1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.336+0000 7f23eaffd700 1 -- 172.21.15.121:0/1145742516 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f23dc004160 con 0x7f23ec1083e0 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.336+0000 7f23eaffd700 1 -- 172.21.15.121:0/1145742516 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23dc004300 con 0x7f23ec1083e0 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.336+0000 7f23eaffd700 1 -- 172.21.15.121:0/1145742516 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f23dc011b20 con 0x7f23ec1083e0 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.337+0000 7f23f2538700 1 -- 172.21.15.121:0/1145742516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 msgr2=0x7f23ec108800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.337+0000 7f23f2538700 1 --2- 172.21.15.121:0/1145742516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec108800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f23dc0098a0 tx=0x7f23dc009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.337+0000 7f23f2538700 1 -- 172.21.15.121:0/1145742516 shutdown_connections 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.337+0000 7f23f2538700 1 --2- 172.21.15.121:0/1145742516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec108800 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.337+0000 7f23f2538700 1 -- 172.21.15.121:0/1145742516 >> 172.21.15.121:0/1145742516 conn(0x7f23ec101b90 msgr2=0x7f23ec103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.337+0000 7f23f2538700 1 -- 172.21.15.121:0/1145742516 shutdown_connections 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.337+0000 7f23f2538700 1 -- 172.21.15.121:0/1145742516 wait complete. 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.338+0000 7f23f2538700 1 Processor -- start 2024-06-10T00:16:45.287 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.338+0000 7f23f2538700 1 -- start start 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.338+0000 7f23f2538700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec19e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.338+0000 7f23f2538700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f23ec106e50 con 0x7f23ec1083e0 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.338+0000 7f23ebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec19e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.338+0000 7f23ebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec19e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60670/0 (socket says 172.21.15.121:60670) 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.338+0000 7f23ebfff700 1 -- 172.21.15.121:0/952446297 learned_addr learned my addr 172.21.15.121:0/952446297 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.339+0000 7f23ebfff700 1 -- 172.21.15.121:0/952446297 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ec19e540 con 0x7f23ec1083e0 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.339+0000 7f23ebfff700 1 --2- 172.21.15.121:0/952446297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec19e000 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f23dc0117e0 tx=0x7f23dc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.339+0000 7f23e97fa700 1 -- 172.21.15.121:0/952446297 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f23dc003bf0 con 0x7f23ec1083e0 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.339+0000 7f23f2538700 1 -- 172.21.15.121:0/952446297 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23ec19e830 con 0x7f23ec1083e0 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.339+0000 7f23e97fa700 1 -- 172.21.15.121:0/952446297 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23dc01a910 con 0x7f23ec1083e0 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.339+0000 7f23e97fa700 1 -- 172.21.15.121:0/952446297 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f23dc024970 con 0x7f23ec1083e0 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.339+0000 7f23f2538700 1 -- 172.21.15.121:0/952446297 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23ec1a14c0 con 0x7f23ec1083e0 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.340+0000 7f23e97fa700 1 -- 172.21.15.121:0/952446297 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f23dc022020 con 0x7f23ec1083e0 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.340+0000 7f23f2538700 1 -- 172.21.15.121:0/952446297 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23ec063af0 con 0x7f23ec1083e0 2024-06-10T00:16:45.288 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.340+0000 7f23e97fa700 1 --2- 172.21.15.121:0/952446297 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23d4038f40 0x7f23d403b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.340+0000 7f23e97fa700 1 -- 172.21.15.121:0/952446297 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f23dc04d2c0 con 0x7f23ec1083e0 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.340+0000 7f23eb7fe700 1 --2- 172.21.15.121:0/952446297 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23d4038f40 0x7f23d403b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.341+0000 7f23eb7fe700 1 --2- 172.21.15.121:0/952446297 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23d4038f40 0x7f23d403b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f23e0006fd0 tx=0x7f23e0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.345+0000 7f23e97fa700 1 -- 172.21.15.121:0/952446297 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f23dc011e20 con 0x7f23ec1083e0 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.553+0000 7f23f2538700 1 -- 172.21.15.121:0/952446297 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f23ec0628d0 con 0x7f23d4038f40 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.619+0000 7f23e97fa700 1 -- 172.21.15.121:0/952446297 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f23dc024cd0 con 0x7f23ec1083e0 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.655+0000 7f23e97fa700 1 -- 172.21.15.121:0/952446297 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f23ec0628d0 con 0x7f23d4038f40 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.657+0000 7f23f2538700 1 -- 172.21.15.121:0/952446297 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23d4038f40 msgr2=0x7f23d403b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.657+0000 7f23f2538700 1 --2- 172.21.15.121:0/952446297 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23d4038f40 0x7f23d403b400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f23e0006fd0 tx=0x7f23e0006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.657+0000 7f23f2538700 1 -- 172.21.15.121:0/952446297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 msgr2=0x7f23ec19e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.657+0000 7f23f2538700 1 --2- 172.21.15.121:0/952446297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec19e000 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f23dc0117e0 tx=0x7f23dc011c10 comp rx=0 tx=0).stop 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.657+0000 7f23f2538700 1 -- 172.21.15.121:0/952446297 shutdown_connections 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.657+0000 7f23f2538700 1 --2- 172.21.15.121:0/952446297 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23d4038f40 0x7f23d403b400 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:45.289 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.657+0000 7f23f2538700 1 --2- 172.21.15.121:0/952446297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23ec1083e0 0x7f23ec19e000 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:45.290 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.657+0000 7f23f2538700 1 -- 172.21.15.121:0/952446297 >> 172.21.15.121:0/952446297 conn(0x7f23ec101b90 msgr2=0x7f23ec194620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:45.290 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.657+0000 7f23f2538700 1 -- 172.21.15.121:0/952446297 shutdown_connections 2024-06-10T00:16:45.290 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:44.657+0000 7f23f2538700 1 -- 172.21.15.121:0/952446297 wait complete. 2024-06-10T00:16:45.290 INFO:teuthology.orchestra.run.smithi121.stdout:Creating initial admin user... 2024-06-10T00:16:45.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:45 smithi121 ceph-mon[32378]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:16:45.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:45 smithi121 ceph-mon[32378]: mgrmap e12: x(active, since 2s) 2024-06-10T00:16:45.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:45.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:45.830 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:46.804 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$PPRUl6ltN8DICjO.UvXxWuv59xtWx32ElRf39ApS4lJTs4kvzFxdm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1717978606, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-10T00:16:46.804 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.660+0000 7f3552402700 1 Processor -- start 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.660+0000 7f3552402700 1 -- start start 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.660+0000 7f3552402700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c10ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.660+0000 7f3552402700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f354c07c640 con 0x7f354c108a70 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.661+0000 7f354bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c10ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.661+0000 7f354bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c10ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60686/0 (socket says 172.21.15.121:60686) 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.661+0000 7f354bfff700 1 -- 172.21.15.121:0/968033199 learned_addr learned my addr 172.21.15.121:0/968033199 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.661+0000 7f354bfff700 1 -- 172.21.15.121:0/968033199 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354c10b3a0 con 0x7f354c108a70 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.661+0000 7f354bfff700 1 --2- 172.21.15.121:0/968033199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c10ae60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f353c0098a0 tx=0x7f353c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661d4ad581c49334 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.661+0000 7f354b7fe700 1 -- 172.21.15.121:0/968033199 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f353c004160 con 0x7f354c108a70 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.661+0000 7f354b7fe700 1 -- 172.21.15.121:0/968033199 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f353c004300 con 0x7f354c108a70 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.661+0000 7f354b7fe700 1 -- 172.21.15.121:0/968033199 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f353c011b20 con 0x7f354c108a70 2024-06-10T00:16:46.805 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.662+0000 7f3552402700 1 -- 172.21.15.121:0/968033199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 msgr2=0x7f354c10ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.662+0000 7f3552402700 1 --2- 172.21.15.121:0/968033199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c10ae60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f353c0098a0 tx=0x7f353c009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.662+0000 7f3552402700 1 -- 172.21.15.121:0/968033199 shutdown_connections 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.662+0000 7f3552402700 1 --2- 172.21.15.121:0/968033199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c10ae60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.662+0000 7f3552402700 1 -- 172.21.15.121:0/968033199 >> 172.21.15.121:0/968033199 conn(0x7f354c1024c0 msgr2=0x7f354c1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.662+0000 7f3552402700 1 -- 172.21.15.121:0/968033199 shutdown_connections 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.662+0000 7f3552402700 1 -- 172.21.15.121:0/968033199 wait complete. 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.663+0000 7f3552402700 1 Processor -- start 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.663+0000 7f3552402700 1 -- start start 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.663+0000 7f3552402700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c19be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.663+0000 7f3552402700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f354c10c340 con 0x7f354c108a70 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.663+0000 7f354bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c19be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:46.806 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.663+0000 7f354bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c19be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60692/0 (socket says 172.21.15.121:60692) 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.663+0000 7f354bfff700 1 -- 172.21.15.121:0/4062617915 learned_addr learned my addr 172.21.15.121:0/4062617915 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.663+0000 7f354bfff700 1 -- 172.21.15.121:0/4062617915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354c19c3d0 con 0x7f354c108a70 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.663+0000 7f354bfff700 1 --2- 172.21.15.121:0/4062617915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c19be90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f353c0117e0 tx=0x7f353c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.664+0000 7f3549ffb700 1 -- 172.21.15.121:0/4062617915 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f353c003ba0 con 0x7f354c108a70 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.664+0000 7f3552402700 1 -- 172.21.15.121:0/4062617915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354c19c660 con 0x7f354c108a70 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.664+0000 7f3549ffb700 1 -- 172.21.15.121:0/4062617915 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f353c01a910 con 0x7f354c108a70 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.664+0000 7f3549ffb700 1 -- 172.21.15.121:0/4062617915 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f353c0248f0 con 0x7f354c108a70 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.664+0000 7f3552402700 1 -- 172.21.15.121:0/4062617915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f354c19cac0 con 0x7f354c108a70 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.664+0000 7f3549ffb700 1 -- 172.21.15.121:0/4062617915 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f353c022020 con 0x7f354c108a70 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.665+0000 7f3549ffb700 1 --2- 172.21.15.121:0/4062617915 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f35380390a0 0x7f353803b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.665+0000 7f3552402700 1 -- 172.21.15.121:0/4062617915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f354c063af0 con 0x7f354c108a70 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.665+0000 7f3549ffb700 1 -- 172.21.15.121:0/4062617915 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f353c04da10 con 0x7f354c108a70 2024-06-10T00:16:46.807 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.665+0000 7f3543fff700 1 --2- 172.21.15.121:0/4062617915 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f35380390a0 0x7f353803b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.665+0000 7f3543fff700 1 --2- 172.21.15.121:0/4062617915 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f35380390a0 0x7f353803b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3534006fd0 tx=0x7f3534006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.670+0000 7f3549ffb700 1 -- 172.21.15.121:0/4062617915 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f353c01aac0 con 0x7f354c108a70 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:45.890+0000 7f3552402700 1 -- 172.21.15.121:0/4062617915 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f354c1036a0 con 0x7f35380390a0 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:46.116+0000 7f3549ffb700 1 -- 172.21.15.121:0/4062617915 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f354c1036a0 con 0x7f35380390a0 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:46.118+0000 7f3552402700 1 -- 172.21.15.121:0/4062617915 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f35380390a0 msgr2=0x7f353803b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:46.118+0000 7f3552402700 1 --2- 172.21.15.121:0/4062617915 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f35380390a0 0x7f353803b560 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3534006fd0 tx=0x7f3534006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:46.118+0000 7f3552402700 1 -- 172.21.15.121:0/4062617915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 msgr2=0x7f354c19be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:46.118+0000 7f3552402700 1 --2- 172.21.15.121:0/4062617915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c19be90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f353c0117e0 tx=0x7f353c011ba0 comp rx=0 tx=0).stop 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:46.118+0000 7f3552402700 1 -- 172.21.15.121:0/4062617915 shutdown_connections 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:46.118+0000 7f3552402700 1 --2- 172.21.15.121:0/4062617915 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f35380390a0 0x7f353803b560 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:46.118+0000 7f3552402700 1 --2- 172.21.15.121:0/4062617915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f354c108a70 0x7f354c19be90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:46.118+0000 7f3552402700 1 -- 172.21.15.121:0/4062617915 >> 172.21.15.121:0/4062617915 conn(0x7f354c1024c0 msgr2=0x7f354c102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:46.808 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:46.118+0000 7f3552402700 1 -- 172.21.15.121:0/4062617915 shutdown_connections 2024-06-10T00:16:46.809 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:46.118+0000 7f3552402700 1 -- 172.21.15.121:0/4062617915 wait complete. 2024-06-10T00:16:46.809 INFO:teuthology.orchestra.run.smithi121.stdout:Fetching dashboard port number... 2024-06-10T00:16:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:47 smithi121 ceph-mon[32378]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:16:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:47 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:48.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stdout 8443 2024-06-10T00:16:48.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.344+0000 7f3e905d6700 1 Processor -- start 2024-06-10T00:16:48.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.344+0000 7f3e905d6700 1 -- start start 2024-06-10T00:16:48.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.345+0000 7f3e905d6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8810ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:48.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.345+0000 7f3e905d6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8807c660 con 0x7f3e88108a30 2024-06-10T00:16:48.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.345+0000 7f3e8e372700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8810ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:48.106 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.345+0000 7f3e8e372700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8810ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60702/0 (socket says 172.21.15.121:60702) 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.345+0000 7f3e8e372700 1 -- 172.21.15.121:0/1973678358 learned_addr learned my addr 172.21.15.121:0/1973678358 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.345+0000 7f3e8e372700 1 -- 172.21.15.121:0/1973678358 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8810b360 con 0x7f3e88108a30 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.345+0000 7f3e8e372700 1 --2- 172.21.15.121:0/1973678358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8810ae20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009e80 tx=0x7f3e8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab026cb5ac6bcf2 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.345+0000 7f3e8d370700 1 -- 172.21.15.121:0/1973678358 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e84017070 con 0x7f3e88108a30 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.345+0000 7f3e8d370700 1 -- 172.21.15.121:0/1973678358 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e8400b8c0 con 0x7f3e88108a30 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.345+0000 7f3e8d370700 1 -- 172.21.15.121:0/1973678358 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e840045c0 con 0x7f3e88108a30 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.346+0000 7f3e905d6700 1 -- 172.21.15.121:0/1973678358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 msgr2=0x7f3e8810ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.346+0000 7f3e905d6700 1 --2- 172.21.15.121:0/1973678358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8810ae20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009e80 tx=0x7f3e8400b330 comp rx=0 tx=0).stop 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.346+0000 7f3e905d6700 1 -- 172.21.15.121:0/1973678358 shutdown_connections 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.346+0000 7f3e905d6700 1 --2- 172.21.15.121:0/1973678358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8810ae20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.346+0000 7f3e905d6700 1 -- 172.21.15.121:0/1973678358 >> 172.21.15.121:0/1973678358 conn(0x7f3e88102460 msgr2=0x7f3e88104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:48.107 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.346+0000 7f3e905d6700 1 -- 172.21.15.121:0/1973678358 shutdown_connections 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.346+0000 7f3e905d6700 1 -- 172.21.15.121:0/1973678358 wait complete. 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.346+0000 7f3e905d6700 1 Processor -- start 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e905d6700 1 -- start start 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e905d6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8819bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e905d6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8810c360 con 0x7f3e88108a30 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e8e372700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8819bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e8e372700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8819bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60716/0 (socket says 172.21.15.121:60716) 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e8e372700 1 -- 172.21.15.121:0/59012055 learned_addr learned my addr 172.21.15.121:0/59012055 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e8e372700 1 -- 172.21.15.121:0/59012055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8819c410 con 0x7f3e88108a30 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e8e372700 1 --2- 172.21.15.121:0/59012055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8819bed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400b7b0 tx=0x7f3e8400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e7f7fe700 1 -- 172.21.15.121:0/59012055 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e84020020 con 0x7f3e88108a30 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e7f7fe700 1 -- 172.21.15.121:0/59012055 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e84003d90 con 0x7f3e88108a30 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e905d6700 1 -- 172.21.15.121:0/59012055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8819c6a0 con 0x7f3e88108a30 2024-06-10T00:16:48.108 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e7f7fe700 1 -- 172.21.15.121:0/59012055 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e84017040 con 0x7f3e88108a30 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.347+0000 7f3e905d6700 1 -- 172.21.15.121:0/59012055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8819f390 con 0x7f3e88108a30 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.348+0000 7f3e7f7fe700 1 -- 172.21.15.121:0/59012055 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f3e84023020 con 0x7f3e88108a30 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.348+0000 7f3e7f7fe700 1 --2- 172.21.15.121:0/59012055 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3e74039050 0x7f3e7403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.348+0000 7f3e7f7fe700 1 -- 172.21.15.121:0/59012055 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3e8400f050 con 0x7f3e88108a30 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.349+0000 7f3e8db71700 1 --2- 172.21.15.121:0/59012055 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3e74039050 0x7f3e7403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.349+0000 7f3e905d6700 1 -- 172.21.15.121:0/59012055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e88063af0 con 0x7f3e88108a30 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.349+0000 7f3e8db71700 1 --2- 172.21.15.121:0/59012055 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3e74039050 0x7f3e7403b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e78006fd0 tx=0x7f3e78006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.354+0000 7f3e7f7fe700 1 -- 172.21.15.121:0/59012055 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3e8402a020 con 0x7f3e88108a30 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.553+0000 7f3e905d6700 1 -- 172.21.15.121:0/59012055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f3e8819f560 con 0x7f3e88108a30 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.554+0000 7f3e7f7fe700 1 -- 172.21.15.121:0/59012055 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f3e8401b730 con 0x7f3e88108a30 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.556+0000 7f3e905d6700 1 -- 172.21.15.121:0/59012055 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3e74039050 msgr2=0x7f3e7403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.556+0000 7f3e905d6700 1 --2- 172.21.15.121:0/59012055 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3e74039050 0x7f3e7403b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e78006fd0 tx=0x7f3e78006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:48.109 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.556+0000 7f3e905d6700 1 -- 172.21.15.121:0/59012055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 msgr2=0x7f3e8819bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.556+0000 7f3e905d6700 1 --2- 172.21.15.121:0/59012055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8819bed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400b7b0 tx=0x7f3e8400bd50 comp rx=0 tx=0).stop 2024-06-10T00:16:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.556+0000 7f3e905d6700 1 -- 172.21.15.121:0/59012055 shutdown_connections 2024-06-10T00:16:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.556+0000 7f3e905d6700 1 --2- 172.21.15.121:0/59012055 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3e74039050 0x7f3e7403b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.556+0000 7f3e905d6700 1 --2- 172.21.15.121:0/59012055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e88108a30 0x7f3e8819bed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.556+0000 7f3e905d6700 1 -- 172.21.15.121:0/59012055 >> 172.21.15.121:0/59012055 conn(0x7f3e88102460 msgr2=0x7f3e88102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.556+0000 7f3e905d6700 1 -- 172.21.15.121:0/59012055 shutdown_connections 2024-06-10T00:16:48.110 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:47.556+0000 7f3e905d6700 1 -- 172.21.15.121:0/59012055 wait complete. 2024-06-10T00:16:48.112 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-10T00:16:48.112 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout disabled 2024-06-10T00:16:48.120 INFO:teuthology.orchestra.run.smithi121.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-10T00:16:48.120 INFO:teuthology.orchestra.run.smithi121.stdout:systemctl: stdout inactive 2024-06-10T00:16:48.120 INFO:teuthology.orchestra.run.smithi121.stdout:firewalld.service is not enabled 2024-06-10T00:16:48.120 INFO:teuthology.orchestra.run.smithi121.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-10T00:16:48.122 INFO:teuthology.orchestra.run.smithi121.stdout:Ceph Dashboard is now available at: 2024-06-10T00:16:48.122 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:16:48.122 INFO:teuthology.orchestra.run.smithi121.stdout: URL: https://smithi121.front.sepia.ceph.com:8443/ 2024-06-10T00:16:48.122 INFO:teuthology.orchestra.run.smithi121.stdout: User: admin 2024-06-10T00:16:48.122 INFO:teuthology.orchestra.run.smithi121.stdout: Password: myapbaaudt 2024-06-10T00:16:48.123 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:16:48.123 INFO:teuthology.orchestra.run.smithi121.stdout:Saving cluster configuration to /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config directory 2024-06-10T00:16:48.123 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling autotune for osd_memory_target 2024-06-10T00:16:48.322 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:48 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/59012055' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-10T00:16:49.347 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.528+0000 7f632ab00700 1 Processor -- start 2024-06-10T00:16:49.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.528+0000 7f632ab00700 1 -- start start 2024-06-10T00:16:49.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.528+0000 7f632ab00700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632407cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:49.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.528+0000 7f632ab00700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f632407d0a0 con 0x7f632407e660 2024-06-10T00:16:49.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.528+0000 7f632889c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632407cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:49.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.529+0000 7f632889c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632407cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60720/0 (socket says 172.21.15.121:60720) 2024-06-10T00:16:49.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.529+0000 7f632889c700 1 -- 172.21.15.121:0/1141828715 learned_addr learned my addr 172.21.15.121:0/1141828715 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:49.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.529+0000 7f632889c700 1 -- 172.21.15.121:0/1141828715 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632407d220 con 0x7f632407e660 2024-06-10T00:16:49.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.529+0000 7f632889c700 1 --2- 172.21.15.121:0/1141828715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632407cb60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f63200098a0 tx=0x7f6320009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9e93cda1798188 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:49.348 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.529+0000 7f631b7fe700 1 -- 172.21.15.121:0/1141828715 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6320004160 con 0x7f632407e660 2024-06-10T00:16:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.529+0000 7f631b7fe700 1 -- 172.21.15.121:0/1141828715 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6320004300 con 0x7f632407e660 2024-06-10T00:16:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.529+0000 7f631b7fe700 1 -- 172.21.15.121:0/1141828715 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6320011b20 con 0x7f632407e660 2024-06-10T00:16:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.530+0000 7f632ab00700 1 -- 172.21.15.121:0/1141828715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 msgr2=0x7f632407cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.530+0000 7f632ab00700 1 --2- 172.21.15.121:0/1141828715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632407cb60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f63200098a0 tx=0x7f6320009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.530+0000 7f632ab00700 1 -- 172.21.15.121:0/1141828715 shutdown_connections 2024-06-10T00:16:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.530+0000 7f632ab00700 1 --2- 172.21.15.121:0/1141828715 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632407cb60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.530+0000 7f632ab00700 1 -- 172.21.15.121:0/1141828715 >> 172.21.15.121:0/1141828715 conn(0x7f63241035d0 msgr2=0x7f63241059f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.530+0000 7f632ab00700 1 -- 172.21.15.121:0/1141828715 shutdown_connections 2024-06-10T00:16:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.530+0000 7f632ab00700 1 -- 172.21.15.121:0/1141828715 wait complete. 2024-06-10T00:16:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.530+0000 7f632ab00700 1 Processor -- start 2024-06-10T00:16:49.349 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.530+0000 7f632ab00700 1 -- start start 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f632ab00700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632419e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f632ab00700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f632410e590 con 0x7f632407e660 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f632889c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632419e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f632889c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632419e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60722/0 (socket says 172.21.15.121:60722) 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f632889c700 1 -- 172.21.15.121:0/1192892467 learned_addr learned my addr 172.21.15.121:0/1192892467 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f632889c700 1 -- 172.21.15.121:0/1192892467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632419e5f0 con 0x7f632407e660 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f632889c700 1 --2- 172.21.15.121:0/1192892467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632419e0b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f63200117e0 tx=0x7f6320011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f6319ffb700 1 -- 172.21.15.121:0/1192892467 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6320003bf0 con 0x7f632407e660 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f6319ffb700 1 -- 172.21.15.121:0/1192892467 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f632001a910 con 0x7f632407e660 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f632ab00700 1 -- 172.21.15.121:0/1192892467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f632419e880 con 0x7f632407e660 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f6319ffb700 1 -- 172.21.15.121:0/1192892467 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6320024970 con 0x7f632407e660 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.531+0000 7f632ab00700 1 -- 172.21.15.121:0/1192892467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f632419ece0 con 0x7f632407e660 2024-06-10T00:16:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.532+0000 7f6319ffb700 1 -- 172.21.15.121:0/1192892467 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f6320022020 con 0x7f632407e660 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.532+0000 7f632ab00700 1 -- 172.21.15.121:0/1192892467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6324063af0 con 0x7f632407e660 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.532+0000 7f6319ffb700 1 --2- 172.21.15.121:0/1192892467 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f630c039150 0x7f630c03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.532+0000 7f6319ffb700 1 -- 172.21.15.121:0/1192892467 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f632004d590 con 0x7f632407e660 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.533+0000 7f631bfff700 1 --2- 172.21.15.121:0/1192892467 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f630c039150 0x7f630c03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.533+0000 7f631bfff700 1 --2- 172.21.15.121:0/1192892467 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f630c039150 0x7f630c03b610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6314006fd0 tx=0x7f6314006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.537+0000 7f6319ffb700 1 -- 172.21.15.121:0/1192892467 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6320011e20 con 0x7f632407e660 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.735+0000 7f632ab00700 1 -- 172.21.15.121:0/1192892467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f63241a1770 con 0x7f632407e660 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.736+0000 7f6319ffb700 1 -- 172.21.15.121:0/1192892467 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f632000b3e0 con 0x7f632407e660 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.738+0000 7f632ab00700 1 -- 172.21.15.121:0/1192892467 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f630c039150 msgr2=0x7f630c03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.738+0000 7f632ab00700 1 --2- 172.21.15.121:0/1192892467 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f630c039150 0x7f630c03b610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6314006fd0 tx=0x7f6314006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.738+0000 7f632ab00700 1 -- 172.21.15.121:0/1192892467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 msgr2=0x7f632419e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.738+0000 7f632ab00700 1 --2- 172.21.15.121:0/1192892467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632419e0b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f63200117e0 tx=0x7f6320011c10 comp rx=0 tx=0).stop 2024-06-10T00:16:49.351 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.738+0000 7f632ab00700 1 -- 172.21.15.121:0/1192892467 shutdown_connections 2024-06-10T00:16:49.352 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.738+0000 7f632ab00700 1 --2- 172.21.15.121:0/1192892467 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f630c039150 0x7f630c03b610 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:49.352 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.738+0000 7f632ab00700 1 --2- 172.21.15.121:0/1192892467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f632407e660 0x7f632419e0b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:49.352 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.738+0000 7f632ab00700 1 -- 172.21.15.121:0/1192892467 >> 172.21.15.121:0/1192892467 conn(0x7f63241035d0 msgr2=0x7f6324104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:49.352 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.738+0000 7f632ab00700 1 -- 172.21.15.121:0/1192892467 shutdown_connections 2024-06-10T00:16:49.352 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:48.738+0000 7f632ab00700 1 -- 172.21.15.121:0/1192892467 wait complete. 2024-06-10T00:16:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:49 smithi121 ceph-mon[32378]: mgrmap e13: x(active, since 6s) 2024-06-10T00:16:50.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.740+0000 7fc4d03c8700 1 Processor -- start 2024-06-10T00:16:50.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.741+0000 7fc4d03c8700 1 -- start start 2024-06-10T00:16:50.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.741+0000 7fc4d03c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c81083e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:50.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.741+0000 7fc4d03c8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c8108920 con 0x7fc4c8107fa0 2024-06-10T00:16:50.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.741+0000 7fc4ce164700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c81083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:50.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.741+0000 7fc4ce164700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c81083e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49512/0 (socket says 172.21.15.121:49512) 2024-06-10T00:16:50.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.741+0000 7fc4ce164700 1 -- 172.21.15.121:0/3387053211 learned_addr learned my addr 172.21.15.121:0/3387053211 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:50.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.741+0000 7fc4ce164700 1 -- 172.21.15.121:0/3387053211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c81090e0 con 0x7fc4c8107fa0 2024-06-10T00:16:50.653 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.741+0000 7fc4ce164700 1 --2- 172.21.15.121:0/3387053211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c81083e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40098a0 tx=0x7fc4c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4313ab03f433c68f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.742+0000 7fc4cd162700 1 -- 172.21.15.121:0/3387053211 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4c4004160 con 0x7fc4c8107fa0 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.742+0000 7fc4cd162700 1 -- 172.21.15.121:0/3387053211 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4c4004300 con 0x7fc4c8107fa0 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.742+0000 7fc4cd162700 1 -- 172.21.15.121:0/3387053211 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4c4011b20 con 0x7fc4c8107fa0 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.742+0000 7fc4d03c8700 1 -- 172.21.15.121:0/3387053211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 msgr2=0x7fc4c81083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.742+0000 7fc4d03c8700 1 --2- 172.21.15.121:0/3387053211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c81083e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40098a0 tx=0x7fc4c4009bb0 comp rx=0 tx=0).stop 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.742+0000 7fc4d03c8700 1 -- 172.21.15.121:0/3387053211 shutdown_connections 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.742+0000 7fc4d03c8700 1 --2- 172.21.15.121:0/3387053211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c81083e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.742+0000 7fc4d03c8700 1 -- 172.21.15.121:0/3387053211 >> 172.21.15.121:0/3387053211 conn(0x7fc4c8103570 msgr2=0x7fc4c81059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.742+0000 7fc4d03c8700 1 -- 172.21.15.121:0/3387053211 shutdown_connections 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.742+0000 7fc4d03c8700 1 -- 172.21.15.121:0/3387053211 wait complete. 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.743+0000 7fc4d03c8700 1 Processor -- start 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.743+0000 7fc4d03c8700 1 -- start start 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.743+0000 7fc4d03c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c819bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:50.654 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.743+0000 7fc4d03c8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c810c3a0 con 0x7fc4c8107fa0 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.743+0000 7fc4ce164700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c819bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.743+0000 7fc4ce164700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c819bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49526/0 (socket says 172.21.15.121:49526) 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.743+0000 7fc4ce164700 1 -- 172.21.15.121:0/1025892281 learned_addr learned my addr 172.21.15.121:0/1025892281 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.744+0000 7fc4ce164700 1 -- 172.21.15.121:0/1025892281 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c819c420 con 0x7fc4c8107fa0 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.744+0000 7fc4ce164700 1 --2- 172.21.15.121:0/1025892281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c819bee0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40117e0 tx=0x7fc4c4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.744+0000 7fc4bf7fe700 1 -- 172.21.15.121:0/1025892281 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4c4003bf0 con 0x7fc4c8107fa0 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.744+0000 7fc4bf7fe700 1 -- 172.21.15.121:0/1025892281 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4c401a910 con 0x7fc4c8107fa0 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.744+0000 7fc4d03c8700 1 -- 172.21.15.121:0/1025892281 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4c819c6b0 con 0x7fc4c8107fa0 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.744+0000 7fc4bf7fe700 1 -- 172.21.15.121:0/1025892281 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc4c40249a0 con 0x7fc4c8107fa0 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.744+0000 7fc4d03c8700 1 -- 172.21.15.121:0/1025892281 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4c819cb10 con 0x7fc4c8107fa0 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.745+0000 7fc4bf7fe700 1 -- 172.21.15.121:0/1025892281 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fc4c4022020 con 0x7fc4c8107fa0 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.745+0000 7fc4d03c8700 1 -- 172.21.15.121:0/1025892281 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4c8063af0 con 0x7fc4c8107fa0 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.745+0000 7fc4bf7fe700 1 --2- 172.21.15.121:0/1025892281 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc4b40390f0 0x7fc4b403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:50.655 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.745+0000 7fc4bf7fe700 1 -- 172.21.15.121:0/1025892281 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc4c404d6d0 con 0x7fc4c8107fa0 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.745+0000 7fc4cd963700 1 --2- 172.21.15.121:0/1025892281 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc4b40390f0 0x7fc4b403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.745+0000 7fc4cd963700 1 --2- 172.21.15.121:0/1025892281 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc4b40390f0 0x7fc4b403b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8006fd0 tx=0x7fc4b8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:49.750+0000 7fc4bf7fe700 1 -- 172.21.15.121:0/1025892281 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc4c4011e20 con 0x7fc4c8107fa0 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.052+0000 7fc4d03c8700 1 -- 172.21.15.121:0/1025892281 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fc4c819f5a0 con 0x7fc4c8107fa0 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.055+0000 7fc4bf7fe700 1 -- 172.21.15.121:0/1025892281 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fc4c400ba00 con 0x7fc4c8107fa0 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.057+0000 7fc4d03c8700 1 -- 172.21.15.121:0/1025892281 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc4b40390f0 msgr2=0x7fc4b403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.057+0000 7fc4d03c8700 1 --2- 172.21.15.121:0/1025892281 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc4b40390f0 0x7fc4b403b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8006fd0 tx=0x7fc4b8006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.057+0000 7fc4d03c8700 1 -- 172.21.15.121:0/1025892281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 msgr2=0x7fc4c819bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.057+0000 7fc4d03c8700 1 --2- 172.21.15.121:0/1025892281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c819bee0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40117e0 tx=0x7fc4c4011c10 comp rx=0 tx=0).stop 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.057+0000 7fc4d03c8700 1 -- 172.21.15.121:0/1025892281 shutdown_connections 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.057+0000 7fc4d03c8700 1 --2- 172.21.15.121:0/1025892281 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc4b40390f0 0x7fc4b403b5b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.057+0000 7fc4d03c8700 1 --2- 172.21.15.121:0/1025892281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4c8107fa0 0x7fc4c819bee0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:50.656 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.057+0000 7fc4d03c8700 1 -- 172.21.15.121:0/1025892281 >> 172.21.15.121:0/1025892281 conn(0x7fc4c8103570 msgr2=0x7fc4c8104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.057+0000 7fc4d03c8700 1 -- 172.21.15.121:0/1025892281 shutdown_connections 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr 2024-06-10T00:16:50.057+0000 7fc4d03c8700 1 -- 172.21.15.121:0/1025892281 wait complete. 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout:Or, if you are only running a single cluster on this host: 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:16:50.657 INFO:teuthology.orchestra.run.smithi121.stdout: ceph telemetry on 2024-06-10T00:16:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:16:50.658 INFO:teuthology.orchestra.run.smithi121.stdout:For more information see: 2024-06-10T00:16:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:16:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-06-10T00:16:50.658 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:16:50.658 INFO:teuthology.orchestra.run.smithi121.stdout:Bootstrap complete. 2024-06-10T00:16:50.691 INFO:tasks.cephadm:Fetching config... 2024-06-10T00:16:50.691 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:16:50.691 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-10T00:16:50.707 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-10T00:16:50.708 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:16:50.708 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-10T00:16:50.763 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-10T00:16:50.763 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:16:50.763 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/keyring of=/dev/stdout 2024-06-10T00:16:50.827 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-10T00:16:50.828 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:16:50.828 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-10T00:16:50.883 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-10T00:16:50.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXqsDLrmdVr/Qt6Pb3RHwIgCkU0CP3j8ySx92StlGDGyCcmdKDt1783xfFYWzKniBtxNe+5TebyJlP1227J/1x7dzojY2n3vS9dEIME3PI4oBZHjL3EDznLavz6GyjTPH2Y8HyL9h93qRWyo5IkVBNMNmungpF9SV+fvtKkZ+2+8673pcm481c7IfAD6Oh+LXZ0/2vlqfYKZCQMOMYli/8xK0wEBrivauzV/QvIAAyo5yi8mNnRP1kwfTbv//VeEhfGcZlkm0o/7ZV7fih4WH+N6Uyv13mH+9W6zKYybXC0WpxoDKBu+072dy1UM0kzNFw+rV/gYeTPJy+fxukYN3IfriFKMNjj9FwOmY7YlJ/JbUU7/khU+OCEumSQ1MKGndu2MkV3yVOAgKQ05pI5LOOOim4dxs6mncEoPDwThMS4eDf/cF1KJFa/Gq7otAKvlc+uMbvTgvZVm0SC1MX/mZYeFdbwDJ4w6426ykIOe0BoPfN4151rDAAf6q3oVIAfAM= ceph-6c432244-26be-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-10T00:16:50.958 INFO:teuthology.orchestra.run.smithi121.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXqsDLrmdVr/Qt6Pb3RHwIgCkU0CP3j8ySx92StlGDGyCcmdKDt1783xfFYWzKniBtxNe+5TebyJlP1227J/1x7dzojY2n3vS9dEIME3PI4oBZHjL3EDznLavz6GyjTPH2Y8HyL9h93qRWyo5IkVBNMNmungpF9SV+fvtKkZ+2+8673pcm481c7IfAD6Oh+LXZ0/2vlqfYKZCQMOMYli/8xK0wEBrivauzV/QvIAAyo5yi8mNnRP1kwfTbv//VeEhfGcZlkm0o/7ZV7fih4WH+N6Uyv13mH+9W6zKYybXC0WpxoDKBu+072dy1UM0kzNFw+rV/gYeTPJy+fxukYN3IfriFKMNjj9FwOmY7YlJ/JbUU7/khU+OCEumSQ1MKGndu2MkV3yVOAgKQ05pI5LOOOim4dxs6mncEoPDwThMS4eDf/cF1KJFa/Gq7otAKvlc+uMbvTgvZVm0SC1MX/mZYeFdbwDJ4w6426ykIOe0BoPfN4151rDAAf6q3oVIAfAM= ceph-6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:16:50.969 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXqsDLrmdVr/Qt6Pb3RHwIgCkU0CP3j8ySx92StlGDGyCcmdKDt1783xfFYWzKniBtxNe+5TebyJlP1227J/1x7dzojY2n3vS9dEIME3PI4oBZHjL3EDznLavz6GyjTPH2Y8HyL9h93qRWyo5IkVBNMNmungpF9SV+fvtKkZ+2+8673pcm481c7IfAD6Oh+LXZ0/2vlqfYKZCQMOMYli/8xK0wEBrivauzV/QvIAAyo5yi8mNnRP1kwfTbv//VeEhfGcZlkm0o/7ZV7fih4WH+N6Uyv13mH+9W6zKYybXC0WpxoDKBu+072dy1UM0kzNFw+rV/gYeTPJy+fxukYN3IfriFKMNjj9FwOmY7YlJ/JbUU7/khU+OCEumSQ1MKGndu2MkV3yVOAgKQ05pI5LOOOim4dxs6mncEoPDwThMS4eDf/cF1KJFa/Gq7otAKvlc+uMbvTgvZVm0SC1MX/mZYeFdbwDJ4w6426ykIOe0BoPfN4151rDAAf6q3oVIAfAM= ceph-6c432244-26be-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-10T00:16:51.008 INFO:teuthology.orchestra.run.smithi135.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXqsDLrmdVr/Qt6Pb3RHwIgCkU0CP3j8ySx92StlGDGyCcmdKDt1783xfFYWzKniBtxNe+5TebyJlP1227J/1x7dzojY2n3vS9dEIME3PI4oBZHjL3EDznLavz6GyjTPH2Y8HyL9h93qRWyo5IkVBNMNmungpF9SV+fvtKkZ+2+8673pcm481c7IfAD6Oh+LXZ0/2vlqfYKZCQMOMYli/8xK0wEBrivauzV/QvIAAyo5yi8mNnRP1kwfTbv//VeEhfGcZlkm0o/7ZV7fih4WH+N6Uyv13mH+9W6zKYybXC0WpxoDKBu+072dy1UM0kzNFw+rV/gYeTPJy+fxukYN3IfriFKMNjj9FwOmY7YlJ/JbUU7/khU+OCEumSQ1MKGndu2MkV3yVOAgKQ05pI5LOOOim4dxs6mncEoPDwThMS4eDf/cF1KJFa/Gq7otAKvlc+uMbvTgvZVm0SC1MX/mZYeFdbwDJ4w6426ykIOe0BoPfN4151rDAAf6q3oVIAfAM= ceph-6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:16:51.020 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXqsDLrmdVr/Qt6Pb3RHwIgCkU0CP3j8ySx92StlGDGyCcmdKDt1783xfFYWzKniBtxNe+5TebyJlP1227J/1x7dzojY2n3vS9dEIME3PI4oBZHjL3EDznLavz6GyjTPH2Y8HyL9h93qRWyo5IkVBNMNmungpF9SV+fvtKkZ+2+8673pcm481c7IfAD6Oh+LXZ0/2vlqfYKZCQMOMYli/8xK0wEBrivauzV/QvIAAyo5yi8mNnRP1kwfTbv//VeEhfGcZlkm0o/7ZV7fih4WH+N6Uyv13mH+9W6zKYybXC0WpxoDKBu+072dy1UM0kzNFw+rV/gYeTPJy+fxukYN3IfriFKMNjj9FwOmY7YlJ/JbUU7/khU+OCEumSQ1MKGndu2MkV3yVOAgKQ05pI5LOOOim4dxs6mncEoPDwThMS4eDf/cF1KJFa/Gq7otAKvlc+uMbvTgvZVm0SC1MX/mZYeFdbwDJ4w6426ykIOe0BoPfN4151rDAAf6q3oVIAfAM= ceph-6c432244-26be-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-10T00:16:51.060 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXqsDLrmdVr/Qt6Pb3RHwIgCkU0CP3j8ySx92StlGDGyCcmdKDt1783xfFYWzKniBtxNe+5TebyJlP1227J/1x7dzojY2n3vS9dEIME3PI4oBZHjL3EDznLavz6GyjTPH2Y8HyL9h93qRWyo5IkVBNMNmungpF9SV+fvtKkZ+2+8673pcm481c7IfAD6Oh+LXZ0/2vlqfYKZCQMOMYli/8xK0wEBrivauzV/QvIAAyo5yi8mNnRP1kwfTbv//VeEhfGcZlkm0o/7ZV7fih4WH+N6Uyv13mH+9W6zKYybXC0WpxoDKBu+072dy1UM0kzNFw+rV/gYeTPJy+fxukYN3IfriFKMNjj9FwOmY7YlJ/JbUU7/khU+OCEumSQ1MKGndu2MkV3yVOAgKQ05pI5LOOOim4dxs6mncEoPDwThMS4eDf/cF1KJFa/Gq7otAKvlc+uMbvTgvZVm0SC1MX/mZYeFdbwDJ4w6426ykIOe0BoPfN4151rDAAf6q3oVIAfAM= ceph-6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:16:51.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-10T00:16:51.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:51 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1025892281' entity='client.admin' 2024-06-10T00:16:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:16:53.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:53 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:53.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:53 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:53.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:53 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi121", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:16:53.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:53 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:53.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:53 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:16:53.817 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:53 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:54.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.212+0000 7f1d9564a700 1 -- 172.21.15.121:0/269522989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d901029f0 msgr2=0x7f1d90104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:54.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.212+0000 7f1d9564a700 1 --2- 172.21.15.121:0/269522989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d901029f0 0x7f1d90104de0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1d80009910 tx=0x7f1d80009c20 comp rx=0 tx=0).stop 2024-06-10T00:16:54.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.213+0000 7f1d9564a700 1 -- 172.21.15.121:0/269522989 shutdown_connections 2024-06-10T00:16:54.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.213+0000 7f1d9564a700 1 --2- 172.21.15.121:0/269522989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d901029f0 0x7f1d90104de0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:54.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.213+0000 7f1d9564a700 1 -- 172.21.15.121:0/269522989 >> 172.21.15.121:0/269522989 conn(0x7f1d900fc400 msgr2=0x7f1d900fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:54.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.213+0000 7f1d9564a700 1 -- 172.21.15.121:0/269522989 shutdown_connections 2024-06-10T00:16:54.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.213+0000 7f1d9564a700 1 -- 172.21.15.121:0/269522989 wait complete. 2024-06-10T00:16:54.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.213+0000 7f1d9564a700 1 Processor -- start 2024-06-10T00:16:54.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.213+0000 7f1d9564a700 1 -- start start 2024-06-10T00:16:54.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.213+0000 7f1d9564a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d901029f0 0x7f1d90197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:54.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.213+0000 7f1d9564a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1d90107f20 con 0x7f1d901029f0 2024-06-10T00:16:54.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.214+0000 7f1d8effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d901029f0 0x7f1d90197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:54.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.214+0000 7f1d8effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d901029f0 0x7f1d90197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49542/0 (socket says 172.21.15.121:49542) 2024-06-10T00:16:54.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.214+0000 7f1d8effd700 1 -- 172.21.15.121:0/2583564561 learned_addr learned my addr 172.21.15.121:0/2583564561 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:54.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.214+0000 7f1d8effd700 1 -- 172.21.15.121:0/2583564561 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d90197a70 con 0x7f1d901029f0 2024-06-10T00:16:54.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.214+0000 7f1d8effd700 1 --2- 172.21.15.121:0/2583564561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d901029f0 0x7f1d90197530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1d80004fa0 tx=0x7f1d80005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:54.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.214+0000 7f1d77fff700 1 -- 172.21.15.121:0/2583564561 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d80027070 con 0x7f1d901029f0 2024-06-10T00:16:54.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.214+0000 7f1d77fff700 1 -- 172.21.15.121:0/2583564561 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d8000be40 con 0x7f1d901029f0 2024-06-10T00:16:54.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.214+0000 7f1d9564a700 1 -- 172.21.15.121:0/2583564561 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d90197d60 con 0x7f1d901029f0 2024-06-10T00:16:54.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.214+0000 7f1d77fff700 1 -- 172.21.15.121:0/2583564561 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d8000f460 con 0x7f1d901029f0 2024-06-10T00:16:54.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.214+0000 7f1d9564a700 1 -- 172.21.15.121:0/2583564561 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d9019a9f0 con 0x7f1d901029f0 2024-06-10T00:16:54.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.215+0000 7f1d77fff700 1 -- 172.21.15.121:0/2583564561 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f1d8002a070 con 0x7f1d901029f0 2024-06-10T00:16:54.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.215+0000 7f1d9564a700 1 -- 172.21.15.121:0/2583564561 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d901086a0 con 0x7f1d901029f0 2024-06-10T00:16:54.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.215+0000 7f1d77fff700 1 --2- 172.21.15.121:0/2583564561 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1d780390f0 0x7f1d7803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:54.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.215+0000 7f1d77fff700 1 -- 172.21.15.121:0/2583564561 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1d8001e030 con 0x7f1d901029f0 2024-06-10T00:16:54.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.215+0000 7f1d8e7fc700 1 --2- 172.21.15.121:0/2583564561 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1d780390f0 0x7f1d7803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:54.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.216+0000 7f1d8e7fc700 1 --2- 172.21.15.121:0/2583564561 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1d780390f0 0x7f1d7803b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1d84006fd0 tx=0x7f1d84006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:54.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.220+0000 7f1d77fff700 1 -- 172.21.15.121:0/2583564561 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1d80013070 con 0x7f1d901029f0 2024-06-10T00:16:54.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.416+0000 7f1d9564a700 1 -- 172.21.15.121:0/2583564561 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f1d9002de20 con 0x7f1d901029f0 2024-06-10T00:16:54.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.419+0000 7f1d77fff700 1 -- 172.21.15.121:0/2583564561 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f1d800292c0 con 0x7f1d901029f0 2024-06-10T00:16:54.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.421+0000 7f1d9564a700 1 -- 172.21.15.121:0/2583564561 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1d780390f0 msgr2=0x7f1d7803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:54.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.421+0000 7f1d9564a700 1 --2- 172.21.15.121:0/2583564561 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1d780390f0 0x7f1d7803b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1d84006fd0 tx=0x7f1d84006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:54.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.421+0000 7f1d9564a700 1 -- 172.21.15.121:0/2583564561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d901029f0 msgr2=0x7f1d90197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:54.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.421+0000 7f1d9564a700 1 --2- 172.21.15.121:0/2583564561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d901029f0 0x7f1d90197530 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1d80004fa0 tx=0x7f1d80005080 comp rx=0 tx=0).stop 2024-06-10T00:16:54.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.421+0000 7f1d9564a700 1 -- 172.21.15.121:0/2583564561 shutdown_connections 2024-06-10T00:16:54.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.421+0000 7f1d9564a700 1 --2- 172.21.15.121:0/2583564561 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1d780390f0 0x7f1d7803b5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:54.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.421+0000 7f1d9564a700 1 --2- 172.21.15.121:0/2583564561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d901029f0 0x7f1d90197530 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:54.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.421+0000 7f1d9564a700 1 -- 172.21.15.121:0/2583564561 >> 172.21.15.121:0/2583564561 conn(0x7f1d900fc400 msgr2=0x7f1d900fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:54.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.421+0000 7f1d9564a700 1 -- 172.21.15.121:0/2583564561 shutdown_connections 2024-06-10T00:16:54.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:54.421+0000 7f1d9564a700 1 -- 172.21.15.121:0/2583564561 wait complete. 2024-06-10T00:16:55.236 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-10T00:16:55.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-10T00:16:55.475 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:16:55.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:55 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2583564561' entity='client.admin' 2024-06-10T00:16:55.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:16:55.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:56.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.853+0000 7f69b4b73700 1 -- 172.21.15.121:0/570515453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69b0101e90 msgr2=0x7f69b01022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:56.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.853+0000 7f69b4b73700 1 --2- 172.21.15.121:0/570515453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69b0101e90 0x7f69b01022b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f69a0009910 tx=0x7f69a0009c20 comp rx=0 tx=0).stop 2024-06-10T00:16:56.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.853+0000 7f69b4b73700 1 -- 172.21.15.121:0/570515453 shutdown_connections 2024-06-10T00:16:56.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.853+0000 7f69b4b73700 1 --2- 172.21.15.121:0/570515453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69b0101e90 0x7f69b01022b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:56.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.853+0000 7f69b4b73700 1 -- 172.21.15.121:0/570515453 >> 172.21.15.121:0/570515453 conn(0x7f69b00fd480 msgr2=0x7f69b00ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:56.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.853+0000 7f69b4b73700 1 -- 172.21.15.121:0/570515453 shutdown_connections 2024-06-10T00:16:56.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.853+0000 7f69b4b73700 1 -- 172.21.15.121:0/570515453 wait complete. 2024-06-10T00:16:56.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.854+0000 7f69b4b73700 1 Processor -- start 2024-06-10T00:16:56.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.854+0000 7f69b4b73700 1 -- start start 2024-06-10T00:16:56.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.854+0000 7f69b4b73700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69b0101e90 0x7f69b0197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:56.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.854+0000 7f69b4b73700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f69b0107ee0 con 0x7f69b0101e90 2024-06-10T00:16:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.854+0000 7f69ae59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69b0101e90 0x7f69b0197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.854+0000 7f69ae59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69b0101e90 0x7f69b0197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49554/0 (socket says 172.21.15.121:49554) 2024-06-10T00:16:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.854+0000 7f69ae59c700 1 -- 172.21.15.121:0/564777534 learned_addr learned my addr 172.21.15.121:0/564777534 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.854+0000 7f69ae59c700 1 -- 172.21.15.121:0/564777534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b0197b50 con 0x7f69b0101e90 2024-06-10T00:16:56.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.855+0000 7f69ae59c700 1 --2- 172.21.15.121:0/564777534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69b0101e90 0x7f69b0197610 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f69a0006010 tx=0x7f69a0005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.855+0000 7f69a7fff700 1 -- 172.21.15.121:0/564777534 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69a0027070 con 0x7f69b0101e90 2024-06-10T00:16:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.855+0000 7f69b4b73700 1 -- 172.21.15.121:0/564777534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69b0197de0 con 0x7f69b0101e90 2024-06-10T00:16:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.855+0000 7f69a7fff700 1 -- 172.21.15.121:0/564777534 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69a000be40 con 0x7f69b0101e90 2024-06-10T00:16:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.855+0000 7f69a7fff700 1 -- 172.21.15.121:0/564777534 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f69a000f460 con 0x7f69b0101e90 2024-06-10T00:16:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.855+0000 7f69b4b73700 1 -- 172.21.15.121:0/564777534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69b019aad0 con 0x7f69b0101e90 2024-06-10T00:16:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.856+0000 7f69a7fff700 1 -- 172.21.15.121:0/564777534 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f69a002a070 con 0x7f69b0101e90 2024-06-10T00:16:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.856+0000 7f69b4b73700 1 -- 172.21.15.121:0/564777534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69b0108660 con 0x7f69b0101e90 2024-06-10T00:16:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.856+0000 7f69a7fff700 1 --2- 172.21.15.121:0/564777534 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f699c0390f0 0x7f699c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.856+0000 7f69a7fff700 1 -- 172.21.15.121:0/564777534 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f69a001e030 con 0x7f69b0101e90 2024-06-10T00:16:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.856+0000 7f69a5d9b700 1 --2- 172.21.15.121:0/564777534 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f699c0390f0 0x7f699c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:56.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.857+0000 7f69a5d9b700 1 --2- 172.21.15.121:0/564777534 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f699c0390f0 0x7f699c03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6998006fd0 tx=0x7f6998006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:56.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:56.861+0000 7f69a7fff700 1 -- 172.21.15.121:0/564777534 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f69a0013070 con 0x7f69b0101e90 2024-06-10T00:16:57.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.060+0000 7f69b4b73700 1 -- 172.21.15.121:0/564777534 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f69b002ddb0 con 0x7f699c0390f0 2024-06-10T00:16:57.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.063+0000 7f69a7fff700 1 -- 172.21.15.121:0/564777534 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f69b002ddb0 con 0x7f699c0390f0 2024-06-10T00:16:57.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.066+0000 7f69b4b73700 1 -- 172.21.15.121:0/564777534 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f699c0390f0 msgr2=0x7f699c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:57.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.066+0000 7f69b4b73700 1 --2- 172.21.15.121:0/564777534 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f699c0390f0 0x7f699c03b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6998006fd0 tx=0x7f6998006e40 comp rx=0 tx=0).stop 2024-06-10T00:16:57.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.066+0000 7f69b4b73700 1 -- 172.21.15.121:0/564777534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69b0101e90 msgr2=0x7f69b0197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:57.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.066+0000 7f69b4b73700 1 --2- 172.21.15.121:0/564777534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69b0101e90 0x7f69b0197610 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f69a0006010 tx=0x7f69a0005050 comp rx=0 tx=0).stop 2024-06-10T00:16:57.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.066+0000 7f69b4b73700 1 -- 172.21.15.121:0/564777534 shutdown_connections 2024-06-10T00:16:57.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.066+0000 7f69b4b73700 1 --2- 172.21.15.121:0/564777534 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f699c0390f0 0x7f699c03b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:57.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.066+0000 7f69b4b73700 1 --2- 172.21.15.121:0/564777534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69b0101e90 0x7f69b0197610 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:57.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.066+0000 7f69b4b73700 1 -- 172.21.15.121:0/564777534 >> 172.21.15.121:0/564777534 conn(0x7f69b00fd480 msgr2=0x7f69b00fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:57.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.066+0000 7f69b4b73700 1 -- 172.21.15.121:0/564777534 shutdown_connections 2024-06-10T00:16:57.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:57.066+0000 7f69b4b73700 1 -- 172.21.15.121:0/564777534 wait complete. 2024-06-10T00:16:57.767 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi135 2024-06-10T00:16:57.767 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-06-10T00:16:57.767 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/etc/ceph/ceph.conf 2024-06-10T00:16:57.787 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-06-10T00:16:57.787 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:16:57.845 INFO:tasks.cephadm:Adding host smithi135 to orchestrator... 2024-06-10T00:16:57.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch host add smithi135 2024-06-10T00:16:58.086 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:16:58.398 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:58 smithi121 ceph-mon[32378]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:16:58.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:58 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:58.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:58 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:16:58.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:58 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:16:58.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:58 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:16:58.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:58 smithi121 ceph-mon[32378]: Updating smithi121:/etc/ceph/ceph.conf 2024-06-10T00:16:58.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:58 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:58.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:58 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:58.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:58 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:16:59.370 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:59 smithi121 ceph-mon[32378]: Updating smithi121:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:16:59.371 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:59 smithi121 ceph-mon[32378]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:16:59.371 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:16:59 smithi121 ceph-mon[32378]: Updating smithi121:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:16:59.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.584+0000 7f67a65f9700 1 -- 172.21.15.121:0/1369735056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67a0105d50 msgr2=0x7f67a0106130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:16:59.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.584+0000 7f67a65f9700 1 --2- 172.21.15.121:0/1369735056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67a0105d50 0x7f67a0106130 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6790009910 tx=0x7f6790009c20 comp rx=0 tx=0).stop 2024-06-10T00:16:59.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.584+0000 7f67a65f9700 1 -- 172.21.15.121:0/1369735056 shutdown_connections 2024-06-10T00:16:59.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.584+0000 7f67a65f9700 1 --2- 172.21.15.121:0/1369735056 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67a0105d50 0x7f67a0106130 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:16:59.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.584+0000 7f67a65f9700 1 -- 172.21.15.121:0/1369735056 >> 172.21.15.121:0/1369735056 conn(0x7f67a0077480 msgr2=0x7f67a0077890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:16:59.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.584+0000 7f67a65f9700 1 -- 172.21.15.121:0/1369735056 shutdown_connections 2024-06-10T00:16:59.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.584+0000 7f67a65f9700 1 -- 172.21.15.121:0/1369735056 wait complete. 2024-06-10T00:16:59.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.585+0000 7f67a65f9700 1 Processor -- start 2024-06-10T00:16:59.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.585+0000 7f67a65f9700 1 -- start start 2024-06-10T00:16:59.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.585+0000 7f67a65f9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67a0105d50 0x7f67a0106b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:59.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.585+0000 7f67a65f9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f67a010b4d0 con 0x7f67a0105d50 2024-06-10T00:16:59.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.585+0000 7f679ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67a0105d50 0x7f67a0106b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:59.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.585+0000 7f679ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67a0105d50 0x7f67a0106b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60420/0 (socket says 172.21.15.121:60420) 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.585+0000 7f679ffff700 1 -- 172.21.15.121:0/1572467783 learned_addr learned my addr 172.21.15.121:0/1572467783 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.586+0000 7f679ffff700 1 -- 172.21.15.121:0/1572467783 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a0108a80 con 0x7f67a0105d50 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.586+0000 7f679ffff700 1 --2- 172.21.15.121:0/1572467783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67a0105d50 0x7f67a0106b90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6790005850 tx=0x7f6790005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.586+0000 7f679d7fa700 1 -- 172.21.15.121:0/1572467783 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6790026070 con 0x7f67a0105d50 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.586+0000 7f67a65f9700 1 -- 172.21.15.121:0/1572467783 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a01070d0 con 0x7f67a0105d50 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.586+0000 7f679d7fa700 1 -- 172.21.15.121:0/1572467783 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f679000bc30 con 0x7f67a0105d50 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.586+0000 7f679d7fa700 1 -- 172.21.15.121:0/1572467783 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6790003a20 con 0x7f67a0105d50 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.586+0000 7f67a65f9700 1 -- 172.21.15.121:0/1572467783 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a0107590 con 0x7f67a0105d50 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.587+0000 7f679d7fa700 1 -- 172.21.15.121:0/1572467783 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f6790029070 con 0x7f67a0105d50 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.587+0000 7f67a65f9700 1 -- 172.21.15.121:0/1572467783 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a01025a0 con 0x7f67a0105d50 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.587+0000 7f679d7fa700 1 --2- 172.21.15.121:0/1572467783 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67880390f0 0x7f678803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.587+0000 7f679d7fa700 1 -- 172.21.15.121:0/1572467783 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f679001e070 con 0x7f67a0105d50 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.587+0000 7f679f7fe700 1 --2- 172.21.15.121:0/1572467783 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67880390f0 0x7f678803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:16:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.588+0000 7f679f7fe700 1 --2- 172.21.15.121:0/1572467783 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67880390f0 0x7f678803b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6794006fd0 tx=0x7f6794006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:16:59.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.592+0000 7f679d7fa700 1 -- 172.21.15.121:0/1572467783 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6790013070 con 0x7f67a0105d50 2024-06-10T00:16:59.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:16:59.792+0000 7f67a65f9700 1 -- 172.21.15.121:0/1572467783 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi135", "target": ["mon-mgr", ""]}) -- 0x7f67a01080c0 con 0x7f67880390f0 2024-06-10T00:17:00.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:00 smithi121 ceph-mon[32378]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi135", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:17:01.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:01 smithi121 ceph-mon[32378]: Deploying cephadm binary to smithi135 2024-06-10T00:17:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:03 smithi121 ceph-mon[32378]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:03.869 INFO:teuthology.orchestra.run.smithi121.stdout:Added host 'smithi135' with addr '172.21.15.135' 2024-06-10T00:17:03.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:03.869+0000 7f679d7fa700 1 -- 172.21.15.121:0/1572467783 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f67a01080c0 con 0x7f67880390f0 2024-06-10T00:17:03.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:03.871+0000 7f67a65f9700 1 -- 172.21.15.121:0/1572467783 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67880390f0 msgr2=0x7f678803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:03.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:03.871+0000 7f67a65f9700 1 --2- 172.21.15.121:0/1572467783 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67880390f0 0x7f678803b5b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6794006fd0 tx=0x7f6794006e40 comp rx=0 tx=0).stop 2024-06-10T00:17:03.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:03.871+0000 7f67a65f9700 1 -- 172.21.15.121:0/1572467783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67a0105d50 msgr2=0x7f67a0106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:03.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:03.871+0000 7f67a65f9700 1 --2- 172.21.15.121:0/1572467783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67a0105d50 0x7f67a0106b90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6790005850 tx=0x7f6790005030 comp rx=0 tx=0).stop 2024-06-10T00:17:03.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:03.871+0000 7f67a65f9700 1 -- 172.21.15.121:0/1572467783 shutdown_connections 2024-06-10T00:17:03.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:03.872+0000 7f67a65f9700 1 --2- 172.21.15.121:0/1572467783 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67880390f0 0x7f678803b5b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:03.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:03.872+0000 7f67a65f9700 1 --2- 172.21.15.121:0/1572467783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67a0105d50 0x7f67a0106b90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:03.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:03.872+0000 7f67a65f9700 1 -- 172.21.15.121:0/1572467783 >> 172.21.15.121:0/1572467783 conn(0x7f67a0077480 msgr2=0x7f67a00ffa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:03.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:03.872+0000 7f67a65f9700 1 -- 172.21.15.121:0/1572467783 shutdown_connections 2024-06-10T00:17:03.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:03.872+0000 7f67a65f9700 1 -- 172.21.15.121:0/1572467783 wait complete. 2024-06-10T00:17:04.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-06-10T00:17:04.683 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:17:05.102 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:04 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:05.102 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:04 smithi121 ceph-mon[32378]: Added host smithi135 2024-06-10T00:17:05.102 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:04 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:05.102 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:04 smithi121 ceph-mon[32378]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:05.102 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:04 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:05.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:06.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.176+0000 7fe5c8460700 1 -- 172.21.15.121:0/2137177617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01020d0 msgr2=0x7fe5c01024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:06.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.176+0000 7fe5c8460700 1 --2- 172.21.15.121:0/2137177617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01020d0 0x7fe5c01024b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00b0a0 tx=0x7fe5bc00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:17:06.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.176+0000 7fe5c8460700 1 -- 172.21.15.121:0/2137177617 shutdown_connections 2024-06-10T00:17:06.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.176+0000 7fe5c8460700 1 --2- 172.21.15.121:0/2137177617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01020d0 0x7fe5c01024b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:06.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.176+0000 7fe5c8460700 1 -- 172.21.15.121:0/2137177617 >> 172.21.15.121:0/2137177617 conn(0x7fe5c00fd980 msgr2=0x7fe5c00ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:06.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.176+0000 7fe5c8460700 1 -- 172.21.15.121:0/2137177617 shutdown_connections 2024-06-10T00:17:06.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.176+0000 7fe5c8460700 1 -- 172.21.15.121:0/2137177617 wait complete. 2024-06-10T00:17:06.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.177+0000 7fe5c8460700 1 Processor -- start 2024-06-10T00:17:06.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.177+0000 7fe5c8460700 1 -- start start 2024-06-10T00:17:06.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.177+0000 7fe5c8460700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01020d0 0x7fe5c019d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:06.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.177+0000 7fe5c8460700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c010c4a0 con 0x7fe5c01020d0 2024-06-10T00:17:06.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.177+0000 7fe5c61fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01020d0 0x7fe5c019d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:06.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.177+0000 7fe5c61fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01020d0 0x7fe5c019d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60430/0 (socket says 172.21.15.121:60430) 2024-06-10T00:17:06.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.177+0000 7fe5c61fc700 1 -- 172.21.15.121:0/3322626414 learned_addr learned my addr 172.21.15.121:0/3322626414 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:17:06.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.177+0000 7fe5c61fc700 1 -- 172.21.15.121:0/3322626414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c019db70 con 0x7fe5c01020d0 2024-06-10T00:17:06.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.178+0000 7fe5c61fc700 1 --2- 172.21.15.121:0/3322626414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01020d0 0x7fe5c019d630 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0093e0 tx=0x7fe5bc003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:06.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.178+0000 7fe5b77fe700 1 -- 172.21.15.121:0/3322626414 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5bc017040 con 0x7fe5c01020d0 2024-06-10T00:17:06.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.178+0000 7fe5b77fe700 1 -- 172.21.15.121:0/3322626414 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5bc022070 con 0x7fe5c01020d0 2024-06-10T00:17:06.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.178+0000 7fe5c8460700 1 -- 172.21.15.121:0/3322626414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c019de00 con 0x7fe5c01020d0 2024-06-10T00:17:06.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.178+0000 7fe5b77fe700 1 -- 172.21.15.121:0/3322626414 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe5bc01d440 con 0x7fe5c01020d0 2024-06-10T00:17:06.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.178+0000 7fe5c8460700 1 -- 172.21.15.121:0/3322626414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c01977b0 con 0x7fe5c01020d0 2024-06-10T00:17:06.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.179+0000 7fe5b77fe700 1 -- 172.21.15.121:0/3322626414 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fe5bc004000 con 0x7fe5c01020d0 2024-06-10T00:17:06.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.179+0000 7fe5c8460700 1 -- 172.21.15.121:0/3322626414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5c01035b0 con 0x7fe5c01020d0 2024-06-10T00:17:06.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.179+0000 7fe5b77fe700 1 --2- 172.21.15.121:0/3322626414 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe5ac039140 0x7fe5ac03b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:06.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.179+0000 7fe5b77fe700 1 -- 172.21.15.121:0/3322626414 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe5bc00f050 con 0x7fe5c01020d0 2024-06-10T00:17:06.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.179+0000 7fe5c59fb700 1 --2- 172.21.15.121:0/3322626414 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe5ac039140 0x7fe5ac03b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:06.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.179+0000 7fe5c59fb700 1 --2- 172.21.15.121:0/3322626414 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe5ac039140 0x7fe5ac03b600 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0006fd0 tx=0x7fe5b0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:06.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.184+0000 7fe5b77fe700 1 -- 172.21.15.121:0/3322626414 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe5bc007e20 con 0x7fe5c01020d0 2024-06-10T00:17:06.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.383+0000 7fe5c8460700 1 -- 172.21.15.121:0/3322626414 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe5c0198ce0 con 0x7fe5ac039140 2024-06-10T00:17:06.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.384+0000 7fe5b77fe700 1 -- 172.21.15.121:0/3322626414 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7fe5c0198ce0 con 0x7fe5ac039140 2024-06-10T00:17:06.385 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:17:06.385 INFO:teuthology.orchestra.run.smithi121.stdout:[{"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}, {"addr": "172.21.15.135", "hostname": "smithi135", "labels": [], "status": ""}] 2024-06-10T00:17:06.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.386+0000 7fe5c8460700 1 -- 172.21.15.121:0/3322626414 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe5ac039140 msgr2=0x7fe5ac03b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:06.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.386+0000 7fe5c8460700 1 --2- 172.21.15.121:0/3322626414 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe5ac039140 0x7fe5ac03b600 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0006fd0 tx=0x7fe5b0006e40 comp rx=0 tx=0).stop 2024-06-10T00:17:06.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.386+0000 7fe5c8460700 1 -- 172.21.15.121:0/3322626414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01020d0 msgr2=0x7fe5c019d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:06.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.386+0000 7fe5c8460700 1 --2- 172.21.15.121:0/3322626414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01020d0 0x7fe5c019d630 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0093e0 tx=0x7fe5bc003ce0 comp rx=0 tx=0).stop 2024-06-10T00:17:06.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.386+0000 7fe5c8460700 1 -- 172.21.15.121:0/3322626414 shutdown_connections 2024-06-10T00:17:06.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.386+0000 7fe5c8460700 1 --2- 172.21.15.121:0/3322626414 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe5ac039140 0x7fe5ac03b600 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:06.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.386+0000 7fe5c8460700 1 --2- 172.21.15.121:0/3322626414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01020d0 0x7fe5c019d630 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:06.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.386+0000 7fe5c8460700 1 -- 172.21.15.121:0/3322626414 >> 172.21.15.121:0/3322626414 conn(0x7fe5c00fd980 msgr2=0x7fe5c0103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:06.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.386+0000 7fe5c8460700 1 -- 172.21.15.121:0/3322626414 shutdown_connections 2024-06-10T00:17:06.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:06.386+0000 7fe5c8460700 1 -- 172.21.15.121:0/3322626414 wait complete. 2024-06-10T00:17:06.938 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2024-06-10T00:17:06.938 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-06-10T00:17:06.938 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2024-06-10T00:17:06.956 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-06-10T00:17:06.956 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:17:07.014 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2024-06-10T00:17:07.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch host add smithi193 2024-06-10T00:17:07.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:06 smithi121 ceph-mon[32378]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:07.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:07.253 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:17:07.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:07 smithi121 ceph-mon[32378]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:17:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.588+0000 7fa9a2986700 1 -- 172.21.15.121:0/4162972609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c105d50 msgr2=0x7fa99c106130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.588+0000 7fa9a2986700 1 --2- 172.21.15.121:0/4162972609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c105d50 0x7fa99c106130 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009910 tx=0x7fa98c009c20 comp rx=0 tx=0).stop 2024-06-10T00:17:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.588+0000 7fa9a2986700 1 -- 172.21.15.121:0/4162972609 shutdown_connections 2024-06-10T00:17:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.588+0000 7fa9a2986700 1 --2- 172.21.15.121:0/4162972609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c105d50 0x7fa99c106130 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.588+0000 7fa9a2986700 1 -- 172.21.15.121:0/4162972609 >> 172.21.15.121:0/4162972609 conn(0x7fa99c077480 msgr2=0x7fa99c077890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.588+0000 7fa9a2986700 1 -- 172.21.15.121:0/4162972609 shutdown_connections 2024-06-10T00:17:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.588+0000 7fa9a2986700 1 -- 172.21.15.121:0/4162972609 wait complete. 2024-06-10T00:17:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.589+0000 7fa9a2986700 1 Processor -- start 2024-06-10T00:17:08.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.589+0000 7fa9a2986700 1 -- start start 2024-06-10T00:17:08.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.589+0000 7fa9a2986700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c105d50 0x7fa99c106b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:08.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.589+0000 7fa9a2986700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c10b4d0 con 0x7fa99c105d50 2024-06-10T00:17:08.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.589+0000 7fa99bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c105d50 0x7fa99c106b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:08.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.589+0000 7fa99bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c105d50 0x7fa99c106b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60460/0 (socket says 172.21.15.121:60460) 2024-06-10T00:17:08.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.589+0000 7fa99bfff700 1 -- 172.21.15.121:0/2851755094 learned_addr learned my addr 172.21.15.121:0/2851755094 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:17:08.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.589+0000 7fa99bfff700 1 -- 172.21.15.121:0/2851755094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99c108a80 con 0x7fa99c105d50 2024-06-10T00:17:08.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.590+0000 7fa99bfff700 1 --2- 172.21.15.121:0/2851755094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c105d50 0x7fa99c106b90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa98c005850 tx=0x7fa98c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:08.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.590+0000 7fa999ffb700 1 -- 172.21.15.121:0/2851755094 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa98c026070 con 0x7fa99c105d50 2024-06-10T00:17:08.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.590+0000 7fa999ffb700 1 -- 172.21.15.121:0/2851755094 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa98c00bc30 con 0x7fa99c105d50 2024-06-10T00:17:08.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.590+0000 7fa9a2986700 1 -- 172.21.15.121:0/2851755094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa99c1070d0 con 0x7fa99c105d50 2024-06-10T00:17:08.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.590+0000 7fa999ffb700 1 -- 172.21.15.121:0/2851755094 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa98c003a20 con 0x7fa99c105d50 2024-06-10T00:17:08.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.590+0000 7fa9a2986700 1 -- 172.21.15.121:0/2851755094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa99c107590 con 0x7fa99c105d50 2024-06-10T00:17:08.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.591+0000 7fa999ffb700 1 -- 172.21.15.121:0/2851755094 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fa98c029070 con 0x7fa99c105d50 2024-06-10T00:17:08.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.591+0000 7fa999ffb700 1 --2- 172.21.15.121:0/2851755094 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9880390f0 0x7fa98803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:08.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.591+0000 7fa999ffb700 1 -- 172.21.15.121:0/2851755094 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa98c01e070 con 0x7fa99c105d50 2024-06-10T00:17:08.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.591+0000 7fa9a2986700 1 -- 172.21.15.121:0/2851755094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa99c1025a0 con 0x7fa99c105d50 2024-06-10T00:17:08.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.591+0000 7fa993fff700 1 --2- 172.21.15.121:0/2851755094 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9880390f0 0x7fa98803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:08.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.591+0000 7fa993fff700 1 --2- 172.21.15.121:0/2851755094 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9880390f0 0x7fa98803b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa984006fd0 tx=0x7fa984006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:08.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.596+0000 7fa999ffb700 1 -- 172.21.15.121:0/2851755094 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa98c030070 con 0x7fa99c105d50 2024-06-10T00:17:08.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:08.796+0000 7fa9a2986700 1 -- 172.21.15.121:0/2851755094 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}) -- 0x7fa99c1080c0 con 0x7fa9880390f0 2024-06-10T00:17:09.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:08 smithi121 ceph-mon[32378]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:10.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:09 smithi121 ceph-mon[32378]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:17:10.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:10.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:10.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:10.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:10.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi135", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:17:10.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:10.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:10 smithi121 ceph-mon[32378]: Deploying cephadm binary to smithi193 2024-06-10T00:17:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:10 smithi121 ceph-mon[32378]: Updating smithi135:/etc/ceph/ceph.conf 2024-06-10T00:17:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:10 smithi121 ceph-mon[32378]: Updating smithi135:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:17:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:10 smithi121 ceph-mon[32378]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:13.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:13.028+0000 7fa999ffb700 1 -- 172.21.15.121:0/2851755094 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fa99c1080c0 con 0x7fa9880390f0 2024-06-10T00:17:13.030 INFO:teuthology.orchestra.run.smithi121.stdout:Added host 'smithi193' with addr '172.21.15.193' 2024-06-10T00:17:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:13.031+0000 7fa9a2986700 1 -- 172.21.15.121:0/2851755094 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9880390f0 msgr2=0x7fa98803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:13.031+0000 7fa9a2986700 1 --2- 172.21.15.121:0/2851755094 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9880390f0 0x7fa98803b5b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa984006fd0 tx=0x7fa984006e40 comp rx=0 tx=0).stop 2024-06-10T00:17:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:13.031+0000 7fa9a2986700 1 -- 172.21.15.121:0/2851755094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c105d50 msgr2=0x7fa99c106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:13.031+0000 7fa9a2986700 1 --2- 172.21.15.121:0/2851755094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c105d50 0x7fa99c106b90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa98c005850 tx=0x7fa98c005030 comp rx=0 tx=0).stop 2024-06-10T00:17:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:13.031+0000 7fa9a2986700 1 -- 172.21.15.121:0/2851755094 shutdown_connections 2024-06-10T00:17:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:13.031+0000 7fa9a2986700 1 --2- 172.21.15.121:0/2851755094 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9880390f0 0x7fa98803b5b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:13.031+0000 7fa9a2986700 1 --2- 172.21.15.121:0/2851755094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c105d50 0x7fa99c106b90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:13.031+0000 7fa9a2986700 1 -- 172.21.15.121:0/2851755094 >> 172.21.15.121:0/2851755094 conn(0x7fa99c077480 msgr2=0x7fa99c100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:13.031+0000 7fa9a2986700 1 -- 172.21.15.121:0/2851755094 shutdown_connections 2024-06-10T00:17:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:13.031+0000 7fa9a2986700 1 -- 172.21.15.121:0/2851755094 wait complete. 2024-06-10T00:17:13.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:13 smithi121 ceph-mon[32378]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:13.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:13.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:13.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:13.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:13.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:13.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-06-10T00:17:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:17:14.224 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:14 smithi121 ceph-mon[32378]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:17:14.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:14 smithi121 ceph-mon[32378]: Updating smithi135:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:17:14.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:14 smithi121 ceph-mon[32378]: Added host smithi193 2024-06-10T00:17:14.225 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:14 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:15.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.274+0000 7fd64b94a700 1 -- 172.21.15.121:0/2447903425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd644103100 msgr2=0x7fd6441034e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:15.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.274+0000 7fd64b94a700 1 --2- 172.21.15.121:0/2447903425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd644103100 0x7fd6441034e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd640009910 tx=0x7fd640009c20 comp rx=0 tx=0).stop 2024-06-10T00:17:15.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.275+0000 7fd64b94a700 1 -- 172.21.15.121:0/2447903425 shutdown_connections 2024-06-10T00:17:15.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.275+0000 7fd64b94a700 1 --2- 172.21.15.121:0/2447903425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd644103100 0x7fd6441034e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:15.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.275+0000 7fd64b94a700 1 -- 172.21.15.121:0/2447903425 >> 172.21.15.121:0/2447903425 conn(0x7fd6440fc8d0 msgr2=0x7fd6440fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:15.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.275+0000 7fd64b94a700 1 -- 172.21.15.121:0/2447903425 shutdown_connections 2024-06-10T00:17:15.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.275+0000 7fd64b94a700 1 -- 172.21.15.121:0/2447903425 wait complete. 2024-06-10T00:17:15.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.275+0000 7fd64b94a700 1 Processor -- start 2024-06-10T00:17:15.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.275+0000 7fd64b94a700 1 -- start start 2024-06-10T00:17:15.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.275+0000 7fd64b94a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd644103100 0x7fd64419d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:15.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.276+0000 7fd64b94a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd64410c550 con 0x7fd644103100 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.276+0000 7fd6496e6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd644103100 0x7fd64419d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.276+0000 7fd6496e6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd644103100 0x7fd64419d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43084/0 (socket says 172.21.15.121:43084) 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.276+0000 7fd6496e6700 1 -- 172.21.15.121:0/3373644938 learned_addr learned my addr 172.21.15.121:0/3373644938 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.276+0000 7fd6496e6700 1 -- 172.21.15.121:0/3373644938 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64419dc00 con 0x7fd644103100 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.276+0000 7fd6496e6700 1 --2- 172.21.15.121:0/3373644938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd644103100 0x7fd64419d6c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd640006010 tx=0x7fd640005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.276+0000 7fd63a7fc700 1 -- 172.21.15.121:0/3373644938 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd640026070 con 0x7fd644103100 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.276+0000 7fd64b94a700 1 -- 172.21.15.121:0/3373644938 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd64419de90 con 0x7fd644103100 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.276+0000 7fd63a7fc700 1 -- 172.21.15.121:0/3373644938 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd64000bc30 con 0x7fd644103100 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.276+0000 7fd63a7fc700 1 -- 172.21.15.121:0/3373644938 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6400039a0 con 0x7fd644103100 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.276+0000 7fd64b94a700 1 -- 172.21.15.121:0/3373644938 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd644197880 con 0x7fd644103100 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.277+0000 7fd63a7fc700 1 -- 172.21.15.121:0/3373644938 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fd640029070 con 0x7fd644103100 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.277+0000 7fd64b94a700 1 -- 172.21.15.121:0/3373644938 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd64406ab10 con 0x7fd644103100 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.277+0000 7fd63a7fc700 1 --2- 172.21.15.121:0/3373644938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6300390f0 0x7fd63003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:15.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.277+0000 7fd63a7fc700 1 -- 172.21.15.121:0/3373644938 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd64001e070 con 0x7fd644103100 2024-06-10T00:17:15.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.277+0000 7fd648ee5700 1 --2- 172.21.15.121:0/3373644938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6300390f0 0x7fd63003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:15.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.278+0000 7fd648ee5700 1 --2- 172.21.15.121:0/3373644938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6300390f0 0x7fd63003b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd634006fd0 tx=0x7fd634006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:15.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.282+0000 7fd63a7fc700 1 -- 172.21.15.121:0/3373644938 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd640013070 con 0x7fd644103100 2024-06-10T00:17:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:15 smithi121 ceph-mon[32378]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:15 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:15.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.481+0000 7fd64b94a700 1 -- 172.21.15.121:0/3373644938 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd644198db0 con 0x7fd6300390f0 2024-06-10T00:17:15.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.482+0000 7fd63a7fc700 1 -- 172.21.15.121:0/3373644938 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7fd644198db0 con 0x7fd6300390f0 2024-06-10T00:17:15.483 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:17:15.483 INFO:teuthology.orchestra.run.smithi121.stdout:[{"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}, {"addr": "172.21.15.135", "hostname": "smithi135", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2024-06-10T00:17:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.484+0000 7fd64b94a700 1 -- 172.21.15.121:0/3373644938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6300390f0 msgr2=0x7fd63003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.484+0000 7fd64b94a700 1 --2- 172.21.15.121:0/3373644938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6300390f0 0x7fd63003b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd634006fd0 tx=0x7fd634006e40 comp rx=0 tx=0).stop 2024-06-10T00:17:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.484+0000 7fd64b94a700 1 -- 172.21.15.121:0/3373644938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd644103100 msgr2=0x7fd64419d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.484+0000 7fd64b94a700 1 --2- 172.21.15.121:0/3373644938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd644103100 0x7fd64419d6c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd640006010 tx=0x7fd640005030 comp rx=0 tx=0).stop 2024-06-10T00:17:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.484+0000 7fd64b94a700 1 -- 172.21.15.121:0/3373644938 shutdown_connections 2024-06-10T00:17:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.484+0000 7fd64b94a700 1 --2- 172.21.15.121:0/3373644938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6300390f0 0x7fd63003b5b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.484+0000 7fd64b94a700 1 --2- 172.21.15.121:0/3373644938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd644103100 0x7fd64419d6c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.484+0000 7fd64b94a700 1 -- 172.21.15.121:0/3373644938 >> 172.21.15.121:0/3373644938 conn(0x7fd6440fc8d0 msgr2=0x7fd644103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.485+0000 7fd64b94a700 1 -- 172.21.15.121:0/3373644938 shutdown_connections 2024-06-10T00:17:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:15.485+0000 7fd64b94a700 1 -- 172.21.15.121:0/3373644938 wait complete. 2024-06-10T00:17:16.053 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-10T00:17:16.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-06-10T00:17:16.283 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:17:16.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:16 smithi121 ceph-mon[32378]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:17:16.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:16.662 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:16 smithi121 ceph-mon[32378]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:17.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.729+0000 7f6c6a743700 1 -- 172.21.15.121:0/2784287250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c641006e0 msgr2=0x7f6c64100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:17.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.729+0000 7f6c6a743700 1 --2- 172.21.15.121:0/2784287250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c641006e0 0x7f6c64100ac0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009920 tx=0x7f6c54009c30 comp rx=0 tx=0).stop 2024-06-10T00:17:17.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.729+0000 7f6c6a743700 1 -- 172.21.15.121:0/2784287250 shutdown_connections 2024-06-10T00:17:17.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.729+0000 7f6c6a743700 1 --2- 172.21.15.121:0/2784287250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c641006e0 0x7f6c64100ac0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:17.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.729+0000 7f6c6a743700 1 -- 172.21.15.121:0/2784287250 >> 172.21.15.121:0/2784287250 conn(0x7f6c64077440 msgr2=0x7f6c64077850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.729+0000 7f6c6a743700 1 -- 172.21.15.121:0/2784287250 shutdown_connections 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.729+0000 7f6c6a743700 1 -- 172.21.15.121:0/2784287250 wait complete. 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.730+0000 7f6c6a743700 1 Processor -- start 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.730+0000 7f6c6a743700 1 -- start start 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.730+0000 7f6c6a743700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c641006e0 0x7f6c64101660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.730+0000 7f6c6a743700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6410c4c0 con 0x7f6c641006e0 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.730+0000 7f6c63fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c641006e0 0x7f6c64101660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.730+0000 7f6c63fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c641006e0 0x7f6c64101660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43094/0 (socket says 172.21.15.121:43094) 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.730+0000 7f6c63fff700 1 -- 172.21.15.121:0/709820168 learned_addr learned my addr 172.21.15.121:0/709820168 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.731+0000 7f6c63fff700 1 -- 172.21.15.121:0/709820168 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c64103550 con 0x7f6c641006e0 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.731+0000 7f6c63fff700 1 --2- 172.21.15.121:0/709820168 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c641006e0 0x7f6c64101660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400b5c0 tx=0x7f6c540055d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.731+0000 7f6c617fa700 1 -- 172.21.15.121:0/709820168 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c54026070 con 0x7f6c641006e0 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.731+0000 7f6c617fa700 1 -- 172.21.15.121:0/709820168 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c5400bc30 con 0x7f6c641006e0 2024-06-10T00:17:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.731+0000 7f6c6a743700 1 -- 172.21.15.121:0/709820168 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c64101c00 con 0x7f6c641006e0 2024-06-10T00:17:17.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.731+0000 7f6c617fa700 1 -- 172.21.15.121:0/709820168 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c54003a40 con 0x7f6c641006e0 2024-06-10T00:17:17.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.731+0000 7f6c6a743700 1 -- 172.21.15.121:0/709820168 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c641020c0 con 0x7f6c641006e0 2024-06-10T00:17:17.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.732+0000 7f6c617fa700 1 -- 172.21.15.121:0/709820168 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f6c54029070 con 0x7f6c641006e0 2024-06-10T00:17:17.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.732+0000 7f6c6a743700 1 -- 172.21.15.121:0/709820168 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c64063af0 con 0x7f6c641006e0 2024-06-10T00:17:17.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.732+0000 7f6c617fa700 1 --2- 172.21.15.121:0/709820168 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6c4c0390f0 0x7f6c4c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:17.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.732+0000 7f6c617fa700 1 -- 172.21.15.121:0/709820168 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6c5401e070 con 0x7f6c641006e0 2024-06-10T00:17:17.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.732+0000 7f6c637fe700 1 --2- 172.21.15.121:0/709820168 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6c4c0390f0 0x7f6c4c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:17.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.733+0000 7f6c637fe700 1 --2- 172.21.15.121:0/709820168 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6c4c0390f0 0x7f6c4c03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c58006fd0 tx=0x7f6c58006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:17.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.737+0000 7f6c617fa700 1 -- 172.21.15.121:0/709820168 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6c54013020 con 0x7f6c641006e0 2024-06-10T00:17:17.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.937+0000 7f6c6a743700 1 -- 172.21.15.121:0/709820168 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f6c640008d0 con 0x7f6c641006e0 2024-06-10T00:17:17.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.989+0000 7f6c617fa700 1 -- 172.21.15.121:0/709820168 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f6c54030020 con 0x7f6c641006e0 2024-06-10T00:17:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.991+0000 7f6c6a743700 1 -- 172.21.15.121:0/709820168 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6c4c0390f0 msgr2=0x7f6c4c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.991+0000 7f6c6a743700 1 --2- 172.21.15.121:0/709820168 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6c4c0390f0 0x7f6c4c03b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c58006fd0 tx=0x7f6c58006e40 comp rx=0 tx=0).stop 2024-06-10T00:17:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.991+0000 7f6c6a743700 1 -- 172.21.15.121:0/709820168 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c641006e0 msgr2=0x7f6c64101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.991+0000 7f6c6a743700 1 --2- 172.21.15.121:0/709820168 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c641006e0 0x7f6c64101660 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6c5400b5c0 tx=0x7f6c540055d0 comp rx=0 tx=0).stop 2024-06-10T00:17:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.991+0000 7f6c6a743700 1 -- 172.21.15.121:0/709820168 shutdown_connections 2024-06-10T00:17:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.991+0000 7f6c6a743700 1 --2- 172.21.15.121:0/709820168 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6c4c0390f0 0x7f6c4c03b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.991+0000 7f6c6a743700 1 --2- 172.21.15.121:0/709820168 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c641006e0 0x7f6c64101660 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.991+0000 7f6c6a743700 1 -- 172.21.15.121:0/709820168 >> 172.21.15.121:0/709820168 conn(0x7f6c64077440 msgr2=0x7f6c64103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.991+0000 7f6c6a743700 1 -- 172.21.15.121:0/709820168 shutdown_connections 2024-06-10T00:17:17.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:17.991+0000 7f6c6a743700 1 -- 172.21.15.121:0/709820168 wait complete. 2024-06-10T00:17:17.992 INFO:teuthology.orchestra.run.smithi121.stderr:adjusted tunables profile to default 2024-06-10T00:17:18.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:17 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/709820168' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-10T00:17:18.565 INFO:tasks.cephadm:Adding mon.a on smithi121 2024-06-10T00:17:18.565 INFO:tasks.cephadm:Adding mon.b on smithi135 2024-06-10T00:17:18.565 INFO:tasks.cephadm:Adding mon.c on smithi193 2024-06-10T00:17:18.566 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch apply mon '3;smithi121:172.21.15.121=a;smithi135:172.21.15.135=b;smithi193:172.21.15.193=c' 2024-06-10T00:17:18.798 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-10T00:17:18.845 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-10T00:17:19.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:18 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/709820168' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-10T00:17:19.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:18 smithi121 ceph-mon[32378]: osdmap e4: 0 total, 0 up, 0 in 2024-06-10T00:17:19.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:18 smithi121 ceph-mon[32378]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:19.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:18 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:19.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:18 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:19.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:18 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:19.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:18 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:19.451 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:18 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:17:19.451 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:18 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:19.451 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:18 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:20.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:19 smithi121 ceph-mon[32378]: Updating smithi193:/etc/ceph/ceph.conf 2024-06-10T00:17:20.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:19 smithi121 ceph-mon[32378]: Updating smithi193:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:17:20.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:19 smithi121 ceph-mon[32378]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:17:20.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:20.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:20.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:20.650 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.649+0000 7f8c59acf700 1 -- 172.21.15.193:0/970618014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c54100fb0 msgr2=0x7f8c54101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:20.650 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.649+0000 7f8c59acf700 1 --2- 172.21.15.193:0/970618014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c54100fb0 0x7f8c54101390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8c44009910 tx=0x7f8c44009c20 comp rx=0 tx=0).stop 2024-06-10T00:17:20.650 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.650+0000 7f8c59acf700 1 -- 172.21.15.193:0/970618014 shutdown_connections 2024-06-10T00:17:20.651 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.650+0000 7f8c59acf700 1 --2- 172.21.15.193:0/970618014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c54100fb0 0x7f8c54101390 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:20.651 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.650+0000 7f8c59acf700 1 -- 172.21.15.193:0/970618014 >> 172.21.15.193:0/970618014 conn(0x7f8c540fc840 msgr2=0x7f8c540fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:20.651 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.650+0000 7f8c59acf700 1 -- 172.21.15.193:0/970618014 shutdown_connections 2024-06-10T00:17:20.651 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.650+0000 7f8c59acf700 1 -- 172.21.15.193:0/970618014 wait complete. 2024-06-10T00:17:20.651 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.650+0000 7f8c59acf700 1 Processor -- start 2024-06-10T00:17:20.651 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.650+0000 7f8c59acf700 1 -- start start 2024-06-10T00:17:20.651 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.651+0000 7f8c59acf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c54100fb0 0x7f8c5419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:20.651 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.651+0000 7f8c59acf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8c5410b380 con 0x7f8c54100fb0 2024-06-10T00:17:20.653 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.651+0000 7f8c537fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c54100fb0 0x7f8c5419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:20.653 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.651+0000 7f8c537fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c54100fb0 0x7f8c5419f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.193:51964/0 (socket says 172.21.15.193:51964) 2024-06-10T00:17:20.653 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.651+0000 7f8c537fe700 1 -- 172.21.15.193:0/1327786492 learned_addr learned my addr 172.21.15.193:0/1327786492 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.651+0000 7f8c537fe700 1 -- 172.21.15.193:0/1327786492 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5419fd60 con 0x7f8c54100fb0 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.652+0000 7f8c537fe700 1 --2- 172.21.15.193:0/1327786492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c54100fb0 0x7f8c5419f820 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8c44005850 tx=0x7f8c44005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.652+0000 7f8c50ff9700 1 -- 172.21.15.193:0/1327786492 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c44026070 con 0x7f8c54100fb0 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.652+0000 7f8c59acf700 1 -- 172.21.15.193:0/1327786492 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c541a0050 con 0x7f8c54100fb0 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.652+0000 7f8c50ff9700 1 -- 172.21.15.193:0/1327786492 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c4400bbd0 con 0x7f8c54100fb0 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.652+0000 7f8c50ff9700 1 -- 172.21.15.193:0/1327786492 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8c44003970 con 0x7f8c54100fb0 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.652+0000 7f8c59acf700 1 -- 172.21.15.193:0/1327786492 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c541999e0 con 0x7f8c54100fb0 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.653+0000 7f8c50ff9700 1 -- 172.21.15.193:0/1327786492 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f8c44029070 con 0x7f8c54100fb0 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.653+0000 7f8c59acf700 1 -- 172.21.15.193:0/1327786492 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c54102490 con 0x7f8c54100fb0 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.653+0000 7f8c50ff9700 1 --2- 172.21.15.193:0/1327786492 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8c3c0390f0 0x7f8c3c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.653+0000 7f8c50ff9700 1 -- 172.21.15.193:0/1327786492 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8c4401e070 con 0x7f8c54100fb0 2024-06-10T00:17:20.654 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.653+0000 7f8c52ffd700 1 --2- 172.21.15.193:0/1327786492 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8c3c0390f0 0x7f8c3c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:20.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.653+0000 7f8c52ffd700 1 --2- 172.21.15.193:0/1327786492 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8c3c0390f0 0x7f8c3c03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8c48006fd0 tx=0x7f8c48006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:20.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.658+0000 7f8c50ff9700 1 -- 172.21.15.193:0/1327786492 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8c44013070 con 0x7f8c54100fb0 2024-06-10T00:17:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.870+0000 7f8c59acf700 1 -- 172.21.15.193:0/1327786492 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi121:172.21.15.121=a;smithi135:172.21.15.135=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}) -- 0x7f8c5419b020 con 0x7f8c3c0390f0 2024-06-10T00:17:20.876 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.876+0000 7f8c50ff9700 1 -- 172.21.15.193:0/1327786492 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8c5419b020 con 0x7f8c3c0390f0 2024-06-10T00:17:20.876 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2024-06-10T00:17:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.878+0000 7f8c59acf700 1 -- 172.21.15.193:0/1327786492 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8c3c0390f0 msgr2=0x7f8c3c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:20.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.878+0000 7f8c59acf700 1 --2- 172.21.15.193:0/1327786492 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8c3c0390f0 0x7f8c3c03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8c48006fd0 tx=0x7f8c48006e40 comp rx=0 tx=0).stop 2024-06-10T00:17:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.878+0000 7f8c59acf700 1 -- 172.21.15.193:0/1327786492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c54100fb0 msgr2=0x7f8c5419f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.878+0000 7f8c59acf700 1 --2- 172.21.15.193:0/1327786492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c54100fb0 0x7f8c5419f820 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8c44005850 tx=0x7f8c44005030 comp rx=0 tx=0).stop 2024-06-10T00:17:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.879+0000 7f8c59acf700 1 -- 172.21.15.193:0/1327786492 shutdown_connections 2024-06-10T00:17:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.879+0000 7f8c59acf700 1 --2- 172.21.15.193:0/1327786492 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8c3c0390f0 0x7f8c3c03b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.879+0000 7f8c59acf700 1 --2- 172.21.15.193:0/1327786492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c54100fb0 0x7f8c5419f820 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.879+0000 7f8c59acf700 1 -- 172.21.15.193:0/1327786492 >> 172.21.15.193:0/1327786492 conn(0x7f8c540fc840 msgr2=0x7f8c5406a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.879+0000 7f8c59acf700 1 -- 172.21.15.193:0/1327786492 shutdown_connections 2024-06-10T00:17:20.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:20.879+0000 7f8c59acf700 1 -- 172.21.15.193:0/1327786492 wait complete. 2024-06-10T00:17:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:21 smithi121 ceph-mon[32378]: Updating smithi193:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:17:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:21 smithi121 ceph-mon[32378]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:21.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:21.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:21.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:21.456 DEBUG:teuthology.orchestra.run.smithi135:mon.b> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.b.service 2024-06-10T00:17:21.458 DEBUG:teuthology.orchestra.run.smithi193:mon.c> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.c.service 2024-06-10T00:17:21.461 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-10T00:17:21.461 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-10T00:17:22.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:22 smithi121 ceph-mon[32378]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi121:172.21.15.121=a;smithi135:172.21.15.135=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:17:22.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:22 smithi121 ceph-mon[32378]: Saving service mon spec with placement smithi121:172.21.15.121=a;smithi135:172.21.15.135=b;smithi193:172.21.15.193=c;count:3 2024-06-10T00:17:22.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:22 smithi121 ceph-mon[32378]: Deploying daemon mon.c on smithi193 2024-06-10T00:17:22.599 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:17:23.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:23 smithi121 ceph-mon[32378]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:25.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.259+0000 7f465c482700 1 -- 172.21.15.193:0/1524746814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4654100ff0 msgr2=0x7f46541013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:25.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.259+0000 7f465c482700 1 --2- 172.21.15.193:0/1524746814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4654100ff0 0x7f46541013d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4650009910 tx=0x7f4650009c20 comp rx=0 tx=0).stop 2024-06-10T00:17:25.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.259+0000 7f465c482700 1 -- 172.21.15.193:0/1524746814 shutdown_connections 2024-06-10T00:17:25.261 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.259+0000 7f465c482700 1 --2- 172.21.15.193:0/1524746814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4654100ff0 0x7f46541013d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.259+0000 7f465c482700 1 -- 172.21.15.193:0/1524746814 >> 172.21.15.193:0/1524746814 conn(0x7f4654077b30 msgr2=0x7f46540feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.259+0000 7f465c482700 1 -- 172.21.15.193:0/1524746814 shutdown_connections 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.259+0000 7f465c482700 1 -- 172.21.15.193:0/1524746814 wait complete. 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.260+0000 7f465c482700 1 Processor -- start 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.260+0000 7f465c482700 1 -- start start 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.260+0000 7f465c482700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4654100ff0 0x7f465419d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.260+0000 7f465c482700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f465410b370 con 0x7f4654100ff0 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.260+0000 7f465a21e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4654100ff0 0x7f465419d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.260+0000 7f465a21e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4654100ff0 0x7f465419d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.193:51986/0 (socket says 172.21.15.193:51986) 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.260+0000 7f465a21e700 1 -- 172.21.15.193:0/2367100741 learned_addr learned my addr 172.21.15.193:0/2367100741 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.261+0000 7f465a21e700 1 -- 172.21.15.193:0/2367100741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465419dbd0 con 0x7f4654100ff0 2024-06-10T00:17:25.262 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.261+0000 7f465a21e700 1 --2- 172.21.15.193:0/2367100741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4654100ff0 0x7f465419d690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4650006010 tx=0x7f4650005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:25.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.261+0000 7f464b7fe700 1 -- 172.21.15.193:0/2367100741 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4650026070 con 0x7f4654100ff0 2024-06-10T00:17:25.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.261+0000 7f464b7fe700 1 -- 172.21.15.193:0/2367100741 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f465000bbd0 con 0x7f4654100ff0 2024-06-10T00:17:25.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.261+0000 7f465c482700 1 -- 172.21.15.193:0/2367100741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f465419de60 con 0x7f4654100ff0 2024-06-10T00:17:25.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.261+0000 7f464b7fe700 1 -- 172.21.15.193:0/2367100741 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4650003910 con 0x7f4654100ff0 2024-06-10T00:17:25.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.261+0000 7f465c482700 1 -- 172.21.15.193:0/2367100741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4654197840 con 0x7f4654100ff0 2024-06-10T00:17:25.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.262+0000 7f464b7fe700 1 -- 172.21.15.193:0/2367100741 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f4650029070 con 0x7f4654100ff0 2024-06-10T00:17:25.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.262+0000 7f465c482700 1 -- 172.21.15.193:0/2367100741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4654102480 con 0x7f4654100ff0 2024-06-10T00:17:25.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.262+0000 7f464b7fe700 1 --2- 172.21.15.193:0/2367100741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f46400390f0 0x7f464003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:25.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.263+0000 7f464b7fe700 1 -- 172.21.15.193:0/2367100741 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f465001e070 con 0x7f4654100ff0 2024-06-10T00:17:25.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.263+0000 7f4659a1d700 1 --2- 172.21.15.193:0/2367100741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f46400390f0 0x7f464003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:25.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.263+0000 7f4659a1d700 1 --2- 172.21.15.193:0/2367100741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f46400390f0 0x7f464003b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4644006fd0 tx=0x7f4644006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:25.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.268+0000 7f464b7fe700 1 -- 172.21.15.193:0/2367100741 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4650013070 con 0x7f4654100ff0 2024-06-10T00:17:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:25 smithi121 ceph-mon[32378]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:25.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.572+0000 7f465c482700 1 -- 172.21.15.193:0/2367100741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f4654063af0 con 0x7f4654100ff0 2024-06-10T00:17:25.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.572+0000 7f464b7fe700 1 -- 172.21.15.193:0/2367100741 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f4650030020 con 0x7f4654100ff0 2024-06-10T00:17:25.572 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:17:25.573 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","modified":"2024-06-10T00:15:39.680682Z","created":"2024-06-10T00:15:39.680682Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-10T00:17:25.574 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.574+0000 7f465c482700 1 -- 172.21.15.193:0/2367100741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f46400390f0 msgr2=0x7f464003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:25.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.574+0000 7f465c482700 1 --2- 172.21.15.193:0/2367100741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f46400390f0 0x7f464003b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4644006fd0 tx=0x7f4644006e40 comp rx=0 tx=0).stop 2024-06-10T00:17:25.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.574+0000 7f465c482700 1 -- 172.21.15.193:0/2367100741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4654100ff0 msgr2=0x7f465419d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:25.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.574+0000 7f465c482700 1 --2- 172.21.15.193:0/2367100741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4654100ff0 0x7f465419d690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4650006010 tx=0x7f4650005030 comp rx=0 tx=0).stop 2024-06-10T00:17:25.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.574+0000 7f465c482700 1 -- 172.21.15.193:0/2367100741 shutdown_connections 2024-06-10T00:17:25.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.574+0000 7f465c482700 1 --2- 172.21.15.193:0/2367100741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f46400390f0 0x7f464003b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:25.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.574+0000 7f465c482700 1 --2- 172.21.15.193:0/2367100741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4654100ff0 0x7f465419d690 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:25.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.574+0000 7f465c482700 1 -- 172.21.15.193:0/2367100741 >> 172.21.15.193:0/2367100741 conn(0x7f4654077b30 msgr2=0x7f46540698c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:25.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.574+0000 7f465c482700 1 -- 172.21.15.193:0/2367100741 shutdown_connections 2024-06-10T00:17:25.575 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:25.575+0000 7f465c482700 1 -- 172.21.15.193:0/2367100741 wait complete. 2024-06-10T00:17:25.576 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-06-10T00:17:25.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:25 smithi193 systemd[1]: Starting Ceph mon.c for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:17:26.350 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:26 smithi121 ceph-mon[32378]: from='client.? 172.21.15.193:0/2367100741' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-10T00:17:26.486 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 podman[36697]: 2024-06-10 00:17:26.285529414 +0000 UTC m=+0.016842958 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:17:26.486 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 podman[36697]: 2024-06-10 00:17:26.389845731 +0000 UTC m=+0.121159283 container create 55317b68c71622ce383a508d371e0305cf07b7233ce1ecba54af4530d8980e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:17:26.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 podman[36697]: 2024-06-10 00:17:26.601572607 +0000 UTC m=+0.332886138 container init 55317b68c71622ce383a508d371e0305cf07b7233ce1ecba54af4530d8980e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:17:26.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 podman[36697]: 2024-06-10 00:17:26.604589193 +0000 UTC m=+0.335902732 container start 55317b68c71622ce383a508d371e0305cf07b7233ce1ecba54af4530d8980e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2) 2024-06-10T00:17:26.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: set uid:gid to 167:167 (ceph:ceph) 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc), process ceph-mon, pid 2 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: pidfile_write: ignore empty --pid-file 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: load: jerasure load: lrc 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: RocksDB version: 7.9.2 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Git sha 0 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Compile date 2024-05-30 15:42:47 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: DB SUMMARY 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: DB Session ID: WTZ6SH1HSS8R0ASUMHIK 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: CURRENT file: CURRENT 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: IDENTITY file: IDENTITY 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-06-10T00:17:26.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-06-10T00:17:26.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.error_if_exists: 0 2024-06-10T00:17:26.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.create_if_missing: 0 2024-06-10T00:17:26.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.paranoid_checks: 1 2024-06-10T00:17:26.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-10T00:17:26.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-10T00:17:26.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-10T00:17:26.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.env: 0x558f4f372c60 2024-06-10T00:17:26.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.fs: PosixFileSystem 2024-06-10T00:17:26.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.info_log: 0x558f50883b20 2024-06-10T00:17:26.883 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_file_opening_threads: 16 2024-06-10T00:17:26.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.statistics: (nil) 2024-06-10T00:17:26.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.use_fsync: 0 2024-06-10T00:17:26.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_log_file_size: 0 2024-06-10T00:17:26.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-10T00:17:26.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-10T00:17:26.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.keep_log_file_num: 1000 2024-06-10T00:17:26.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.recycle_log_file_num: 0 2024-06-10T00:17:26.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.allow_fallocate: 1 2024-06-10T00:17:26.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.allow_mmap_reads: 0 2024-06-10T00:17:26.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.allow_mmap_writes: 0 2024-06-10T00:17:26.885 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.use_direct_reads: 0 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.create_missing_column_families: 0 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.db_log_dir: 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.wal_dir: 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.advise_random_on_open: 1 2024-06-10T00:17:26.886 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.db_write_buffer_size: 0 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.write_buffer_manager: 0x558f508541e0 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.rate_limiter: (nil) 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.wal_recovery_mode: 2 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.enable_thread_tracking: 0 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.enable_pipelined_write: 0 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.unordered_write: 0 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-10T00:17:26.887 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.row_cache: None 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.wal_filter: None 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.allow_ingest_behind: 0 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.two_write_queues: 0 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.manual_wal_flush: 0 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.wal_compression: 0 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.atomic_flush: 0 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.log_readahead_size: 0 2024-06-10T00:17:26.888 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.best_efforts_recovery: 0 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.allow_data_in_errors: 0 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.db_host_id: __hostname__ 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_background_jobs: 2 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_background_compactions: -1 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_subcompactions: 1 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_total_wal_size: 0 2024-06-10T00:17:26.889 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_open_files: -1 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bytes_per_sync: 0 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_readahead_size: 0 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_background_flushes: -1 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Compression algorithms supported: 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: kZSTD supported: 0 2024-06-10T00:17:26.890 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: kXpressCompression supported: 0 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: kLZ4HCCompression supported: 1 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: kZlibCompression supported: 1 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: kSnappyCompression supported: 1 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: kLZ4Compression supported: 1 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: kBZip2Compression supported: 0 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.merge_operator: 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_filter: None 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_filter_factory: None 2024-06-10T00:17:26.891 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.sst_partitioner_factory: None 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558f50883c40) 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout: cache_index_and_filter_blocks: 1 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout: pin_top_level_index_and_filter: 1 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout: index_type: 0 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout: data_block_index_type: 0 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout: index_shortening: 1 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout: checksum: 4 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout: no_block_cache: 0 2024-06-10T00:17:26.892 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache: 0x558f50885090 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache_name: BinnedLRUCache 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache_options: 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: capacity : 536870912 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: num_shard_bits : 4 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: strict_capacity_limit : 0 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: high_pri_pool_ratio: 0.000 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache_compressed: (nil) 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: persistent_cache: (nil) 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_size: 4096 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_size_deviation: 10 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_restart_interval: 16 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: index_block_restart_interval: 1 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: metadata_block_size: 4096 2024-06-10T00:17:26.893 INFO:journalctl@ceph.mon.c.smithi193.stdout: partition_filters: 0 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: use_delta_encoding: 1 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: filter_policy: bloomfilter 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: whole_key_filtering: 1 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: verify_compression: 0 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: read_amp_bytes_per_bit: 0 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: format_version: 5 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: enable_index_compression: 1 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_align: 0 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: max_auto_readahead_size: 262144 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: prepopulate_block_cache: 0 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: initial_auto_readahead_size: 8192 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout: num_file_reads_for_auto_readahead: 2 2024-06-10T00:17:26.894 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.write_buffer_size: 33554432 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_write_buffer_number: 2 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compression: NoCompression 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bottommost_compression: Disabled 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.prefix_extractor: nullptr 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.num_levels: 7 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-10T00:17:26.895 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compression_opts.level: 32767 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compression_opts.strategy: 0 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compression_opts.enabled: false 2024-06-10T00:17:26.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.target_file_size_base: 67108864 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-10T00:17:26.897 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.arena_block_size: 1048576 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.disable_auto_compactions: 0 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-10T00:17:26.898 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.table_properties_collectors: 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.inplace_update_support: 0 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.bloom_locality: 0 2024-06-10T00:17:26.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.max_successive_merges: 0 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.paranoid_file_checks: 0 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.force_consistency_checks: 1 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.report_bg_io_stats: 0 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.ttl: 2592000 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.enable_blob_files: false 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.min_blob_size: 0 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.blob_file_size: 268435456 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-10T00:17:26.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.blob_file_starting_level: 0 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 82f9bae5-4b0b-4d76-ace7-a05cd9fb7c97 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717978646635541, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717978646636137, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717978646, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "82f9bae5-4b0b-4d76-ace7-a05cd9fb7c97", "db_session_id": "WTZ6SH1HSS8R0ASUMHIK", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717978646636241, "job": 1, "event": "recovery_finished"} 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-06-10T00:17:26.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558f50974000 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: DB pointer 0x558f5095e000 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** DB Stats ** 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-10T00:17:26.902 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** Compaction Stats [default] ** 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** Compaction Stats [default] ** 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-06-10T00:17:26.903 INFO:journalctl@ceph.mon.c.smithi193.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval compaction: 0.00 GB write, 0.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: Block cache BinnedLRUCache@0x558f50885090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-06-10T00:17:26.904 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: using public_addr v2:172.21.15.193:0/0 -> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: starting mon.c rank -1 at public addrs [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] at bind addrs [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(???) e0 preinit fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).mds e1 new map 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).mds e1 print_map 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout: e1 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout: legacy client fscid: -1 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout: No filesystems configured 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-06-10T00:17:26.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:17:26.906 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:26.907 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:17:26.907 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.907 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.907 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi135", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Deploying cephadm binary to smithi193 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Updating smithi135:/etc/ceph/ceph.conf 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Updating smithi135:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:26.908 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Updating smithi135:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Added host smithi193 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/709820168' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/709820168' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: osdmap e4: 0 total, 0 up, 0 in 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:26.909 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Updating smithi193:/etc/ceph/ceph.conf 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Updating smithi193:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.910 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Updating smithi193:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi121:172.21.15.121=a;smithi135:172.21.15.135=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Saving service mon spec with placement smithi121:172.21.15.121=a;smithi135:172.21.15.135=b;smithi193:172.21.15.193=c;count:3 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: Deploying daemon mon.c on smithi193 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:26.911 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: from='client.? 172.21.15.193:0/2367100741' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-10T00:17:26.912 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 ceph-mon[36716]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-06-10T00:17:26.912 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 bash[36697]: 55317b68c71622ce383a508d371e0305cf07b7233ce1ecba54af4530d8980e5f 2024-06-10T00:17:26.912 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:26 smithi193 systemd[1]: Started Ceph mon.c for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:17:27.242 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-10T00:17:27.242 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-10T00:17:27.472 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:17:31.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.793+0000 7f62f17aa700 1 -- 172.21.15.193:0/508451249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ec101120 msgr2=0x7f62d4005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:31.794 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.793+0000 7f62f17aa700 1 --2- 172.21.15.193:0/508451249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ec101120 0x7f62d4005610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f62e0005fa0 tx=0x7f62e000ffc0 comp rx=0 tx=0).stop 2024-06-10T00:17:31.794 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.793+0000 7f62f17aa700 1 -- 172.21.15.193:0/508451249 shutdown_connections 2024-06-10T00:17:31.794 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.793+0000 7f62f17aa700 1 --2- 172.21.15.193:0/508451249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ec101120 0x7f62d4005610 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:31.794 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.793+0000 7f62f17aa700 1 -- 172.21.15.193:0/508451249 >> 172.21.15.193:0/508451249 conn(0x7f62ec0fc9d0 msgr2=0x7f62ec0fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:31.794 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.793+0000 7f62f17aa700 1 -- 172.21.15.193:0/508451249 shutdown_connections 2024-06-10T00:17:31.794 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.793+0000 7f62f17aa700 1 -- 172.21.15.193:0/508451249 wait complete. 2024-06-10T00:17:31.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.794+0000 7f62f17aa700 1 Processor -- start 2024-06-10T00:17:31.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.794+0000 7f62f17aa700 1 -- start start 2024-06-10T00:17:31.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.794+0000 7f62f17aa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ec101120 0x7f62ec19d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:31.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62f17aa700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62ec19dc90 0x7f62ec197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:31.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62f17aa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec108f50 con 0x7f62ec101120 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62f17aa700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec108dd0 con 0x7f62ec19dc90 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62ea7fc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62ec19dc90 0x7f62ec197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62ea7fc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62ec19dc90 0x7f62ec197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:44644/0 (socket says 172.21.15.193:44644) 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62ea7fc700 1 -- 172.21.15.193:0/1369574966 learned_addr learned my addr 172.21.15.193:0/1369574966 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62ea7fc700 1 -- 172.21.15.193:0/1369574966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62ec19dc90 msgr2=0x7f62ec197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62ea7fc700 1 -- 172.21.15.193:0/1369574966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62ec19dc90 msgr2=0x7f62ec197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62ea7fc700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62ec19dc90 0x7f62ec197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62ea7fc700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62ec19dc90 0x7f62ec197910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62eaffd700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ec101120 0x7f62ec19d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62eaffd700 1 -- 172.21.15.193:0/1369574966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62ec19dc90 msgr2=0x7f62ec197910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62eaffd700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62ec19dc90 0x7f62ec197910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.795+0000 7f62eaffd700 1 -- 172.21.15.193:0/1369574966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec19e2d0 con 0x7f62ec101120 2024-06-10T00:17:31.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.796+0000 7f62eaffd700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ec101120 0x7f62ec19d750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f62e0000c00 tx=0x7f62e0004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:31.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.796+0000 7f62d3fff700 1 -- 172.21.15.193:0/1369574966 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f62e00053b0 con 0x7f62ec101120 2024-06-10T00:17:31.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.796+0000 7f62d3fff700 1 -- 172.21.15.193:0/1369574966 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62e0005550 con 0x7f62ec101120 2024-06-10T00:17:31.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.796+0000 7f62f17aa700 1 -- 172.21.15.193:0/1369574966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec197e50 con 0x7f62ec101120 2024-06-10T00:17:31.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.796+0000 7f62d3fff700 1 -- 172.21.15.193:0/1369574966 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f62e0011670 con 0x7f62ec101120 2024-06-10T00:17:31.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.796+0000 7f62f17aa700 1 -- 172.21.15.193:0/1369574966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec1983b0 con 0x7f62ec101120 2024-06-10T00:17:31.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.797+0000 7f62d3fff700 1 -- 172.21.15.193:0/1369574966 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f62e00118d0 con 0x7f62ec101120 2024-06-10T00:17:31.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.797+0000 7f62f17aa700 1 -- 172.21.15.193:0/1369574966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ec102600 con 0x7f62ec101120 2024-06-10T00:17:31.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.797+0000 7f62d3fff700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62d8038eb0 0x7f62d803b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:31.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.797+0000 7f62d3fff700 1 -- 172.21.15.193:0/1369574966 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f62e004e840 con 0x7f62ec101120 2024-06-10T00:17:31.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.797+0000 7f62ea7fc700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62d8038eb0 0x7f62d803b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:31.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.798+0000 7f62ea7fc700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62d8038eb0 0x7f62d803b370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009180 tx=0x7f62dc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:31.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:31.803+0000 7f62d3fff700 1 -- 172.21.15.193:0/1369574966 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f62e0022070 con 0x7f62ec101120 2024-06-10T00:17:32.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.083+0000 7f62f17aa700 1 -- 172.21.15.193:0/1369574966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f62ec198e40 con 0x7f62ec101120 2024-06-10T00:17:32.084 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:17:32.084 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.084 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: mon.a calling monitor election 2024-06-10T00:17:32.084 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.084 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:32.084 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.084 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: mon.c calling monitor election 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: monmap epoch 2 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: last_changed 2024-06-10T00:17:26.660781+0000 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: created 2024-06-10T00:15:39.680682+0000 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: min_mon_release 19 (squid) 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: election_strategy: 1 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-06-10T00:17:32.085 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: fsmap 2024-06-10T00:17:32.086 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: osdmap e4: 0 total, 0 up, 0 in 2024-06-10T00:17:32.086 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: mgrmap e13: x(active, since 49s) 2024-06-10T00:17:32.086 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: overall HEALTH_OK 2024-06-10T00:17:32.086 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:32.086 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:32.086 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:32.086 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:32.086 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:32.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.084+0000 7f62d3fff700 1 -- 172.21.15.193:0/1369574966 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f62e0019bf0 con 0x7f62ec101120 2024-06-10T00:17:32.087 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:17:32.087 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","modified":"2024-06-10T00:17:26.660781Z","created":"2024-06-10T00:15:39.680682Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-10T00:17:32.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.086+0000 7f62f17aa700 1 -- 172.21.15.193:0/1369574966 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62d8038eb0 msgr2=0x7f62d803b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:32.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.086+0000 7f62f17aa700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62d8038eb0 0x7f62d803b370 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f62dc009180 tx=0x7f62dc008040 comp rx=0 tx=0).stop 2024-06-10T00:17:32.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.086+0000 7f62f17aa700 1 -- 172.21.15.193:0/1369574966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ec101120 msgr2=0x7f62ec19d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:32.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.086+0000 7f62f17aa700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ec101120 0x7f62ec19d750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f62e0000c00 tx=0x7f62e0004d60 comp rx=0 tx=0).stop 2024-06-10T00:17:32.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.086+0000 7f62f17aa700 1 -- 172.21.15.193:0/1369574966 shutdown_connections 2024-06-10T00:17:32.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.086+0000 7f62f17aa700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62d8038eb0 0x7f62d803b370 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:32.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.086+0000 7f62f17aa700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ec101120 0x7f62ec19d750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:32.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.086+0000 7f62f17aa700 1 --2- 172.21.15.193:0/1369574966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62ec19dc90 0x7f62ec197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:32.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.086+0000 7f62f17aa700 1 -- 172.21.15.193:0/1369574966 >> 172.21.15.193:0/1369574966 conn(0x7f62ec0fc9d0 msgr2=0x7f62ec0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:32.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.086+0000 7f62f17aa700 1 -- 172.21.15.193:0/1369574966 shutdown_connections 2024-06-10T00:17:32.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:32.086+0000 7f62f17aa700 1 -- 172.21.15.193:0/1369574966 wait complete. 2024-06-10T00:17:32.088 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-06-10T00:17:32.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:17:32.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: mon.a calling monitor election 2024-06-10T00:17:32.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:32.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: mon.c calling monitor election 2024-06-10T00:17:32.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:32.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:32.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-10T00:17:32.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: monmap epoch 2 2024-06-10T00:17:32.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:32.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: last_changed 2024-06-10T00:17:26.660781+0000 2024-06-10T00:17:32.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: created 2024-06-10T00:15:39.680682+0000 2024-06-10T00:17:32.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: min_mon_release 19 (squid) 2024-06-10T00:17:32.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: election_strategy: 1 2024-06-10T00:17:32.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-06-10T00:17:32.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-06-10T00:17:32.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: fsmap 2024-06-10T00:17:32.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: osdmap e4: 0 total, 0 up, 0 in 2024-06-10T00:17:32.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: mgrmap e13: x(active, since 49s) 2024-06-10T00:17:32.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: overall HEALTH_OK 2024-06-10T00:17:32.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:32.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:32.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:32.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:32.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:33.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:32 smithi193 ceph-mon[36716]: Deploying daemon mon.b on smithi135 2024-06-10T00:17:33.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:32 smithi193 ceph-mon[36716]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:33.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:32 smithi193 ceph-mon[36716]: from='client.? 172.21.15.193:0/1369574966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-10T00:17:33.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:32 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:33.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:32 smithi121 ceph-mon[32378]: Deploying daemon mon.b on smithi135 2024-06-10T00:17:33.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:32 smithi121 ceph-mon[32378]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:33.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:32 smithi121 ceph-mon[32378]: from='client.? 172.21.15.193:0/1369574966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-10T00:17:33.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:32 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:33.673 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-10T00:17:33.674 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-10T00:17:33.914 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:17:35.265 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.263+0000 7f9dc47a8700 1 -- 172.21.15.193:0/868377791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dbc102200 msgr2=0x7f9dbc1025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:35.265 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.263+0000 7f9dc47a8700 1 --2- 172.21.15.193:0/868377791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dbc102200 0x7f9dbc1025e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9db8009910 tx=0x7f9db8009c20 comp rx=0 tx=0).stop 2024-06-10T00:17:35.265 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.263+0000 7f9dc47a8700 1 -- 172.21.15.193:0/868377791 shutdown_connections 2024-06-10T00:17:35.265 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.263+0000 7f9dc47a8700 1 --2- 172.21.15.193:0/868377791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dbc102200 0x7f9dbc1025e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.263+0000 7f9dc47a8700 1 -- 172.21.15.193:0/868377791 >> 172.21.15.193:0/868377791 conn(0x7f9dbc0fda90 msgr2=0x7f9dbc0ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.263+0000 7f9dc47a8700 1 -- 172.21.15.193:0/868377791 shutdown_connections 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.263+0000 7f9dc47a8700 1 -- 172.21.15.193:0/868377791 wait complete. 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc47a8700 1 Processor -- start 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc47a8700 1 -- start start 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc47a8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dbc102200 0x7f9dbc074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc47a8700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc074cd0 0x7f9dbc075150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc47a8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9dbc10c5d0 con 0x7f9dbc102200 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc47a8700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9dbc10c450 con 0x7f9dbc074cd0 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc1d43700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc074cd0 0x7f9dbc075150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc1d43700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc074cd0 0x7f9dbc075150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:44652/0 (socket says 172.21.15.193:44652) 2024-06-10T00:17:35.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc1d43700 1 -- 172.21.15.193:0/2146437832 learned_addr learned my addr 172.21.15.193:0/2146437832 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc1d43700 1 -- 172.21.15.193:0/2146437832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc074cd0 msgr2=0x7f9dbc075150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc1d43700 1 -- 172.21.15.193:0/2146437832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc074cd0 msgr2=0x7f9dbc075150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc1d43700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc074cd0 0x7f9dbc075150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc1d43700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc074cd0 0x7f9dbc075150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.264+0000 7f9dc2544700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dbc102200 0x7f9dbc074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.265+0000 7f9dc2544700 1 -- 172.21.15.193:0/2146437832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc074cd0 msgr2=0x7f9dbc075150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.265+0000 7f9dc2544700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc074cd0 0x7f9dbc075150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.265+0000 7f9dc2544700 1 -- 172.21.15.193:0/2146437832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dbc0766b0 con 0x7f9dbc102200 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.265+0000 7f9dc2544700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dbc102200 0x7f9dbc074790 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9db8014000 tx=0x7f9db8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.265+0000 7f9db37fe700 1 -- 172.21.15.193:0/2146437832 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9db8005dc0 con 0x7f9dbc102200 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.265+0000 7f9db37fe700 1 -- 172.21.15.193:0/2146437832 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9db800be40 con 0x7f9dbc102200 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.265+0000 7f9db37fe700 1 -- 172.21.15.193:0/2146437832 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f9db800f460 con 0x7f9dbc102200 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.265+0000 7f9dc47a8700 1 -- 172.21.15.193:0/2146437832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dbc1a3f20 con 0x7f9dbc102200 2024-06-10T00:17:35.267 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.265+0000 7f9dc47a8700 1 -- 172.21.15.193:0/2146437832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dbc1a43d0 con 0x7f9dbc102200 2024-06-10T00:17:35.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.266+0000 7f9db37fe700 1 -- 172.21.15.193:0/2146437832 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f9db802a070 con 0x7f9dbc102200 2024-06-10T00:17:35.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.266+0000 7f9dc47a8700 1 -- 172.21.15.193:0/2146437832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dbc1036e0 con 0x7f9dbc102200 2024-06-10T00:17:35.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.266+0000 7f9db37fe700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f9da8038e90 0x7f9da803b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:35.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.266+0000 7f9db37fe700 1 -- 172.21.15.193:0/2146437832 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9db801e030 con 0x7f9dbc102200 2024-06-10T00:17:35.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.266+0000 7f9dc1d43700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f9da8038e90 0x7f9da803b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:35.269 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.267+0000 7f9dc1d43700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f9da8038e90 0x7f9da803b350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9dac009180 tx=0x7f9dac008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:35.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.272+0000 7f9db37fe700 1 -- 172.21.15.193:0/2146437832 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9db8013070 con 0x7f9dbc102200 2024-06-10T00:17:35.335 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.335+0000 7f9db37fe700 1 -- 172.21.15.193:0/2146437832 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db802be20 con 0x7f9dbc102200 2024-06-10T00:17:35.335 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:35 smithi193 ceph-mon[36716]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:35.351 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:35 smithi135 ceph-mon[36424]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-06-10T00:17:35.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:35 smithi121 ceph-mon[32378]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:35.546 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:35.546+0000 7f9dc47a8700 1 -- 172.21.15.193:0/2146437832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9dbc1a4720 con 0x7f9dbc102200 2024-06-10T00:17:35.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:35 smithi135 bash[36405]: 37fc9b55c2e6556e87d6494e2489de8ccaec4b01e63e193a5d13496a1258ec74 2024-06-10T00:17:35.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:35 smithi135 systemd[1]: Started Ceph mon.b for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:17:40.344 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.344+0000 7f9db37fe700 1 -- 172.21.15.193:0/2146437832 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7f9db8031070 con 0x7f9dbc102200 2024-06-10T00:17:40.344 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-06-10T00:17:40.344 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":3,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","modified":"2024-06-10T00:17:35.334701Z","created":"2024-06-10T00:15:39.680682Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:3300","nonce":0},{"type":"v1","addr":"172.21.15.135:6789","nonce":0}]},"addr":"172.21.15.135:6789/0","public_addr":"172.21.15.135:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-10T00:17:40.346 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.347+0000 7f9dc47a8700 1 -- 172.21.15.193:0/2146437832 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f9da8038e90 msgr2=0x7f9da803b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:40.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.347+0000 7f9dc47a8700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f9da8038e90 0x7f9da803b350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9dac009180 tx=0x7f9dac008040 comp rx=0 tx=0).stop 2024-06-10T00:17:40.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.347+0000 7f9dc47a8700 1 -- 172.21.15.193:0/2146437832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dbc102200 msgr2=0x7f9dbc074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:40.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.347+0000 7f9dc47a8700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dbc102200 0x7f9dbc074790 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9db8014000 tx=0x7f9db8005030 comp rx=0 tx=0).stop 2024-06-10T00:17:40.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.347+0000 7f9dc47a8700 1 -- 172.21.15.193:0/2146437832 shutdown_connections 2024-06-10T00:17:40.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.347+0000 7f9dc47a8700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f9da8038e90 0x7f9da803b350 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:40.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.347+0000 7f9dc47a8700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dbc102200 0x7f9dbc074790 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:40.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.347+0000 7f9dc47a8700 1 --2- 172.21.15.193:0/2146437832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dbc074cd0 0x7f9dbc075150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:40.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.347+0000 7f9dc47a8700 1 -- 172.21.15.193:0/2146437832 >> 172.21.15.193:0/2146437832 conn(0x7f9dbc0fda90 msgr2=0x7f9dbc103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:40.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.347+0000 7f9dc47a8700 1 -- 172.21.15.193:0/2146437832 shutdown_connections 2024-06-10T00:17:40.347 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:40.347+0000 7f9dc47a8700 1 -- 172.21.15.193:0/2146437832 wait complete. 2024-06-10T00:17:40.348 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 3 2024-06-10T00:17:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:17:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: mon.c calling monitor election 2024-06-10T00:17:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: mon.a calling monitor election 2024-06-10T00:17:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: from='client.? 172.21.15.193:0/2146437832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-10T00:17:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: monmap epoch 3 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: last_changed 2024-06-10T00:17:35.334701+0000 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: created 2024-06-10T00:15:39.680682+0000 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: min_mon_release 19 (squid) 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: election_strategy: 1 2024-06-10T00:17:40.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-06-10T00:17:40.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-06-10T00:17:40.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: 2: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-06-10T00:17:40.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: fsmap 2024-06-10T00:17:40.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: osdmap e4: 0 total, 0 up, 0 in 2024-06-10T00:17:40.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: mgrmap e13: x(active, since 58s) 2024-06-10T00:17:40.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: overall HEALTH_OK 2024-06-10T00:17:40.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:40.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:17:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: mon.c calling monitor election 2024-06-10T00:17:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: mon.a calling monitor election 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: from='client.? 172.21.15.193:0/2146437832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-10T00:17:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: monmap epoch 3 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: last_changed 2024-06-10T00:17:35.334701+0000 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: created 2024-06-10T00:15:39.680682+0000 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: min_mon_release 19 (squid) 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: election_strategy: 1 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: 2: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: fsmap 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: osdmap e4: 0 total, 0 up, 0 in 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: mgrmap e13: x(active, since 58s) 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: overall HEALTH_OK 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:40.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:40.928 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-10T00:17:40.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-06-10T00:17:41.175 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:17:41.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:41 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:41.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:41 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:41.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:41 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:41.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:41 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:41.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:41 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:41 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:41 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:41 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:42.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:17:42.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: mon.c calling monitor election 2024-06-10T00:17:42.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: mon.a calling monitor election 2024-06-10T00:17:42.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:42.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:17:42.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='client.? 172.21.15.193:0/2146437832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-10T00:17:42.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:42.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:42.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:42.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:42.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:42.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:42.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:42.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:42.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-10T00:17:42.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: monmap epoch 3 2024-06-10T00:17:42.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:42.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: last_changed 2024-06-10T00:17:35.334701+0000 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: created 2024-06-10T00:15:39.680682+0000 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: min_mon_release 19 (squid) 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: election_strategy: 1 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: 2: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: fsmap 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: osdmap e4: 0 total, 0 up, 0 in 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: mgrmap e13: x(active, since 58s) 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: overall HEALTH_OK 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:42.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:42.761 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:42.761 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:42.761 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:42.761 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:42.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.854+0000 7fc66609c700 1 -- 172.21.15.121:0/4143418551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc660103060 msgr2=0x7fc660103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:42.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.854+0000 7fc66609c700 1 --2- 172.21.15.121:0/4143418551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc660103060 0x7fc660103440 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc654009910 tx=0x7fc654009c20 comp rx=0 tx=0).stop 2024-06-10T00:17:42.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.854+0000 7fc66609c700 1 -- 172.21.15.121:0/4143418551 shutdown_connections 2024-06-10T00:17:42.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.854+0000 7fc66609c700 1 --2- 172.21.15.121:0/4143418551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc660103060 0x7fc660103440 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:42.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.854+0000 7fc66609c700 1 -- 172.21.15.121:0/4143418551 >> 172.21.15.121:0/4143418551 conn(0x7fc6600fc8d0 msgr2=0x7fc6600fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.854+0000 7fc66609c700 1 -- 172.21.15.121:0/4143418551 shutdown_connections 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.854+0000 7fc66609c700 1 -- 172.21.15.121:0/4143418551 wait complete. 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.855+0000 7fc66609c700 1 Processor -- start 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.855+0000 7fc66609c700 1 -- start start 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.855+0000 7fc66609c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc660103060 0x7fc66019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.855+0000 7fc66609c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66019de60 0x7fc660197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.855+0000 7fc66609c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc660198020 0x7fc6601984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.855+0000 7fc66609c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc66010c500 con 0x7fc660198020 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.855+0000 7fc66609c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc66010c380 con 0x7fc66019de60 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.855+0000 7fc66609c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc66010c680 con 0x7fc660103060 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.855+0000 7fc65ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc660198020 0x7fc6601984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:42.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.855+0000 7fc65ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc660198020 0x7fc6601984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43382/0 (socket says 172.21.15.121:43382) 2024-06-10T00:17:42.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.855+0000 7fc65ffff700 1 -- 172.21.15.121:0/2517688651 learned_addr learned my addr 172.21.15.121:0/2517688651 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:17:42.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65effd700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66019de60 0x7fc660197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:42.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65f7fe700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc660103060 0x7fc66019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:42.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65ffff700 1 -- 172.21.15.121:0/2517688651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc660103060 msgr2=0x7fc66019d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:42.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65ffff700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc660103060 0x7fc66019d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:42.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65ffff700 1 -- 172.21.15.121:0/2517688651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66019de60 msgr2=0x7fc660197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:42.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65ffff700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66019de60 0x7fc660197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:42.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65ffff700 1 -- 172.21.15.121:0/2517688651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6601a47f0 con 0x7fc660198020 2024-06-10T00:17:42.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65effd700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66019de60 0x7fc660197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:17:42.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65ffff700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc660198020 0x7fc6601984d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc65800ba50 tx=0x7fc65800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:42.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65cff9700 1 -- 172.21.15.121:0/2517688651 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658019070 con 0x7fc660198020 2024-06-10T00:17:42.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65cff9700 1 -- 172.21.15.121:0/2517688651 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc658014440 con 0x7fc660198020 2024-06-10T00:17:42.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc66609c700 1 -- 172.21.15.121:0/2517688651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6601a4a20 con 0x7fc660198020 2024-06-10T00:17:42.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc65cff9700 1 -- 172.21.15.121:0/2517688651 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658007590 con 0x7fc660198020 2024-06-10T00:17:42.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.856+0000 7fc66609c700 1 -- 172.21.15.121:0/2517688651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6601a4f10 con 0x7fc660198020 2024-06-10T00:17:42.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.857+0000 7fc65cff9700 1 -- 172.21.15.121:0/2517688651 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fc65801c070 con 0x7fc660198020 2024-06-10T00:17:42.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.857+0000 7fc66609c700 1 -- 172.21.15.121:0/2517688651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc660063af0 con 0x7fc660198020 2024-06-10T00:17:42.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.857+0000 7fc65cff9700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc648039580 0x7fc64803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:42.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.858+0000 7fc65cff9700 1 -- 172.21.15.121:0/2517688651 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc65804c8f0 con 0x7fc660198020 2024-06-10T00:17:42.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.858+0000 7fc65f7fe700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc648039580 0x7fc64803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:42.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.858+0000 7fc65f7fe700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc648039580 0x7fc64803ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc654009750 tx=0x7fc654005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:42.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:42.867+0000 7fc65cff9700 1 -- 172.21.15.121:0/2517688651 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc658014910 con 0x7fc660198020 2024-06-10T00:17:43.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.074+0000 7fc66609c700 1 -- 172.21.15.121:0/2517688651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fc6601a51a0 con 0x7fc660198020 2024-06-10T00:17:43.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.074+0000 7fc65cff9700 1 -- 172.21.15.121:0/2517688651 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7fc65800e090 con 0x7fc660198020 2024-06-10T00:17:43.078 INFO:teuthology.orchestra.run.smithi121.stdout:# minimal ceph.conf for 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:43.078 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-06-10T00:17:43.078 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:43.078 INFO:teuthology.orchestra.run.smithi121.stdout: mon_host = [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-06-10T00:17:43.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.076+0000 7fc66609c700 1 -- 172.21.15.121:0/2517688651 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc648039580 msgr2=0x7fc64803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:43.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.076+0000 7fc66609c700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc648039580 0x7fc64803ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc654009750 tx=0x7fc654005fb0 comp rx=0 tx=0).stop 2024-06-10T00:17:43.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.076+0000 7fc66609c700 1 -- 172.21.15.121:0/2517688651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc660198020 msgr2=0x7fc6601984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:43.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.076+0000 7fc66609c700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc660198020 0x7fc6601984d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc65800ba50 tx=0x7fc65800be10 comp rx=0 tx=0).stop 2024-06-10T00:17:43.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.076+0000 7fc66609c700 1 -- 172.21.15.121:0/2517688651 shutdown_connections 2024-06-10T00:17:43.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.076+0000 7fc66609c700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc648039580 0x7fc64803ba40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:43.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.076+0000 7fc66609c700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc660198020 0x7fc6601984d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:43.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.076+0000 7fc66609c700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc660103060 0x7fc66019d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:43.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.076+0000 7fc66609c700 1 --2- 172.21.15.121:0/2517688651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66019de60 0x7fc660197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:43.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.076+0000 7fc66609c700 1 -- 172.21.15.121:0/2517688651 >> 172.21.15.121:0/2517688651 conn(0x7fc6600fc8d0 msgr2=0x7fc660105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:43.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.077+0000 7fc66609c700 1 -- 172.21.15.121:0/2517688651 shutdown_connections 2024-06-10T00:17:43.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:43.077+0000 7fc66609c700 1 -- 172.21.15.121:0/2517688651 wait complete. 2024-06-10T00:17:43.532 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: mon.b calling monitor election 2024-06-10T00:17:43.532 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: mon.b calling monitor election 2024-06-10T00:17:43.532 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: mon.c calling monitor election 2024-06-10T00:17:43.532 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: mon.a calling monitor election 2024-06-10T00:17:43.532 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-10T00:17:43.532 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: monmap epoch 3 2024-06-10T00:17:43.532 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:43.532 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: last_changed 2024-06-10T00:17:35.334701+0000 2024-06-10T00:17:43.532 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: created 2024-06-10T00:15:39.680682+0000 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: min_mon_release 19 (squid) 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: election_strategy: 1 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: 2: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: fsmap 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: osdmap e4: 0 total, 0 up, 0 in 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: mgrmap e13: x(active, since 60s) 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: overall HEALTH_OK 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.533 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: Reconfiguring mon.a (unknown last config time)... 2024-06-10T00:17:43.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:43.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-10T00:17:43.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:43.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: Reconfiguring daemon mon.a on smithi121 2024-06-10T00:17:43.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2517688651' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:43.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: mon.b calling monitor election 2024-06-10T00:17:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: mon.b calling monitor election 2024-06-10T00:17:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: mon.c calling monitor election 2024-06-10T00:17:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: mon.a calling monitor election 2024-06-10T00:17:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-10T00:17:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: monmap epoch 3 2024-06-10T00:17:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: last_changed 2024-06-10T00:17:35.334701+0000 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: created 2024-06-10T00:15:39.680682+0000 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: min_mon_release 19 (squid) 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: election_strategy: 1 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: 2: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: fsmap 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: osdmap e4: 0 total, 0 up, 0 in 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: mgrmap e13: x(active, since 60s) 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: overall HEALTH_OK 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: Reconfiguring mon.a (unknown last config time)... 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: Reconfiguring daemon mon.a on smithi121 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2517688651' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:43.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: mon.b calling monitor election 2024-06-10T00:17:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: mon.b calling monitor election 2024-06-10T00:17:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: mon.c calling monitor election 2024-06-10T00:17:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: mon.a calling monitor election 2024-06-10T00:17:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: monmap epoch 3 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: last_changed 2024-06-10T00:17:35.334701+0000 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: created 2024-06-10T00:15:39.680682+0000 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: min_mon_release 19 (squid) 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: election_strategy: 1 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.a 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: 2: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: fsmap 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: osdmap e4: 0 total, 0 up, 0 in 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: mgrmap e13: x(active, since 60s) 2024-06-10T00:17:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: overall HEALTH_OK 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: Reconfiguring mon.a (unknown last config time)... 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: Reconfiguring daemon mon.a on smithi121 2024-06-10T00:17:43.760 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2517688651' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:43.761 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:17:43.824 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-10T00:17:43.824 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:17:43.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.conf 2024-06-10T00:17:43.852 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:17:43.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:17:43.917 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-06-10T00:17:43.917 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.conf 2024-06-10T00:17:43.950 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-06-10T00:17:43.950 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:17:44.017 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-06-10T00:17:44.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2024-06-10T00:17:44.048 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-06-10T00:17:44.048 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:17:44.117 INFO:tasks.cephadm:Adding mgr.x on smithi121 2024-06-10T00:17:44.118 INFO:tasks.cephadm:Adding mgr.y on smithi135 2024-06-10T00:17:44.118 INFO:tasks.cephadm:Adding mgr.z on smithi193 2024-06-10T00:17:44.118 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch apply mgr '3;smithi121=x;smithi135=y;smithi193=z' 2024-06-10T00:17:44.389 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:17:44.613 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:44 smithi193 ceph-mon[36716]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:44.651 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:44 smithi121 ceph-mon[32378]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:44.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:44 smithi135 ceph-mon[36424]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:45.822 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.822+0000 7fb0a5102700 1 -- 172.21.15.193:0/2318752545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0a0100f10 msgr2=0x7fb0a0101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:45.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.822+0000 7fb0a5102700 1 --2- 172.21.15.193:0/2318752545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0a0100f10 0x7fb0a0101330 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb09800b0a0 tx=0x7fb09800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:17:45.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.822+0000 7fb0a5102700 1 -- 172.21.15.193:0/2318752545 shutdown_connections 2024-06-10T00:17:45.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.822+0000 7fb0a5102700 1 --2- 172.21.15.193:0/2318752545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0a0100f10 0x7fb0a0101330 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:45.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.822+0000 7fb0a5102700 1 -- 172.21.15.193:0/2318752545 >> 172.21.15.193:0/2318752545 conn(0x7fb0a00fc4d0 msgr2=0x7fb0a00fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:45.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.822+0000 7fb0a5102700 1 -- 172.21.15.193:0/2318752545 shutdown_connections 2024-06-10T00:17:45.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.822+0000 7fb0a5102700 1 -- 172.21.15.193:0/2318752545 wait complete. 2024-06-10T00:17:45.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.823+0000 7fb0a5102700 1 Processor -- start 2024-06-10T00:17:45.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.823+0000 7fb0a5102700 1 -- start start 2024-06-10T00:17:45.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.823+0000 7fb0a5102700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0a0100f10 0x7fb0a0197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:45.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.823+0000 7fb0a5102700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0a0197d50 0x7fb0a019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:45.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.823+0000 7fb0a5102700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0a01981d0 0x7fb0a019f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:45.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.823+0000 7fb0a5102700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a0106ed0 con 0x7fb0a0197d50 2024-06-10T00:17:45.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb0a5102700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a0106d50 con 0x7fb0a01981d0 2024-06-10T00:17:45.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb0a5102700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a0107050 con 0x7fb0a0100f10 2024-06-10T00:17:45.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb09ed9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0a0100f10 0x7fb0a0197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:45.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb09ed9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0a0100f10 0x7fb0a0197810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:39242/0 (socket says 172.21.15.193:39242) 2024-06-10T00:17:45.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb09e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0a0197d50 0x7fb0a019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb09ed9d700 1 -- 172.21.15.193:0/2741915234 learned_addr learned my addr 172.21.15.193:0/2741915234 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb09f59e700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0a01981d0 0x7fb0a019f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb09ed9d700 1 -- 172.21.15.193:0/2741915234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0a01981d0 msgr2=0x7fb0a019f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb09ed9d700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0a01981d0 0x7fb0a019f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb09ed9d700 1 -- 172.21.15.193:0/2741915234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0a0197d50 msgr2=0x7fb0a019cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb09ed9d700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0a0197d50 0x7fb0a019cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb09ed9d700 1 -- 172.21.15.193:0/2741915234 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a019fc20 con 0x7fb0a0100f10 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb09ed9d700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0a0100f10 0x7fb0a0197810 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb098015040 tx=0x7fb098009740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb097fff700 1 -- 172.21.15.193:0/2741915234 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb098017040 con 0x7fb0a0100f10 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb097fff700 1 -- 172.21.15.193:0/2741915234 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb09800bd60 con 0x7fb0a0100f10 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb0a5102700 1 -- 172.21.15.193:0/2741915234 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0a019feb0 con 0x7fb0a0100f10 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb097fff700 1 -- 172.21.15.193:0/2741915234 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0980128a0 con 0x7fb0a0100f10 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.824+0000 7fb0a5102700 1 -- 172.21.15.193:0/2741915234 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0a01a0380 con 0x7fb0a0100f10 2024-06-10T00:17:45.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.825+0000 7fb097fff700 1 -- 172.21.15.193:0/2741915234 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fb098012ae0 con 0x7fb0a0100f10 2024-06-10T00:17:45.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.825+0000 7fb0a5102700 1 -- 172.21.15.193:0/2741915234 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0a0063af0 con 0x7fb0a0100f10 2024-06-10T00:17:45.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.825+0000 7fb097fff700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb08c039580 0x7fb08c03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:45.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.825+0000 7fb097fff700 1 -- 172.21.15.193:0/2741915234 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb09800f020 con 0x7fb0a0100f10 2024-06-10T00:17:45.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.826+0000 7fb09e59c700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb08c039580 0x7fb08c03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:45.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.826+0000 7fb09e59c700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb08c039580 0x7fb08c03ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb088009ad0 tx=0x7fb088009210 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:45.830 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:45.831+0000 7fb097fff700 1 -- 172.21.15.193:0/2741915234 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb098029020 con 0x7fb0a0100f10 2024-06-10T00:17:45.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:45.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:45.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:45 smithi193 ceph-mon[36716]: Reconfiguring mon.b (monmap changed)... 2024-06-10T00:17:45.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:45.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-10T00:17:45.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:45.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:45 smithi193 ceph-mon[36716]: Reconfiguring daemon mon.b on smithi135 2024-06-10T00:17:46.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:46.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:46.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:45 smithi135 ceph-mon[36424]: Reconfiguring mon.b (monmap changed)... 2024-06-10T00:17:46.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:46.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-10T00:17:46.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:46.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:45 smithi135 ceph-mon[36424]: Reconfiguring daemon mon.b on smithi135 2024-06-10T00:17:46.039 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.039+0000 7fb0a5102700 1 -- 172.21.15.193:0/2741915234 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi121=x;smithi135=y;smithi193=z", "target": ["mon-mgr", ""]}) -- 0x7fb0a0062960 con 0x7fb08c039580 2024-06-10T00:17:46.047 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.048+0000 7fb097fff700 1 -- 172.21.15.193:0/2741915234 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb0a0062960 con 0x7fb08c039580 2024-06-10T00:17:46.047 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mgr update... 2024-06-10T00:17:46.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 -- 172.21.15.193:0/2741915234 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb08c039580 msgr2=0x7fb08c03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:46.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb08c039580 0x7fb08c03ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb088009ad0 tx=0x7fb088009210 comp rx=0 tx=0).stop 2024-06-10T00:17:46.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 -- 172.21.15.193:0/2741915234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0a0100f10 msgr2=0x7fb0a0197810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:46.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0a0100f10 0x7fb0a0197810 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb098015040 tx=0x7fb098009740 comp rx=0 tx=0).stop 2024-06-10T00:17:46.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 -- 172.21.15.193:0/2741915234 shutdown_connections 2024-06-10T00:17:46.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb08c039580 0x7fb08c03ba40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:46.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0a01981d0 0x7fb0a019f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:46.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0a0100f10 0x7fb0a0197810 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:46.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 --2- 172.21.15.193:0/2741915234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0a0197d50 0x7fb0a019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:46.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 -- 172.21.15.193:0/2741915234 >> 172.21.15.193:0/2741915234 conn(0x7fb0a00fc4d0 msgr2=0x7fb0a006ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:46.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 -- 172.21.15.193:0/2741915234 shutdown_connections 2024-06-10T00:17:46.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:17:46.050+0000 7fb0a5102700 1 -- 172.21.15.193:0/2741915234 wait complete. 2024-06-10T00:17:46.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:46.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:46.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:45 smithi121 ceph-mon[32378]: Reconfiguring mon.b (monmap changed)... 2024-06-10T00:17:46.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:46.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-10T00:17:46.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:46.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:45 smithi121 ceph-mon[32378]: Reconfiguring daemon mon.b on smithi135 2024-06-10T00:17:46.651 DEBUG:teuthology.orchestra.run.smithi135:mgr.y> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.y.service 2024-06-10T00:17:46.653 DEBUG:teuthology.orchestra.run.smithi193:mgr.z> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.z.service 2024-06-10T00:17:46.656 INFO:tasks.cephadm:Deploying OSDs... 2024-06-10T00:17:46.656 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:17:46.656 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-06-10T00:17:46.676 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-10T00:17:46.677 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-06-10T00:17:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-10T00:17:46.735 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:46.735 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 674 Links: 1 2024-06-10T00:17:46.736 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:46.736 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:46.736 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-06-10 00:16:53.481146538 +0000 2024-06-10T00:17:46.736 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-06-10 00:11:45.685886470 +0000 2024-06-10T00:17:46.736 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-06-10 00:11:45.685886470 +0000 2024-06-10T00:17:46.736 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-06-10 00:11:45.685886470 +0000 2024-06-10T00:17:46.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-10T00:17:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-06-10T00:17:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-06-10T00:17:46.804 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000232372 s, 2.2 MB/s 2024-06-10T00:17:46.806 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-10T00:17:46.864 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-06-10T00:17:46.922 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-10T00:17:46.922 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:46.922 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 694 Links: 1 2024-06-10T00:17:46.922 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:46.922 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:46.922 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-06-10 00:16:53.506146218 +0000 2024-06-10T00:17:46.922 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-06-10 00:11:46.005882278 +0000 2024-06-10T00:17:46.922 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-06-10 00:11:46.005882278 +0000 2024-06-10T00:17:46.922 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-06-10 00:11:46.005882278 +0000 2024-06-10T00:17:46.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-10T00:17:46.987 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-06-10T00:17:46.987 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-06-10T00:17:46.988 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000220552 s, 2.3 MB/s 2024-06-10T00:17:46.989 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-10T00:17:47.046 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-06-10T00:17:47.104 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-10T00:17:47.104 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:47.104 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 714 Links: 1 2024-06-10T00:17:47.104 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:47.104 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:47.104 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-06-10 00:16:53.530145911 +0000 2024-06-10T00:17:47.104 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-06-10 00:11:46.323878114 +0000 2024-06-10T00:17:47.104 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-06-10 00:11:46.323878114 +0000 2024-06-10T00:17:47.104 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-06-10 00:11:46.323878114 +0000 2024-06-10T00:17:47.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-10T00:17:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:47 smithi193 ceph-mon[36716]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:47 smithi193 ceph-mon[36716]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi121=x;smithi135=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:17:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:47 smithi193 ceph-mon[36716]: Saving service mgr spec with placement smithi121=x;smithi135=y;smithi193=z;count:3 2024-06-10T00:17:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:47 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:47 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:47 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:47 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:47 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-10T00:17:47.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:47 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:47.170 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:47 smithi121 ceph-mon[32378]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:47.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:47 smithi121 ceph-mon[32378]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi121=x;smithi135=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:17:47.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:47 smithi121 ceph-mon[32378]: Saving service mgr spec with placement smithi121=x;smithi135=y;smithi193=z;count:3 2024-06-10T00:17:47.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:47 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:47.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:47 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:47.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:47 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:47.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:47 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:47.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:47 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-10T00:17:47.171 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:47 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:47.172 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-06-10T00:17:47.172 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-06-10T00:17:47.172 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000201382 s, 2.5 MB/s 2024-06-10T00:17:47.172 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-10T00:17:47.229 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-06-10T00:17:47.286 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-10T00:17:47.286 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:47.286 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 731 Links: 1 2024-06-10T00:17:47.286 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:47.286 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:47.286 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-06-10 00:16:53.556145578 +0000 2024-06-10T00:17:47.286 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-06-10 00:11:46.656873751 +0000 2024-06-10T00:17:47.286 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-06-10 00:11:46.656873751 +0000 2024-06-10T00:17:47.286 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-06-10 00:11:46.656873751 +0000 2024-06-10T00:17:47.287 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-10T00:17:47.350 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-06-10T00:17:47.351 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-06-10T00:17:47.351 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000236722 s, 2.2 MB/s 2024-06-10T00:17:47.352 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-10T00:17:47.409 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-06-10T00:17:47.409 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2024-06-10T00:17:47.427 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-10T00:17:47.427 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_1 2024-06-10T00:17:47.485 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-10T00:17:47.485 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:47.486 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-06-10T00:17:47.486 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:47.486 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:47.486 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-06-10 00:17:34.840132974 +0000 2024-06-10T00:17:47.486 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-06-10 00:11:52.705811231 +0000 2024-06-10T00:17:47.486 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-06-10 00:11:52.705811231 +0000 2024-06-10T00:17:47.486 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-06-10 00:11:52.705811231 +0000 2024-06-10T00:17:47.486 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-10T00:17:47.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:47 smithi135 ceph-mon[36424]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:47 smithi135 ceph-mon[36424]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi121=x;smithi135=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:17:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:47 smithi135 ceph-mon[36424]: Saving service mgr spec with placement smithi121=x;smithi135=y;smithi193=z;count:3 2024-06-10T00:17:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:47 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:47 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:47 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:47 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-10T00:17:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:47 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-10T00:17:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:47 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:47.534 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-06-10T00:17:47.534 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-06-10T00:17:47.534 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000232615 s, 2.2 MB/s 2024-06-10T00:17:47.535 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-10T00:17:47.593 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_2 2024-06-10T00:17:47.650 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-10T00:17:47.650 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:47.650 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-10T00:17:47.650 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:47.650 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:47.650 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-06-10 00:17:34.841132952 +0000 2024-06-10T00:17:47.650 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-06-10 00:11:53.034803923 +0000 2024-06-10T00:17:47.650 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-06-10 00:11:53.034803923 +0000 2024-06-10T00:17:47.651 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-06-10 00:11:53.034803923 +0000 2024-06-10T00:17:47.651 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-10T00:17:47.715 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-06-10T00:17:47.715 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-06-10T00:17:47.715 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.00022072 s, 2.3 MB/s 2024-06-10T00:17:47.716 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-10T00:17:47.773 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_3 2024-06-10T00:17:47.830 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-10T00:17:47.830 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:47.830 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-06-10T00:17:47.830 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:47.830 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:47.830 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-06-10 00:17:34.841132952 +0000 2024-06-10T00:17:47.830 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-06-10 00:11:53.367796526 +0000 2024-06-10T00:17:47.830 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-06-10 00:11:53.368796504 +0000 2024-06-10T00:17:47.830 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-06-10 00:11:53.367796526 +0000 2024-06-10T00:17:47.831 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-10T00:17:47.893 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-06-10T00:17:47.893 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-06-10T00:17:47.893 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000271318 s, 1.9 MB/s 2024-06-10T00:17:47.894 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-10T00:17:47.951 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_4 2024-06-10T00:17:48.006 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-10T00:17:48.006 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:48.006 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-06-10T00:17:48.006 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:48.006 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:48.006 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-06-10 00:17:34.841132952 +0000 2024-06-10T00:17:48.006 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-06-10 00:11:53.701789107 +0000 2024-06-10T00:17:48.006 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-06-10 00:11:53.701789107 +0000 2024-06-10T00:17:48.006 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-06-10 00:11:53.701789107 +0000 2024-06-10T00:17:48.006 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-10T00:17:48.068 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-06-10T00:17:48.068 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-06-10T00:17:48.068 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000206041 s, 2.5 MB/s 2024-06-10T00:17:48.070 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-10T00:17:48.125 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-06-10T00:17:48.126 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-06-10T00:17:48.144 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-10T00:17:48.144 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-06-10T00:17:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-10T00:17:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:48.202 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-06-10T00:17:48.202 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:48.202 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:48.202 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-06-10 00:17:25.600647811 +0000 2024-06-10T00:17:48.202 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-06-10 00:11:53.857833261 +0000 2024-06-10T00:17:48.202 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-06-10 00:11:53.857833261 +0000 2024-06-10T00:17:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-06-10 00:11:53.857833261 +0000 2024-06-10T00:17:48.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-10T00:17:48.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:48 smithi135 ceph-mon[36424]: Reconfiguring mon.c (monmap changed)... 2024-06-10T00:17:48.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:48 smithi135 ceph-mon[36424]: Reconfiguring daemon mon.c on smithi193 2024-06-10T00:17:48.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:48 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:48.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:48 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:48.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:48 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:48.268 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-06-10T00:17:48.268 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-06-10T00:17:48.268 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000283966 s, 1.8 MB/s 2024-06-10T00:17:48.269 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-10T00:17:48.327 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-06-10T00:17:48.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:48 smithi193 ceph-mon[36716]: Reconfiguring mon.c (monmap changed)... 2024-06-10T00:17:48.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:48 smithi193 ceph-mon[36716]: Reconfiguring daemon mon.c on smithi193 2024-06-10T00:17:48.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:48 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:48.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:48 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:48.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:48 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:48.384 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-10T00:17:48.384 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:48.384 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-06-10T00:17:48.384 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:48.384 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:48.384 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-06-10 00:17:25.601647796 +0000 2024-06-10T00:17:48.384 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-06-10 00:11:54.205827881 +0000 2024-06-10T00:17:48.384 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-06-10 00:11:54.205827881 +0000 2024-06-10T00:17:48.384 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-06-10 00:11:54.205827881 +0000 2024-06-10T00:17:48.385 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-10T00:17:48.446 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-06-10T00:17:48.446 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-06-10T00:17:48.447 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000206077 s, 2.5 MB/s 2024-06-10T00:17:48.448 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-10T00:17:48.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:48 smithi121 ceph-mon[32378]: Reconfiguring mon.c (monmap changed)... 2024-06-10T00:17:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:48 smithi121 ceph-mon[32378]: Reconfiguring daemon mon.c on smithi193 2024-06-10T00:17:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:48 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:48 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:48 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:48.504 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-06-10T00:17:48.559 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-10T00:17:48.559 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:48.560 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-06-10T00:17:48.560 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:48.560 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:48.560 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-06-10 00:17:25.601647796 +0000 2024-06-10T00:17:48.560 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-06-10 00:11:54.535822779 +0000 2024-06-10T00:17:48.560 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-06-10 00:11:54.535822779 +0000 2024-06-10T00:17:48.560 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-06-10 00:11:54.535822779 +0000 2024-06-10T00:17:48.560 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-10T00:17:48.622 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-06-10T00:17:48.622 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-06-10T00:17:48.622 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.00020638 s, 2.5 MB/s 2024-06-10T00:17:48.623 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-10T00:17:48.679 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-06-10T00:17:48.734 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-10T00:17:48.734 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-10T00:17:48.735 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-06-10T00:17:48.735 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:17:48.735 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-06-10T00:17:48.735 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-06-10 00:17:25.602647781 +0000 2024-06-10T00:17:48.735 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-06-10 00:11:54.876817507 +0000 2024-06-10T00:17:48.735 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-06-10 00:11:54.876817507 +0000 2024-06-10T00:17:48.735 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-06-10 00:11:54.876817507 +0000 2024-06-10T00:17:48.735 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-10T00:17:48.796 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-06-10T00:17:48.797 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-06-10T00:17:48.797 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000222295 s, 2.3 MB/s 2024-06-10T00:17:48.798 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-10T00:17:48.853 INFO:tasks.cephadm:Deploying osd.0 on smithi121 with /dev/vg_nvme/lv_4... 2024-06-10T00:17:48.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-10T00:17:49.085 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:17:49.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:49 smithi121 ceph-mon[32378]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:49.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:49.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:49.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:49.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-10T00:17:49.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-10T00:17:49.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-10T00:17:49.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:49.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:49 smithi193 ceph-mon[36716]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-10T00:17:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-10T00:17:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-10T00:17:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:49.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:49 smithi135 ceph-mon[36424]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-10T00:17:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-10T00:17:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-10T00:17:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:50.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:50 smithi193 ceph-mon[36716]: Deploying daemon mgr.z on smithi193 2024-06-10T00:17:50.325 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:50 smithi121 ceph-mon[32378]: Deploying daemon mgr.z on smithi193 2024-06-10T00:17:50.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:50 smithi135 ceph-mon[36424]: Deploying daemon mgr.z on smithi193 2024-06-10T00:17:50.842 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:50 smithi193 systemd[1]: Started Ceph mgr.z for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:17:50.977 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-10T00:17:50.978 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-10T00:17:50.978 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-06-10T00:17:50.978 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-06-10T00:17:50.978 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128521 s, 816 MB/s 2024-06-10T00:17:50.978 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-06-10T00:17:51.127 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-mon[36716]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:51.127 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:51.127 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:51.127 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:51.127 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-10T00:17:51.127 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-10T00:17:51.127 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-10T00:17:51.128 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:51.128 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:50 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:50.842+0000 7fb6412e3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-10T00:17:51.128 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:51.025+0000 7fb6412e3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:17:51.128 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:51.127+0000 7fb6412e3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-10T00:17:51.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:51 smithi121 ceph-mon[32378]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:51.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:51.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:51.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:51.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-10T00:17:51.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-10T00:17:51.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-10T00:17:51.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:51 smithi135 ceph-mon[36424]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-10T00:17:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-10T00:17:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-10T00:17:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:51.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_4 2024-06-10T00:17:51.808 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:17:51.880 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:51.432+0000 7fb6412e3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-10T00:17:51.881 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:51 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:51.517+0000 7fb6412e3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-10T00:17:52.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:52 smithi121 ceph-mon[32378]: Deploying daemon mgr.y on smithi135 2024-06-10T00:17:52.366 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:52 smithi135 ceph-mon[36424]: Deploying daemon mgr.y on smithi135 2024-06-10T00:17:52.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:52 smithi193 ceph-mon[36716]: Deploying daemon mgr.y on smithi135 2024-06-10T00:17:52.939 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:52 smithi135 systemd[1]: Starting Ceph mgr.y for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:17:53.061 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:52 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:52.726+0000 7fb6412e3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-10T00:17:53.061 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:52 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:52.898+0000 7fb6412e3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-10T00:17:53.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.228+0000 7f83aee6b700 1 -- 172.21.15.121:0/3142739400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a81013e0 msgr2=0x7f83a8101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:53.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.228+0000 7f83aee6b700 1 --2- 172.21.15.121:0/3142739400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a81013e0 0x7f83a8101800 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f839c0099c0 tx=0x7f839c009cd0 comp rx=0 tx=0).stop 2024-06-10T00:17:53.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.229+0000 7f83aee6b700 1 -- 172.21.15.121:0/3142739400 shutdown_connections 2024-06-10T00:17:53.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.229+0000 7f83aee6b700 1 --2- 172.21.15.121:0/3142739400 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83a8102f80 0x7f83a8109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:53.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.229+0000 7f83aee6b700 1 --2- 172.21.15.121:0/3142739400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a81013e0 0x7f83a8101800 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:53.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.229+0000 7f83aee6b700 1 --2- 172.21.15.121:0/3142739400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83a81025e0 0x7f83a8102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:53.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.229+0000 7f83aee6b700 1 -- 172.21.15.121:0/3142739400 >> 172.21.15.121:0/3142739400 conn(0x7f83a80fc980 msgr2=0x7f83a80fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:17:53.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.229+0000 7f83aee6b700 1 -- 172.21.15.121:0/3142739400 shutdown_connections 2024-06-10T00:17:53.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.229+0000 7f83aee6b700 1 -- 172.21.15.121:0/3142739400 wait complete. 2024-06-10T00:17:53.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.229+0000 7f83aee6b700 1 Processor -- start 2024-06-10T00:17:53.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.229+0000 7f83aee6b700 1 -- start start 2024-06-10T00:17:53.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.229+0000 7f83aee6b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a81013e0 0x7f83a81975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:53.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83aee6b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83a81025e0 0x7f83a8197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:53.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83aee6b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83a8102f80 0x7f83a819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:53.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83aee6b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f83a810cfc0 con 0x7f83a81013e0 2024-06-10T00:17:53.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83aee6b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f83a810ce40 con 0x7f83a81025e0 2024-06-10T00:17:53.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83aee6b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f83a810d140 con 0x7f83a8102f80 2024-06-10T00:17:53.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83acc07700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a81013e0 0x7f83a81975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:53.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83acc07700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a81013e0 0x7f83a81975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51532/0 (socket says 172.21.15.121:51532) 2024-06-10T00:17:53.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83acc07700 1 -- 172.21.15.121:0/1910410304 learned_addr learned my addr 172.21.15.121:0/1910410304 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:17:53.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83a7fff700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83a81025e0 0x7f83a8197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:53.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83ad408700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83a8102f80 0x7f83a819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:53.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83acc07700 1 -- 172.21.15.121:0/1910410304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83a8102f80 msgr2=0x7f83a819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:53.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83acc07700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83a8102f80 0x7f83a819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:53.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83acc07700 1 -- 172.21.15.121:0/1910410304 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83a81025e0 msgr2=0x7f83a8197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:17:53.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83acc07700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83a81025e0 0x7f83a8197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:17:53.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83acc07700 1 -- 172.21.15.121:0/1910410304 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a819f210 con 0x7f83a81013e0 2024-06-10T00:17:53.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83acc07700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a81013e0 0x7f83a81975b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c010 tx=0x7f839c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:53.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83a5ffb700 1 -- 172.21.15.121:0/1910410304 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c027070 con 0x7f83a81013e0 2024-06-10T00:17:53.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83a5ffb700 1 -- 172.21.15.121:0/1910410304 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f839c004130 con 0x7f83a81013e0 2024-06-10T00:17:53.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83a5ffb700 1 -- 172.21.15.121:0/1910410304 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c022e20 con 0x7f83a81013e0 2024-06-10T00:17:53.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.230+0000 7f83aee6b700 1 -- 172.21.15.121:0/1910410304 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a819f4a0 con 0x7f83a81013e0 2024-06-10T00:17:53.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.231+0000 7f83aee6b700 1 -- 172.21.15.121:0/1910410304 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a819f9d0 con 0x7f83a81013e0 2024-06-10T00:17:53.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.231+0000 7f83a5ffb700 1 -- 172.21.15.121:0/1910410304 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f839c02a070 con 0x7f83a81013e0 2024-06-10T00:17:53.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.231+0000 7f83aee6b700 1 -- 172.21.15.121:0/1910410304 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a8069180 con 0x7f83a81013e0 2024-06-10T00:17:53.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.231+0000 7f83a5ffb700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8398039330 0x7f839803b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:17:53.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.232+0000 7f83a5ffb700 1 -- 172.21.15.121:0/1910410304 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f839c04cfb0 con 0x7f83a81013e0 2024-06-10T00:17:53.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.232+0000 7f83a7fff700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8398039330 0x7f839803b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:17:53.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.232+0000 7f83a7fff700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8398039330 0x7f839803b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8394004630 tx=0x7f839400b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:17:53.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.236+0000 7f83a5ffb700 1 -- 172.21.15.121:0/1910410304 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f839c013070 con 0x7f83a81013e0 2024-06-10T00:17:53.237 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:53 smithi135 ceph-mon[36424]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:53.237 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:52 smithi135 podman[37795]: 2024-06-10 00:17:52.841707301 +0000 UTC m=+0.017467340 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:17:53.237 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:52 smithi135 podman[37795]: 2024-06-10 00:17:52.944595187 +0000 UTC m=+0.120355223 container create a4bbc28e66c799635b9e3120da4decd0f74a4ff1f148a805ccbbcbf33b0ec96d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:17:53.237 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:53 smithi135 podman[37795]: 2024-06-10 00:17:53.155212228 +0000 UTC m=+0.330972267 container init a4bbc28e66c799635b9e3120da4decd0f74a4ff1f148a805ccbbcbf33b0ec96d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc) 2024-06-10T00:17:53.238 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:53 smithi135 podman[37795]: 2024-06-10 00:17:53.157743218 +0000 UTC m=+0.333503265 container start a4bbc28e66c799635b9e3120da4decd0f74a4ff1f148a805ccbbcbf33b0ec96d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240528) 2024-06-10T00:17:53.238 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:53 smithi135 ceph-mgr[37815]: -- 172.21.15.135:0/2542132198 <== mon.2 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c7b21605a0 con 0x55c7b2134800 2024-06-10T00:17:53.238 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:53 smithi135 bash[37795]: a4bbc28e66c799635b9e3120da4decd0f74a4ff1f148a805ccbbcbf33b0ec96d 2024-06-10T00:17:53.337 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:53 smithi193 ceph-mon[36716]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:53.337 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:53.070+0000 7fb6412e3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-10T00:17:53.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:53.446+0000 7f83aee6b700 1 -- 172.21.15.121:0/1910410304 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f83a80628d0 con 0x7f8398039330 2024-06-10T00:17:53.447 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:53 smithi121 ceph-mon[32378]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:53.502 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:53 smithi135 systemd[1]: Started Ceph mgr.y for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:17:53.502 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:53 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:53.311+0000 7f3bf4b83200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-10T00:17:53.631 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:53.365+0000 7fb6412e3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-10T00:17:53.631 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:53.458+0000 7fb6412e3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-10T00:17:53.757 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:53 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:53.502+0000 7f3bf4b83200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:17:53.757 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:53 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:53.611+0000 7f3bf4b83200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-10T00:17:54.131 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:53.670+0000 7fb6412e3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-10T00:17:54.131 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:53.866+0000 7fb6412e3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:17:54.257 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:53 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:53.940+0000 7f3bf4b83200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-10T00:17:54.257 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:54 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:54.041+0000 7f3bf4b83200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-10T00:17:54.351 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:54 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.352 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:54 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.352 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:54 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.352 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:54 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.352 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:54 smithi121 ceph-mon[32378]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:17:54.352 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:54 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:17:54.352 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:54 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:17:54.352 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:54 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:54.352 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:54 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:54.352 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:54 smithi121 ceph-mon[32378]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:54.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-mon[36716]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:17:54.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:17:54.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:17:54.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:54.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:54.506 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-mon[36716]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:54.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:54 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:54 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:54 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:54 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:54 smithi135 ceph-mon[36424]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:17:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:54 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:17:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:54 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:17:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:54 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:54 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:17:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:54 smithi135 ceph-mon[36424]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:54.880 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:54.505+0000 7fb6412e3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-10T00:17:55.228 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:54 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:54.937+0000 7fb6412e3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-10T00:17:55.229 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:55.032+0000 7fb6412e3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-10T00:17:55.229 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:55.228+0000 7fb6412e3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-10T00:17:55.614 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:55.273+0000 7f3bf4b83200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-10T00:17:55.614 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:55.442+0000 7f3bf4b83200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-10T00:17:55.614 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:55.614+0000 7f3bf4b83200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-10T00:17:55.898 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:55.898 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:55.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:55.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:55.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:55.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:55.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:55.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: Reconfiguring mgr.x (unknown last config time)... 2024-06-10T00:17:55.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-10T00:17:55.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-10T00:17:55.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:55.899 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-mon[36424]: Reconfiguring daemon mgr.x on smithi121 2024-06-10T00:17:55.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:55.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:55.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:55.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:55.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:55.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:55.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:55.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: Reconfiguring mgr.x (unknown last config time)... 2024-06-10T00:17:55.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-10T00:17:55.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-10T00:17:55.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:55.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:55 smithi121 ceph-mon[32378]: Reconfiguring daemon mgr.x on smithi121 2024-06-10T00:17:56.074 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:55.746+0000 7fb6412e3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-10T00:17:56.074 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:55.837+0000 7fb6412e3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-10T00:17:56.074 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:56.074 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:56.074 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:56.074 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:56.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:56.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:17:56.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:56.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: Reconfiguring mgr.x (unknown last config time)... 2024-06-10T00:17:56.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-10T00:17:56.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-10T00:17:56.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:17:56.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:55 smithi193 ceph-mon[36716]: Reconfiguring daemon mgr.x on smithi121 2024-06-10T00:17:56.195 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:55.898+0000 7f3bf4b83200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-10T00:17:56.195 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:55.990+0000 7f3bf4b83200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-10T00:17:56.195 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:56 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:56.195+0000 7f3bf4b83200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-10T00:17:56.380 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:56.074+0000 7fb6412e3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-10T00:17:56.381 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:56.205+0000 7fb6412e3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-10T00:17:56.381 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:56.298+0000 7fb6412e3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-10T00:17:56.507 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:56 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:56.384+0000 7f3bf4b83200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:17:56.697 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:56.412+0000 7fb6412e3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-10T00:17:56.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:56 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3288628414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d80f9176-d911-4798-98cd-02587aa53bb5"}]: dispatch 2024-06-10T00:17:56.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:56 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3288628414' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d80f9176-d911-4798-98cd-02587aa53bb5"}]': finished 2024-06-10T00:17:56.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:56 smithi121 ceph-mon[32378]: osdmap e5: 1 total, 0 up, 1 in 2024-06-10T00:17:56.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:56 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:17:56.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:56 smithi121 ceph-mon[32378]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:56.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:56 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/791938764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:17:56.984 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:56 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3288628414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d80f9176-d911-4798-98cd-02587aa53bb5"}]: dispatch 2024-06-10T00:17:56.984 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:56 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3288628414' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d80f9176-d911-4798-98cd-02587aa53bb5"}]': finished 2024-06-10T00:17:56.984 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:56 smithi135 ceph-mon[36424]: osdmap e5: 1 total, 0 up, 1 in 2024-06-10T00:17:56.984 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:56 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:17:56.985 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:56 smithi135 ceph-mon[36424]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:56.985 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:56 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/791938764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:17:57.130 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:56.712+0000 7fb6412e3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-10T00:17:57.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:56 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3288628414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d80f9176-d911-4798-98cd-02587aa53bb5"}]: dispatch 2024-06-10T00:17:57.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:56 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3288628414' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d80f9176-d911-4798-98cd-02587aa53bb5"}]': finished 2024-06-10T00:17:57.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:56 smithi193 ceph-mon[36716]: osdmap e5: 1 total, 0 up, 1 in 2024-06-10T00:17:57.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:56 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:17:57.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:56 smithi193 ceph-mon[36716]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:57.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:56 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/791938764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:17:57.257 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:56 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:56.984+0000 7f3bf4b83200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-10T00:17:57.630 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:57 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:57.302+0000 7fb6412e3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-10T00:17:57.631 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:17:57 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:17:57.514+0000 7fb6412e3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-10T00:17:57.695 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:57 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:57.401+0000 7f3bf4b83200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-10T00:17:57.695 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:57 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:57.496+0000 7f3bf4b83200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-10T00:17:57.695 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:57 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:57.695+0000 7f3bf4b83200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-10T00:17:58.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:17:58.076+0000 7f83a5ffb700 1 -- 172.21.15.121:0/1910410304 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 14) ==== 92061+0+0 (secure 0 0 0) 0x7f839c003c50 con 0x7f83a81013e0 2024-06-10T00:17:58.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:58 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:58.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:58 smithi193 ceph-mon[36716]: Standby manager daemon z started 2024-06-10T00:17:58.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:58 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-10T00:17:58.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:58 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:17:58.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:58 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-10T00:17:58.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:58 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:17:58.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:58 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:58.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:58 smithi121 ceph-mon[32378]: Standby manager daemon z started 2024-06-10T00:17:58.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:58 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-10T00:17:58.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:58 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:17:58.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:58 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-10T00:17:58.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:58 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:17:58.507 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:58.217+0000 7f3bf4b83200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-10T00:17:58.507 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:58.307+0000 7f3bf4b83200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-10T00:17:58.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:17:58.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-mon[36424]: Standby manager daemon z started 2024-06-10T00:17:58.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-10T00:17:58.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:17:58.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-10T00:17:58.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.193:0/613112461' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:17:58.892 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:58.551+0000 7f3bf4b83200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-10T00:17:58.892 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:58.683+0000 7f3bf4b83200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-10T00:17:58.892 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:58.776+0000 7f3bf4b83200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-10T00:17:58.892 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:58.892+0000 7f3bf4b83200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-10T00:17:59.194 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:59 smithi135 ceph-mon[36424]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:59.194 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:59 smithi135 ceph-mon[36424]: mgrmap e14: x(active, since 76s), standbys: z 2024-06-10T00:17:59.194 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:17:59 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-10T00:17:59.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:59 smithi121 ceph-mon[32378]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:59.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:59 smithi121 ceph-mon[32378]: mgrmap e14: x(active, since 76s), standbys: z 2024-06-10T00:17:59.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:17:59 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-10T00:17:59.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:59 smithi193 ceph-mon[36716]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:17:59.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:59 smithi193 ceph-mon[36716]: mgrmap e14: x(active, since 76s), standbys: z 2024-06-10T00:17:59.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:17:59 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-10T00:17:59.507 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:59 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:59.194+0000 7f3bf4b83200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-10T00:18:00.074 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:17:59 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:17:59.792+0000 7f3bf4b83200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-10T00:18:00.074 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:18:00 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:18:00.008+0000 7f3bf4b83200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-10T00:18:00.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:00.084+0000 7f83a5ffb700 1 -- 172.21.15.121:0/1910410304 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f839c00c2d0 con 0x7f83a81013e0 2024-06-10T00:18:00.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:00 smithi121 ceph-mon[32378]: Standby manager daemon y started 2024-06-10T00:18:00.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:00 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-10T00:18:00.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:00 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:18:00.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:00 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-10T00:18:00.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:00 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:18:00.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:00 smithi193 ceph-mon[36716]: Standby manager daemon y started 2024-06-10T00:18:00.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:00 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-10T00:18:00.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:00 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:18:00.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:00 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-10T00:18:00.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:00 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:18:00.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:00 smithi135 ceph-mon[36424]: Standby manager daemon y started 2024-06-10T00:18:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:00 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-10T00:18:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:00 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:18:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:00 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-10T00:18:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:00 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.135:0/2717301340' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:18:01.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:01 smithi193 ceph-mon[36716]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:01.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:01 smithi193 ceph-mon[36716]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-06-10T00:18:01.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:01 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-10T00:18:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:01 smithi121 ceph-mon[32378]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:01 smithi121 ceph-mon[32378]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-06-10T00:18:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:01 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-10T00:18:01.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:01 smithi135 ceph-mon[36424]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:01.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:01 smithi135 ceph-mon[36424]: mgrmap e15: x(active, since 78s), standbys: z, y 2024-06-10T00:18:01.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:01 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-10T00:18:02.525 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:02 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:02.525 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:02 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:02.525 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:02 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:02.526 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:02 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:02.526 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:02 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:02.526 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:02 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:02.526 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:02 smithi121 ceph-mon[32378]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:02.707 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:02 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:02.707 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:02 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:02.707 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:02 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:02.707 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:02 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:02.707 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:02 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:02.707 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:02 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:02.707 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:02 smithi135 ceph-mon[36424]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:02.837 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:02 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:02.837 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:02 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:02.837 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:02 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:02.837 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:02 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:02.837 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:02 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:02.837 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:02 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:02.838 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:02 smithi193 ceph-mon[36716]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:04.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-10T00:18:04.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:04.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-10T00:18:04.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:04.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-10T00:18:04.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:04.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:04 smithi121 ceph-mon[32378]: Deploying daemon osd.0 on smithi121 2024-06-10T00:18:04.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:04 smithi121 ceph-mon[32378]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:05.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:04 smithi193 ceph-mon[36716]: Deploying daemon osd.0 on smithi121 2024-06-10T00:18:05.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:04 smithi193 ceph-mon[36716]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:05.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:04 smithi135 ceph-mon[36424]: Deploying daemon osd.0 on smithi121 2024-06-10T00:18:05.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:04 smithi135 ceph-mon[36424]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:07.374 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:07 smithi121 ceph-mon[32378]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:07.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:07 smithi193 ceph-mon[36716]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:07.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:07 smithi135 ceph-mon[36424]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:09.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:09 smithi193 ceph-mon[36716]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:09.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:09.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:09.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:09.432 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:09 smithi121 ceph-mon[32378]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:09.432 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:09.432 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:09.432 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:09 smithi135 ceph-mon[36424]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:11.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:11 smithi193 ceph-mon[36716]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:11 smithi121 ceph-mon[32378]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:11.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:11 smithi135 ceph-mon[36424]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:12.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.599+0000 7f83a5ffb700 1 -- 172.21.15.121:0/1910410304 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f83a80628d0 con 0x7f8398039330 2024-06-10T00:18:12.599 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 0 on host 'smithi121' 2024-06-10T00:18:12.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 -- 172.21.15.121:0/1910410304 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8398039330 msgr2=0x7f839803b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:12.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8398039330 0x7f839803b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8394004630 tx=0x7f839400b3c0 comp rx=0 tx=0).stop 2024-06-10T00:18:12.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 -- 172.21.15.121:0/1910410304 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a81013e0 msgr2=0x7f83a81975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:12.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a81013e0 0x7f83a81975b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f839c00c010 tx=0x7f839c005e80 comp rx=0 tx=0).stop 2024-06-10T00:18:12.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 -- 172.21.15.121:0/1910410304 shutdown_connections 2024-06-10T00:18:12.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8398039330 0x7f839803b7f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:12.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f83a8102f80 0x7f83a819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:12.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83a81013e0 0x7f83a81975b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:12.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 --2- 172.21.15.121:0/1910410304 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f83a81025e0 0x7f83a8197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:12.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 -- 172.21.15.121:0/1910410304 >> 172.21.15.121:0/1910410304 conn(0x7f83a80fc980 msgr2=0x7f83a8105960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:18:12.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 -- 172.21.15.121:0/1910410304 shutdown_connections 2024-06-10T00:18:12.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:18:12.601+0000 7f83aee6b700 1 -- 172.21.15.121:0/1910410304 wait complete. 2024-06-10T00:18:12.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:12 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:12.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:12 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:12.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:12 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:12.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:12 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:12.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:12 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:12.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:12 smithi121 ceph-mon[32378]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:12.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:12 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:12.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:12 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:12.858 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:12 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:12.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:12 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:12.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:12 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:12.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:12 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:12.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:12 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:12.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:12 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:12.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:12 smithi193 ceph-mon[36716]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:12.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:12 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:12.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:12 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:12.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:12 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:13.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:12 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:13.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:12 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:13.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:12 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:13.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:12 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:13.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:12 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:13.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:12 smithi135 ceph-mon[36424]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:13.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:12 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:13.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:12 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:13.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:12 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:13.189 DEBUG:teuthology.orchestra.run.smithi121:osd.0> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.0.service 2024-06-10T00:18:13.192 INFO:tasks.cephadm:Deploying osd.1 on smithi135 with /dev/vg_nvme/lv_4... 2024-06-10T00:18:13.192 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-10T00:18:13.433 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:18:13.817 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:13 smithi135 ceph-mon[36424]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-10T00:18:13.898 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:13 smithi121 ceph-mon[32378]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-10T00:18:14.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:13 smithi193 ceph-mon[36716]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-10T00:18:14.737 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:14 smithi135 ceph-mon[36424]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-10T00:18:14.738 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:14 smithi135 ceph-mon[36424]: osdmap e6: 1 total, 0 up, 1 in 2024-06-10T00:18:14.738 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:14 smithi135 ceph-mon[36424]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:18:14.738 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:14 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:14.738 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:14 smithi135 ceph-mon[36424]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:14.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:14 smithi121 ceph-mon[32378]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-10T00:18:14.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:14 smithi121 ceph-mon[32378]: osdmap e6: 1 total, 0 up, 1 in 2024-06-10T00:18:14.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:14 smithi121 ceph-mon[32378]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:18:14.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:14 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:14.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:14 smithi121 ceph-mon[32378]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:15.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:14 smithi193 ceph-mon[36716]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-10T00:18:15.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:14 smithi193 ceph-mon[36716]: osdmap e6: 1 total, 0 up, 1 in 2024-06-10T00:18:15.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:14 smithi193 ceph-mon[36716]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:18:15.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:14 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:15.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:14 smithi193 ceph-mon[36716]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:15.279 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-10T00:18:15.279 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-10T00:18:15.279 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-06-10T00:18:15.279 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-06-10T00:18:15.279 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126436 s, 829 MB/s 2024-06-10T00:18:15.279 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-06-10T00:18:15.424 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:18:15 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0[46997]: 2024-06-10T00:18:15.075+0000 7efc79128700 -1 osd.0 0 waiting for initial osdmap 2024-06-10T00:18:15.425 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:18:15 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0[46997]: 2024-06-10T00:18:15.078+0000 7efc70f13700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:18:15.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:15 smithi121 ceph-mon[32378]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:18:15.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:15 smithi121 ceph-mon[32378]: osdmap e7: 1 total, 0 up, 1 in 2024-06-10T00:18:15.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:15 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:15.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:15 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:15.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:15 smithi121 ceph-mon[32378]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' 2024-06-10T00:18:15.829 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:15 smithi135 ceph-mon[36424]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:18:15.829 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:15 smithi135 ceph-mon[36424]: osdmap e7: 1 total, 0 up, 1 in 2024-06-10T00:18:15.829 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:15 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:15.829 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:15 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:15.829 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:15 smithi135 ceph-mon[36424]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' 2024-06-10T00:18:15.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_4 2024-06-10T00:18:16.094 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:18:16.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:15 smithi193 ceph-mon[36716]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:18:16.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:15 smithi193 ceph-mon[36716]: osdmap e7: 1 total, 0 up, 1 in 2024-06-10T00:18:16.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:15 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:16.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:15 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:16.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:15 smithi193 ceph-mon[36716]: from='osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040]' entity='osd.0' 2024-06-10T00:18:16.868 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040] boot 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: osdmap e8: 1 total, 1 up, 1 in 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi121 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi121 to 17088M 2024-06-10T00:18:16.869 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:16.870 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:16.870 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:16.870 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:16.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:18:16.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:18:16.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040] boot 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: osdmap e8: 1 total, 1 up, 1 in 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi121 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi121 to 17088M 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:16.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:16.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: osd.0 [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040] boot 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: osdmap e8: 1 total, 1 up, 1 in 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi121 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:18:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi121 to 17088M 2024-06-10T00:18:17.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:17.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:17.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:17.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:17.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.674+0000 7fd12dac1700 1 -- 172.21.15.135:0/3572378983 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd128103b00 msgr2=0x7fd128103f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:17.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.674+0000 7fd12dac1700 1 --2- 172.21.15.135:0/3572378983 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd128103b00 0x7fd128103f20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd1180099c0 tx=0x7fd118009cd0 comp rx=0 tx=0).stop 2024-06-10T00:18:17.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.674+0000 7fd12dac1700 1 -- 172.21.15.135:0/3572378983 shutdown_connections 2024-06-10T00:18:17.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.674+0000 7fd12dac1700 1 --2- 172.21.15.135:0/3572378983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128100f40 0x7fd12810aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:17.676 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.674+0000 7fd12dac1700 1 --2- 172.21.15.135:0/3572378983 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd128103b00 0x7fd128103f20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.674+0000 7fd12dac1700 1 --2- 172.21.15.135:0/3572378983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd128100580 0x7fd128100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.674+0000 7fd12dac1700 1 -- 172.21.15.135:0/3572378983 >> 172.21.15.135:0/3572378983 conn(0x7fd1280fc100 msgr2=0x7fd1280fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.674+0000 7fd12dac1700 1 -- 172.21.15.135:0/3572378983 shutdown_connections 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.674+0000 7fd12dac1700 1 -- 172.21.15.135:0/3572378983 wait complete. 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.675+0000 7fd12dac1700 1 Processor -- start 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.675+0000 7fd12dac1700 1 -- start start 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.675+0000 7fd12dac1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd128100580 0x7fd128198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.675+0000 7fd12dac1700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd128100f40 0x7fd128198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.675+0000 7fd12dac1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128103b00 0x7fd12819fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.675+0000 7fd12dac1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd12810e200 con 0x7fd128100580 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.675+0000 7fd12dac1700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd12810e080 con 0x7fd128100f40 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.675+0000 7fd12dac1700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd12810e380 con 0x7fd128103b00 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.675+0000 7fd126ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd128100f40 0x7fd128198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:18:17.677 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.675+0000 7fd126ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd128100f40 0x7fd128198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:56264/0 (socket says 172.21.15.135:56264) 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.675+0000 7fd126ffd700 1 -- 172.21.15.135:0/2449895710 learned_addr learned my addr 172.21.15.135:0/2449895710 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd1277fe700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd128100580 0x7fd128198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd127fff700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128103b00 0x7fd12819fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd126ffd700 1 -- 172.21.15.135:0/2449895710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128103b00 msgr2=0x7fd12819fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd126ffd700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128103b00 0x7fd12819fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd126ffd700 1 -- 172.21.15.135:0/2449895710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd128100580 msgr2=0x7fd128198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd126ffd700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd128100580 0x7fd128198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd126ffd700 1 -- 172.21.15.135:0/2449895710 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1281a04e0 con 0x7fd128100f40 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd127fff700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128103b00 0x7fd12819fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd126ffd700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd128100f40 0x7fd128198dc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00b500 tx=0x7fd11c00b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd124ff9700 1 -- 172.21.15.135:0/2449895710 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c015020 con 0x7fd128100f40 2024-06-10T00:18:17.678 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd124ff9700 1 -- 172.21.15.135:0/2449895710 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd11c011d10 con 0x7fd128100f40 2024-06-10T00:18:17.679 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd12dac1700 1 -- 172.21.15.135:0/2449895710 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1281a07d0 con 0x7fd128100f40 2024-06-10T00:18:17.679 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd124ff9700 1 -- 172.21.15.135:0/2449895710 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c00f2e0 con 0x7fd128100f40 2024-06-10T00:18:17.679 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.676+0000 7fd12dac1700 1 -- 172.21.15.135:0/2449895710 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1281a0d90 con 0x7fd128100f40 2024-06-10T00:18:17.679 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.677+0000 7fd12dac1700 1 -- 172.21.15.135:0/2449895710 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd128069180 con 0x7fd128100f40 2024-06-10T00:18:17.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.678+0000 7fd124ff9700 1 -- 172.21.15.135:0/2449895710 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7fd11c018020 con 0x7fd128100f40 2024-06-10T00:18:17.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.679+0000 7fd124ff9700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd1100a2b90 0x7fd1100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:18:17.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.679+0000 7fd124ff9700 1 -- 172.21.15.135:0/2449895710 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fd11c0cd100 con 0x7fd128100f40 2024-06-10T00:18:17.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.680+0000 7fd1277fe700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd1100a2b90 0x7fd1100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:18:17.680 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.680+0000 7fd1277fe700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd1100a2b90 0x7fd1100a5050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd11800b580 tx=0x7fd118005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:18:17.683 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.683+0000 7fd124ff9700 1 -- 172.21.15.135:0/2449895710 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd11c01f020 con 0x7fd128100f40 2024-06-10T00:18:17.885 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:17.886+0000 7fd12dac1700 1 -- 172.21.15.135:0/2449895710 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd1280628d0 con 0x7fd1100a2b90 2024-06-10T00:18:18.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:18 smithi135 ceph-mon[36424]: osdmap e9: 1 total, 1 up, 1 in 2024-06-10T00:18:18.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:18 smithi135 ceph-mon[36424]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:18:18.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:18 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:18:18.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:18 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:18:18.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:18 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:18.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:18 smithi135 ceph-mon[36424]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:18.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:18 smithi121 ceph-mon[32378]: osdmap e9: 1 total, 1 up, 1 in 2024-06-10T00:18:18.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:18 smithi121 ceph-mon[32378]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:18:18.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:18 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:18:18.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:18 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:18:18.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:18 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:18.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:18 smithi121 ceph-mon[32378]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:19.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:18 smithi193 ceph-mon[36716]: osdmap e9: 1 total, 1 up, 1 in 2024-06-10T00:18:19.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:18 smithi193 ceph-mon[36716]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:18:19.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:18 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:18:19.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:18 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:18:19.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:18 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:19.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:18 smithi193 ceph-mon[36716]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:21.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:21 smithi135 ceph-mon[36424]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:21 smithi135 ceph-mon[36424]: from='client.? 172.21.15.135:0/1679661236' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3887c2a6-d2b6-43e9-9632-c73154511128"}]: dispatch 2024-06-10T00:18:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:21 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3887c2a6-d2b6-43e9-9632-c73154511128"}]: dispatch 2024-06-10T00:18:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:21 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3887c2a6-d2b6-43e9-9632-c73154511128"}]': finished 2024-06-10T00:18:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:21 smithi135 ceph-mon[36424]: osdmap e10: 2 total, 1 up, 2 in 2024-06-10T00:18:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:21.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:21 smithi193 ceph-mon[36716]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:21 smithi193 ceph-mon[36716]: from='client.? 172.21.15.135:0/1679661236' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3887c2a6-d2b6-43e9-9632-c73154511128"}]: dispatch 2024-06-10T00:18:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:21 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3887c2a6-d2b6-43e9-9632-c73154511128"}]: dispatch 2024-06-10T00:18:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:21 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3887c2a6-d2b6-43e9-9632-c73154511128"}]': finished 2024-06-10T00:18:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:21 smithi193 ceph-mon[36716]: osdmap e10: 2 total, 1 up, 2 in 2024-06-10T00:18:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:21 smithi121 ceph-mon[32378]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:21 smithi121 ceph-mon[32378]: from='client.? 172.21.15.135:0/1679661236' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3887c2a6-d2b6-43e9-9632-c73154511128"}]: dispatch 2024-06-10T00:18:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:21 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3887c2a6-d2b6-43e9-9632-c73154511128"}]: dispatch 2024-06-10T00:18:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:21 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3887c2a6-d2b6-43e9-9632-c73154511128"}]': finished 2024-06-10T00:18:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:21 smithi121 ceph-mon[32378]: osdmap e10: 2 total, 1 up, 2 in 2024-06-10T00:18:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:22.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:22 smithi193 ceph-mon[36716]: from='client.? 172.21.15.135:0/3526871634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:18:22.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:22 smithi121 ceph-mon[32378]: from='client.? 172.21.15.135:0/3526871634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:18:22.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:22 smithi135 ceph-mon[36424]: from='client.? 172.21.15.135:0/3526871634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:18:23.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:23 smithi193 ceph-mon[36716]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:23 smithi121 ceph-mon[32378]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:23.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:23 smithi135 ceph-mon[36424]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:25.352 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:25 smithi135 ceph-mon[36424]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:25.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:25 smithi193 ceph-mon[36716]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:25 smithi121 ceph-mon[32378]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:27.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:27 smithi135 ceph-mon[36424]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:27.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:27 smithi193 ceph-mon[36716]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:27.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:27 smithi121 ceph-mon[32378]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:28.365 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-10T00:18:28.365 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:28.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-10T00:18:28.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:28.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-10T00:18:28.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:29.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:29 smithi135 ceph-mon[36424]: Deploying daemon osd.1 on smithi135 2024-06-10T00:18:29.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:29 smithi135 ceph-mon[36424]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:29.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:29 smithi193 ceph-mon[36716]: Deploying daemon osd.1 on smithi135 2024-06-10T00:18:29.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:29 smithi193 ceph-mon[36716]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:29 smithi121 ceph-mon[32378]: Deploying daemon osd.1 on smithi135 2024-06-10T00:18:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:29 smithi121 ceph-mon[32378]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:31.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:31 smithi135 ceph-mon[36424]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:31.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:31 smithi193 ceph-mon[36716]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:31.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:31 smithi121 ceph-mon[32378]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:33.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:33 smithi193 ceph-mon[36716]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:33.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:33 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:33.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:33 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:33.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:33 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:33.448 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:33 smithi135 ceph-mon[36424]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:33.448 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:33 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:33.448 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:33 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:33.448 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:33 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:33.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:33 smithi121 ceph-mon[32378]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:33 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:33 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:33 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:35.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:35 smithi193 ceph-mon[36716]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:35.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:35 smithi121 ceph-mon[32378]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:35.498 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:35 smithi135 ceph-mon[36424]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:36.149 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 1 on host 'smithi135' 2024-06-10T00:18:36.150 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.149+0000 7fd124ff9700 1 -- 172.21.15.135:0/2449895710 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd1280628d0 con 0x7fd1100a2b90 2024-06-10T00:18:36.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.151+0000 7fd12dac1700 1 -- 172.21.15.135:0/2449895710 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd1100a2b90 msgr2=0x7fd1100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:36.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.151+0000 7fd12dac1700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd1100a2b90 0x7fd1100a5050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd11800b580 tx=0x7fd118005fb0 comp rx=0 tx=0).stop 2024-06-10T00:18:36.151 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.151+0000 7fd12dac1700 1 -- 172.21.15.135:0/2449895710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd128100f40 msgr2=0x7fd128198dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:36.152 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.151+0000 7fd12dac1700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd128100f40 0x7fd128198dc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00b500 tx=0x7fd11c00b810 comp rx=0 tx=0).stop 2024-06-10T00:18:36.152 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.151+0000 7fd12dac1700 1 -- 172.21.15.135:0/2449895710 shutdown_connections 2024-06-10T00:18:36.152 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.151+0000 7fd12dac1700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd1100a2b90 0x7fd1100a5050 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:36.152 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.151+0000 7fd12dac1700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd128103b00 0x7fd12819fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:36.152 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.151+0000 7fd12dac1700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd128100580 0x7fd128198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:36.152 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.151+0000 7fd12dac1700 1 --2- 172.21.15.135:0/2449895710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd128100f40 0x7fd128198dc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:36.152 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.151+0000 7fd12dac1700 1 -- 172.21.15.135:0/2449895710 >> 172.21.15.135:0/2449895710 conn(0x7fd1280fc100 msgr2=0x7fd128106c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:18:36.152 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.152+0000 7fd12dac1700 1 -- 172.21.15.135:0/2449895710 shutdown_connections 2024-06-10T00:18:36.152 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:18:36.152+0000 7fd12dac1700 1 -- 172.21.15.135:0/2449895710 wait complete. 2024-06-10T00:18:36.453 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.453 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.453 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:36.454 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:36.454 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.454 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:36 smithi135 ceph-mon[36424]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:36.454 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.454 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.454 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:36.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:36 smithi193 ceph-mon[36716]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:36.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:36.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:36.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:36 smithi121 ceph-mon[32378]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:36.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:36.645 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:36.774 DEBUG:teuthology.orchestra.run.smithi135:osd.1> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.1.service 2024-06-10T00:18:36.776 INFO:tasks.cephadm:Deploying osd.2 on smithi193 with /dev/vg_nvme/lv_4... 2024-06-10T00:18:36.776 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-10T00:18:37.017 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:18:37.442 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:37 smithi193 ceph-mon[36716]: from='osd.1 [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-10T00:18:37.442 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:37 smithi193 ceph-mon[36716]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-10T00:18:37.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:37 smithi135 ceph-mon[36424]: from='osd.1 [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-10T00:18:37.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:37 smithi135 ceph-mon[36424]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-10T00:18:37.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:37 smithi121 ceph-mon[32378]: from='osd.1 [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-10T00:18:37.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:37 smithi121 ceph-mon[32378]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-10T00:18:38.352 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:38 smithi193 ceph-mon[36716]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-10T00:18:38.352 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:38 smithi193 ceph-mon[36716]: from='osd.1 [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:18:38.352 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:38 smithi193 ceph-mon[36716]: osdmap e11: 2 total, 1 up, 2 in 2024-06-10T00:18:38.352 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:38 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:38.352 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:38 smithi193 ceph-mon[36716]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:18:38.353 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:38 smithi193 ceph-mon[36716]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:38.353 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:38 smithi193 ceph-mon[36716]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:18:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:38 smithi135 ceph-mon[36424]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-10T00:18:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:38 smithi135 ceph-mon[36424]: from='osd.1 [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:18:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:38 smithi135 ceph-mon[36424]: osdmap e11: 2 total, 1 up, 2 in 2024-06-10T00:18:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:38 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:38 smithi135 ceph-mon[36424]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:18:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:38 smithi135 ceph-mon[36424]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:38 smithi135 ceph-mon[36424]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:18:38.508 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:18:38 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1[40983]: 2024-06-10T00:18:38.241+0000 7fb4aee8e700 -1 osd.1 0 waiting for initial osdmap 2024-06-10T00:18:38.508 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:18:38 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1[40983]: 2024-06-10T00:18:38.244+0000 7fb4aac81700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:18:38.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:38 smithi121 ceph-mon[32378]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-10T00:18:38.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:38 smithi121 ceph-mon[32378]: from='osd.1 [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:18:38.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:38 smithi121 ceph-mon[32378]: osdmap e11: 2 total, 1 up, 2 in 2024-06-10T00:18:38.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:38 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:38.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:38 smithi121 ceph-mon[32378]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:18:38.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:38 smithi121 ceph-mon[32378]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:38.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:38 smithi121 ceph-mon[32378]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:18:38.918 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-10T00:18:38.919 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-10T00:18:38.919 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-06-10T00:18:38.919 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-06-10T00:18:38.919 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121394 s, 864 MB/s 2024-06-10T00:18:38.919 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-06-10T00:18:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:39 smithi135 ceph-mon[36424]: osdmap e12: 2 total, 1 up, 2 in 2024-06-10T00:18:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:39.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:39 smithi193 ceph-mon[36716]: osdmap e12: 2 total, 1 up, 2 in 2024-06-10T00:18:39.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:39.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:39.527 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:39.557 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_4 2024-06-10T00:18:39.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:39 smithi121 ceph-mon[32378]: osdmap e12: 2 total, 1 up, 2 in 2024-06-10T00:18:39.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:39.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:39.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:39.802 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:18:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:18:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:18:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: osd.1 [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323] boot 2024-06-10T00:18:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: osdmap e13: 2 total, 2 up, 2 in 2024-06-10T00:18:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi135 2024-06-10T00:18:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:18:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi135 to 17088M 2024-06-10T00:18:40.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:40.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:40.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:40 smithi135 ceph-mon[36424]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:40.623 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: osd.1 [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323] boot 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: osdmap e13: 2 total, 2 up, 2 in 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi135 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi135 to 17088M 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:40.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.625 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:40 smithi193 ceph-mon[36716]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: osd.1 [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323] boot 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: osdmap e13: 2 total, 2 up, 2 in 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi135 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi135 to 17088M 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:18:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:40.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:40 smithi121 ceph-mon[32378]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-06-10T00:18:41.391 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.389+0000 7fd680938700 1 -- 172.21.15.193:0/88825494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd67c102f80 msgr2=0x7fd67c109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:41.391 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.389+0000 7fd680938700 1 --2- 172.21.15.193:0/88825494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd67c102f80 0x7fd67c109830 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd67400b0a0 tx=0x7fd67400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:18:41.391 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.389+0000 7fd680938700 1 -- 172.21.15.193:0/88825494 shutdown_connections 2024-06-10T00:18:41.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.389+0000 7fd680938700 1 --2- 172.21.15.193:0/88825494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd67c102f80 0x7fd67c109830 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:41.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.389+0000 7fd680938700 1 --2- 172.21.15.193:0/88825494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd67c1013e0 0x7fd67c101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:41.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.389+0000 7fd680938700 1 --2- 172.21.15.193:0/88825494 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd67c1025e0 0x7fd67c102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:41.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.389+0000 7fd680938700 1 -- 172.21.15.193:0/88825494 >> 172.21.15.193:0/88825494 conn(0x7fd67c0fc980 msgr2=0x7fd67c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:18:41.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.389+0000 7fd680938700 1 -- 172.21.15.193:0/88825494 shutdown_connections 2024-06-10T00:18:41.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.389+0000 7fd680938700 1 -- 172.21.15.193:0/88825494 wait complete. 2024-06-10T00:18:41.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.389+0000 7fd680938700 1 Processor -- start 2024-06-10T00:18:41.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.389+0000 7fd680938700 1 -- start start 2024-06-10T00:18:41.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd680938700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd67c1013e0 0x7fd67c105320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:18:41.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd680938700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd67c1025e0 0x7fd67c1039c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:18:41.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd680938700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd67c102f80 0x7fd67c103f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd680938700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c10cfa0 con 0x7fd67c1025e0 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd680938700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c10ce20 con 0x7fd67c102f80 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd680938700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c10d120 con 0x7fd67c1013e0 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd67a59c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd67c1013e0 0x7fd67c105320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd67a59c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd67c1013e0 0x7fd67c105320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:54672/0 (socket says 172.21.15.193:54672) 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd67a59c700 1 -- 172.21.15.193:0/981292159 learned_addr learned my addr 172.21.15.193:0/981292159 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd67ad9d700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd67c102f80 0x7fd67c103f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd679d9b700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd67c1025e0 0x7fd67c1039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd67a59c700 1 -- 172.21.15.193:0/981292159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd67c102f80 msgr2=0x7fd67c103f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd67a59c700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd67c102f80 0x7fd67c103f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd67a59c700 1 -- 172.21.15.193:0/981292159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd67c1025e0 msgr2=0x7fd67c1039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:41.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd67a59c700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd67c1025e0 0x7fd67c1039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:41.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd67a59c700 1 -- 172.21.15.193:0/981292159 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67c1047d0 con 0x7fd67c1013e0 2024-06-10T00:18:41.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd679d9b700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd67c1025e0 0x7fd67c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:18:41.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd67ad9d700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd67c102f80 0x7fd67c103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:18:41.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd67a59c700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd67c1013e0 0x7fd67c105320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00d980 tx=0x7fd66c00dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:18:41.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd66b7fe700 1 -- 172.21.15.193:0/981292159 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c017030 con 0x7fd67c1013e0 2024-06-10T00:18:41.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd66b7fe700 1 -- 172.21.15.193:0/981292159 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd66c004500 con 0x7fd67c1013e0 2024-06-10T00:18:41.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd66b7fe700 1 -- 172.21.15.193:0/981292159 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd66c003b20 con 0x7fd67c1013e0 2024-06-10T00:18:41.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd680938700 1 -- 172.21.15.193:0/981292159 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67c1a27b0 con 0x7fd67c1013e0 2024-06-10T00:18:41.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.390+0000 7fd680938700 1 -- 172.21.15.193:0/981292159 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67c1a2d10 con 0x7fd67c1013e0 2024-06-10T00:18:41.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.391+0000 7fd680938700 1 -- 172.21.15.193:0/981292159 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd67c069180 con 0x7fd67c1013e0 2024-06-10T00:18:41.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.392+0000 7fd66b7fe700 1 -- 172.21.15.193:0/981292159 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7fd66c01a070 con 0x7fd67c1013e0 2024-06-10T00:18:41.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.393+0000 7fd66b7fe700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6640a2c10 0x7fd6640a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:18:41.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.393+0000 7fd66b7fe700 1 -- 172.21.15.193:0/981292159 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fd66c0cd170 con 0x7fd67c1013e0 2024-06-10T00:18:41.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.393+0000 7fd679d9b700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6640a2c10 0x7fd6640a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:18:41.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.394+0000 7fd679d9b700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6640a2c10 0x7fd6640a50d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd670005950 tx=0x7fd670005f50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:18:41.397 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.398+0000 7fd66b7fe700 1 -- 172.21.15.193:0/981292159 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd66c009940 con 0x7fd67c1013e0 2024-06-10T00:18:41.599 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:41.600+0000 7fd680938700 1 -- 172.21.15.193:0/981292159 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd67c0feb00 con 0x7fd6640a2c10 2024-06-10T00:18:41.600 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:41 smithi193 ceph-mon[36716]: osdmap e14: 2 total, 2 up, 2 in 2024-06-10T00:18:41.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:41 smithi121 ceph-mon[32378]: osdmap e14: 2 total, 2 up, 2 in 2024-06-10T00:18:41.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:41 smithi135 ceph-mon[36424]: osdmap e14: 2 total, 2 up, 2 in 2024-06-10T00:18:42.530 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:42 smithi193 ceph-mon[36716]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:18:42.530 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:18:42.530 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:18:42.530 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:42.530 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:42 smithi193 ceph-mon[36716]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:42.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:42 smithi121 ceph-mon[32378]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:18:42.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:18:42.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:18:42.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:42.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:42 smithi121 ceph-mon[32378]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:42.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:42 smithi135 ceph-mon[36424]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:18:42.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:18:42.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:18:42.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:42.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:42 smithi135 ceph-mon[36424]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:45 smithi193 ceph-mon[36716]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:45 smithi193 ceph-mon[36716]: from='client.? 172.21.15.193:0/4189665041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d37465a0-81c9-499f-a0f3-fe5cdc1538ab"}]: dispatch 2024-06-10T00:18:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:45 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d37465a0-81c9-499f-a0f3-fe5cdc1538ab"}]: dispatch 2024-06-10T00:18:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:45 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d37465a0-81c9-499f-a0f3-fe5cdc1538ab"}]': finished 2024-06-10T00:18:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:45 smithi193 ceph-mon[36716]: osdmap e15: 3 total, 2 up, 3 in 2024-06-10T00:18:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:18:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:45 smithi193 ceph-mon[36716]: from='client.? 172.21.15.193:0/3096967479' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:18:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:45 smithi121 ceph-mon[32378]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:45 smithi121 ceph-mon[32378]: from='client.? 172.21.15.193:0/4189665041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d37465a0-81c9-499f-a0f3-fe5cdc1538ab"}]: dispatch 2024-06-10T00:18:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:45 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d37465a0-81c9-499f-a0f3-fe5cdc1538ab"}]: dispatch 2024-06-10T00:18:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:45 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d37465a0-81c9-499f-a0f3-fe5cdc1538ab"}]': finished 2024-06-10T00:18:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:45 smithi121 ceph-mon[32378]: osdmap e15: 3 total, 2 up, 3 in 2024-06-10T00:18:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:18:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:45 smithi121 ceph-mon[32378]: from='client.? 172.21.15.193:0/3096967479' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:18:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:45 smithi135 ceph-mon[36424]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:45 smithi135 ceph-mon[36424]: from='client.? 172.21.15.193:0/4189665041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d37465a0-81c9-499f-a0f3-fe5cdc1538ab"}]: dispatch 2024-06-10T00:18:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:45 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d37465a0-81c9-499f-a0f3-fe5cdc1538ab"}]: dispatch 2024-06-10T00:18:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:45 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d37465a0-81c9-499f-a0f3-fe5cdc1538ab"}]': finished 2024-06-10T00:18:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:45 smithi135 ceph-mon[36424]: osdmap e15: 3 total, 2 up, 3 in 2024-06-10T00:18:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:18:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:45 smithi135 ceph-mon[36424]: from='client.? 172.21.15.193:0/3096967479' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:18:47.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:47 smithi193 ceph-mon[36716]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:47.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:47 smithi121 ceph-mon[32378]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:47.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:47 smithi135 ceph-mon[36424]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:49.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:49 smithi193 ceph-mon[36716]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:49.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:49 smithi121 ceph-mon[32378]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:49 smithi135 ceph-mon[36424]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:51.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:51 smithi193 ceph-mon[36716]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:51 smithi121 ceph-mon[32378]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:51.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:51 smithi135 ceph-mon[36424]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:52.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-10T00:18:52.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:52.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-10T00:18:52.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:52.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-10T00:18:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:18:53.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:53 smithi193 ceph-mon[36716]: Deploying daemon osd.2 on smithi193 2024-06-10T00:18:53.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:53 smithi193 ceph-mon[36716]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:53.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:53 smithi121 ceph-mon[32378]: Deploying daemon osd.2 on smithi193 2024-06-10T00:18:53.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:53 smithi121 ceph-mon[32378]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:53.494 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:53 smithi135 ceph-mon[36424]: Deploying daemon osd.2 on smithi193 2024-06-10T00:18:53.494 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:53 smithi135 ceph-mon[36424]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:55.205 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:55 smithi193 ceph-mon[36716]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:55.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:55 smithi121 ceph-mon[32378]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:55.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:55 smithi135 ceph-mon[36424]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:57.329 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:57 smithi193 ceph-mon[36716]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:57.329 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:57 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:57.329 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:57 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:57.329 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:57 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:57 smithi121 ceph-mon[32378]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:57 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:57 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:57 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:57.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:57 smithi135 ceph-mon[36424]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:57.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:57 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:57.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:57 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:18:57.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:57 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:18:59.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:18:59 smithi193 ceph-mon[36716]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:59.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:18:59 smithi121 ceph-mon[32378]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:59.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:18:59 smithi135 ceph-mon[36424]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:18:59.615 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.614+0000 7fd66b7fe700 1 -- 172.21.15.193:0/981292159 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd67c0feb00 con 0x7fd6640a2c10 2024-06-10T00:18:59.615 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 2 on host 'smithi193' 2024-06-10T00:18:59.617 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 -- 172.21.15.193:0/981292159 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6640a2c10 msgr2=0x7fd6640a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:59.617 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6640a2c10 0x7fd6640a50d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd670005950 tx=0x7fd670005f50 comp rx=0 tx=0).stop 2024-06-10T00:18:59.617 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 -- 172.21.15.193:0/981292159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd67c1013e0 msgr2=0x7fd67c105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:18:59.617 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd67c1013e0 0x7fd67c105320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd66c00d980 tx=0x7fd66c00dd40 comp rx=0 tx=0).stop 2024-06-10T00:18:59.617 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 -- 172.21.15.193:0/981292159 shutdown_connections 2024-06-10T00:18:59.617 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd6640a2c10 0x7fd6640a50d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:59.617 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd67c102f80 0x7fd67c103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:59.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd67c1013e0 0x7fd67c105320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:59.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 --2- 172.21.15.193:0/981292159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd67c1025e0 0x7fd67c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:18:59.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 -- 172.21.15.193:0/981292159 >> 172.21.15.193:0/981292159 conn(0x7fd67c0fc980 msgr2=0x7fd67c0fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:18:59.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 -- 172.21.15.193:0/981292159 shutdown_connections 2024-06-10T00:18:59.618 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:18:59.617+0000 7fd680938700 1 -- 172.21.15.193:0/981292159 wait complete. 2024-06-10T00:19:00.471 DEBUG:teuthology.orchestra.run.smithi193:osd.2> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.2.service 2024-06-10T00:19:00.473 INFO:tasks.cephadm:Deploying osd.3 on smithi121 with /dev/vg_nvme/lv_3... 2024-06-10T00:19:00.474 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-10T00:19:00.726 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:19:00.848 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:00.848 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:00.848 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:00 smithi121 ceph-mon[32378]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:19:00.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:00 smithi121 ceph-mon[32378]: from='osd.2 [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-10T00:19:00.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:00 smithi121 ceph-mon[32378]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-10T00:19:00.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:00.849 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:00 smithi193 ceph-mon[36716]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:19:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:00 smithi193 ceph-mon[36716]: from='osd.2 [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-10T00:19:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:00 smithi193 ceph-mon[36716]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-10T00:19:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:00.881 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:19:00 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2[41840]: 2024-06-10T00:19:00.509+0000 7f3557c207c0 -1 osd.2 0 log_to_monitors true 2024-06-10T00:19:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:00 smithi135 ceph-mon[36424]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:19:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:00 smithi135 ceph-mon[36424]: from='osd.2 [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-10T00:19:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:00 smithi135 ceph-mon[36424]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-10T00:19:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:01.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:01 smithi193 ceph-mon[36716]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-10T00:19:01.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:01 smithi193 ceph-mon[36716]: osdmap e16: 3 total, 2 up, 3 in 2024-06-10T00:19:01.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:01 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:01.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:01 smithi193 ceph-mon[36716]: from='osd.2 [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:19:01.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:01 smithi193 ceph-mon[36716]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:19:01.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:01 smithi121 ceph-mon[32378]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-10T00:19:01.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:01 smithi121 ceph-mon[32378]: osdmap e16: 3 total, 2 up, 3 in 2024-06-10T00:19:01.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:01 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:01.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:01 smithi121 ceph-mon[32378]: from='osd.2 [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:19:01.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:01 smithi121 ceph-mon[32378]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:19:02.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:01 smithi135 ceph-mon[36424]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-10T00:19:02.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:01 smithi135 ceph-mon[36424]: osdmap e16: 3 total, 2 up, 3 in 2024-06-10T00:19:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:01 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:01 smithi135 ceph-mon[36424]: from='osd.2 [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:19:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:01 smithi135 ceph-mon[36424]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:19:02.341 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:19:02 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2[41840]: 2024-06-10T00:19:02.023+0000 7f354d284700 -1 osd.2 0 waiting for initial osdmap 2024-06-10T00:19:02.341 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:19:02 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2[41840]: 2024-06-10T00:19:02.027+0000 7f354506f700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:19:02.659 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-10T00:19:02.659 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-10T00:19:02.659 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-06-10T00:19:02.659 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-06-10T00:19:02.660 INFO:teuthology.orchestra.run.smithi121.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111372 s, 942 MB/s 2024-06-10T00:19:02.660 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-06-10T00:19:02.874 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:02 smithi193 ceph-mon[36716]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:19:02.874 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:02 smithi193 ceph-mon[36716]: osdmap e17: 3 total, 2 up, 3 in 2024-06-10T00:19:02.874 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:02 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:02.874 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:02 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:02.874 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:02 smithi193 ceph-mon[36716]: from='osd.2 ' entity='osd.2' 2024-06-10T00:19:02.874 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:02 smithi193 ceph-mon[36716]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:19:02.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:02 smithi121 ceph-mon[32378]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:19:02.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:02 smithi121 ceph-mon[32378]: osdmap e17: 3 total, 2 up, 3 in 2024-06-10T00:19:02.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:02 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:02.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:02 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:02.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:02 smithi121 ceph-mon[32378]: from='osd.2 ' entity='osd.2' 2024-06-10T00:19:02.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:02 smithi121 ceph-mon[32378]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:19:03.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:02 smithi135 ceph-mon[36424]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:19:03.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:02 smithi135 ceph-mon[36424]: osdmap e17: 3 total, 2 up, 3 in 2024-06-10T00:19:03.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:02 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:03.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:02 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:03.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:02 smithi135 ceph-mon[36424]: from='osd.2 ' entity='osd.2' 2024-06-10T00:19:03.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:02 smithi135 ceph-mon[36424]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-06-10T00:19:03.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_3 2024-06-10T00:19:03.507 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:19:03.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:03 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:19:03.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:03 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:19:03.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:03.841 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:03 smithi121 ceph-mon[32378]: osd.2 [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346] boot 2024-06-10T00:19:03.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:03 smithi121 ceph-mon[32378]: osdmap e18: 3 total, 3 up, 3 in 2024-06-10T00:19:03.842 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:03.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:03 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:19:03.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:03 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:19:03.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:03.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:03 smithi193 ceph-mon[36716]: osd.2 [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346] boot 2024-06-10T00:19:03.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:03 smithi193 ceph-mon[36716]: osdmap e18: 3 total, 3 up, 3 in 2024-06-10T00:19:03.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:04.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:03 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:19:04.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:03 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:19:04.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:04.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:03 smithi135 ceph-mon[36424]: osd.2 [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346] boot 2024-06-10T00:19:04.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:03 smithi135 ceph-mon[36424]: osdmap e18: 3 total, 3 up, 3 in 2024-06-10T00:19:04.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:19:05.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.047+0000 7fe8246a6700 1 -- 172.21.15.121:0/3426273012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81c1025e0 msgr2=0x7fe81c102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.047+0000 7fe8246a6700 1 --2- 172.21.15.121:0/3426273012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81c1025e0 0x7fe81c102a40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe8100099c0 tx=0x7fe810009cd0 comp rx=0 tx=0).stop 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.047+0000 7fe8246a6700 1 -- 172.21.15.121:0/3426273012 shutdown_connections 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.047+0000 7fe8246a6700 1 --2- 172.21.15.121:0/3426273012 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe81c102f80 0x7fe81c109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.047+0000 7fe8246a6700 1 --2- 172.21.15.121:0/3426273012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c1013e0 0x7fe81c101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.047+0000 7fe8246a6700 1 --2- 172.21.15.121:0/3426273012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81c1025e0 0x7fe81c102a40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.047+0000 7fe8246a6700 1 -- 172.21.15.121:0/3426273012 >> 172.21.15.121:0/3426273012 conn(0x7fe81c0fc980 msgr2=0x7fe81c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.047+0000 7fe8246a6700 1 -- 172.21.15.121:0/3426273012 shutdown_connections 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.047+0000 7fe8246a6700 1 -- 172.21.15.121:0/3426273012 wait complete. 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe8246a6700 1 Processor -- start 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe8246a6700 1 -- start start 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe8246a6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe81c1013e0 0x7fe81c1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:05.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe8246a6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c1025e0 0x7fe81c198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe8246a6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81c102f80 0x7fe81c19fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe8246a6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c10cfd0 con 0x7fe81c102f80 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe8246a6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c10ce50 con 0x7fe81c1013e0 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe8246a6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c10d150 con 0x7fe81c1025e0 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe822c43700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81c102f80 0x7fe81c19fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe822c43700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81c102f80 0x7fe81c19fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52920/0 (socket says 172.21.15.121:52920) 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe822c43700 1 -- 172.21.15.121:0/3970156965 learned_addr learned my addr 172.21.15.121:0/3970156965 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe821c41700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c1025e0 0x7fe81c198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe822442700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe81c1013e0 0x7fe81c1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe822c43700 1 -- 172.21.15.121:0/3970156965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c1025e0 msgr2=0x7fe81c198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe822c43700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c1025e0 0x7fe81c198d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:05.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe822c43700 1 -- 172.21.15.121:0/3970156965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe81c1013e0 msgr2=0x7fe81c1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe822c43700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe81c1013e0 0x7fe81c1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.048+0000 7fe822c43700 1 -- 172.21.15.121:0/3970156965 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c1a0440 con 0x7fe81c102f80 2024-06-10T00:19:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.049+0000 7fe822c43700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81c102f80 0x7fe81c19fda0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00d9b0 tx=0x7fe80c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:19:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.049+0000 7fe81b7fe700 1 -- 172.21.15.121:0/3970156965 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c016070 con 0x7fe81c102f80 2024-06-10T00:19:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.049+0000 7fe8246a6700 1 -- 172.21.15.121:0/3970156965 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81c1a0730 con 0x7fe81c102f80 2024-06-10T00:19:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.049+0000 7fe81b7fe700 1 -- 172.21.15.121:0/3970156965 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe80c004d10 con 0x7fe81c102f80 2024-06-10T00:19:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.049+0000 7fe81b7fe700 1 -- 172.21.15.121:0/3970156965 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c005560 con 0x7fe81c102f80 2024-06-10T00:19:05.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.049+0000 7fe8246a6700 1 -- 172.21.15.121:0/3970156965 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81c1a0c20 con 0x7fe81c102f80 2024-06-10T00:19:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.050+0000 7fe8246a6700 1 -- 172.21.15.121:0/3970156965 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe81c069180 con 0x7fe81c102f80 2024-06-10T00:19:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.051+0000 7fe81b7fe700 1 -- 172.21.15.121:0/3970156965 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7fe80c004750 con 0x7fe81c102f80 2024-06-10T00:19:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.051+0000 7fe81b7fe700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe8040a2b90 0x7fe8040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.052+0000 7fe822442700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe8040a2b90 0x7fe8040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.052+0000 7fe81b7fe700 1 -- 172.21.15.121:0/3970156965 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7fe80c0cddb0 con 0x7fe81c102f80 2024-06-10T00:19:05.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.052+0000 7fe822442700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe8040a2b90 0x7fe8040a5050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe81c102440 tx=0x7fe81400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:19:05.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.055+0000 7fe81b7fe700 1 -- 172.21.15.121:0/3970156965 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe80c01f020 con 0x7fe81c102f80 2024-06-10T00:19:05.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi193 2024-06-10T00:19:05.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:05.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi193 to 17088M 2024-06-10T00:19:05.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:05.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:05.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:05.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: osdmap e19: 3 total, 3 up, 3 in 2024-06-10T00:19:05.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-10T00:19:05.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:05.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:05.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:05.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:05.259+0000 7fe8246a6700 1 -- 172.21.15.121:0/3970156965 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe81c1a0f40 con 0x7fe8040a2b90 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi193 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi193 to 17088M 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: osdmap e19: 3 total, 3 up, 3 in 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-10T00:19:05.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:05.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:05.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:05.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi193 2024-06-10T00:19:05.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:05.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi193 to 17088M 2024-06-10T00:19:05.511 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.511 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:05.511 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:05.511 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: pgmap v80: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:05.511 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: osdmap e19: 3 total, 3 up, 3 in 2024-06-10T00:19:05.511 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:05.511 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-06-10T00:19:05.511 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:05.511 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:05.511 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:05.511 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:06.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-10T00:19:06.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:06 smithi121 ceph-mon[32378]: osdmap e20: 3 total, 3 up, 3 in 2024-06-10T00:19:06.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-10T00:19:06.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:19:06.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:19:06.160 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:06.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-10T00:19:06.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:06 smithi193 ceph-mon[36716]: osdmap e20: 3 total, 3 up, 3 in 2024-06-10T00:19:06.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-10T00:19:06.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:19:06.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:19:06.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:06.449 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:19:06 smithi121 sudo[51776]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-10T00:19:06.449 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:19:06 smithi121 sudo[51776]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:06.449 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:19:06 smithi121 sudo[51776]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:06.449 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:19:06 smithi121 sudo[51776]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:06.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-06-10T00:19:06.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:06 smithi135 ceph-mon[36424]: osdmap e20: 3 total, 3 up, 3 in 2024-06-10T00:19:06.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-10T00:19:06.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:19:06.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:19:06.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:06.934 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:19:06 smithi121 sudo[51782]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-10T00:19:06.934 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:19:06 smithi121 sudo[51782]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:06.934 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:19:06 smithi121 sudo[51782]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:06.937 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:19:06 smithi121 sudo[51782]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:07.170 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:19:06 smithi135 sudo[44807]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-10T00:19:07.171 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:19:06 smithi135 sudo[44807]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:07.171 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:19:06 smithi135 sudo[44807]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:07.171 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:19:06 smithi135 sudo[44807]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:07.171 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:07 smithi135 ceph-mon[36424]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:19:07.171 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:07 smithi135 ceph-mon[36424]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:07.171 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:07 smithi135 ceph-mon[36424]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:19:07.171 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:07 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-10T00:19:07.171 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:07 smithi135 ceph-mon[36424]: osdmap e21: 3 total, 3 up, 3 in 2024-06-10T00:19:07.171 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:07 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:07.194 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:07 smithi121 ceph-mon[32378]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:19:07.215 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:07 smithi121 ceph-mon[32378]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:07.215 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:07 smithi121 ceph-mon[32378]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:19:07.215 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:07 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-10T00:19:07.215 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:07 smithi121 ceph-mon[32378]: osdmap e21: 3 total, 3 up, 3 in 2024-06-10T00:19:07.215 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:07 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:07.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:07 smithi193 ceph-mon[36716]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:19:07.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:07 smithi193 ceph-mon[36716]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:07.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:07 smithi193 ceph-mon[36716]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:19:07.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:07 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-10T00:19:07.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:07 smithi193 ceph-mon[36716]: osdmap e21: 3 total, 3 up, 3 in 2024-06-10T00:19:07.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:07 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:07.507 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:19:07 smithi135 sudo[44810]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-10T00:19:07.508 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:19:07 smithi135 sudo[44810]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:07.508 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:19:07 smithi135 sudo[44810]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:07.508 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:19:07 smithi135 sudo[44810]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:07.694 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:19:07 smithi193 sudo[45446]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-10T00:19:07.695 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:19:07 smithi193 sudo[45446]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:07.695 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:19:07 smithi193 sudo[45446]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:07.695 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:19:07 smithi193 sudo[45446]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:08.130 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:19:07 smithi193 sudo[45449]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-10T00:19:08.131 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:19:07 smithi193 sudo[45449]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:08.131 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:19:07 smithi193 sudo[45449]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:08.131 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:19:07 smithi193 sudo[45449]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:08.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:07 smithi121 sudo[51858]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-10T00:19:08.220 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:07 smithi121 sudo[51858]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:08.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:07 smithi121 sudo[51858]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:08.221 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:07 smithi121 sudo[51858]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:08.481 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 ceph-mon[36716]: osdmap e22: 3 total, 3 up, 3 in 2024-06-10T00:19:08.481 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:08.482 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:08.482 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 ceph-mon[36716]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-10T00:19:08.482 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 sudo[45452]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-10T00:19:08.482 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 sudo[45452]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:08.482 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 sudo[45452]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:08.482 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 sudo[45452]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:08.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:08 smithi135 ceph-mon[36424]: osdmap e22: 3 total, 3 up, 3 in 2024-06-10T00:19:08.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:08.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:08.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:08 smithi135 ceph-mon[36424]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-10T00:19:08.598 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:08 smithi121 sudo[51861]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-10T00:19:08.598 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:08 smithi121 sudo[51861]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:08.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:08 smithi121 sudo[51861]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:08.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:08 smithi121 sudo[51861]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:08.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:08 smithi121 ceph-mon[32378]: osdmap e22: 3 total, 3 up, 3 in 2024-06-10T00:19:08.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:08.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:08.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:08 smithi121 ceph-mon[32378]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-10T00:19:08.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:08.732+0000 7fe81b7fe700 1 -- 172.21.15.121:0/3970156965 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fe80c098140 con 0x7fe81c102f80 2024-06-10T00:19:08.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 sudo[45455]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-10T00:19:08.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 sudo[45455]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:08.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 sudo[45455]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:08.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:08 smithi193 sudo[45455]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:09.213 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:08 smithi135 sudo[44813]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-10T00:19:09.214 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:08 smithi135 sudo[44813]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:09.214 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:08 smithi135 sudo[44813]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:09.214 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:08 smithi135 sudo[44813]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:09.214 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 sudo[44816]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-10T00:19:09.214 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 sudo[44816]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-10T00:19:09.214 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 sudo[44816]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-06-10T00:19:09.214 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 sudo[44816]: pam_unix(sudo:session): session closed for user root 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1966891762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e4414c2-8c5d-4b4a-895e-b3d0157ab36a"}]: dispatch 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1966891762' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e4414c2-8c5d-4b4a-895e-b3d0157ab36a"}]': finished 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: osdmap e23: 4 total, 3 up, 4 in 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:19:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-10T00:19:09.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-10T00:19:09.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-06-10T00:19:09.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:19:09.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:19:09.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:19:09.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2184737609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:19:09.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:09 smithi135 ceph-mon[36424]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-10T00:19:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-10T00:19:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:19:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1966891762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e4414c2-8c5d-4b4a-895e-b3d0157ab36a"}]: dispatch 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1966891762' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e4414c2-8c5d-4b4a-895e-b3d0157ab36a"}]': finished 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: osdmap e23: 4 total, 3 up, 4 in 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:19:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:19:09.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:19:09.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2184737609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:19:09.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:09 smithi193 ceph-mon[36716]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-10T00:19:09.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:09.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-10T00:19:09.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:19:09.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1966891762' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e4414c2-8c5d-4b4a-895e-b3d0157ab36a"}]: dispatch 2024-06-10T00:19:09.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1966891762' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e4414c2-8c5d-4b4a-895e-b3d0157ab36a"}]': finished 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: osdmap e23: 4 total, 3 up, 4 in 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:19:09.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:19:09.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2184737609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:19:09.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:09 smithi121 ceph-mon[32378]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-10T00:19:10.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:10 smithi135 ceph-mon[36424]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-10T00:19:10.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:10 smithi135 ceph-mon[36424]: Cluster is now healthy 2024-06-10T00:19:10.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:10 smithi135 ceph-mon[36424]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:10.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:10 smithi193 ceph-mon[36716]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-10T00:19:10.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:10 smithi193 ceph-mon[36716]: Cluster is now healthy 2024-06-10T00:19:10.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:10 smithi193 ceph-mon[36716]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:10.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:10 smithi121 ceph-mon[32378]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-10T00:19:10.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:10 smithi121 ceph-mon[32378]: Cluster is now healthy 2024-06-10T00:19:10.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:10 smithi121 ceph-mon[32378]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:13.353 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:13 smithi121 ceph-mon[32378]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:13.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:13 smithi193 ceph-mon[36716]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:13.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:13 smithi135 ceph-mon[36424]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:15.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:15 smithi193 ceph-mon[36716]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:15.417 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:15 smithi121 ceph-mon[32378]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:15.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:15 smithi135 ceph-mon[36424]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:16.369 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-10T00:19:16.370 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:16.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-10T00:19:16.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:16.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-10T00:19:16.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:17.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:17 smithi193 ceph-mon[36716]: Deploying daemon osd.3 on smithi121 2024-06-10T00:19:17.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:17 smithi193 ceph-mon[36716]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:17.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:17 smithi121 ceph-mon[32378]: Deploying daemon osd.3 on smithi121 2024-06-10T00:19:17.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:17 smithi121 ceph-mon[32378]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:17 smithi135 ceph-mon[36424]: Deploying daemon osd.3 on smithi121 2024-06-10T00:19:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:17 smithi135 ceph-mon[36424]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:19.143 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:19 smithi121 ceph-mon[32378]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:19.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:19 smithi193 ceph-mon[36716]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:19.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:19 smithi135 ceph-mon[36424]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:21 smithi193 ceph-mon[36716]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:21 smithi121 ceph-mon[32378]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:21.409 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:21.410 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:21 smithi135 ceph-mon[36424]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:23.296 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:23 smithi121 ceph-mon[32378]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:23.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:23 smithi193 ceph-mon[36716]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:23.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:23 smithi135 ceph-mon[36424]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:24.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.417+0000 7fe81b7fe700 1 -- 172.21.15.121:0/3970156965 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe81c1a0f40 con 0x7fe8040a2b90 2024-06-10T00:19:24.417 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 3 on host 'smithi121' 2024-06-10T00:19:24.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 -- 172.21.15.121:0/3970156965 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe8040a2b90 msgr2=0x7fe8040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:24.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe8040a2b90 0x7fe8040a5050 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe81c102440 tx=0x7fe81400b040 comp rx=0 tx=0).stop 2024-06-10T00:19:24.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 -- 172.21.15.121:0/3970156965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81c102f80 msgr2=0x7fe81c19fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:24.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81c102f80 0x7fe81c19fda0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00d9b0 tx=0x7fe80c00dcc0 comp rx=0 tx=0).stop 2024-06-10T00:19:24.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 -- 172.21.15.121:0/3970156965 shutdown_connections 2024-06-10T00:19:24.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe8040a2b90 0x7fe8040a5050 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:24.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81c102f80 0x7fe81c19fda0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:24.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe81c1013e0 0x7fe81c1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:24.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 --2- 172.21.15.121:0/3970156965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe81c1025e0 0x7fe81c198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:24.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 -- 172.21.15.121:0/3970156965 >> 172.21.15.121:0/3970156965 conn(0x7fe81c0fc980 msgr2=0x7fe81c0fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:19:24.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 -- 172.21.15.121:0/3970156965 shutdown_connections 2024-06-10T00:19:24.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:19:24.421+0000 7fe8246a6700 1 -- 172.21.15.121:0/3970156965 wait complete. 2024-06-10T00:19:24.679 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:24 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:24.679 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:24 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:24.680 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:24 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:24.680 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:24 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:24.680 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:24 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:24.680 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:24 smithi121 ceph-mon[32378]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:24.680 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:24 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:24.680 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:24 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:24.680 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:24 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:24.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:24 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:24.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:24 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:24.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:24 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:24.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:24 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:24.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:24 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:24.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:24 smithi193 ceph-mon[36716]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:24.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:24 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:24.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:24 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:24.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:24 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:25.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:24 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:25.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:24 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:25.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:24 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:25.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:24 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:25.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:24 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:25.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:24 smithi135 ceph-mon[36424]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:25.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:24 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:25.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:24 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:25.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:24 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:25.072 DEBUG:teuthology.orchestra.run.smithi121:osd.3> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.3.service 2024-06-10T00:19:25.074 INFO:tasks.cephadm:Deploying osd.4 on smithi135 with /dev/vg_nvme/lv_3... 2024-06-10T00:19:25.074 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-10T00:19:25.318 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:19:25.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:25 smithi121 ceph-mon[32378]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-10T00:19:25.708 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:25 smithi135 ceph-mon[36424]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-10T00:19:25.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:25 smithi193 ceph-mon[36716]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-10T00:19:26.767 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:26 smithi121 ceph-mon[32378]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-10T00:19:26.767 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:26 smithi121 ceph-mon[32378]: osdmap e24: 4 total, 3 up, 4 in 2024-06-10T00:19:26.768 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:26 smithi121 ceph-mon[32378]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:19:26.768 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:26 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:26.768 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:26 smithi121 ceph-mon[32378]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:26.768 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:19:26 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3[53872]: 2024-06-10T00:19:26.610+0000 7fc43d05b700 -1 osd.3 0 waiting for initial osdmap 2024-06-10T00:19:26.769 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:19:26 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3[53872]: 2024-06-10T00:19:26.616+0000 7fc438e4e700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:19:26.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:26 smithi193 ceph-mon[36716]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-10T00:19:26.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:26 smithi193 ceph-mon[36716]: osdmap e24: 4 total, 3 up, 4 in 2024-06-10T00:19:26.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:26 smithi193 ceph-mon[36716]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:19:26.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:26.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:26 smithi193 ceph-mon[36716]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:27.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:26 smithi135 ceph-mon[36424]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-10T00:19:27.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:26 smithi135 ceph-mon[36424]: osdmap e24: 4 total, 3 up, 4 in 2024-06-10T00:19:27.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:26 smithi135 ceph-mon[36424]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:19:27.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:26 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:27.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:26 smithi135 ceph-mon[36424]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-06-10T00:19:27.064 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-10T00:19:27.064 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-10T00:19:27.064 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-06-10T00:19:27.064 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-06-10T00:19:27.064 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112341 s, 933 MB/s 2024-06-10T00:19:27.064 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-06-10T00:19:27.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:27 smithi121 ceph-mon[32378]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:19:27.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:27 smithi121 ceph-mon[32378]: osdmap e25: 4 total, 3 up, 4 in 2024-06-10T00:19:27.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:27 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:27.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:27 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:27.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:27 smithi121 ceph-mon[32378]: osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067] boot 2024-06-10T00:19:27.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:27 smithi121 ceph-mon[32378]: osdmap e26: 4 total, 4 up, 4 in 2024-06-10T00:19:27.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:27 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:27.703 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:27 smithi135 ceph-mon[36424]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:19:27.703 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:27 smithi135 ceph-mon[36424]: osdmap e25: 4 total, 3 up, 4 in 2024-06-10T00:19:27.703 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:27 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:27.703 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:27 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:27.703 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:27 smithi135 ceph-mon[36424]: osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067] boot 2024-06-10T00:19:27.703 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:27 smithi135 ceph-mon[36424]: osdmap e26: 4 total, 4 up, 4 in 2024-06-10T00:19:27.703 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:27 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:27.704 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_3 2024-06-10T00:19:27.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:27 smithi193 ceph-mon[36716]: from='osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:19:27.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:27 smithi193 ceph-mon[36716]: osdmap e25: 4 total, 3 up, 4 in 2024-06-10T00:19:27.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:27 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:27.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:27 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:27.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:27 smithi193 ceph-mon[36716]: osd.3 [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067] boot 2024-06-10T00:19:27.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:27 smithi193 ceph-mon[36716]: osdmap e26: 4 total, 4 up, 4 in 2024-06-10T00:19:27.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:27 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:19:27.937 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:19:28.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:19:28.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:19:28.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:28.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi121 2024-06-10T00:19:28.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:28.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:28.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi121 to 8544M 2024-06-10T00:19:28.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:28.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:28.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:28 smithi193 ceph-mon[36716]: osdmap e27: 4 total, 4 up, 4 in 2024-06-10T00:19:28.909 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:19:28.909 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:19:28.909 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:28.909 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi121 2024-06-10T00:19:28.909 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:28.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:28.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi121 to 8544M 2024-06-10T00:19:28.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:28.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:28.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.910 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:28 smithi135 ceph-mon[36424]: osdmap e27: 4 total, 4 up, 4 in 2024-06-10T00:19:28.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:19:28.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:19:28.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:28.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi121 2024-06-10T00:19:28.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:28.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:28.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi121 to 8544M 2024-06-10T00:19:28.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:28.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:28.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:28.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:28 smithi121 ceph-mon[32378]: osdmap e27: 4 total, 4 up, 4 in 2024-06-10T00:19:29.434 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.432+0000 7f8282e4b700 1 -- 172.21.15.135:0/2108863485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f827c102520 msgr2=0x7f827c102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:29.434 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.432+0000 7f8282e4b700 1 --2- 172.21.15.135:0/2108863485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f827c102520 0x7f827c102940 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f82700099c0 tx=0x7f8270009cd0 comp rx=0 tx=0).stop 2024-06-10T00:19:29.434 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.432+0000 7f8282e4b700 1 -- 172.21.15.135:0/2108863485 shutdown_connections 2024-06-10T00:19:29.434 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.432+0000 7f8282e4b700 1 --2- 172.21.15.135:0/2108863485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f827c1040c0 0x7f827c10a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:29.434 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.432+0000 7f8282e4b700 1 --2- 172.21.15.135:0/2108863485 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f827c102520 0x7f827c102940 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:29.434 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.432+0000 7f8282e4b700 1 --2- 172.21.15.135:0/2108863485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f827c103720 0x7f827c103b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.432+0000 7f8282e4b700 1 -- 172.21.15.135:0/2108863485 >> 172.21.15.135:0/2108863485 conn(0x7f827c0fdae0 msgr2=0x7f827c0fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.432+0000 7f8282e4b700 1 -- 172.21.15.135:0/2108863485 shutdown_connections 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.432+0000 7f8282e4b700 1 -- 172.21.15.135:0/2108863485 wait complete. 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f8282e4b700 1 Processor -- start 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f8282e4b700 1 -- start start 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f8282e4b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f827c102520 0x7f827c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f8282e4b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f827c103720 0x7f827c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f8282e4b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f827c1040c0 0x7f827c19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f8282e4b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f827c10e100 con 0x7f827c1040c0 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f8282e4b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f827c10df80 con 0x7f827c103720 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f8282e4b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f827c10e280 con 0x7f827c102520 2024-06-10T00:19:29.435 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f827bfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f827c103720 0x7f827c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f827bfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f827c103720 0x7f827c198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:37236/0 (socket says 172.21.15.135:37236) 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f827bfff700 1 -- 172.21.15.135:0/544341802 learned_addr learned my addr 172.21.15.135:0/544341802 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f82813e8700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f827c1040c0 0x7f827c19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f8280be7700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f827c102520 0x7f827c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f827bfff700 1 -- 172.21.15.135:0/544341802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f827c102520 msgr2=0x7f827c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f827bfff700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f827c102520 0x7f827c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f827bfff700 1 -- 172.21.15.135:0/544341802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f827c1040c0 msgr2=0x7f827c19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f827bfff700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f827c1040c0 0x7f827c19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.433+0000 7f827bfff700 1 -- 172.21.15.135:0/544341802 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827c1a02e0 con 0x7f827c103720 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.434+0000 7f82813e8700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f827c1040c0 0x7f827c19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.434+0000 7f827bfff700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f827c103720 0x7f827c198bc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f826800c9b0 tx=0x7f826800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.434+0000 7f8279ffb700 1 -- 172.21.15.135:0/544341802 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268007b00 con 0x7f827c103720 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.434+0000 7f8279ffb700 1 -- 172.21.15.135:0/544341802 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f826800ce50 con 0x7f827c103720 2024-06-10T00:19:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.434+0000 7f8282e4b700 1 -- 172.21.15.135:0/544341802 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f827c1a05d0 con 0x7f827c103720 2024-06-10T00:19:29.437 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.434+0000 7f8279ffb700 1 -- 172.21.15.135:0/544341802 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268018660 con 0x7f827c103720 2024-06-10T00:19:29.437 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.434+0000 7f8282e4b700 1 -- 172.21.15.135:0/544341802 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f827c1a0b10 con 0x7f827c103720 2024-06-10T00:19:29.437 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.435+0000 7f8282e4b700 1 -- 172.21.15.135:0/544341802 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f827c069180 con 0x7f827c103720 2024-06-10T00:19:29.438 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.436+0000 7f8279ffb700 1 -- 172.21.15.135:0/544341802 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8268018800 con 0x7f827c103720 2024-06-10T00:19:29.438 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.436+0000 7f8279ffb700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f826c0a2c20 0x7f826c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:29.438 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.437+0000 7f8279ffb700 1 -- 172.21.15.135:0/544341802 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7f82680cdd50 con 0x7f827c103720 2024-06-10T00:19:29.438 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.437+0000 7f8280be7700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f826c0a2c20 0x7f826c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:29.438 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.437+0000 7f8280be7700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f826c0a2c20 0x7f826c0a50e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8270003890 tx=0x7f8270005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:19:29.441 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.441+0000 7f8279ffb700 1 -- 172.21.15.135:0/544341802 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f82680d1030 con 0x7f827c103720 2024-06-10T00:19:29.649 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:29.650+0000 7f8282e4b700 1 -- 172.21.15.135:0/544341802 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f827c0628d0 con 0x7f826c0a2c20 2024-06-10T00:19:30.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:30 smithi121 ceph-mon[32378]: osdmap e28: 4 total, 4 up, 4 in 2024-06-10T00:19:30.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:30 smithi121 ceph-mon[32378]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:19:30.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:30 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:19:30.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:30 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:19:30.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:30 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:30.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:30 smithi121 ceph-mon[32378]: osdmap e29: 4 total, 4 up, 4 in 2024-06-10T00:19:30.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:30 smithi121 ceph-mon[32378]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:30.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:30 smithi193 ceph-mon[36716]: osdmap e28: 4 total, 4 up, 4 in 2024-06-10T00:19:30.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:30 smithi193 ceph-mon[36716]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:19:30.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:30 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:19:30.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:30 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:19:30.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:30 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:30.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:30 smithi193 ceph-mon[36716]: osdmap e29: 4 total, 4 up, 4 in 2024-06-10T00:19:30.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:30 smithi193 ceph-mon[36716]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:30.892 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:30 smithi135 ceph-mon[36424]: osdmap e28: 4 total, 4 up, 4 in 2024-06-10T00:19:30.893 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:30 smithi135 ceph-mon[36424]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:19:30.893 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:30 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:19:30.893 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:30 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:19:30.893 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:30 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:30.893 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:30 smithi135 ceph-mon[36424]: osdmap e29: 4 total, 4 up, 4 in 2024-06-10T00:19:30.893 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:30 smithi135 ceph-mon[36424]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:32.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:31 smithi135 ceph-mon[36424]: osdmap e30: 4 total, 4 up, 4 in 2024-06-10T00:19:32.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:31 smithi193 ceph-mon[36716]: osdmap e30: 4 total, 4 up, 4 in 2024-06-10T00:19:32.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:31 smithi121 ceph-mon[32378]: osdmap e30: 4 total, 4 up, 4 in 2024-06-10T00:19:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:32 smithi135 ceph-mon[36424]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-10T00:19:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:32 smithi135 ceph-mon[36424]: from='client.? 172.21.15.135:0/2572190501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6e3f25-d136-4698-8657-ca62ff8d4ffe"}]: dispatch 2024-06-10T00:19:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:32 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6e3f25-d136-4698-8657-ca62ff8d4ffe"}]: dispatch 2024-06-10T00:19:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:32 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a6e3f25-d136-4698-8657-ca62ff8d4ffe"}]': finished 2024-06-10T00:19:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:32 smithi135 ceph-mon[36424]: osdmap e31: 5 total, 4 up, 5 in 2024-06-10T00:19:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:32 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:33.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:32 smithi193 ceph-mon[36716]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-10T00:19:33.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:32 smithi193 ceph-mon[36716]: from='client.? 172.21.15.135:0/2572190501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6e3f25-d136-4698-8657-ca62ff8d4ffe"}]: dispatch 2024-06-10T00:19:33.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:32 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6e3f25-d136-4698-8657-ca62ff8d4ffe"}]: dispatch 2024-06-10T00:19:33.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:32 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a6e3f25-d136-4698-8657-ca62ff8d4ffe"}]': finished 2024-06-10T00:19:33.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:32 smithi193 ceph-mon[36716]: osdmap e31: 5 total, 4 up, 5 in 2024-06-10T00:19:33.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:32 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:32 smithi121 ceph-mon[32378]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-10T00:19:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:32 smithi121 ceph-mon[32378]: from='client.? 172.21.15.135:0/2572190501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6e3f25-d136-4698-8657-ca62ff8d4ffe"}]: dispatch 2024-06-10T00:19:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:32 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a6e3f25-d136-4698-8657-ca62ff8d4ffe"}]: dispatch 2024-06-10T00:19:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:32 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a6e3f25-d136-4698-8657-ca62ff8d4ffe"}]': finished 2024-06-10T00:19:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:32 smithi121 ceph-mon[32378]: osdmap e31: 5 total, 4 up, 5 in 2024-06-10T00:19:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:32 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:34.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:33 smithi135 ceph-mon[36424]: from='client.? 172.21.15.135:0/478524477' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:19:34.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:33 smithi193 ceph-mon[36716]: from='client.? 172.21.15.135:0/478524477' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:19:34.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:33 smithi121 ceph-mon[32378]: from='client.? 172.21.15.135:0/478524477' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:19:35.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:34 smithi135 ceph-mon[36424]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-06-10T00:19:35.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:34 smithi193 ceph-mon[36716]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-06-10T00:19:35.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:34 smithi121 ceph-mon[32378]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-06-10T00:19:37.352 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:37 smithi135 ceph-mon[36424]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-06-10T00:19:37.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:37 smithi193 ceph-mon[36716]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-06-10T00:19:37.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:37 smithi121 ceph-mon[32378]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-06-10T00:19:39.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:39 smithi135 ceph-mon[36424]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-10T00:19:39.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:39 smithi193 ceph-mon[36716]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-10T00:19:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:39 smithi121 ceph-mon[32378]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-10T00:19:40.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-10T00:19:40.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:40.354 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:40 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-10T00:19:40.354 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:40 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:40.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-10T00:19:40.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:41.368 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:41 smithi135 ceph-mon[36424]: Deploying daemon osd.4 on smithi135 2024-06-10T00:19:41.368 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:41 smithi135 ceph-mon[36424]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:41.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:41 smithi193 ceph-mon[36716]: Deploying daemon osd.4 on smithi135 2024-06-10T00:19:41.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:41 smithi193 ceph-mon[36716]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:41.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:41 smithi121 ceph-mon[32378]: Deploying daemon osd.4 on smithi135 2024-06-10T00:19:41.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:41 smithi121 ceph-mon[32378]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:43.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:43 smithi193 ceph-mon[36716]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:43.423 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:43 smithi135 ceph-mon[36424]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:43.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:43 smithi121 ceph-mon[32378]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:45.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:45 smithi193 ceph-mon[36716]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:45 smithi121 ceph-mon[32378]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:45 smithi135 ceph-mon[36424]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:46.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:46.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:46.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:46.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:46.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:46.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:46.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:46.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:46.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:47.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:47 smithi135 ceph-mon[36424]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:47.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:47 smithi193 ceph-mon[36716]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:47 smithi121 ceph-mon[32378]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:48.611 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.611+0000 7f8279ffb700 1 -- 172.21.15.135:0/544341802 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f827c0628d0 con 0x7f826c0a2c20 2024-06-10T00:19:48.611 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 4 on host 'smithi135' 2024-06-10T00:19:48.613 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 -- 172.21.15.135:0/544341802 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f826c0a2c20 msgr2=0x7f826c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:48.613 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f826c0a2c20 0x7f826c0a50e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8270003890 tx=0x7f8270005fb0 comp rx=0 tx=0).stop 2024-06-10T00:19:48.613 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 -- 172.21.15.135:0/544341802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f827c103720 msgr2=0x7f827c198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:48.613 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f827c103720 0x7f827c198bc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f826800c9b0 tx=0x7f826800ccc0 comp rx=0 tx=0).stop 2024-06-10T00:19:48.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 -- 172.21.15.135:0/544341802 shutdown_connections 2024-06-10T00:19:48.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f826c0a2c20 0x7f826c0a50e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:48.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f827c1040c0 0x7f827c19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:48.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f827c102520 0x7f827c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:48.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 --2- 172.21.15.135:0/544341802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f827c103720 0x7f827c198bc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:48.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 -- 172.21.15.135:0/544341802 >> 172.21.15.135:0/544341802 conn(0x7f827c0fdae0 msgr2=0x7f827c1069c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:19:48.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 -- 172.21.15.135:0/544341802 shutdown_connections 2024-06-10T00:19:48.614 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:19:48.613+0000 7f8282e4b700 1 -- 172.21.15.135:0/544341802 wait complete. 2024-06-10T00:19:49.242 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:49 smithi135 ceph-mon[36424]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:49.242 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.242 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.242 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.242 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.242 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:49 smithi135 ceph-mon[36424]: from='osd.4 [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-10T00:19:49.242 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:49 smithi135 ceph-mon[36424]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-10T00:19:49.267 DEBUG:teuthology.orchestra.run.smithi135:osd.4> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.4.service 2024-06-10T00:19:49.269 INFO:tasks.cephadm:Deploying osd.5 on smithi193 with /dev/vg_nvme/lv_3... 2024-06-10T00:19:49.270 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-10T00:19:49.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:49 smithi193 ceph-mon[36716]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:49.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:49 smithi193 ceph-mon[36716]: from='osd.4 [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-10T00:19:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:49 smithi193 ceph-mon[36716]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-10T00:19:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:49 smithi121 ceph-mon[32378]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:49 smithi121 ceph-mon[32378]: from='osd.4 [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-10T00:19:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:49 smithi121 ceph-mon[32378]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-10T00:19:49.513 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:19:50.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:50 smithi121 ceph-mon[32378]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-10T00:19:50.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:50 smithi121 ceph-mon[32378]: from='osd.4 [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:19:50.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:50 smithi121 ceph-mon[32378]: osdmap e32: 5 total, 4 up, 5 in 2024-06-10T00:19:50.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:50 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:50.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:50 smithi121 ceph-mon[32378]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:19:50.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:50 smithi121 ceph-mon[32378]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:50.986 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:50 smithi193 ceph-mon[36716]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-10T00:19:50.986 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:50 smithi193 ceph-mon[36716]: from='osd.4 [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:19:50.986 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:50 smithi193 ceph-mon[36716]: osdmap e32: 5 total, 4 up, 5 in 2024-06-10T00:19:50.986 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:50 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:50.986 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:50 smithi193 ceph-mon[36716]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:19:50.986 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:50 smithi193 ceph-mon[36716]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:51.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:50 smithi135 ceph-mon[36424]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-10T00:19:51.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:50 smithi135 ceph-mon[36424]: from='osd.4 [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:19:51.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:50 smithi135 ceph-mon[36424]: osdmap e32: 5 total, 4 up, 5 in 2024-06-10T00:19:51.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:50 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:51.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:50 smithi135 ceph-mon[36424]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:19:51.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:50 smithi135 ceph-mon[36424]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-06-10T00:19:51.008 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:19:50 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4[47587]: 2024-06-10T00:19:50.692+0000 7f5c0b171700 -1 osd.4 0 waiting for initial osdmap 2024-06-10T00:19:51.008 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:19:50 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4[47587]: 2024-06-10T00:19:50.698+0000 7f5c06f64700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:19:51.541 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-10T00:19:51.541 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-10T00:19:51.541 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-06-10T00:19:51.541 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-06-10T00:19:51.541 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111111 s, 944 MB/s 2024-06-10T00:19:51.541 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-06-10T00:19:51.865 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:51 smithi135 ceph-mon[36424]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:19:51.865 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:51 smithi135 ceph-mon[36424]: osdmap e33: 5 total, 4 up, 5 in 2024-06-10T00:19:51.865 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:51.865 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:51.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:51 smithi121 ceph-mon[32378]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:19:51.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:51 smithi121 ceph-mon[32378]: osdmap e33: 5 total, 4 up, 5 in 2024-06-10T00:19:51.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:51.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:52.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:51 smithi193 ceph-mon[36716]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:19:52.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:51 smithi193 ceph-mon[36716]: osdmap e33: 5 total, 4 up, 5 in 2024-06-10T00:19:52.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:52.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:52.218 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_3 2024-06-10T00:19:52.462 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:19:52.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:19:52.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:19:52.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: osd.4 [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459] boot 2024-06-10T00:19:52.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: osdmap e34: 5 total, 5 up, 5 in 2024-06-10T00:19:52.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:52.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:19:52.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi135 2024-06-10T00:19:52.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:52.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:52.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:52.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:52.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi135 to 8544M 2024-06-10T00:19:52.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:52.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:52.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:52.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:52.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:52.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:52.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:52.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:19:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:19:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: osd.4 [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459] boot 2024-06-10T00:19:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: osdmap e34: 5 total, 5 up, 5 in 2024-06-10T00:19:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:19:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi135 2024-06-10T00:19:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:52.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:52.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:52.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi135 to 8544M 2024-06-10T00:19:52.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:52.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:52.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:52.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:52.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:52.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:52.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:52.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:53.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: osd.4 [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459] boot 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: osdmap e34: 5 total, 5 up, 5 in 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi135 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi135 to 8544M 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:53.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:53.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:19:53.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:53.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:19:53.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:19:53.856 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:53 smithi193 ceph-mon[36716]: osdmap e35: 5 total, 5 up, 5 in 2024-06-10T00:19:53.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:53 smithi121 ceph-mon[32378]: osdmap e35: 5 total, 5 up, 5 in 2024-06-10T00:19:54.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:53 smithi135 ceph-mon[36424]: osdmap e35: 5 total, 5 up, 5 in 2024-06-10T00:19:54.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.078+0000 7f057bb7e700 1 -- 172.21.15.193:0/2823694770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0574102f60 msgr2=0x7f0574109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:54.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.078+0000 7f057bb7e700 1 --2- 172.21.15.193:0/2823694770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0574102f60 0x7f0574109810 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f056c00b0a0 tx=0x7f056c00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:19:54.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.078+0000 7f057bb7e700 1 -- 172.21.15.193:0/2823694770 shutdown_connections 2024-06-10T00:19:54.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.078+0000 7f057bb7e700 1 --2- 172.21.15.193:0/2823694770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0574102f60 0x7f0574109810 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:54.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.078+0000 7f057bb7e700 1 --2- 172.21.15.193:0/2823694770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f05741013c0 0x7f05741017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:54.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.078+0000 7f057bb7e700 1 --2- 172.21.15.193:0/2823694770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05741025c0 0x7f0574102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:54.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.078+0000 7f057bb7e700 1 -- 172.21.15.193:0/2823694770 >> 172.21.15.193:0/2823694770 conn(0x7f05740fc980 msgr2=0x7f05740feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:19:54.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.078+0000 7f057bb7e700 1 -- 172.21.15.193:0/2823694770 shutdown_connections 2024-06-10T00:19:54.081 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.078+0000 7f057bb7e700 1 -- 172.21.15.193:0/2823694770 wait complete. 2024-06-10T00:19:54.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f057bb7e700 1 Processor -- start 2024-06-10T00:19:54.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f057bb7e700 1 -- start start 2024-06-10T00:19:54.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f057bb7e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05741013c0 0x7f05741985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:54.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f057bb7e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05741025c0 0x7f0574198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:54.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f057bb7e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0574102f60 0x7f057419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:54.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f057bb7e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f057410cf90 con 0x7f05741013c0 2024-06-10T00:19:54.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f057bb7e700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f057410ce10 con 0x7f0574102f60 2024-06-10T00:19:54.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f057bb7e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f057410d110 con 0x7f05741025c0 2024-06-10T00:19:54.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f0579119700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05741025c0 0x7f0574198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:54.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f0579119700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05741025c0 0x7f0574198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:34860/0 (socket says 172.21.15.193:34860) 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f0579119700 1 -- 172.21.15.193:0/2527680580 learned_addr learned my addr 172.21.15.193:0/2527680580 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f057991a700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05741013c0 0x7f05741985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.079+0000 7f057a11b700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0574102f60 0x7f057419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f0579119700 1 -- 172.21.15.193:0/2527680580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0574102f60 msgr2=0x7f057419fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f0579119700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0574102f60 0x7f057419fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f0579119700 1 -- 172.21.15.193:0/2527680580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05741013c0 msgr2=0x7f05741985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f0579119700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05741013c0 0x7f05741985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f0579119700 1 -- 172.21.15.193:0/2527680580 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05741a0200 con 0x7f05741025c0 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f057991a700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05741013c0 0x7f05741985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f0579119700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05741025c0 0x7f0574198ae0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f057000c950 tx=0x7f057000cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f056affd700 1 -- 172.21.15.193:0/2527680580 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0570007ac0 con 0x7f05741025c0 2024-06-10T00:19:54.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f057bb7e700 1 -- 172.21.15.193:0/2527680580 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05741a04f0 con 0x7f05741025c0 2024-06-10T00:19:54.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f056affd700 1 -- 172.21.15.193:0/2527680580 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0570004d10 con 0x7f05741025c0 2024-06-10T00:19:54.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f056affd700 1 -- 172.21.15.193:0/2527680580 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0570005680 con 0x7f05741025c0 2024-06-10T00:19:54.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.080+0000 7f057bb7e700 1 -- 172.21.15.193:0/2527680580 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05741a0a80 con 0x7f05741025c0 2024-06-10T00:19:54.084 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.081+0000 7f057bb7e700 1 -- 172.21.15.193:0/2527680580 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0574069180 con 0x7f05741025c0 2024-06-10T00:19:54.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.082+0000 7f056affd700 1 -- 172.21.15.193:0/2527680580 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0570004750 con 0x7f05741025c0 2024-06-10T00:19:54.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.082+0000 7f056affd700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f05600a2c20 0x7f05600a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:19:54.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.083+0000 7f056affd700 1 -- 172.21.15.193:0/2527680580 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f05700cdc60 con 0x7f05741025c0 2024-06-10T00:19:54.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.083+0000 7f057991a700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f05600a2c20 0x7f05600a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:19:54.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.083+0000 7f057991a700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f05600a2c20 0x7f05600a50e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0574102420 tx=0x7f056400a680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:19:54.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.087+0000 7f056affd700 1 -- 172.21.15.193:0/2527680580 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0570014030 con 0x7f05741025c0 2024-06-10T00:19:54.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:19:54.289+0000 7f057bb7e700 1 -- 172.21.15.193:0/2527680580 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f05740fea90 con 0x7f05600a2c20 2024-06-10T00:19:55.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:54 smithi135 ceph-mon[36424]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:19:55.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:54 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:19:55.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:54 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:19:55.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:54 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:55.069 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:54 smithi193 ceph-mon[36716]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:19:55.069 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:54 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:19:55.069 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:54 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:19:55.070 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:54 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:55.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:54 smithi121 ceph-mon[32378]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:19:55.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:54 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:19:55.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:54 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:19:55.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:54 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:19:55.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:55 smithi193 ceph-mon[36716]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:19:56.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:55 smithi135 ceph-mon[36424]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:19:56.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:55 smithi121 ceph-mon[32378]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:19:57.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:56 smithi135 ceph-mon[36424]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:19:57.100 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:56 smithi121 ceph-mon[32378]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:19:57.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:56 smithi193 ceph-mon[36716]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:19:57.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:57 smithi193 ceph-mon[36716]: from='client.? 172.21.15.193:0/2148428866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62361748-4968-4095-9e32-756c6cfdfaf0"}]: dispatch 2024-06-10T00:19:57.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:57 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62361748-4968-4095-9e32-756c6cfdfaf0"}]: dispatch 2024-06-10T00:19:57.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:57 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62361748-4968-4095-9e32-756c6cfdfaf0"}]': finished 2024-06-10T00:19:57.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:57 smithi193 ceph-mon[36716]: osdmap e36: 6 total, 5 up, 6 in 2024-06-10T00:19:57.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:57 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:19:58.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:57 smithi135 ceph-mon[36424]: from='client.? 172.21.15.193:0/2148428866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62361748-4968-4095-9e32-756c6cfdfaf0"}]: dispatch 2024-06-10T00:19:58.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:57 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62361748-4968-4095-9e32-756c6cfdfaf0"}]: dispatch 2024-06-10T00:19:58.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:57 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62361748-4968-4095-9e32-756c6cfdfaf0"}]': finished 2024-06-10T00:19:58.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:57 smithi135 ceph-mon[36424]: osdmap e36: 6 total, 5 up, 6 in 2024-06-10T00:19:58.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:57 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:19:58.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:57 smithi121 ceph-mon[32378]: from='client.? 172.21.15.193:0/2148428866' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62361748-4968-4095-9e32-756c6cfdfaf0"}]: dispatch 2024-06-10T00:19:58.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:57 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62361748-4968-4095-9e32-756c6cfdfaf0"}]: dispatch 2024-06-10T00:19:58.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:57 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62361748-4968-4095-9e32-756c6cfdfaf0"}]': finished 2024-06-10T00:19:58.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:57 smithi121 ceph-mon[32378]: osdmap e36: 6 total, 5 up, 6 in 2024-06-10T00:19:58.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:57 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:19:59.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:58 smithi135 ceph-mon[36424]: from='client.? 172.21.15.193:0/3472856938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:19:59.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:19:58 smithi135 ceph-mon[36424]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:19:59.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:58 smithi193 ceph-mon[36716]: from='client.? 172.21.15.193:0/3472856938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:19:59.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:19:58 smithi193 ceph-mon[36716]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:19:59.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:58 smithi121 ceph-mon[32378]: from='client.? 172.21.15.193:0/3472856938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:19:59.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:19:58 smithi121 ceph-mon[32378]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:00.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:00 smithi193 ceph-mon[36716]: overall HEALTH_OK 2024-06-10T00:20:00.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:00 smithi121 ceph-mon[32378]: overall HEALTH_OK 2024-06-10T00:20:00.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:00 smithi135 ceph-mon[36424]: overall HEALTH_OK 2024-06-10T00:20:01.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:01 smithi193 ceph-mon[36716]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:01 smithi121 ceph-mon[32378]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:01.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:01 smithi135 ceph-mon[36424]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:03.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:03 smithi193 ceph-mon[36716]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:03 smithi121 ceph-mon[32378]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:03.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:03 smithi135 ceph-mon[36424]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:05.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:05 smithi193 ceph-mon[36716]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:05.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-10T00:20:05.366 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:05 smithi121 ceph-mon[32378]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-10T00:20:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:05.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:05 smithi135 ceph-mon[36424]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:05.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-10T00:20:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:06.291 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:06 smithi193 ceph-mon[36716]: Deploying daemon osd.5 on smithi193 2024-06-10T00:20:06.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:06 smithi121 ceph-mon[32378]: Deploying daemon osd.5 on smithi193 2024-06-10T00:20:06.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:06 smithi135 ceph-mon[36424]: Deploying daemon osd.5 on smithi193 2024-06-10T00:20:07.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:07 smithi193 ceph-mon[36716]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:07.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:07 smithi121 ceph-mon[32378]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:07.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:07 smithi135 ceph-mon[36424]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:09.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:09 smithi193 ceph-mon[36716]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:09.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:09 smithi121 ceph-mon[32378]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:09 smithi135 ceph-mon[36424]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:11.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:10 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:11.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:10 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:11.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:10 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:11.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:10 smithi193 ceph-mon[36716]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:11.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:10 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:11.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:10 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:11.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:10 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:11.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:10 smithi121 ceph-mon[32378]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:11.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:10 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:11.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:10 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:11.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:10 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:11.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:10 smithi135 ceph-mon[36424]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:13.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.153+0000 7f056affd700 1 -- 172.21.15.193:0/2527680580 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f05740fea90 con 0x7f05600a2c20 2024-06-10T00:20:13.153 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 5 on host 'smithi193' 2024-06-10T00:20:13.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 -- 172.21.15.193:0/2527680580 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f05600a2c20 msgr2=0x7f05600a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:20:13.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f05600a2c20 0x7f05600a50e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0574102420 tx=0x7f056400a680 comp rx=0 tx=0).stop 2024-06-10T00:20:13.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 -- 172.21.15.193:0/2527680580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05741025c0 msgr2=0x7f0574198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:20:13.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05741025c0 0x7f0574198ae0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f057000c950 tx=0x7f057000cd10 comp rx=0 tx=0).stop 2024-06-10T00:20:13.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 -- 172.21.15.193:0/2527680580 shutdown_connections 2024-06-10T00:20:13.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f05600a2c20 0x7f05600a50e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:13.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0574102f60 0x7f057419fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:13.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05741013c0 0x7f05741985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:13.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 --2- 172.21.15.193:0/2527680580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05741025c0 0x7f0574198ae0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:13.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 -- 172.21.15.193:0/2527680580 >> 172.21.15.193:0/2527680580 conn(0x7f05740fc980 msgr2=0x7f05740fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:20:13.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 -- 172.21.15.193:0/2527680580 shutdown_connections 2024-06-10T00:20:13.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:20:13.155+0000 7f057bb7e700 1 -- 172.21.15.193:0/2527680580 wait complete. 2024-06-10T00:20:13.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:13 smithi193 ceph-mon[36716]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:13.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:13 smithi121 ceph-mon[32378]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:13.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:13 smithi135 ceph-mon[36424]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:14.149 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.5.service 2024-06-10T00:20:14.151 INFO:tasks.cephadm:Deploying osd.6 on smithi121 with /dev/vg_nvme/lv_2... 2024-06-10T00:20:14.151 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-10T00:20:14.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:14 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:14.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:14 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:14.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:14 smithi193 ceph-mon[36716]: from='osd.5 [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-10T00:20:14.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:14 smithi193 ceph-mon[36716]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-10T00:20:14.400 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:20:14.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:14 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:14.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:14 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:14.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:14 smithi121 ceph-mon[32378]: from='osd.5 [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-10T00:20:14.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:14 smithi121 ceph-mon[32378]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-10T00:20:14.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:14 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:14.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:14 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:14.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:14 smithi135 ceph-mon[36424]: from='osd.5 [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-10T00:20:14.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:14 smithi135 ceph-mon[36424]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-10T00:20:15.237 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:15 smithi193 ceph-mon[36716]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:15.237 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:15 smithi193 ceph-mon[36716]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-10T00:20:15.237 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:15 smithi193 ceph-mon[36716]: osdmap e37: 6 total, 5 up, 6 in 2024-06-10T00:20:15.237 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:15 smithi193 ceph-mon[36716]: from='osd.5 [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:20:15.237 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:15 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:15.237 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:15 smithi193 ceph-mon[36716]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:20:15.237 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:15 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:15.238 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:15 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:15 smithi121 ceph-mon[32378]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:15 smithi121 ceph-mon[32378]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-10T00:20:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:15 smithi121 ceph-mon[32378]: osdmap e37: 6 total, 5 up, 6 in 2024-06-10T00:20:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:15 smithi121 ceph-mon[32378]: from='osd.5 [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:20:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:15 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:15.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:15 smithi121 ceph-mon[32378]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:20:15.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:15 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:15.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:15 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:15.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:15 smithi135 ceph-mon[36424]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:15.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:15 smithi135 ceph-mon[36424]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-10T00:20:15.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:15 smithi135 ceph-mon[36424]: osdmap e37: 6 total, 5 up, 6 in 2024-06-10T00:20:15.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:15 smithi135 ceph-mon[36424]: from='osd.5 [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:20:15.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:15 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:15.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:15 smithi135 ceph-mon[36424]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:20:15.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:15 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:15.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:15 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:15.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:20:15 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:20:15.536+0000 7f8d4eab3700 -1 osd.5 0 waiting for initial osdmap 2024-06-10T00:20:15.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:20:15 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:20:15.542+0000 7f8d4689e700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:20:16.269 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:16 smithi193 ceph-mon[36716]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:20:16.269 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:16 smithi193 ceph-mon[36716]: osdmap e38: 6 total, 5 up, 6 in 2024-06-10T00:20:16.269 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:16.269 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:16.269 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:16 smithi193 ceph-mon[36716]: from='osd.5 ' entity='osd.5' 2024-06-10T00:20:16.270 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:16 smithi193 ceph-mon[36716]: osd.5 [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206] boot 2024-06-10T00:20:16.270 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:16 smithi193 ceph-mon[36716]: osdmap e39: 6 total, 6 up, 6 in 2024-06-10T00:20:16.270 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:16 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:16.383 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-10T00:20:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-10T00:20:16.384 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-06-10T00:20:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-06-10T00:20:16.384 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119419 s, 878 MB/s 2024-06-10T00:20:16.384 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-06-10T00:20:16.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:16 smithi121 ceph-mon[32378]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:20:16.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:16 smithi121 ceph-mon[32378]: osdmap e38: 6 total, 5 up, 6 in 2024-06-10T00:20:16.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:16.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:16.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:16 smithi121 ceph-mon[32378]: from='osd.5 ' entity='osd.5' 2024-06-10T00:20:16.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:16 smithi121 ceph-mon[32378]: osd.5 [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206] boot 2024-06-10T00:20:16.384 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:16 smithi121 ceph-mon[32378]: osdmap e39: 6 total, 6 up, 6 in 2024-06-10T00:20:16.385 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:16 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:16.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:16 smithi135 ceph-mon[36424]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:20:16.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:16 smithi135 ceph-mon[36424]: osdmap e38: 6 total, 5 up, 6 in 2024-06-10T00:20:16.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:16.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:16.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:16 smithi135 ceph-mon[36424]: from='osd.5 ' entity='osd.5' 2024-06-10T00:20:16.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:16 smithi135 ceph-mon[36424]: osd.5 [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206] boot 2024-06-10T00:20:16.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:16 smithi135 ceph-mon[36424]: osdmap e39: 6 total, 6 up, 6 in 2024-06-10T00:20:16.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:16 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:20:17.032 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_2 2024-06-10T00:20:17.283 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:20:17.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:17 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:20:17.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:17 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:20:17.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:17 smithi193 ceph-mon[36716]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:17.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:17 smithi193 ceph-mon[36716]: osdmap e40: 6 total, 6 up, 6 in 2024-06-10T00:20:17.395 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:17 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:20:17.396 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:17 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:20:17.396 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:17 smithi121 ceph-mon[32378]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:17.396 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:17 smithi121 ceph-mon[32378]: osdmap e40: 6 total, 6 up, 6 in 2024-06-10T00:20:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:17 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:20:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:17 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:20:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:17 smithi135 ceph-mon[36424]: pgmap v135: 1 pgs: 1 peering; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-06-10T00:20:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:17 smithi135 ceph-mon[36424]: osdmap e40: 6 total, 6 up, 6 in 2024-06-10T00:20:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.846+0000 7f5aec6f4700 1 -- 172.21.15.121:0/3265540109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae41036f0 msgr2=0x7f5ae4103b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:20:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.846+0000 7f5aec6f4700 1 --2- 172.21.15.121:0/3265540109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae41036f0 0x7f5ae4103b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5ad40099c0 tx=0x7f5ad4009cd0 comp rx=0 tx=0).stop 2024-06-10T00:20:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 -- 172.21.15.121:0/3265540109 shutdown_connections 2024-06-10T00:20:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 --2- 172.21.15.121:0/3265540109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ae4104090 0x7f5ae410a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 --2- 172.21.15.121:0/3265540109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ae41024f0 0x7f5ae4102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 --2- 172.21.15.121:0/3265540109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae41036f0 0x7f5ae4103b50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 -- 172.21.15.121:0/3265540109 >> 172.21.15.121:0/3265540109 conn(0x7f5ae40fda90 msgr2=0x7f5ae40ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:20:18.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 -- 172.21.15.121:0/3265540109 shutdown_connections 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 -- 172.21.15.121:0/3265540109 wait complete. 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 Processor -- start 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 -- start start 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ae41024f0 0x7f5ae4076230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ae41036f0 0x7f5ae4074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4104090 0x7f5ae4074dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae410e1d0 con 0x7f5ae4104090 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae410e050 con 0x7f5ae41036f0 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.847+0000 7f5aec6f4700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae410e350 con 0x7f5ae41024f0 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aeac91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4104090 0x7f5ae4074dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aeac91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4104090 0x7f5ae4074dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34986/0 (socket says 172.21.15.121:34986) 2024-06-10T00:20:18.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aeac91700 1 -- 172.21.15.121:0/240655124 learned_addr learned my addr 172.21.15.121:0/240655124 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aea490700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ae41024f0 0x7f5ae4076230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5ae9c8f700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ae41036f0 0x7f5ae4074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aeac91700 1 -- 172.21.15.121:0/240655124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ae41024f0 msgr2=0x7f5ae4076230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aeac91700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ae41024f0 0x7f5ae4076230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aeac91700 1 -- 172.21.15.121:0/240655124 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ae41036f0 msgr2=0x7f5ae4074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aeac91700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ae41036f0 0x7f5ae4074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aeac91700 1 -- 172.21.15.121:0/240655124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae4075670 con 0x7f5ae4104090 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aeac91700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4104090 0x7f5ae4074dc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00b970 tx=0x7f5adc00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5adb7fe700 1 -- 172.21.15.121:0/240655124 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc00d860 con 0x7f5ae4104090 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5adb7fe700 1 -- 172.21.15.121:0/240655124 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5adc00e430 con 0x7f5ae4104090 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aec6f4700 1 -- 172.21.15.121:0/240655124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae41a29e0 con 0x7f5ae4104090 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5adb7fe700 1 -- 172.21.15.121:0/240655124 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc017660 con 0x7f5ae4104090 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.848+0000 7f5aec6f4700 1 -- 172.21.15.121:0/240655124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae41a2fa0 con 0x7f5ae4104090 2024-06-10T00:20:18.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.849+0000 7f5aec6f4700 1 -- 172.21.15.121:0/240655124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae4069180 con 0x7f5ae4104090 2024-06-10T00:20:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.850+0000 7f5adb7fe700 1 -- 172.21.15.121:0/240655124 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f5adc00e5e0 con 0x7f5ae4104090 2024-06-10T00:20:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.851+0000 7f5adb7fe700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5ad00a2ca0 0x7f5ad00a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:20:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.851+0000 7f5aea490700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5ad00a2ca0 0x7f5ad00a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:20:18.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.851+0000 7f5adb7fe700 1 -- 172.21.15.121:0/240655124 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7f5adc0cdca0 con 0x7f5ae4104090 2024-06-10T00:20:18.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.852+0000 7f5aea490700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5ad00a2ca0 0x7f5ad00a5160 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4103550 tx=0x7f5ae0009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:20:18.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:18.856+0000 7f5adb7fe700 1 -- 172.21.15.121:0/240655124 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5adc013030 con 0x7f5ae4104090 2024-06-10T00:20:19.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:19.062+0000 7f5aec6f4700 1 -- 172.21.15.121:0/240655124 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5ae40628d0 con 0x7f5ad00a2ca0 2024-06-10T00:20:19.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:19.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi193 2024-06-10T00:20:19.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:19.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:19.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi193 to 8544M 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: osdmap e41: 6 total, 6 up, 6 in 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:20:19.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:19 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi193 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi193 to 8544M 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.453 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: osdmap e41: 6 total, 6 up, 6 in 2024-06-10T00:20:19.454 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:19.454 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:19.454 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:19.454 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.454 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:20:19.454 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:20:19.454 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:19 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi193 2024-06-10T00:20:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi193 to 8544M 2024-06-10T00:20:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: osdmap e41: 6 total, 6 up, 6 in 2024-06-10T00:20:19.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:19.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:19.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:19.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:19.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:20:19.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:20:19.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:19 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:20.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:20 smithi121 ceph-mon[32378]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:20:20.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:20 smithi121 ceph-mon[32378]: osdmap e42: 6 total, 6 up, 6 in 2024-06-10T00:20:20.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:20 smithi193 ceph-mon[36716]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:20:20.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:20 smithi193 ceph-mon[36716]: osdmap e42: 6 total, 6 up, 6 in 2024-06-10T00:20:20.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:20 smithi135 ceph-mon[36424]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:20:20.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:20 smithi135 ceph-mon[36424]: osdmap e42: 6 total, 6 up, 6 in 2024-06-10T00:20:21.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:21 smithi193 ceph-mon[36716]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:21 smithi193 ceph-mon[36716]: osdmap e43: 6 total, 6 up, 6 in 2024-06-10T00:20:21.408 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:21 smithi135 ceph-mon[36424]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:21.408 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:21 smithi135 ceph-mon[36424]: osdmap e43: 6 total, 6 up, 6 in 2024-06-10T00:20:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:21 smithi121 ceph-mon[32378]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:21 smithi121 ceph-mon[32378]: osdmap e43: 6 total, 6 up, 6 in 2024-06-10T00:20:22.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:22 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1121266350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3541db4e-839e-4ad2-bd35-2566a649d87f"}]: dispatch 2024-06-10T00:20:22.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:22 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1121266350' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3541db4e-839e-4ad2-bd35-2566a649d87f"}]': finished 2024-06-10T00:20:22.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:22 smithi193 ceph-mon[36716]: osdmap e44: 7 total, 6 up, 7 in 2024-06-10T00:20:22.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:22 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:22.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:22 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1121266350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3541db4e-839e-4ad2-bd35-2566a649d87f"}]: dispatch 2024-06-10T00:20:22.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:22 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1121266350' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3541db4e-839e-4ad2-bd35-2566a649d87f"}]': finished 2024-06-10T00:20:22.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:22 smithi121 ceph-mon[32378]: osdmap e44: 7 total, 6 up, 7 in 2024-06-10T00:20:22.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:22 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:22.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:22 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1121266350' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3541db4e-839e-4ad2-bd35-2566a649d87f"}]: dispatch 2024-06-10T00:20:22.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:22 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1121266350' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3541db4e-839e-4ad2-bd35-2566a649d87f"}]': finished 2024-06-10T00:20:22.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:22 smithi135 ceph-mon[36424]: osdmap e44: 7 total, 6 up, 7 in 2024-06-10T00:20:22.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:22 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:23.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:23 smithi193 ceph-mon[36716]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-10T00:20:23.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:23 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/995339107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:20:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:23 smithi121 ceph-mon[32378]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-10T00:20:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:23 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/995339107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:20:23.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:23 smithi135 ceph-mon[36424]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-10T00:20:23.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:23 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/995339107' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:20:25.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:25 smithi193 ceph-mon[36716]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-06-10T00:20:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:25 smithi121 ceph-mon[32378]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-06-10T00:20:25.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:25 smithi135 ceph-mon[36424]: pgmap v145: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-06-10T00:20:27.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:27 smithi121 ceph-mon[32378]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-06-10T00:20:27.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:27 smithi193 ceph-mon[36716]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-06-10T00:20:27.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:27 smithi135 ceph-mon[36424]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-06-10T00:20:29.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:29 smithi193 ceph-mon[36716]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-10T00:20:29.447 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:29 smithi121 ceph-mon[32378]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-10T00:20:29.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:29 smithi135 ceph-mon[36424]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-10T00:20:30.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:30 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-10T00:20:30.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:30 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:30.434 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:30 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-10T00:20:30.434 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:30 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:30.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:30 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-10T00:20:30.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:30 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:31.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:31 smithi121 ceph-mon[32378]: Deploying daemon osd.6 on smithi121 2024-06-10T00:20:31.363 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:31 smithi121 ceph-mon[32378]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:31.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:31 smithi193 ceph-mon[36716]: Deploying daemon osd.6 on smithi121 2024-06-10T00:20:31.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:31 smithi193 ceph-mon[36716]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:31.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:31 smithi135 ceph-mon[36424]: Deploying daemon osd.6 on smithi121 2024-06-10T00:20:31.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:31 smithi135 ceph-mon[36424]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:33.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:33 smithi193 ceph-mon[36716]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:33.386 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:33 smithi121 ceph-mon[32378]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:33.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:33 smithi135 ceph-mon[36424]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:35.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:35 smithi193 ceph-mon[36716]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:35.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:35 smithi121 ceph-mon[32378]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:35.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:35 smithi135 ceph-mon[36424]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:36.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:36.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:36.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:36.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:36 smithi121 ceph-mon[32378]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:36.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:36.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:36.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:36.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:36 smithi135 ceph-mon[36424]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:36.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:36 smithi193 ceph-mon[36716]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:39.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.135+0000 7f5adb7fe700 1 -- 172.21.15.121:0/240655124 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5ae40628d0 con 0x7f5ad00a2ca0 2024-06-10T00:20:39.135 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 6 on host 'smithi121' 2024-06-10T00:20:39.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 -- 172.21.15.121:0/240655124 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5ad00a2ca0 msgr2=0x7f5ad00a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:20:39.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5ad00a2ca0 0x7f5ad00a5160 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4103550 tx=0x7f5ae0009190 comp rx=0 tx=0).stop 2024-06-10T00:20:39.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 -- 172.21.15.121:0/240655124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4104090 msgr2=0x7f5ae4074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:20:39.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4104090 0x7f5ae4074dc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5adc00b970 tx=0x7f5adc00bc80 comp rx=0 tx=0).stop 2024-06-10T00:20:39.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 -- 172.21.15.121:0/240655124 shutdown_connections 2024-06-10T00:20:39.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5ad00a2ca0 0x7f5ad00a5160 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:39.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4104090 0x7f5ae4074dc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:39.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ae41024f0 0x7f5ae4076230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 --2- 172.21.15.121:0/240655124 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ae41036f0 0x7f5ae4074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 -- 172.21.15.121:0/240655124 >> 172.21.15.121:0/240655124 conn(0x7f5ae40fda90 msgr2=0x7f5ae4106b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:20:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 -- 172.21.15.121:0/240655124 shutdown_connections 2024-06-10T00:20:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:20:39.137+0000 7f5aec6f4700 1 -- 172.21.15.121:0/240655124 wait complete. 2024-06-10T00:20:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:39 smithi121 ceph-mon[32378]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:39 smithi121 ceph-mon[32378]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-10T00:20:39.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:39 smithi135 ceph-mon[36424]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:39 smithi135 ceph-mon[36424]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-10T00:20:39.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:39 smithi193 ceph-mon[36716]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:20:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:39 smithi193 ceph-mon[36716]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-10T00:20:39.913 DEBUG:teuthology.orchestra.run.smithi121:osd.6> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.6.service 2024-06-10T00:20:39.915 INFO:tasks.cephadm:Deploying osd.7 on smithi135 with /dev/vg_nvme/lv_2... 2024-06-10T00:20:39.916 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-10T00:20:40.160 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:20:41.378 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:41 smithi135 ceph-mon[36424]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:41.378 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:41 smithi135 ceph-mon[36424]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-10T00:20:41.378 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:41 smithi135 ceph-mon[36424]: osdmap e45: 7 total, 6 up, 7 in 2024-06-10T00:20:41.378 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:41 smithi135 ceph-mon[36424]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:20:41.378 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:41 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:41.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:41 smithi193 ceph-mon[36716]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:41.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:41 smithi193 ceph-mon[36716]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-10T00:20:41.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:41 smithi193 ceph-mon[36716]: osdmap e45: 7 total, 6 up, 7 in 2024-06-10T00:20:41.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:41 smithi193 ceph-mon[36716]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:20:41.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:41 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:41.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:41 smithi121 ceph-mon[32378]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:41.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:41 smithi121 ceph-mon[32378]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-10T00:20:41.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:41 smithi121 ceph-mon[32378]: osdmap e45: 7 total, 6 up, 7 in 2024-06-10T00:20:41.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:41 smithi121 ceph-mon[32378]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:20:41.399 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:41 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:41.667 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:20:41 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:20:41.532+0000 7f8bdad15700 -1 osd.6 0 waiting for initial osdmap 2024-06-10T00:20:41.667 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:20:41 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:20:41.542+0000 7f8bd2b00700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:20:42.245 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-10T00:20:42.245 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-10T00:20:42.246 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-06-10T00:20:42.246 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-06-10T00:20:42.246 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119609 s, 877 MB/s 2024-06-10T00:20:42.246 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-06-10T00:20:42.273 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:42 smithi135 ceph-mon[36424]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:20:42.273 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:42 smithi135 ceph-mon[36424]: osdmap e46: 7 total, 6 up, 7 in 2024-06-10T00:20:42.273 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:42.273 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:42.273 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:42 smithi135 ceph-mon[36424]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' 2024-06-10T00:20:42.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:42 smithi121 ceph-mon[32378]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:20:42.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:42 smithi121 ceph-mon[32378]: osdmap e46: 7 total, 6 up, 7 in 2024-06-10T00:20:42.356 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:42.356 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:42.356 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:42 smithi121 ceph-mon[32378]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' 2024-06-10T00:20:42.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:42 smithi193 ceph-mon[36716]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:20:42.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:42 smithi193 ceph-mon[36716]: osdmap e46: 7 total, 6 up, 7 in 2024-06-10T00:20:42.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:42.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:42.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:42 smithi193 ceph-mon[36716]: from='osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681]' entity='osd.6' 2024-06-10T00:20:42.946 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_2 2024-06-10T00:20:43.191 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:20:43.249 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:20:43.249 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:20:43.249 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681] boot 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: osdmap e47: 7 total, 7 up, 7 in 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:43.250 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681] boot 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: osdmap e47: 7 total, 7 up, 7 in 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:43.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:43.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:20:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:20:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-06-10T00:20:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: osd.6 [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681] boot 2024-06-10T00:20:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: osdmap e47: 7 total, 7 up, 7 in 2024-06-10T00:20:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:20:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:43.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:43.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:43.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:20:43.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:43.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:43.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:20:44.300 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:44 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi121 2024-06-10T00:20:44.300 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:44 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi121 to 5696M 2024-06-10T00:20:44.300 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:44.300 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:44 smithi135 ceph-mon[36424]: osdmap e48: 7 total, 7 up, 7 in 2024-06-10T00:20:44.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:44 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi121 2024-06-10T00:20:44.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:44 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi121 to 5696M 2024-06-10T00:20:44.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:44.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:44 smithi193 ceph-mon[36716]: osdmap e48: 7 total, 7 up, 7 in 2024-06-10T00:20:44.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:44 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi121 2024-06-10T00:20:44.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:44 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi121 to 5696M 2024-06-10T00:20:44.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:20:44.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:44 smithi121 ceph-mon[32378]: osdmap e48: 7 total, 7 up, 7 in 2024-06-10T00:20:44.806 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.806+0000 7f22cacb2700 1 -- 172.21.15.135:0/2599741729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22c4102f80 msgr2=0x7f22c4109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:20:44.806 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.806+0000 7f22cacb2700 1 --2- 172.21.15.135:0/2599741729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22c4102f80 0x7f22c4109830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00b0d0 tx=0x7f22bc00b3e0 comp rx=0 tx=0).stop 2024-06-10T00:20:44.806 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.806+0000 7f22cacb2700 1 -- 172.21.15.135:0/2599741729 shutdown_connections 2024-06-10T00:20:44.807 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.806+0000 7f22cacb2700 1 --2- 172.21.15.135:0/2599741729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22c4102f80 0x7f22c4109830 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:44.807 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.806+0000 7f22cacb2700 1 --2- 172.21.15.135:0/2599741729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22c41013e0 0x7f22c4101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:44.807 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.806+0000 7f22cacb2700 1 --2- 172.21.15.135:0/2599741729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22c41025e0 0x7f22c4102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:44.807 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.806+0000 7f22cacb2700 1 -- 172.21.15.135:0/2599741729 >> 172.21.15.135:0/2599741729 conn(0x7f22c40fc980 msgr2=0x7f22c40fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:20:44.807 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22cacb2700 1 -- 172.21.15.135:0/2599741729 shutdown_connections 2024-06-10T00:20:44.807 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22cacb2700 1 -- 172.21.15.135:0/2599741729 wait complete. 2024-06-10T00:20:44.809 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22cacb2700 1 Processor -- start 2024-06-10T00:20:44.809 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22cacb2700 1 -- start start 2024-06-10T00:20:44.809 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22cacb2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22c41013e0 0x7f22c4198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:20:44.809 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22cacb2700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22c41025e0 0x7f22c4198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:20:44.809 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22cacb2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22c4102f80 0x7f22c419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:20:44.809 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22cacb2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f22c410d020 con 0x7f22c4102f80 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22cacb2700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f22c410cea0 con 0x7f22c41013e0 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22cacb2700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f22c410d1a0 con 0x7f22c41025e0 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22c8a4e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22c41013e0 0x7f22c4198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22c8a4e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22c41013e0 0x7f22c4198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:41182/0 (socket says 172.21.15.135:41182) 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22c8a4e700 1 -- 172.21.15.135:0/2046461152 learned_addr learned my addr 172.21.15.135:0/2046461152 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.807+0000 7f22c924f700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22c4102f80 0x7f22c419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22c3fff700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22c41025e0 0x7f22c4198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22c8a4e700 1 -- 172.21.15.135:0/2046461152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22c41025e0 msgr2=0x7f22c4198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22c8a4e700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22c41025e0 0x7f22c4198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22c8a4e700 1 -- 172.21.15.135:0/2046461152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22c4102f80 msgr2=0x7f22c419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22c8a4e700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22c4102f80 0x7f22c419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22c8a4e700 1 -- 172.21.15.135:0/2046461152 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c41a02b0 con 0x7f22c41013e0 2024-06-10T00:20:44.810 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22c8a4e700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22c41013e0 0x7f22c4198650 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f22b800d980 tx=0x7f22b800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:20:44.811 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22c1ffb700 1 -- 172.21.15.135:0/2046461152 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b8017070 con 0x7f22c41013e0 2024-06-10T00:20:44.811 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22c1ffb700 1 -- 172.21.15.135:0/2046461152 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22b8009e10 con 0x7f22c41013e0 2024-06-10T00:20:44.811 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22c1ffb700 1 -- 172.21.15.135:0/2046461152 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b800f620 con 0x7f22c41013e0 2024-06-10T00:20:44.811 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22cacb2700 1 -- 172.21.15.135:0/2046461152 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c41a05a0 con 0x7f22c41013e0 2024-06-10T00:20:44.811 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.808+0000 7f22cacb2700 1 -- 172.21.15.135:0/2046461152 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c41a0a90 con 0x7f22c41013e0 2024-06-10T00:20:44.811 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.809+0000 7f22cacb2700 1 -- 172.21.15.135:0/2046461152 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22c4069180 con 0x7f22c41013e0 2024-06-10T00:20:44.812 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.810+0000 7f22c1ffb700 1 -- 172.21.15.135:0/2046461152 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f22b8010710 con 0x7f22c41013e0 2024-06-10T00:20:44.812 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.811+0000 7f22c1ffb700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f22b40a2c20 0x7f22b40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:20:44.812 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.811+0000 7f22c1ffb700 1 -- 172.21.15.135:0/2046461152 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f22b80cded0 con 0x7f22c41013e0 2024-06-10T00:20:44.812 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.811+0000 7f22c3fff700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f22b40a2c20 0x7f22b40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:20:44.812 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.811+0000 7f22c3fff700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f22b40a2c20 0x7f22b40a50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f22b0005ea0 tx=0x7f22b0005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:20:44.815 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:44.815+0000 7f22c1ffb700 1 -- 172.21.15.135:0/2046461152 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f22b809bb40 con 0x7f22c41013e0 2024-06-10T00:20:45.018 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:20:45.018+0000 7f22cacb2700 1 -- 172.21.15.135:0/2046461152 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f22c406b2d0 con 0x7f22b40a2c20 2024-06-10T00:20:45.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:45 smithi193 ceph-mon[36716]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:20:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:20:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:45 smithi121 ceph-mon[32378]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:20:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:20:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:45 smithi135 ceph-mon[36424]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:20:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:20:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:46.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:46 smithi135 ceph-mon[36424]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:20:46.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:46 smithi193 ceph-mon[36716]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:20:46.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:46 smithi121 ceph-mon[32378]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:20:47.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:47 smithi193 ceph-mon[36716]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:47 smithi121 ceph-mon[32378]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:47.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:47 smithi135 ceph-mon[36424]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:48.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:48 smithi193 ceph-mon[36716]: from='client.? 172.21.15.135:0/602952616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3979b209-73a0-4891-85dc-b7aad76f0ff0"}]: dispatch 2024-06-10T00:20:48.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:48 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3979b209-73a0-4891-85dc-b7aad76f0ff0"}]: dispatch 2024-06-10T00:20:48.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:48 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3979b209-73a0-4891-85dc-b7aad76f0ff0"}]': finished 2024-06-10T00:20:48.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:48 smithi193 ceph-mon[36716]: osdmap e49: 8 total, 7 up, 8 in 2024-06-10T00:20:48.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:48 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:20:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:48 smithi121 ceph-mon[32378]: from='client.? 172.21.15.135:0/602952616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3979b209-73a0-4891-85dc-b7aad76f0ff0"}]: dispatch 2024-06-10T00:20:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:48 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3979b209-73a0-4891-85dc-b7aad76f0ff0"}]: dispatch 2024-06-10T00:20:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:48 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3979b209-73a0-4891-85dc-b7aad76f0ff0"}]': finished 2024-06-10T00:20:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:48 smithi121 ceph-mon[32378]: osdmap e49: 8 total, 7 up, 8 in 2024-06-10T00:20:48.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:48 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:20:48.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:48 smithi135 ceph-mon[36424]: from='client.? 172.21.15.135:0/602952616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3979b209-73a0-4891-85dc-b7aad76f0ff0"}]: dispatch 2024-06-10T00:20:48.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:48 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3979b209-73a0-4891-85dc-b7aad76f0ff0"}]: dispatch 2024-06-10T00:20:48.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:48 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3979b209-73a0-4891-85dc-b7aad76f0ff0"}]': finished 2024-06-10T00:20:48.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:48 smithi135 ceph-mon[36424]: osdmap e49: 8 total, 7 up, 8 in 2024-06-10T00:20:48.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:48 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:20:49.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:49 smithi193 ceph-mon[36716]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:49 smithi193 ceph-mon[36716]: from='client.? 172.21.15.135:0/2799032617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:20:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:49 smithi121 ceph-mon[32378]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:49 smithi121 ceph-mon[32378]: from='client.? 172.21.15.135:0/2799032617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:20:49.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:49 smithi135 ceph-mon[36424]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:49.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:49 smithi135 ceph-mon[36424]: from='client.? 172.21.15.135:0/2799032617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:20:51.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:51 smithi193 ceph-mon[36716]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:51.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:51 smithi121 ceph-mon[32378]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:51.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:51 smithi135 ceph-mon[36424]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:53.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:53 smithi135 ceph-mon[36424]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:53.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:53 smithi193 ceph-mon[36716]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:53.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:53 smithi121 ceph-mon[32378]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:55.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:55 smithi193 ceph-mon[36716]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:55.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:55 smithi121 ceph-mon[32378]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:55.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:55 smithi135 ceph-mon[36424]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:56.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:56 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-10T00:20:56.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:56 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:56.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:56 smithi121 ceph-mon[32378]: Deploying daemon osd.7 on smithi135 2024-06-10T00:20:56.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:56 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-10T00:20:56.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:56 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:56.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:56 smithi135 ceph-mon[36424]: Deploying daemon osd.7 on smithi135 2024-06-10T00:20:56.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:56 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-10T00:20:56.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:56 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:20:56.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:56 smithi193 ceph-mon[36716]: Deploying daemon osd.7 on smithi135 2024-06-10T00:20:57.386 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:57 smithi135 ceph-mon[36424]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:57.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:57 smithi121 ceph-mon[32378]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:57.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:57 smithi193 ceph-mon[36716]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:59.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:20:59 smithi121 ceph-mon[32378]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:59.476 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:20:59 smithi135 ceph-mon[36424]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:20:59.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:20:59 smithi193 ceph-mon[36716]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:01.281 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:01 smithi135 ceph-mon[36424]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:01.281 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:01 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:01.281 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:01 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:01.281 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:01 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:01 smithi121 ceph-mon[32378]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:01 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:01 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:01 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:01.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:01 smithi193 ceph-mon[36716]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:01.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:01 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:01.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:01 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:01.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:01 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:03.181 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:03 smithi135 ceph-mon[36424]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:03 smithi121 ceph-mon[32378]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:03.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:03 smithi193 ceph-mon[36716]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:04.335 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 7 on host 'smithi135' 2024-06-10T00:21:04.335 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.335+0000 7f22c1ffb700 1 -- 172.21.15.135:0/2046461152 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f22c406b2d0 con 0x7f22b40a2c20 2024-06-10T00:21:04.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 -- 172.21.15.135:0/2046461152 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f22b40a2c20 msgr2=0x7f22b40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:04.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f22b40a2c20 0x7f22b40a50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f22b0005ea0 tx=0x7f22b0005e30 comp rx=0 tx=0).stop 2024-06-10T00:21:04.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 -- 172.21.15.135:0/2046461152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22c41013e0 msgr2=0x7f22c4198650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:04.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22c41013e0 0x7f22c4198650 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f22b800d980 tx=0x7f22b800dc90 comp rx=0 tx=0).stop 2024-06-10T00:21:04.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 -- 172.21.15.135:0/2046461152 shutdown_connections 2024-06-10T00:21:04.337 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f22b40a2c20 0x7f22b40a50e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:04.338 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22c4102f80 0x7f22c419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:04.338 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22c41013e0 0x7f22c4198650 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:04.338 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 --2- 172.21.15.135:0/2046461152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22c41025e0 0x7f22c4198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:04.338 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 -- 172.21.15.135:0/2046461152 >> 172.21.15.135:0/2046461152 conn(0x7f22c40fc980 msgr2=0x7f22c406abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:21:04.338 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 -- 172.21.15.135:0/2046461152 shutdown_connections 2024-06-10T00:21:04.338 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:21:04.337+0000 7f22cacb2700 1 -- 172.21.15.135:0/2046461152 wait complete. 2024-06-10T00:21:04.519 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:04 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.519 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:04 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.519 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:04 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:04.520 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:04 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:04.520 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:04 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.520 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:04 smithi135 ceph-mon[36424]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:04.520 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:04 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.520 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:04 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.520 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:04 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:04.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:04 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:04 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:04 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:04 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:04 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:04 smithi121 ceph-mon[32378]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:04 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:04 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:04 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:04 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:04 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:04 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:04 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:04 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:04 smithi193 ceph-mon[36716]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:04 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:04 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:04 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:04.955 DEBUG:teuthology.orchestra.run.smithi135:osd.7> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.7.service 2024-06-10T00:21:04.957 INFO:tasks.cephadm:Deploying osd.8 on smithi193 with /dev/vg_nvme/lv_2... 2024-06-10T00:21:04.957 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-10T00:21:05.204 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:21:05.580 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:05 smithi193 ceph-mon[36716]: from='osd.7 [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-10T00:21:05.580 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:05 smithi193 ceph-mon[36716]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-10T00:21:05.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:05 smithi135 ceph-mon[36424]: from='osd.7 [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-10T00:21:05.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:05 smithi135 ceph-mon[36424]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-10T00:21:05.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:05 smithi121 ceph-mon[32378]: from='osd.7 [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-10T00:21:05.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:05 smithi121 ceph-mon[32378]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-10T00:21:06.490 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:06 smithi135 ceph-mon[36424]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-10T00:21:06.490 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:06 smithi135 ceph-mon[36424]: from='osd.7 [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:21:06.490 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:06 smithi135 ceph-mon[36424]: osdmap e50: 8 total, 7 up, 8 in 2024-06-10T00:21:06.490 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:06.490 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:06 smithi135 ceph-mon[36424]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:21:06.490 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:06 smithi135 ceph-mon[36424]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:06.491 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:21:06 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:21:06.479+0000 7f4dd3919700 -1 osd.7 0 waiting for initial osdmap 2024-06-10T00:21:06.491 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:21:06 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:21:06.490+0000 7f4dcf70c700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:21:06.619 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:06 smithi193 ceph-mon[36716]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-10T00:21:06.619 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:06 smithi193 ceph-mon[36716]: from='osd.7 [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:21:06.619 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:06 smithi193 ceph-mon[36716]: osdmap e50: 8 total, 7 up, 8 in 2024-06-10T00:21:06.619 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:06.619 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:06 smithi193 ceph-mon[36716]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:21:06.620 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:06 smithi193 ceph-mon[36716]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:06.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:06 smithi121 ceph-mon[32378]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-10T00:21:06.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:06 smithi121 ceph-mon[32378]: from='osd.7 [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:21:06.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:06 smithi121 ceph-mon[32378]: osdmap e50: 8 total, 7 up, 8 in 2024-06-10T00:21:06.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:06.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:06 smithi121 ceph-mon[32378]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:21:06.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:06 smithi121 ceph-mon[32378]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-06-10T00:21:07.205 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-10T00:21:07.205 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-10T00:21:07.205 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-06-10T00:21:07.205 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-06-10T00:21:07.205 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012604 s, 832 MB/s 2024-06-10T00:21:07.205 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-06-10T00:21:07.671 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:07 smithi135 ceph-mon[36424]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:21:07.671 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:07 smithi135 ceph-mon[36424]: osdmap e51: 8 total, 7 up, 8 in 2024-06-10T00:21:07.671 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:07 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:07.671 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:07 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:07.837 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:07 smithi193 ceph-mon[36716]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:21:07.837 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:07 smithi193 ceph-mon[36716]: osdmap e51: 8 total, 7 up, 8 in 2024-06-10T00:21:07.837 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:07 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:07.837 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:07 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:07.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_2 2024-06-10T00:21:07.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:07 smithi121 ceph-mon[32378]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:21:07.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:07 smithi121 ceph-mon[32378]: osdmap e51: 8 total, 7 up, 8 in 2024-06-10T00:21:07.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:07 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:07.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:07 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:08.113 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:21:08.489 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:21:08.489 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:21:08.489 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: osd.7 [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559] boot 2024-06-10T00:21:08.490 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: osdmap e52: 8 total, 8 up, 8 in 2024-06-10T00:21:08.490 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:08.490 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi135 2024-06-10T00:21:08.490 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.490 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.490 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:08.490 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:08.490 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:08.490 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi135 to 5696M 2024-06-10T00:21:08.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:08.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:08.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:08 smithi193 ceph-mon[36716]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:21:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:21:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: osd.7 [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559] boot 2024-06-10T00:21:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: osdmap e52: 8 total, 8 up, 8 in 2024-06-10T00:21:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi135 2024-06-10T00:21:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:08.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi135 to 5696M 2024-06-10T00:21:08.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:08.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:08.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:08 smithi135 ceph-mon[36424]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:08.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:21:08.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:21:08.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: osd.7 [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559] boot 2024-06-10T00:21:08.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: osdmap e52: 8 total, 8 up, 8 in 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi135 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi135 to 5696M 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:08.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:08 smithi121 ceph-mon[32378]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:09.572 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:09 smithi193 ceph-mon[36716]: osdmap e53: 8 total, 8 up, 8 in 2024-06-10T00:21:09.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:09 smithi135 ceph-mon[36424]: osdmap e53: 8 total, 8 up, 8 in 2024-06-10T00:21:09.800 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.800+0000 7f231e482700 1 -- 172.21.15.193:0/3263617300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2318105910 msgr2=0x7f2318107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:09.800 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.800+0000 7f231e482700 1 --2- 172.21.15.193:0/3263617300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2318105910 0x7f2318107d00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2308009990 tx=0x7f2308009ca0 comp rx=0 tx=0).stop 2024-06-10T00:21:09.800 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.800+0000 7f231e482700 1 -- 172.21.15.193:0/3263617300 shutdown_connections 2024-06-10T00:21:09.800 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.800+0000 7f231e482700 1 --2- 172.21.15.193:0/3263617300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2318108240 0x7f231810a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:09.800 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.800+0000 7f231e482700 1 --2- 172.21.15.193:0/3263617300 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2318102fe0 0x7f23181053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:09.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.800+0000 7f231e482700 1 --2- 172.21.15.193:0/3263617300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2318105910 0x7f2318107d00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:09.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.800+0000 7f231e482700 1 -- 172.21.15.193:0/3263617300 >> 172.21.15.193:0/3263617300 conn(0x7f23180fc990 msgr2=0x7f23180fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:21:09.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.800+0000 7f231e482700 1 -- 172.21.15.193:0/3263617300 shutdown_connections 2024-06-10T00:21:09.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.800+0000 7f231e482700 1 -- 172.21.15.193:0/3263617300 wait complete. 2024-06-10T00:21:09.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.801+0000 7f231e482700 1 Processor -- start 2024-06-10T00:21:09.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.801+0000 7f231e482700 1 -- start start 2024-06-10T00:21:09.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.801+0000 7f231e482700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2318102fe0 0x7f23181986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:21:09.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.801+0000 7f231e482700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2318105910 0x7f2318198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:21:09.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.801+0000 7f231e482700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2318108240 0x7f231819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:21:09.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.801+0000 7f231e482700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f231810e140 con 0x7f2318102fe0 2024-06-10T00:21:09.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.801+0000 7f231e482700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f231810dfc0 con 0x7f2318108240 2024-06-10T00:21:09.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.801+0000 7f231e482700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f231810e2c0 con 0x7f2318105910 2024-06-10T00:21:09.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.801+0000 7f23177fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2318105910 0x7f2318198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:21:09.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f23177fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2318105910 0x7f2318198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:36552/0 (socket says 172.21.15.193:36552) 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f23177fe700 1 -- 172.21.15.193:0/2622009781 learned_addr learned my addr 172.21.15.193:0/2622009781 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f231ca1f700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2318108240 0x7f231819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f2317fff700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2318102fe0 0x7f23181986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f23177fe700 1 -- 172.21.15.193:0/2622009781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2318108240 msgr2=0x7f231819fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f23177fe700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2318108240 0x7f231819fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f23177fe700 1 -- 172.21.15.193:0/2622009781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2318102fe0 msgr2=0x7f23181986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f23177fe700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2318102fe0 0x7f23181986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f23177fe700 1 -- 172.21.15.193:0/2622009781 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23181a0300 con 0x7f2318105910 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f2317fff700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2318102fe0 0x7f23181986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f23177fe700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2318105910 0x7f2318198be0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f23080047b0 tx=0x7f2308009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f23157fa700 1 -- 172.21.15.193:0/2622009781 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308027070 con 0x7f2318105910 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f231e482700 1 -- 172.21.15.193:0/2622009781 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23181a0590 con 0x7f2318105910 2024-06-10T00:21:09.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f23157fa700 1 -- 172.21.15.193:0/2622009781 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2308022470 con 0x7f2318105910 2024-06-10T00:21:09.805 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f23157fa700 1 -- 172.21.15.193:0/2622009781 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230800f6c0 con 0x7f2318105910 2024-06-10T00:21:09.805 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.802+0000 7f231e482700 1 -- 172.21.15.193:0/2622009781 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23181a0ac0 con 0x7f2318105910 2024-06-10T00:21:09.805 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.803+0000 7f231e482700 1 -- 172.21.15.193:0/2622009781 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2318069180 con 0x7f2318105910 2024-06-10T00:21:09.805 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.804+0000 7f23157fa700 1 -- 172.21.15.193:0/2622009781 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f230802a070 con 0x7f2318105910 2024-06-10T00:21:09.805 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.805+0000 7f23157fa700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23000a2c20 0x7f23000a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:21:09.806 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.805+0000 7f23157fa700 1 -- 172.21.15.193:0/2622009781 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7f23080cf580 con 0x7f2318105910 2024-06-10T00:21:09.806 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.805+0000 7f2317fff700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23000a2c20 0x7f23000a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:21:09.806 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.805+0000 7f2317fff700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23000a2c20 0x7f23000a50e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f230c005fd0 tx=0x7f230c005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:21:09.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:09.809+0000 7f23157fa700 1 -- 172.21.15.193:0/2622009781 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f230805e850 con 0x7f2318105910 2024-06-10T00:21:09.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:09 smithi121 ceph-mon[32378]: osdmap e53: 8 total, 8 up, 8 in 2024-06-10T00:21:10.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:10.016+0000 7f231e482700 1 -- 172.21.15.193:0/2622009781 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f23180628d0 con 0x7f23000a2c20 2024-06-10T00:21:10.825 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:10 smithi193 ceph-mon[36716]: osdmap e54: 8 total, 8 up, 8 in 2024-06-10T00:21:10.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:10 smithi193 ceph-mon[36716]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:21:10.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:10 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:21:10.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:10 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:21:10.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:10 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:10.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:10 smithi193 ceph-mon[36716]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:10.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:10 smithi121 ceph-mon[32378]: osdmap e54: 8 total, 8 up, 8 in 2024-06-10T00:21:10.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:10 smithi121 ceph-mon[32378]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:21:10.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:10 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:21:10.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:10 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:21:10.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:10 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:10.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:10 smithi121 ceph-mon[32378]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:11.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:10 smithi135 ceph-mon[36424]: osdmap e54: 8 total, 8 up, 8 in 2024-06-10T00:21:11.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:10 smithi135 ceph-mon[36424]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:21:11.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:10 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:21:11.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:10 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:21:11.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:10 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:11.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:10 smithi135 ceph-mon[36424]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:11.613 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:11 smithi193 ceph-mon[36716]: osdmap e55: 8 total, 8 up, 8 in 2024-06-10T00:21:11.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:11 smithi121 ceph-mon[32378]: osdmap e55: 8 total, 8 up, 8 in 2024-06-10T00:21:12.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:11 smithi135 ceph-mon[36424]: osdmap e55: 8 total, 8 up, 8 in 2024-06-10T00:21:12.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:12 smithi193 ceph-mon[36716]: osdmap e56: 8 total, 8 up, 8 in 2024-06-10T00:21:12.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:12 smithi193 ceph-mon[36716]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:12.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:12 smithi121 ceph-mon[32378]: osdmap e56: 8 total, 8 up, 8 in 2024-06-10T00:21:12.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:12 smithi121 ceph-mon[32378]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:13.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:12 smithi135 ceph-mon[36424]: osdmap e56: 8 total, 8 up, 8 in 2024-06-10T00:21:13.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:12 smithi135 ceph-mon[36424]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:13.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:13 smithi193 ceph-mon[36716]: from='client.? 172.21.15.193:0/1863988346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a099d63-2664-4c39-a992-2095f55cdfa7"}]: dispatch 2024-06-10T00:21:13.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:13 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a099d63-2664-4c39-a992-2095f55cdfa7"}]: dispatch 2024-06-10T00:21:13.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:13 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a099d63-2664-4c39-a992-2095f55cdfa7"}]': finished 2024-06-10T00:21:13.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:13 smithi193 ceph-mon[36716]: osdmap e57: 9 total, 8 up, 9 in 2024-06-10T00:21:13.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:13 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:13.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:13 smithi193 ceph-mon[36716]: from='client.? 172.21.15.193:0/1820981528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:21:13.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:13 smithi121 ceph-mon[32378]: from='client.? 172.21.15.193:0/1863988346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a099d63-2664-4c39-a992-2095f55cdfa7"}]: dispatch 2024-06-10T00:21:13.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:13 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a099d63-2664-4c39-a992-2095f55cdfa7"}]: dispatch 2024-06-10T00:21:13.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:13 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a099d63-2664-4c39-a992-2095f55cdfa7"}]': finished 2024-06-10T00:21:13.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:13 smithi121 ceph-mon[32378]: osdmap e57: 9 total, 8 up, 9 in 2024-06-10T00:21:13.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:13.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:13 smithi121 ceph-mon[32378]: from='client.? 172.21.15.193:0/1820981528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:21:14.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:13 smithi135 ceph-mon[36424]: from='client.? 172.21.15.193:0/1863988346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a099d63-2664-4c39-a992-2095f55cdfa7"}]: dispatch 2024-06-10T00:21:14.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:13 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6a099d63-2664-4c39-a992-2095f55cdfa7"}]: dispatch 2024-06-10T00:21:14.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:13 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6a099d63-2664-4c39-a992-2095f55cdfa7"}]': finished 2024-06-10T00:21:14.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:13 smithi135 ceph-mon[36424]: osdmap e57: 9 total, 8 up, 9 in 2024-06-10T00:21:14.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:13 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:14.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:13 smithi135 ceph-mon[36424]: from='client.? 172.21.15.193:0/1820981528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:21:14.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:14 smithi193 ceph-mon[36716]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:14.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:14 smithi121 ceph-mon[32378]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:15.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:14 smithi135 ceph-mon[36424]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:17.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:17 smithi193 ceph-mon[36716]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:17.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:17 smithi121 ceph-mon[32378]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:17 smithi135 ceph-mon[36424]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:19.216 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:19 smithi193 ceph-mon[36716]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:19.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:19 smithi121 ceph-mon[32378]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:19.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:19 smithi135 ceph-mon[36424]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:21 smithi193 ceph-mon[36716]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-10T00:21:21.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:21.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:21 smithi193 ceph-mon[36716]: Deploying daemon osd.8 on smithi193 2024-06-10T00:21:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:21 smithi121 ceph-mon[32378]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-10T00:21:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:21 smithi121 ceph-mon[32378]: Deploying daemon osd.8 on smithi193 2024-06-10T00:21:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:21 smithi135 ceph-mon[36424]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-06-10T00:21:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:21 smithi135 ceph-mon[36424]: Deploying daemon osd.8 on smithi193 2024-06-10T00:21:23.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:23 smithi193 ceph-mon[36716]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:23 smithi121 ceph-mon[32378]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:23.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:23 smithi135 ceph-mon[36424]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:25.319 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:25 smithi193 ceph-mon[36716]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:25 smithi121 ceph-mon[32378]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:25.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:25 smithi135 ceph-mon[36424]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:26 smithi193 ceph-mon[36716]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:27.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:26 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:27.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:26 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:27.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:26 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:27.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:26 smithi121 ceph-mon[32378]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:27.260 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:26 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:27.260 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:26 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:27.261 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:26 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:27.261 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:26 smithi135 ceph-mon[36424]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:29.373 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:29 smithi193 ceph-mon[36716]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:29.429 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.429+0000 7f23157fa700 1 -- 172.21.15.193:0/2622009781 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f23180628d0 con 0x7f23000a2c20 2024-06-10T00:21:29.429 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 8 on host 'smithi193' 2024-06-10T00:21:29.431 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 -- 172.21.15.193:0/2622009781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23000a2c20 msgr2=0x7f23000a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:29.431 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23000a2c20 0x7f23000a50e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f230c005fd0 tx=0x7f230c005e30 comp rx=0 tx=0).stop 2024-06-10T00:21:29.431 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 -- 172.21.15.193:0/2622009781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2318105910 msgr2=0x7f2318198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:29.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2318105910 0x7f2318198be0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f23080047b0 tx=0x7f2308009f50 comp rx=0 tx=0).stop 2024-06-10T00:21:29.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 -- 172.21.15.193:0/2622009781 shutdown_connections 2024-06-10T00:21:29.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f23000a2c20 0x7f23000a50e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:29.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2318108240 0x7f231819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:29.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2318102fe0 0x7f23181986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:29.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 --2- 172.21.15.193:0/2622009781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2318105910 0x7f2318198be0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:29.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 -- 172.21.15.193:0/2622009781 >> 172.21.15.193:0/2622009781 conn(0x7f23180fc990 msgr2=0x7f2318103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:21:29.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 -- 172.21.15.193:0/2622009781 shutdown_connections 2024-06-10T00:21:29.432 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:21:29.431+0000 7f231e482700 1 -- 172.21.15.193:0/2622009781 wait complete. 2024-06-10T00:21:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:29 smithi121 ceph-mon[32378]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:29.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:29 smithi135 ceph-mon[36424]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:30.137 DEBUG:teuthology.orchestra.run.smithi193:osd.8> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.8.service 2024-06-10T00:21:30.139 INFO:tasks.cephadm:Deploying osd.9 on smithi121 with /dev/vg_nvme/lv_1... 2024-06-10T00:21:30.140 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-10T00:21:30.390 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:21:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:30 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:30 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:30 smithi121 ceph-mon[32378]: from='osd.8 [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-10T00:21:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:30 smithi121 ceph-mon[32378]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-10T00:21:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:30 smithi121 ceph-mon[32378]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:30 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:30 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:30.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:30 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:30.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:30 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:30.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:30 smithi135 ceph-mon[36424]: from='osd.8 [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-10T00:21:30.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:30 smithi135 ceph-mon[36424]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-10T00:21:30.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:30 smithi135 ceph-mon[36424]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:30.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:30 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:30.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:30 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:30.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:30 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:30.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:30 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:30.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:30 smithi193 ceph-mon[36716]: from='osd.8 [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-10T00:21:30.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:30 smithi193 ceph-mon[36716]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-06-10T00:21:30.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:30 smithi193 ceph-mon[36716]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:30.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:30 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:30.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:30 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:31.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:31 smithi121 ceph-mon[32378]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-10T00:21:31.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:31 smithi121 ceph-mon[32378]: osdmap e58: 9 total, 8 up, 9 in 2024-06-10T00:21:31.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:31 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:31.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:31 smithi121 ceph-mon[32378]: from='osd.8 [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:21:31.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:31 smithi121 ceph-mon[32378]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:21:31.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:31 smithi135 ceph-mon[36424]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-10T00:21:31.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:31 smithi135 ceph-mon[36424]: osdmap e58: 9 total, 8 up, 9 in 2024-06-10T00:21:31.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:31 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:31.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:31 smithi135 ceph-mon[36424]: from='osd.8 [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:21:31.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:31 smithi135 ceph-mon[36424]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:21:31.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:31 smithi193 ceph-mon[36716]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-06-10T00:21:31.913 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:31 smithi193 ceph-mon[36716]: osdmap e58: 9 total, 8 up, 9 in 2024-06-10T00:21:31.914 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:31 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:31.914 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:31 smithi193 ceph-mon[36716]: from='osd.8 [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:21:31.914 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:31 smithi193 ceph-mon[36716]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:21:31.914 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:21:31 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:21:31.423+0000 7fa5194a9700 -1 osd.8 0 waiting for initial osdmap 2024-06-10T00:21:31.914 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:21:31 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:21:31.433+0000 7fa51529c700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:21:32.421 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-10T00:21:32.421 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-10T00:21:32.421 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-06-10T00:21:32.421 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-06-10T00:21:32.421 INFO:teuthology.orchestra.run.smithi121.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112174 s, 935 MB/s 2024-06-10T00:21:32.421 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-06-10T00:21:32.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:32 smithi121 ceph-mon[32378]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:21:32.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:32 smithi121 ceph-mon[32378]: osdmap e59: 9 total, 8 up, 9 in 2024-06-10T00:21:32.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:32 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:32.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:32 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:32.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:32 smithi121 ceph-mon[32378]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:32.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:32 smithi135 ceph-mon[36424]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:21:32.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:32 smithi135 ceph-mon[36424]: osdmap e59: 9 total, 8 up, 9 in 2024-06-10T00:21:32.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:32 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:32.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:32 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:32.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:32 smithi135 ceph-mon[36424]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:32.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:32 smithi193 ceph-mon[36716]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:21:32.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:32 smithi193 ceph-mon[36716]: osdmap e59: 9 total, 8 up, 9 in 2024-06-10T00:21:32.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:32 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:32.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:32 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:32.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:32 smithi193 ceph-mon[36716]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-06-10T00:21:33.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi121:vg_nvme/lv_1 2024-06-10T00:21:33.337 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:21:33.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:33 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:21:33.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:33 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:21:33.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:33 smithi121 ceph-mon[32378]: osd.8 [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186] boot 2024-06-10T00:21:33.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:33 smithi121 ceph-mon[32378]: osdmap e60: 9 total, 9 up, 9 in 2024-06-10T00:21:33.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:33 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:33.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:33 smithi121 ceph-mon[32378]: osdmap e61: 9 total, 9 up, 9 in 2024-06-10T00:21:33.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:33 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:21:33.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:33 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:21:33.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:33 smithi135 ceph-mon[36424]: osd.8 [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186] boot 2024-06-10T00:21:33.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:33 smithi135 ceph-mon[36424]: osdmap e60: 9 total, 9 up, 9 in 2024-06-10T00:21:33.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:33 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:33.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:33 smithi135 ceph-mon[36424]: osdmap e61: 9 total, 9 up, 9 in 2024-06-10T00:21:33.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:33 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:21:33.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:33 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:21:33.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:33 smithi193 ceph-mon[36716]: osd.8 [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186] boot 2024-06-10T00:21:33.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:33 smithi193 ceph-mon[36716]: osdmap e60: 9 total, 9 up, 9 in 2024-06-10T00:21:33.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:33 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:21:33.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:33 smithi193 ceph-mon[36716]: osdmap e61: 9 total, 9 up, 9 in 2024-06-10T00:21:35.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.050+0000 7fb0daa45700 1 -- 172.21.15.121:0/3709268506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0d41013c0 msgr2=0x7fb0d41017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:35.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.050+0000 7fb0daa45700 1 --2- 172.21.15.121:0/3709268506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0d41013c0 0x7fb0d41017e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40099c0 tx=0x7fb0c4009cd0 comp rx=0 tx=0).stop 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.050+0000 7fb0daa45700 1 -- 172.21.15.121:0/3709268506 shutdown_connections 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.050+0000 7fb0daa45700 1 --2- 172.21.15.121:0/3709268506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d4102f60 0x7fb0d4109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.050+0000 7fb0daa45700 1 --2- 172.21.15.121:0/3709268506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0d41013c0 0x7fb0d41017e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.050+0000 7fb0daa45700 1 --2- 172.21.15.121:0/3709268506 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0d41025c0 0x7fb0d4102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.050+0000 7fb0daa45700 1 -- 172.21.15.121:0/3709268506 >> 172.21.15.121:0/3709268506 conn(0x7fb0d40fc960 msgr2=0x7fb0d40feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.050+0000 7fb0daa45700 1 -- 172.21.15.121:0/3709268506 shutdown_connections 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.050+0000 7fb0daa45700 1 -- 172.21.15.121:0/3709268506 wait complete. 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.050+0000 7fb0daa45700 1 Processor -- start 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.050+0000 7fb0daa45700 1 -- start start 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0daa45700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d41013c0 0x7fb0d4105300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0daa45700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0d41025c0 0x7fb0d41039a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0daa45700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0d4102f60 0x7fb0d4103ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:21:35.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0daa45700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d410cfb0 con 0x7fb0d4102f60 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0daa45700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d410ce30 con 0x7fb0d41025c0 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0daa45700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d410d130 con 0x7fb0d41013c0 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d8fe2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0d4102f60 0x7fb0d4103ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d8fe2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0d4102f60 0x7fb0d4103ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38928/0 (socket says 172.21.15.121:38928) 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d8fe2700 1 -- 172.21.15.121:0/2684225081 learned_addr learned my addr 172.21.15.121:0/2684225081 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d3fff700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d41013c0 0x7fb0d4105300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d37fe700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0d41025c0 0x7fb0d41039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d8fe2700 1 -- 172.21.15.121:0/2684225081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d41013c0 msgr2=0x7fb0d4105300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d8fe2700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d41013c0 0x7fb0d4105300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d8fe2700 1 -- 172.21.15.121:0/2684225081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0d41025c0 msgr2=0x7fb0d41039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d8fe2700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0d41025c0 0x7fb0d41039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d8fe2700 1 -- 172.21.15.121:0/2684225081 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d41047b0 con 0x7fb0d4102f60 2024-06-10T00:21:35.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d8fe2700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0d4102f60 0x7fb0d4103ee0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc007ee0 tx=0x7fb0cc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:21:35.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d17fa700 1 -- 172.21.15.121:0/2684225081 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc00dbd0 con 0x7fb0d4102f60 2024-06-10T00:21:35.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d17fa700 1 -- 172.21.15.121:0/2684225081 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0cc00dd70 con 0x7fb0d4102f60 2024-06-10T00:21:35.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0d17fa700 1 -- 172.21.15.121:0/2684225081 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc0133f0 con 0x7fb0d4102f60 2024-06-10T00:21:35.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0daa45700 1 -- 172.21.15.121:0/2684225081 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d41a28c0 con 0x7fb0d4102f60 2024-06-10T00:21:35.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.051+0000 7fb0daa45700 1 -- 172.21.15.121:0/2684225081 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d41a2e20 con 0x7fb0d4102f60 2024-06-10T00:21:35.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.052+0000 7fb0daa45700 1 -- 172.21.15.121:0/2684225081 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d4069180 con 0x7fb0d4102f60 2024-06-10T00:21:35.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.054+0000 7fb0d17fa700 1 -- 172.21.15.121:0/2684225081 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb0cc01a020 con 0x7fb0d4102f60 2024-06-10T00:21:35.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.054+0000 7fb0d17fa700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0bc0a2ca0 0x7fb0bc0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:21:35.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.054+0000 7fb0d3fff700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0bc0a2ca0 0x7fb0bc0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:21:35.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.054+0000 7fb0d17fa700 1 -- 172.21.15.121:0/2684225081 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fb0cc0ce3d0 con 0x7fb0d4102f60 2024-06-10T00:21:35.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.055+0000 7fb0d3fff700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0bc0a2ca0 0x7fb0bc0a5160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4000c00 tx=0x7fb0c4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:21:35.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.058+0000 7fb0d17fa700 1 -- 172.21.15.121:0/2684225081 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb0cc010070 con 0x7fb0d4102f60 2024-06-10T00:21:35.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi193 2024-06-10T00:21:35.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:35.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:35.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:35.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi193 to 5696M 2024-06-10T00:21:35.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:35.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:35.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:35.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:35.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:35.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:35.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:34 smithi193 ceph-mon[36716]: osdmap e62: 9 total, 9 up, 9 in 2024-06-10T00:21:35.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi193 2024-06-10T00:21:35.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:35.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:35.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:35.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi193 to 5696M 2024-06-10T00:21:35.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:35.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:35.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:35.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:35.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:35.202 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:35.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:34 smithi121 ceph-mon[32378]: osdmap e62: 9 total, 9 up, 9 in 2024-06-10T00:21:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi193 2024-06-10T00:21:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:21:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi193 to 5696M 2024-06-10T00:21:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:35.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:35.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:35.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:35.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:35.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:35.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:34 smithi135 ceph-mon[36424]: osdmap e62: 9 total, 9 up, 9 in 2024-06-10T00:21:35.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:35.269+0000 7fb0daa45700 1 -- 172.21.15.121:0/2684225081 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb0d40feb30 con 0x7fb0bc0a2ca0 2024-06-10T00:21:36.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:36 smithi121 ceph-mon[32378]: osdmap e63: 9 total, 9 up, 9 in 2024-06-10T00:21:36.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:21:36.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:21:36.147 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:36 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:36.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:36 smithi193 ceph-mon[36716]: osdmap e63: 9 total, 9 up, 9 in 2024-06-10T00:21:36.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:21:36.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:21:36.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:36 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:36.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:36 smithi135 ceph-mon[36424]: osdmap e63: 9 total, 9 up, 9 in 2024-06-10T00:21:36.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:21:36.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:21:36.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:36 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:37.198 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:37 smithi121 ceph-mon[32378]: from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:21:37.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:37 smithi121 ceph-mon[32378]: osdmap e64: 9 total, 9 up, 9 in 2024-06-10T00:21:37.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:37 smithi121 ceph-mon[32378]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:37.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:37 smithi193 ceph-mon[36716]: from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:21:37.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:37 smithi193 ceph-mon[36716]: osdmap e64: 9 total, 9 up, 9 in 2024-06-10T00:21:37.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:37 smithi193 ceph-mon[36716]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:37.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:37 smithi135 ceph-mon[36424]: from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:21:37.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:37 smithi135 ceph-mon[36424]: osdmap e64: 9 total, 9 up, 9 in 2024-06-10T00:21:37.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:37 smithi135 ceph-mon[36424]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:38.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:38 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1367208565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "109bd90d-d878-46b2-99c3-af0792345523"}]: dispatch 2024-06-10T00:21:38.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:38 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1367208565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "109bd90d-d878-46b2-99c3-af0792345523"}]': finished 2024-06-10T00:21:38.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:38 smithi121 ceph-mon[32378]: osdmap e65: 10 total, 9 up, 10 in 2024-06-10T00:21:38.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:38 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:38.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:38 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1367208565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "109bd90d-d878-46b2-99c3-af0792345523"}]: dispatch 2024-06-10T00:21:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:38 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1367208565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "109bd90d-d878-46b2-99c3-af0792345523"}]': finished 2024-06-10T00:21:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:38 smithi135 ceph-mon[36424]: osdmap e65: 10 total, 9 up, 10 in 2024-06-10T00:21:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:38 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:38.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:38 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1367208565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "109bd90d-d878-46b2-99c3-af0792345523"}]: dispatch 2024-06-10T00:21:38.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:38 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1367208565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "109bd90d-d878-46b2-99c3-af0792345523"}]': finished 2024-06-10T00:21:38.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:38 smithi193 ceph-mon[36716]: osdmap e65: 10 total, 9 up, 10 in 2024-06-10T00:21:38.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:38 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:39 smithi121 ceph-mon[32378]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-06-10T00:21:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:39 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3055827617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:21:39.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:39 smithi135 ceph-mon[36424]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-06-10T00:21:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:39 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3055827617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:21:39.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:39 smithi193 ceph-mon[36716]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-06-10T00:21:39.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:39 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3055827617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:21:41.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:41 smithi121 ceph-mon[32378]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-06-10T00:21:41.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:41 smithi135 ceph-mon[36424]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-06-10T00:21:41.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:41 smithi193 ceph-mon[36716]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-06-10T00:21:43.296 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:43 smithi121 ceph-mon[32378]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-06-10T00:21:43.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:43 smithi135 ceph-mon[36424]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-06-10T00:21:43.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:43 smithi193 ceph-mon[36716]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-06-10T00:21:45.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:45 smithi121 ceph-mon[32378]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-06-10T00:21:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:45 smithi135 ceph-mon[36424]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-06-10T00:21:45.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:45 smithi193 ceph-mon[36716]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-06-10T00:21:46.447 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-10T00:21:46.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:46.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-10T00:21:46.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:46.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-06-10T00:21:46.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:47.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:47 smithi121 ceph-mon[32378]: Deploying daemon osd.9 on smithi121 2024-06-10T00:21:47.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:47 smithi121 ceph-mon[32378]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-10T00:21:47.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:47 smithi135 ceph-mon[36424]: Deploying daemon osd.9 on smithi121 2024-06-10T00:21:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:47 smithi135 ceph-mon[36424]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-10T00:21:47.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:47 smithi193 ceph-mon[36716]: Deploying daemon osd.9 on smithi121 2024-06-10T00:21:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:47 smithi193 ceph-mon[36716]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-10T00:21:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:49 smithi121 ceph-mon[32378]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:49.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:49 smithi135 ceph-mon[36424]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:49.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:49 smithi193 ceph-mon[36716]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:51.398 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:51 smithi121 ceph-mon[32378]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:51.433 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:51 smithi193 ceph-mon[36716]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:51.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:51 smithi135 ceph-mon[36424]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:52 smithi121 ceph-mon[32378]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:52 smithi135 ceph-mon[36424]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:53.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:53.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:53.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:53.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:52 smithi193 ceph-mon[36716]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:55.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:55 smithi121 ceph-mon[32378]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:55.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:55.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:55.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:55.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:55.124 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:55 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:55.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:55 smithi193 ceph-mon[36716]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:55.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:55.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:55.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:55.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:55.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:55 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:55 smithi135 ceph-mon[36424]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:21:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:21:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:55 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:55.725 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 9 on host 'smithi121' 2024-06-10T00:21:55.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.725+0000 7fb0d17fa700 1 -- 172.21.15.121:0/2684225081 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb0d40feb30 con 0x7fb0bc0a2ca0 2024-06-10T00:21:55.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.727+0000 7fb0daa45700 1 -- 172.21.15.121:0/2684225081 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0bc0a2ca0 msgr2=0x7fb0bc0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:55.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.727+0000 7fb0daa45700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0bc0a2ca0 0x7fb0bc0a5160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4000c00 tx=0x7fb0c4005fb0 comp rx=0 tx=0).stop 2024-06-10T00:21:55.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.727+0000 7fb0daa45700 1 -- 172.21.15.121:0/2684225081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0d4102f60 msgr2=0x7fb0d4103ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:21:55.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.727+0000 7fb0daa45700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0d4102f60 0x7fb0d4103ee0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc007ee0 tx=0x7fb0cc00d3b0 comp rx=0 tx=0).stop 2024-06-10T00:21:55.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.728+0000 7fb0daa45700 1 -- 172.21.15.121:0/2684225081 shutdown_connections 2024-06-10T00:21:55.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.728+0000 7fb0daa45700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0bc0a2ca0 0x7fb0bc0a5160 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:55.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.728+0000 7fb0daa45700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0d4102f60 0x7fb0d4103ee0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:55.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.728+0000 7fb0daa45700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0d41013c0 0x7fb0d4105300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:55.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.728+0000 7fb0daa45700 1 --2- 172.21.15.121:0/2684225081 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb0d41025c0 0x7fb0d41039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:21:55.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.728+0000 7fb0daa45700 1 -- 172.21.15.121:0/2684225081 >> 172.21.15.121:0/2684225081 conn(0x7fb0d40fc960 msgr2=0x7fb0d40fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:21:55.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.728+0000 7fb0daa45700 1 -- 172.21.15.121:0/2684225081 shutdown_connections 2024-06-10T00:21:55.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:21:55.728+0000 7fb0daa45700 1 -- 172.21.15.121:0/2684225081 wait complete. 2024-06-10T00:21:56.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:56 smithi121 ceph-mon[32378]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-10T00:21:56.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:56 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:56.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:56 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:56.307 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:56 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:56.337 DEBUG:teuthology.orchestra.run.smithi121:osd.9> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.9.service 2024-06-10T00:21:56.339 INFO:tasks.cephadm:Deploying osd.10 on smithi135 with /dev/vg_nvme/lv_1... 2024-06-10T00:21:56.339 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-10T00:21:56.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:56 smithi193 ceph-mon[36716]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-10T00:21:56.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:56 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:56.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:56 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:56.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:56 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:56.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:56 smithi135 ceph-mon[36424]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-06-10T00:21:56.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:56 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:56.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:56 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:21:56.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:56 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:21:56.584 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:21:57.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:57 smithi135 ceph-mon[36424]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:57.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:57 smithi135 ceph-mon[36424]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-10T00:21:57.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:57 smithi135 ceph-mon[36424]: osdmap e66: 10 total, 9 up, 10 in 2024-06-10T00:21:57.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:57 smithi135 ceph-mon[36424]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:21:57.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:57 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:57 smithi121 ceph-mon[32378]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:57 smithi121 ceph-mon[32378]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-10T00:21:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:57 smithi121 ceph-mon[32378]: osdmap e66: 10 total, 9 up, 10 in 2024-06-10T00:21:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:57 smithi121 ceph-mon[32378]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:21:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:57 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:57.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:21:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:21:57.136+0000 7f774637f700 -1 osd.9 0 waiting for initial osdmap 2024-06-10T00:21:57.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:21:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:21:57.148+0000 7f7742172700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:21:57.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:57 smithi193 ceph-mon[36716]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:57 smithi193 ceph-mon[36716]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-06-10T00:21:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:57 smithi193 ceph-mon[36716]: osdmap e66: 10 total, 9 up, 10 in 2024-06-10T00:21:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:57 smithi193 ceph-mon[36716]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-06-10T00:21:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:57 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:58.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:58 smithi135 ceph-mon[36424]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:21:58.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:58 smithi135 ceph-mon[36424]: osdmap e67: 10 total, 9 up, 10 in 2024-06-10T00:21:58.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:58 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:58.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:58 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:58.311 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:58 smithi121 ceph-mon[32378]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:21:58.311 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:58 smithi121 ceph-mon[32378]: osdmap e67: 10 total, 9 up, 10 in 2024-06-10T00:21:58.311 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:58 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:58.311 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:58 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:58.506 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-10T00:21:58.507 INFO:teuthology.orchestra.run.smithi135.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-10T00:21:58.507 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10+0 records in 2024-06-10T00:21:58.507 INFO:teuthology.orchestra.run.smithi135.stderr:10+0 records out 2024-06-10T00:21:58.507 INFO:teuthology.orchestra.run.smithi135.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116437 s, 901 MB/s 2024-06-10T00:21:58.507 INFO:teuthology.orchestra.run.smithi135.stderr:--> Zapping successful for: 2024-06-10T00:21:58.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:58 smithi193 ceph-mon[36716]: from='osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-06-10T00:21:58.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:58 smithi193 ceph-mon[36716]: osdmap e67: 10 total, 9 up, 10 in 2024-06-10T00:21:58.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:58 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:58.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:58 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:59.198 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:59 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:21:59.198 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:59 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:21:59.199 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:59 smithi135 ceph-mon[36424]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:59.199 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:59 smithi135 ceph-mon[36424]: osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801] boot 2024-06-10T00:21:59.199 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:59 smithi135 ceph-mon[36424]: osdmap e68: 10 total, 10 up, 10 in 2024-06-10T00:21:59.199 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:21:59 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:59.228 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi135:vg_nvme/lv_1 2024-06-10T00:21:59.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:59 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:21:59.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:59 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:21:59.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:59 smithi121 ceph-mon[32378]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:59.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:59 smithi121 ceph-mon[32378]: osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801] boot 2024-06-10T00:21:59.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:59 smithi121 ceph-mon[32378]: osdmap e68: 10 total, 10 up, 10 in 2024-06-10T00:21:59.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:21:59 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:21:59.469 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:21:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:59 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:21:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:59 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:21:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:59 smithi193 ceph-mon[36716]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-06-10T00:21:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:59 smithi193 ceph-mon[36716]: osd.9 [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801] boot 2024-06-10T00:21:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:59 smithi193 ceph-mon[36716]: osdmap e68: 10 total, 10 up, 10 in 2024-06-10T00:21:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:21:59 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:22:00.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:00 smithi121 ceph-mon[32378]: osdmap e69: 10 total, 10 up, 10 in 2024-06-10T00:22:00.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:00.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:00.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:00.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:00.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:00.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:00 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:00.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:00 smithi135 ceph-mon[36424]: osdmap e69: 10 total, 10 up, 10 in 2024-06-10T00:22:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:00 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:00 smithi193 ceph-mon[36716]: osdmap e69: 10 total, 10 up, 10 in 2024-06-10T00:22:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:00 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:01.205 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.204+0000 7fba4e671700 1 -- 172.21.15.135:0/1710070764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba48104080 msgr2=0x7fba4810a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:01.205 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.204+0000 7fba4e671700 1 --2- 172.21.15.135:0/1710070764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba48104080 0x7fba4810a930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fba4000b0d0 tx=0x7fba4000b3e0 comp rx=0 tx=0).stop 2024-06-10T00:22:01.205 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.204+0000 7fba4e671700 1 -- 172.21.15.135:0/1710070764 shutdown_connections 2024-06-10T00:22:01.205 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.204+0000 7fba4e671700 1 --2- 172.21.15.135:0/1710070764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba48104080 0x7fba4810a930 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:01.205 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.204+0000 7fba4e671700 1 --2- 172.21.15.135:0/1710070764 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba481024e0 0x7fba48102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:01.205 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.204+0000 7fba4e671700 1 --2- 172.21.15.135:0/1710070764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba481036e0 0x7fba48103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:01.205 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.204+0000 7fba4e671700 1 -- 172.21.15.135:0/1710070764 >> 172.21.15.135:0/1710070764 conn(0x7fba480fda80 msgr2=0x7fba480ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:01.205 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.204+0000 7fba4e671700 1 -- 172.21.15.135:0/1710070764 shutdown_connections 2024-06-10T00:22:01.205 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.204+0000 7fba4e671700 1 -- 172.21.15.135:0/1710070764 wait complete. 2024-06-10T00:22:01.206 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:01 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi121 2024-06-10T00:22:01.206 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:01 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi121 to 4272M 2024-06-10T00:22:01.206 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:01 smithi135 ceph-mon[36424]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:01.208 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.205+0000 7fba4e671700 1 Processor -- start 2024-06-10T00:22:01.208 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.205+0000 7fba4e671700 1 -- start start 2024-06-10T00:22:01.208 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.205+0000 7fba4e671700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba481024e0 0x7fba481986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:01.208 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.205+0000 7fba4e671700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba481036e0 0x7fba48198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:01.208 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.205+0000 7fba4e671700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba48104080 0x7fba4819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:01.208 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.205+0000 7fba4e671700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fba4810e150 con 0x7fba48104080 2024-06-10T00:22:01.208 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.205+0000 7fba4e671700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fba4810dfd0 con 0x7fba481036e0 2024-06-10T00:22:01.208 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba4e671700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba4810e2d0 con 0x7fba481024e0 2024-06-10T00:22:01.208 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba477fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba481036e0 0x7fba48198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:01.208 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba477fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba481036e0 0x7fba48198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:36066/0 (socket says 172.21.15.135:36066) 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba477fe700 1 -- 172.21.15.135:0/551594671 learned_addr learned my addr 172.21.15.135:0/551594671 (peer_addr_for_me v2:172.21.15.135:0/0) 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba47fff700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba481024e0 0x7fba481986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba4cc0e700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba48104080 0x7fba4819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba477fe700 1 -- 172.21.15.135:0/551594671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba481024e0 msgr2=0x7fba481986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba477fe700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba481024e0 0x7fba481986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba477fe700 1 -- 172.21.15.135:0/551594671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba48104080 msgr2=0x7fba4819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba477fe700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba48104080 0x7fba4819fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba477fe700 1 -- 172.21.15.135:0/551594671 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba481a0340 con 0x7fba481036e0 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba477fe700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba481036e0 0x7fba48198c20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00c920 tx=0x7fba3c00cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba457fa700 1 -- 172.21.15.135:0/551594671 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c004a30 con 0x7fba481036e0 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba4e671700 1 -- 172.21.15.135:0/551594671 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba481a0630 con 0x7fba481036e0 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba457fa700 1 -- 172.21.15.135:0/551594671 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba3c005500 con 0x7fba481036e0 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba457fa700 1 -- 172.21.15.135:0/551594671 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c007d40 con 0x7fba481036e0 2024-06-10T00:22:01.209 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.206+0000 7fba4e671700 1 -- 172.21.15.135:0/551594671 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba481a0c40 con 0x7fba481036e0 2024-06-10T00:22:01.210 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.207+0000 7fba4e671700 1 -- 172.21.15.135:0/551594671 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba48069180 con 0x7fba481036e0 2024-06-10T00:22:01.210 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.209+0000 7fba457fa700 1 -- 172.21.15.135:0/551594671 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fba3c004bd0 con 0x7fba481036e0 2024-06-10T00:22:01.210 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.209+0000 7fba457fa700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fba300a2ca0 0x7fba300a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:01.210 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.209+0000 7fba457fa700 1 -- 172.21.15.135:0/551594671 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7fba3c0ce6c0 con 0x7fba481036e0 2024-06-10T00:22:01.211 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.209+0000 7fba47fff700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fba300a2ca0 0x7fba300a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:01.211 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.210+0000 7fba47fff700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fba300a2ca0 0x7fba300a5160 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fba48103540 tx=0x7fba38009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:01.212 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.213+0000 7fba457fa700 1 -- 172.21.15.135:0/551594671 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fba3c014030 con 0x7fba481036e0 2024-06-10T00:22:01.417 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:01.417+0000 7fba4e671700 1 -- 172.21.15.135:0/551594671 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fba480628d0 con 0x7fba300a2ca0 2024-06-10T00:22:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:01 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi121 2024-06-10T00:22:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:01 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi121 to 4272M 2024-06-10T00:22:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:01 smithi121 ceph-mon[32378]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:01.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:01 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi121 2024-06-10T00:22:01.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:01 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi121 to 4272M 2024-06-10T00:22:01.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:01 smithi193 ceph-mon[36716]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:02.360 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:02 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:22:02.360 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:02 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:22:02.361 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:02 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:02.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:02 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:22:02.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:02 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:22:02.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:02 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:02.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:02 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:22:02.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:02 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:22:02.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:02 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:03.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:03 smithi135 ceph-mon[36424]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:22:03.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:03 smithi135 ceph-mon[36424]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:03 smithi121 ceph-mon[32378]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:22:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:03 smithi121 ceph-mon[32378]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:03.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:03 smithi193 ceph-mon[36716]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi135:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:22:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:03 smithi193 ceph-mon[36716]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:05 smithi121 ceph-mon[32378]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:05 smithi121 ceph-mon[32378]: from='client.? 172.21.15.135:0/2325877105' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e56ceb-7b01-4bec-9d2d-bff37873aa75"}]: dispatch 2024-06-10T00:22:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:05 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e56ceb-7b01-4bec-9d2d-bff37873aa75"}]: dispatch 2024-06-10T00:22:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:05 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66e56ceb-7b01-4bec-9d2d-bff37873aa75"}]': finished 2024-06-10T00:22:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:05 smithi121 ceph-mon[32378]: osdmap e70: 11 total, 10 up, 11 in 2024-06-10T00:22:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:05 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:05 smithi121 ceph-mon[32378]: from='client.? 172.21.15.135:0/3301916507' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:22:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:05 smithi135 ceph-mon[36424]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:05 smithi135 ceph-mon[36424]: from='client.? 172.21.15.135:0/2325877105' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e56ceb-7b01-4bec-9d2d-bff37873aa75"}]: dispatch 2024-06-10T00:22:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:05 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e56ceb-7b01-4bec-9d2d-bff37873aa75"}]: dispatch 2024-06-10T00:22:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:05 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66e56ceb-7b01-4bec-9d2d-bff37873aa75"}]': finished 2024-06-10T00:22:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:05 smithi135 ceph-mon[36424]: osdmap e70: 11 total, 10 up, 11 in 2024-06-10T00:22:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:05 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:05 smithi135 ceph-mon[36424]: from='client.? 172.21.15.135:0/3301916507' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:22:05.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:05 smithi193 ceph-mon[36716]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:05 smithi193 ceph-mon[36716]: from='client.? 172.21.15.135:0/2325877105' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e56ceb-7b01-4bec-9d2d-bff37873aa75"}]: dispatch 2024-06-10T00:22:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:05 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66e56ceb-7b01-4bec-9d2d-bff37873aa75"}]: dispatch 2024-06-10T00:22:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:05 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66e56ceb-7b01-4bec-9d2d-bff37873aa75"}]': finished 2024-06-10T00:22:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:05 smithi193 ceph-mon[36716]: osdmap e70: 11 total, 10 up, 11 in 2024-06-10T00:22:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:05 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:05 smithi193 ceph-mon[36716]: from='client.? 172.21.15.135:0/3301916507' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:22:07.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:07 smithi121 ceph-mon[32378]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:07.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:07 smithi135 ceph-mon[36424]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:07 smithi193 ceph-mon[36716]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:09.231 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:09 smithi135 ceph-mon[36424]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:09.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:09 smithi121 ceph-mon[32378]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:09.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:09 smithi193 ceph-mon[36716]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:11.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:11 smithi135 ceph-mon[36424]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:11 smithi121 ceph-mon[32378]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:11.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:11 smithi193 ceph-mon[36716]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:12.430 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:12 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-10T00:22:12.430 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:12 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:12.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:12 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-10T00:22:12.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:12 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:12.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:12 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-06-10T00:22:12.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:12 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:13.427 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:13 smithi135 ceph-mon[36424]: Deploying daemon osd.10 on smithi135 2024-06-10T00:22:13.427 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:13 smithi135 ceph-mon[36424]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:13.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:13 smithi121 ceph-mon[32378]: Deploying daemon osd.10 on smithi135 2024-06-10T00:22:13.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:13 smithi121 ceph-mon[32378]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:13.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:13 smithi193 ceph-mon[36716]: Deploying daemon osd.10 on smithi135 2024-06-10T00:22:13.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:13 smithi193 ceph-mon[36716]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:15.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:15 smithi135 ceph-mon[36424]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:15 smithi121 ceph-mon[32378]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:15.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:15 smithi193 ceph-mon[36716]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:17.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:17 smithi121 ceph-mon[32378]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:17.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:17 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:17.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:17 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:17.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:17 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:17 smithi135 ceph-mon[36424]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:17 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:17 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:17 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:17.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:17 smithi193 ceph-mon[36716]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:17 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:17 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:17 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:19.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:19 smithi121 ceph-mon[32378]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:19.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:19 smithi135 ceph-mon[36424]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:19.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:19 smithi193 ceph-mon[36716]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:20.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.989+0000 7fba457fa700 1 -- 172.21.15.135:0/551594671 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fba480628d0 con 0x7fba300a2ca0 2024-06-10T00:22:20.991 INFO:teuthology.orchestra.run.smithi135.stdout:Created osd(s) 10 on host 'smithi135' 2024-06-10T00:22:20.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 -- 172.21.15.135:0/551594671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fba300a2ca0 msgr2=0x7fba300a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:20.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fba300a2ca0 0x7fba300a5160 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fba48103540 tx=0x7fba38009190 comp rx=0 tx=0).stop 2024-06-10T00:22:20.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 -- 172.21.15.135:0/551594671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba481036e0 msgr2=0x7fba48198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:20.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba481036e0 0x7fba48198c20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00c920 tx=0x7fba3c00cc30 comp rx=0 tx=0).stop 2024-06-10T00:22:20.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 -- 172.21.15.135:0/551594671 shutdown_connections 2024-06-10T00:22:20.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fba300a2ca0 0x7fba300a5160 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:20.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba48104080 0x7fba4819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:20.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba481024e0 0x7fba481986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:20.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 --2- 172.21.15.135:0/551594671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba481036e0 0x7fba48198c20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:20.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 -- 172.21.15.135:0/551594671 >> 172.21.15.135:0/551594671 conn(0x7fba480fda80 msgr2=0x7fba48106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:20.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 -- 172.21.15.135:0/551594671 shutdown_connections 2024-06-10T00:22:20.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-06-10T00:22:20.992+0000 7fba4e671700 1 -- 172.21.15.135:0/551594671 wait complete. 2024-06-10T00:22:21.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:21 smithi135 ceph-mon[36424]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:21 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:21 smithi193 ceph-mon[36716]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:21 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:21 smithi121 ceph-mon[32378]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:21.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:21 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:21.643 DEBUG:teuthology.orchestra.run.smithi135:osd.10> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.10.service 2024-06-10T00:22:21.645 INFO:tasks.cephadm:Deploying osd.11 on smithi193 with /dev/vg_nvme/lv_1... 2024-06-10T00:22:21.646 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-10T00:22:21.888 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:22:22.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:22 smithi135 ceph-mon[36424]: from='osd.10 [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-10T00:22:22.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:22 smithi135 ceph-mon[36424]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-10T00:22:22.340 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:22 smithi193 ceph-mon[36716]: from='osd.10 [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-10T00:22:22.340 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:22 smithi193 ceph-mon[36716]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-10T00:22:22.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:22 smithi121 ceph-mon[32378]: from='osd.10 [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-10T00:22:22.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:22 smithi121 ceph-mon[32378]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-06-10T00:22:23.105 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:23 smithi135 ceph-mon[36424]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-10T00:22:23.105 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:23 smithi135 ceph-mon[36424]: osdmap e71: 11 total, 10 up, 11 in 2024-06-10T00:22:23.105 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:23 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:23.106 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:23 smithi135 ceph-mon[36424]: from='osd.10 [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:22:23.106 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:23 smithi135 ceph-mon[36424]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:22:23.106 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:23 smithi135 ceph-mon[36424]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:23.333 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:23 smithi193 ceph-mon[36716]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-10T00:22:23.333 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:23 smithi193 ceph-mon[36716]: osdmap e71: 11 total, 10 up, 11 in 2024-06-10T00:22:23.333 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:23 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:23.334 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:23 smithi193 ceph-mon[36716]: from='osd.10 [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:22:23.334 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:23 smithi193 ceph-mon[36716]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:22:23.334 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:23 smithi193 ceph-mon[36716]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:23 smithi121 ceph-mon[32378]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-06-10T00:22:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:23 smithi121 ceph-mon[32378]: osdmap e71: 11 total, 10 up, 11 in 2024-06-10T00:22:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:23 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:23 smithi121 ceph-mon[32378]: from='osd.10 [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:22:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:23 smithi121 ceph-mon[32378]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]: dispatch 2024-06-10T00:22:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:23 smithi121 ceph-mon[32378]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:23.686 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:22:23 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:22:23.572+0000 7ff54f1ae700 -1 osd.10 0 waiting for initial osdmap 2024-06-10T00:22:23.686 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:22:23 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:22:23.585+0000 7ff546f99700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:22:24.062 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-10T00:22:24.063 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-10T00:22:24.063 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-06-10T00:22:24.063 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-06-10T00:22:24.063 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121958 s, 860 MB/s 2024-06-10T00:22:24.063 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-06-10T00:22:24.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:24 smithi135 ceph-mon[36424]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:22:24.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:24 smithi135 ceph-mon[36424]: osdmap e72: 11 total, 10 up, 11 in 2024-06-10T00:22:24.377 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:24 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:24.378 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:24 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:24.378 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:24 smithi135 ceph-mon[36424]: from='osd.10 ' entity='osd.10' 2024-06-10T00:22:24.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:24 smithi193 ceph-mon[36716]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:22:24.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:24 smithi193 ceph-mon[36716]: osdmap e72: 11 total, 10 up, 11 in 2024-06-10T00:22:24.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:24 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:24.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:24 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:24.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:24 smithi193 ceph-mon[36716]: from='osd.10 ' entity='osd.10' 2024-06-10T00:22:24.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:24 smithi121 ceph-mon[32378]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi135", "root=default"]}]': finished 2024-06-10T00:22:24.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:24 smithi121 ceph-mon[32378]: osdmap e72: 11 total, 10 up, 11 in 2024-06-10T00:22:24.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:24 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:24.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:24 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:24.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:24 smithi121 ceph-mon[32378]: from='osd.10 ' entity='osd.10' 2024-06-10T00:22:24.880 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_1 2024-06-10T00:22:25.122 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: osd.10 [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287] boot 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: osdmap e73: 11 total, 11 up, 11 in 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:25.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:25.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:25 smithi193 ceph-mon[36716]: osdmap e74: 11 total, 11 up, 11 in 2024-06-10T00:22:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:22:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:22:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: osd.10 [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287] boot 2024-06-10T00:22:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: osdmap e73: 11 total, 11 up, 11 in 2024-06-10T00:22:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:25.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:25.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:25 smithi121 ceph-mon[32378]: osdmap e74: 11 total, 11 up, 11 in 2024-06-10T00:22:25.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: osd.10 [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287] boot 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: osdmap e73: 11 total, 11 up, 11 in 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:25.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:25.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:25.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:25 smithi135 ceph-mon[36424]: osdmap e74: 11 total, 11 up, 11 in 2024-06-10T00:22:26.334 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:26 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi135 2024-06-10T00:22:26.334 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:26 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi135 to 4272M 2024-06-10T00:22:26.335 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:26 smithi193 ceph-mon[36716]: osdmap e75: 11 total, 11 up, 11 in 2024-06-10T00:22:26.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:26 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi135 2024-06-10T00:22:26.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:26 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi135 to 4272M 2024-06-10T00:22:26.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:26 smithi121 ceph-mon[32378]: osdmap e75: 11 total, 11 up, 11 in 2024-06-10T00:22:26.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:26 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi135 2024-06-10T00:22:26.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:26 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi135 to 4272M 2024-06-10T00:22:26.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:26 smithi135 ceph-mon[36424]: osdmap e75: 11 total, 11 up, 11 in 2024-06-10T00:22:26.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.869+0000 7fcf67ee6700 1 -- 172.21.15.193:0/4039204740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf60102510 msgr2=0x7fcf60102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:26.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.869+0000 7fcf67ee6700 1 --2- 172.21.15.193:0/4039204740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf60102510 0x7fcf60102930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009990 tx=0x7fcf50009ca0 comp rx=0 tx=0).stop 2024-06-10T00:22:26.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.869+0000 7fcf67ee6700 1 -- 172.21.15.193:0/4039204740 shutdown_connections 2024-06-10T00:22:26.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.869+0000 7fcf67ee6700 1 --2- 172.21.15.193:0/4039204740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf601040b0 0x7fcf6010a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:26.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.869+0000 7fcf67ee6700 1 --2- 172.21.15.193:0/4039204740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf60102510 0x7fcf60102930 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.869+0000 7fcf67ee6700 1 --2- 172.21.15.193:0/4039204740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf60103710 0x7fcf60103b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.869+0000 7fcf67ee6700 1 -- 172.21.15.193:0/4039204740 >> 172.21.15.193:0/4039204740 conn(0x7fcf600fdab0 msgr2=0x7fcf600ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.869+0000 7fcf67ee6700 1 -- 172.21.15.193:0/4039204740 shutdown_connections 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.869+0000 7fcf67ee6700 1 -- 172.21.15.193:0/4039204740 wait complete. 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf67ee6700 1 Processor -- start 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf67ee6700 1 -- start start 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf67ee6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf60102510 0x7fcf601986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf67ee6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf60103710 0x7fcf60198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf67ee6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf601040b0 0x7fcf6019fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf67ee6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6010e100 con 0x7fcf60102510 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf67ee6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6010df80 con 0x7fcf60103710 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf67ee6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6010e280 con 0x7fcf601040b0 2024-06-10T00:22:26.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf66483700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf601040b0 0x7fcf6019fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf66483700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf601040b0 0x7fcf6019fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:49302/0 (socket says 172.21.15.193:49302) 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf66483700 1 -- 172.21.15.193:0/1530274191 learned_addr learned my addr 172.21.15.193:0/1530274191 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf65c82700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf60102510 0x7fcf601986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf65481700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf60103710 0x7fcf60198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf66483700 1 -- 172.21.15.193:0/1530274191 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf60103710 msgr2=0x7fcf60198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf66483700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf60103710 0x7fcf60198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf66483700 1 -- 172.21.15.193:0/1530274191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf60102510 msgr2=0x7fcf601986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf66483700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf60102510 0x7fcf601986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf66483700 1 -- 172.21.15.193:0/1530274191 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf601a0310 con 0x7fcf601040b0 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.870+0000 7fcf65c82700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf60102510 0x7fcf601986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.871+0000 7fcf66483700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf601040b0 0x7fcf6019fc70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007d40 tx=0x7fcf5800a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.871+0000 7fcf56ffd700 1 -- 172.21.15.193:0/1530274191 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58016020 con 0x7fcf601040b0 2024-06-10T00:22:26.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.871+0000 7fcf56ffd700 1 -- 172.21.15.193:0/1530274191 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf5800ad60 con 0x7fcf601040b0 2024-06-10T00:22:26.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.871+0000 7fcf56ffd700 1 -- 172.21.15.193:0/1530274191 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58011780 con 0x7fcf601040b0 2024-06-10T00:22:26.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.871+0000 7fcf67ee6700 1 -- 172.21.15.193:0/1530274191 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf601a0600 con 0x7fcf601040b0 2024-06-10T00:22:26.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.871+0000 7fcf67ee6700 1 -- 172.21.15.193:0/1530274191 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf601a0af0 con 0x7fcf601040b0 2024-06-10T00:22:26.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.872+0000 7fcf67ee6700 1 -- 172.21.15.193:0/1530274191 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf60069180 con 0x7fcf601040b0 2024-06-10T00:22:26.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.873+0000 7fcf56ffd700 1 -- 172.21.15.193:0/1530274191 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fcf58019020 con 0x7fcf601040b0 2024-06-10T00:22:26.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.873+0000 7fcf56ffd700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fcf4c0a2c20 0x7fcf4c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:26.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.874+0000 7fcf56ffd700 1 -- 172.21.15.193:0/1530274191 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6723+0+0 (secure 0 0 0) 0x7fcf580ceb50 con 0x7fcf601040b0 2024-06-10T00:22:26.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.874+0000 7fcf65c82700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fcf4c0a2c20 0x7fcf4c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:26.875 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.874+0000 7fcf65c82700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fcf4c0a2c20 0x7fcf4c0a50e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcf500039c0 tx=0x7fcf50005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:26.878 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:26.878+0000 7fcf56ffd700 1 -- 172.21.15.193:0/1530274191 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcf58020020 con 0x7fcf601040b0 2024-06-10T00:22:27.083 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:27.084+0000 7fcf67ee6700 1 -- 172.21.15.193:0/1530274191 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fcf600628d0 con 0x7fcf4c0a2c20 2024-06-10T00:22:27.350 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:27 smithi121 ceph-mon[32378]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:27.350 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:27 smithi121 ceph-mon[32378]: osdmap e76: 11 total, 11 up, 11 in 2024-06-10T00:22:27.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:27 smithi193 ceph-mon[36716]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:27.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:27 smithi193 ceph-mon[36716]: osdmap e76: 11 total, 11 up, 11 in 2024-06-10T00:22:27.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:27 smithi135 ceph-mon[36424]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:27.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:27 smithi135 ceph-mon[36424]: osdmap e76: 11 total, 11 up, 11 in 2024-06-10T00:22:28.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:28 smithi193 ceph-mon[36716]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:22:28.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:22:28.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:22:28.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:28 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:28.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:28 smithi193 ceph-mon[36716]: osdmap e77: 11 total, 11 up, 11 in 2024-06-10T00:22:28.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:28 smithi121 ceph-mon[32378]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:22:28.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:22:28.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:22:28.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:28 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:28.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:28 smithi121 ceph-mon[32378]: osdmap e77: 11 total, 11 up, 11 in 2024-06-10T00:22:28.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:28 smithi135 ceph-mon[36424]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:22:28.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-10T00:22:28.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-10T00:22:28.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:28 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:28.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:28 smithi135 ceph-mon[36424]: osdmap e77: 11 total, 11 up, 11 in 2024-06-10T00:22:29.361 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:29 smithi193 ceph-mon[36716]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-10T00:22:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:29 smithi121 ceph-mon[32378]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-10T00:22:29.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:29 smithi135 ceph-mon[36424]: pgmap v240: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-06-10T00:22:30.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:30 smithi193 ceph-mon[36716]: from='client.? 172.21.15.193:0/2880061054' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b8b392a-5b98-4b28-8301-8ceecd572573"}]: dispatch 2024-06-10T00:22:30.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:30 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b8b392a-5b98-4b28-8301-8ceecd572573"}]: dispatch 2024-06-10T00:22:30.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:30 smithi193 ceph-mon[36716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b8b392a-5b98-4b28-8301-8ceecd572573"}]': finished 2024-06-10T00:22:30.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:30 smithi193 ceph-mon[36716]: osdmap e78: 12 total, 11 up, 12 in 2024-06-10T00:22:30.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:30 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:30 smithi121 ceph-mon[32378]: from='client.? 172.21.15.193:0/2880061054' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b8b392a-5b98-4b28-8301-8ceecd572573"}]: dispatch 2024-06-10T00:22:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:30 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b8b392a-5b98-4b28-8301-8ceecd572573"}]: dispatch 2024-06-10T00:22:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:30 smithi121 ceph-mon[32378]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b8b392a-5b98-4b28-8301-8ceecd572573"}]': finished 2024-06-10T00:22:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:30 smithi121 ceph-mon[32378]: osdmap e78: 12 total, 11 up, 12 in 2024-06-10T00:22:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:30 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:30.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:30 smithi135 ceph-mon[36424]: from='client.? 172.21.15.193:0/2880061054' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b8b392a-5b98-4b28-8301-8ceecd572573"}]: dispatch 2024-06-10T00:22:30.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:30 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3b8b392a-5b98-4b28-8301-8ceecd572573"}]: dispatch 2024-06-10T00:22:30.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:30 smithi135 ceph-mon[36424]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3b8b392a-5b98-4b28-8301-8ceecd572573"}]': finished 2024-06-10T00:22:30.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:30 smithi135 ceph-mon[36424]: osdmap e78: 12 total, 11 up, 12 in 2024-06-10T00:22:30.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:30 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:31.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:31 smithi193 ceph-mon[36716]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-06-10T00:22:31.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:31 smithi193 ceph-mon[36716]: from='client.? 172.21.15.193:0/3001329727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:22:31.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:31 smithi121 ceph-mon[32378]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-06-10T00:22:31.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:31 smithi121 ceph-mon[32378]: from='client.? 172.21.15.193:0/3001329727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:22:31.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:31 smithi135 ceph-mon[36424]: pgmap v242: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 142 KiB/s, 0 objects/s recovering 2024-06-10T00:22:31.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:31 smithi135 ceph-mon[36424]: from='client.? 172.21.15.193:0/3001329727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-10T00:22:33.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:33 smithi193 ceph-mon[36716]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-10T00:22:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:33 smithi121 ceph-mon[32378]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-10T00:22:33.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:33 smithi135 ceph-mon[36424]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-10T00:22:35.300 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:35 smithi193 ceph-mon[36716]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-06-10T00:22:35.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:35 smithi121 ceph-mon[32378]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-06-10T00:22:35.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:35 smithi135 ceph-mon[36424]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-06-10T00:22:37.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:37 smithi193 ceph-mon[36716]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:37.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:37 smithi121 ceph-mon[32378]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:37.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:37 smithi135 ceph-mon[36424]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:38.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:38 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-10T00:22:38.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:38 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:38.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:38 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-10T00:22:38.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:38 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:38.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:38 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-06-10T00:22:38.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:38 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:39.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:39 smithi193 ceph-mon[36716]: Deploying daemon osd.11 on smithi193 2024-06-10T00:22:39.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:39 smithi193 ceph-mon[36716]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:39 smithi121 ceph-mon[32378]: Deploying daemon osd.11 on smithi193 2024-06-10T00:22:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:39 smithi121 ceph-mon[32378]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:39.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:39 smithi135 ceph-mon[36424]: Deploying daemon osd.11 on smithi193 2024-06-10T00:22:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:39 smithi135 ceph-mon[36424]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:41.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:41 smithi193 ceph-mon[36716]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:41.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:41 smithi121 ceph-mon[32378]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:41.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:41 smithi135 ceph-mon[36424]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:43.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:43 smithi193 ceph-mon[36716]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:43 smithi121 ceph-mon[32378]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:43.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:43 smithi135 ceph-mon[36424]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:44 smithi193 ceph-mon[36716]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:44.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:44 smithi121 ceph-mon[32378]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:44.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:44 smithi135 ceph-mon[36424]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:46.854 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.854+0000 7fcf56ffd700 1 -- 172.21.15.193:0/1530274191 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fcf600628d0 con 0x7fcf4c0a2c20 2024-06-10T00:22:46.855 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 11 on host 'smithi193' 2024-06-10T00:22:46.857 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 -- 172.21.15.193:0/1530274191 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fcf4c0a2c20 msgr2=0x7fcf4c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:46.857 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fcf4c0a2c20 0x7fcf4c0a50e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcf500039c0 tx=0x7fcf50005fb0 comp rx=0 tx=0).stop 2024-06-10T00:22:46.857 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 -- 172.21.15.193:0/1530274191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf601040b0 msgr2=0x7fcf6019fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:46.857 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf601040b0 0x7fcf6019fc70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007d40 tx=0x7fcf5800a340 comp rx=0 tx=0).stop 2024-06-10T00:22:46.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 -- 172.21.15.193:0/1530274191 shutdown_connections 2024-06-10T00:22:46.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fcf4c0a2c20 0x7fcf4c0a50e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:46.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf601040b0 0x7fcf6019fc70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:46.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf60102510 0x7fcf601986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:46.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 --2- 172.21.15.193:0/1530274191 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf60103710 0x7fcf60198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:46.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 -- 172.21.15.193:0/1530274191 >> 172.21.15.193:0/1530274191 conn(0x7fcf600fdab0 msgr2=0x7fcf60106a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:46.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 -- 172.21.15.193:0/1530274191 shutdown_connections 2024-06-10T00:22:46.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:22:46.857+0000 7fcf67ee6700 1 -- 172.21.15.193:0/1530274191 wait complete. 2024-06-10T00:22:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:47 smithi121 ceph-mon[32378]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:47 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:47 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:47.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:47 smithi135 ceph-mon[36424]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:47 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:47 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:47.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:47 smithi193 ceph-mon[36716]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:47.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:47 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:47.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:47 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:47.603 DEBUG:teuthology.orchestra.run.smithi193:osd.11> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.11.service 2024-06-10T00:22:47.606 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-06-10T00:22:47.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-10T00:22:47.860 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:22:48.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:48 smithi121 ceph-mon[32378]: from='osd.11 [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-10T00:22:48.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:48 smithi121 ceph-mon[32378]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-10T00:22:48.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:48 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:48.266 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:48 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:48.394 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:48 smithi193 ceph-mon[36716]: from='osd.11 [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-10T00:22:48.394 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:48 smithi193 ceph-mon[36716]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-10T00:22:48.394 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:48 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:48.394 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:48 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:48.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:48 smithi135 ceph-mon[36424]: from='osd.11 [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-10T00:22:48.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:48 smithi135 ceph-mon[36424]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-06-10T00:22:48.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:48 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:48.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:48 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:49.314 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:49 smithi121 ceph-mon[32378]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:49.314 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:49 smithi121 ceph-mon[32378]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-10T00:22:49.314 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:49 smithi121 ceph-mon[32378]: from='osd.11 [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:22:49.314 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:49 smithi121 ceph-mon[32378]: osdmap e79: 12 total, 11 up, 12 in 2024-06-10T00:22:49.314 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:49.314 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:49 smithi121 ceph-mon[32378]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:22:49.421 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:49 smithi193 ceph-mon[36716]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:49.421 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:49 smithi193 ceph-mon[36716]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-10T00:22:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:49 smithi193 ceph-mon[36716]: from='osd.11 [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:22:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:49 smithi193 ceph-mon[36716]: osdmap e79: 12 total, 11 up, 12 in 2024-06-10T00:22:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:49.422 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:49 smithi193 ceph-mon[36716]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:22:49.422 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:22:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:22:49.150+0000 7fb437783700 -1 osd.11 0 waiting for initial osdmap 2024-06-10T00:22:49.422 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:22:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:22:49.162+0000 7fb433576700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-10T00:22:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:49 smithi135 ceph-mon[36424]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:49 smithi135 ceph-mon[36424]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-06-10T00:22:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:49 smithi135 ceph-mon[36424]: from='osd.11 [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:22:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:49 smithi135 ceph-mon[36424]: osdmap e79: 12 total, 11 up, 12 in 2024-06-10T00:22:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:49 smithi135 ceph-mon[36424]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-06-10T00:22:49.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.671+0000 7fb1b9e55700 1 -- 172.21.15.121:0/916516688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1b41007b0 msgr2=0x7fb1b4100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:49.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.671+0000 7fb1b9e55700 1 --2- 172.21.15.121:0/916516688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1b41007b0 0x7fb1b4100bd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40099c0 tx=0x7fb1a4009cd0 comp rx=0 tx=0).stop 2024-06-10T00:22:49.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.671+0000 7fb1b9e55700 1 -- 172.21.15.121:0/916516688 shutdown_connections 2024-06-10T00:22:49.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.671+0000 7fb1b9e55700 1 --2- 172.21.15.121:0/916516688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1b41041d0 0x7fb1b410aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:49.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.671+0000 7fb1b9e55700 1 --2- 172.21.15.121:0/916516688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1b41007b0 0x7fb1b4100bd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:49.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.671+0000 7fb1b9e55700 1 --2- 172.21.15.121:0/916516688 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1b4103830 0x7fb1b4103c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:49.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.671+0000 7fb1b9e55700 1 -- 172.21.15.121:0/916516688 >> 172.21.15.121:0/916516688 conn(0x7fb1b40fc0b0 msgr2=0x7fb1b40fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:49.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.671+0000 7fb1b9e55700 1 -- 172.21.15.121:0/916516688 shutdown_connections 2024-06-10T00:22:49.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.671+0000 7fb1b9e55700 1 -- 172.21.15.121:0/916516688 wait complete. 2024-06-10T00:22:49.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b9e55700 1 Processor -- start 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b9e55700 1 -- start start 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b9e55700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1b41007b0 0x7fb1b41986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b9e55700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1b4103830 0x7fb1b4198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b9e55700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1b41041d0 0x7fb1b419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b9e55700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b410e0c0 con 0x7fb1b41007b0 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b9e55700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b410df40 con 0x7fb1b41041d0 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b9e55700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b410e240 con 0x7fb1b4103830 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1b41007b0 0x7fb1b41986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1b41007b0 0x7fb1b41986b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59700/0 (socket says 172.21.15.121:59700) 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b37fe700 1 -- 172.21.15.121:0/3291913869 learned_addr learned my addr 172.21.15.121:0/3291913869 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b2ffd700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1b4103830 0x7fb1b4198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b3fff700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1b41041d0 0x7fb1b419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b37fe700 1 -- 172.21.15.121:0/3291913869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1b4103830 msgr2=0x7fb1b4198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b37fe700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1b4103830 0x7fb1b4198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b37fe700 1 -- 172.21.15.121:0/3291913869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1b41041d0 msgr2=0x7fb1b419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b37fe700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1b41041d0 0x7fb1b419fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b37fe700 1 -- 172.21.15.121:0/3291913869 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b41a0310 con 0x7fb1b41007b0 2024-06-10T00:22:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.672+0000 7fb1b2ffd700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1b4103830 0x7fb1b4198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:22:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.673+0000 7fb1b37fe700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1b41007b0 0x7fb1b41986b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb1a400fab0 tx=0x7fb1a400fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.673+0000 7fb1b0ff9700 1 -- 172.21.15.121:0/3291913869 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4027070 con 0x7fb1b41007b0 2024-06-10T00:22:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.673+0000 7fb1b0ff9700 1 -- 172.21.15.121:0/3291913869 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a400fd00 con 0x7fb1b41007b0 2024-06-10T00:22:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.673+0000 7fb1b9e55700 1 -- 172.21.15.121:0/3291913869 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b41a05a0 con 0x7fb1b41007b0 2024-06-10T00:22:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.673+0000 7fb1b0ff9700 1 -- 172.21.15.121:0/3291913869 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a40173f0 con 0x7fb1b41007b0 2024-06-10T00:22:49.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.673+0000 7fb1b9e55700 1 -- 172.21.15.121:0/3291913869 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b41a0a80 con 0x7fb1b41007b0 2024-06-10T00:22:49.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.674+0000 7fb1b9e55700 1 -- 172.21.15.121:0/3291913869 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1b4069180 con 0x7fb1b41007b0 2024-06-10T00:22:49.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.675+0000 7fb1b0ff9700 1 -- 172.21.15.121:0/3291913869 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb1a402a070 con 0x7fb1b41007b0 2024-06-10T00:22:49.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.676+0000 7fb1b0ff9700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb19c0a2c20 0x7fb19c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:49.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.676+0000 7fb1b2ffd700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb19c0a2c20 0x7fb19c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:49.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.676+0000 7fb1b0ff9700 1 -- 172.21.15.121:0/3291913869 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7fb1a40cf960 con 0x7fb1b41007b0 2024-06-10T00:22:49.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.676+0000 7fb1b2ffd700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb19c0a2c20 0x7fb19c0a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8005950 tx=0x7fb1a800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:49.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.679+0000 7fb1b0ff9700 1 -- 172.21.15.121:0/3291913869 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb1a4031080 con 0x7fb1b41007b0 2024-06-10T00:22:49.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.875+0000 7fb1b9e55700 1 -- 172.21.15.121:0/3291913869 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fb1b40630a0 con 0x7fb1b41007b0 2024-06-10T00:22:49.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.876+0000 7fb1b0ff9700 1 -- 172.21.15.121:0/3291913869 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7fb1a4013070 con 0x7fb1b41007b0 2024-06-10T00:22:49.876 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:22:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 -- 172.21.15.121:0/3291913869 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb19c0a2c20 msgr2=0x7fb19c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb19c0a2c20 0x7fb19c0a50e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8005950 tx=0x7fb1a800b410 comp rx=0 tx=0).stop 2024-06-10T00:22:49.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 -- 172.21.15.121:0/3291913869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1b41007b0 msgr2=0x7fb1b41986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:49.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1b41007b0 0x7fb1b41986b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb1a400fab0 tx=0x7fb1a400fae0 comp rx=0 tx=0).stop 2024-06-10T00:22:49.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 -- 172.21.15.121:0/3291913869 shutdown_connections 2024-06-10T00:22:49.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb19c0a2c20 0x7fb19c0a50e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:49.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb1b41041d0 0x7fb1b419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:49.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1b41007b0 0x7fb1b41986b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:49.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 --2- 172.21.15.121:0/3291913869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1b4103830 0x7fb1b4198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:49.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 -- 172.21.15.121:0/3291913869 >> 172.21.15.121:0/3291913869 conn(0x7fb1b40fc0b0 msgr2=0x7fb1b4106b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:49.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 -- 172.21.15.121:0/3291913869 shutdown_connections 2024-06-10T00:22:49.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:49.878+0000 7fb1b9e55700 1 -- 172.21.15.121:0/3291913869 wait complete. 2024-06-10T00:22:50.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:50 smithi193 ceph-mon[36716]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:22:50.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:50 smithi193 ceph-mon[36716]: osdmap e80: 12 total, 11 up, 12 in 2024-06-10T00:22:50.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:50 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:50.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:50 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:50.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:50 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3291913869' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-10T00:22:50.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:50 smithi121 ceph-mon[32378]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:22:50.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:50 smithi121 ceph-mon[32378]: osdmap e80: 12 total, 11 up, 12 in 2024-06-10T00:22:50.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:50 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:50.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:50 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:50.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:50 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3291913869' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-10T00:22:50.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:50 smithi135 ceph-mon[36424]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-06-10T00:22:50.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:50 smithi135 ceph-mon[36424]: osdmap e80: 12 total, 11 up, 12 in 2024-06-10T00:22:50.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:50 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:50.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:50 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:50.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:50 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3291913869' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-10T00:22:50.576 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1717978944,"num_in_osds":12,"osd_in_since":1717978949,"num_remapped_pgs":0} 2024-06-10T00:22:51.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:51 smithi193 ceph-mon[36716]: purged_snaps scrub starts 2024-06-10T00:22:51.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:51 smithi193 ceph-mon[36716]: purged_snaps scrub ok 2024-06-10T00:22:51.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:51 smithi193 ceph-mon[36716]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:51.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:51.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:51 smithi193 ceph-mon[36716]: osd.11 [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609] boot 2024-06-10T00:22:51.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:51 smithi193 ceph-mon[36716]: osdmap e81: 12 total, 12 up, 12 in 2024-06-10T00:22:51.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:51 smithi135 ceph-mon[36424]: purged_snaps scrub starts 2024-06-10T00:22:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:51 smithi135 ceph-mon[36424]: purged_snaps scrub ok 2024-06-10T00:22:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:51 smithi135 ceph-mon[36424]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:51 smithi135 ceph-mon[36424]: osd.11 [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609] boot 2024-06-10T00:22:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:51 smithi135 ceph-mon[36424]: osdmap e81: 12 total, 12 up, 12 in 2024-06-10T00:22:51.409 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:51 smithi121 ceph-mon[32378]: purged_snaps scrub starts 2024-06-10T00:22:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:51 smithi121 ceph-mon[32378]: purged_snaps scrub ok 2024-06-10T00:22:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:51 smithi121 ceph-mon[32378]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-06-10T00:22:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:51 smithi121 ceph-mon[32378]: osd.11 [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609] boot 2024-06-10T00:22:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:51 smithi121 ceph-mon[32378]: osdmap e81: 12 total, 12 up, 12 in 2024-06-10T00:22:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:22:51.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-10T00:22:51.833 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:22:52.243 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: osdmap e82: 12 total, 12 up, 12 in 2024-06-10T00:22:52.243 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.243 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.243 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.243 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.243 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.243 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:52.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:52.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:52.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:52.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:52.244 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:52 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: osdmap e82: 12 total, 12 up, 12 in 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:52.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:52.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:52.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:52 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: osdmap e82: 12 total, 12 up, 12 in 2024-06-10T00:22:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:22:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:52.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:52.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:22:52.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:22:52.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:22:52.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:52 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:22:53.286 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:53 smithi121 ceph-mon[32378]: Detected new or changed devices on smithi193 2024-06-10T00:22:53.287 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:53 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi193 to 4272M 2024-06-10T00:22:53.287 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:53 smithi121 ceph-mon[32378]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:22:53.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:53 smithi193 ceph-mon[36716]: Detected new or changed devices on smithi193 2024-06-10T00:22:53.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:53 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi193 to 4272M 2024-06-10T00:22:53.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:53 smithi193 ceph-mon[36716]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:22:53.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:53 smithi135 ceph-mon[36424]: Detected new or changed devices on smithi193 2024-06-10T00:22:53.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:53 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi193 to 4272M 2024-06-10T00:22:53.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:53 smithi135 ceph-mon[36424]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:22:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.515+0000 7fd2c90da700 1 -- 172.21.15.121:0/1041441101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2c4105930 msgr2=0x7fd2c4107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.515+0000 7fd2c90da700 1 --2- 172.21.15.121:0/1041441101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2c4105930 0x7fd2c4107d20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0098f0 tx=0x7fd2ac009c00 comp rx=0 tx=0).stop 2024-06-10T00:22:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.515+0000 7fd2c90da700 1 -- 172.21.15.121:0/1041441101 shutdown_connections 2024-06-10T00:22:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.515+0000 7fd2c90da700 1 --2- 172.21.15.121:0/1041441101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2c4108260 0x7fd2c410a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.515+0000 7fd2c90da700 1 --2- 172.21.15.121:0/1041441101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2c4103000 0x7fd2c41053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.515+0000 7fd2c90da700 1 --2- 172.21.15.121:0/1041441101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2c4105930 0x7fd2c4107d20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.515+0000 7fd2c90da700 1 -- 172.21.15.121:0/1041441101 >> 172.21.15.121:0/1041441101 conn(0x7fd2c40fc990 msgr2=0x7fd2c40fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.515+0000 7fd2c90da700 1 -- 172.21.15.121:0/1041441101 shutdown_connections 2024-06-10T00:22:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.515+0000 7fd2c90da700 1 -- 172.21.15.121:0/1041441101 wait complete. 2024-06-10T00:22:53.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c90da700 1 Processor -- start 2024-06-10T00:22:53.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c90da700 1 -- start start 2024-06-10T00:22:53.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c90da700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2c4103000 0x7fd2c4198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:53.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c90da700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2c4105930 0x7fd2c4198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:53.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c90da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2c4108260 0x7fd2c419fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:53.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c90da700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c410e0c0 con 0x7fd2c4108260 2024-06-10T00:22:53.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c90da700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c410df40 con 0x7fd2c4105930 2024-06-10T00:22:53.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c90da700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c410e240 con 0x7fd2c4103000 2024-06-10T00:22:53.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2c4108260 0x7fd2c419fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:53.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2c4108260 0x7fd2c419fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59710/0 (socket says 172.21.15.121:59710) 2024-06-10T00:22:53.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c359e700 1 -- 172.21.15.121:0/1815452522 learned_addr learned my addr 172.21.15.121:0/1815452522 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:22:53.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c2d9d700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2c4103000 0x7fd2c4198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:53.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.516+0000 7fd2c259c700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2c4105930 0x7fd2c4198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:53.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2c359e700 1 -- 172.21.15.121:0/1815452522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2c4103000 msgr2=0x7fd2c4198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:53.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2c359e700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2c4103000 0x7fd2c4198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:53.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2c359e700 1 -- 172.21.15.121:0/1815452522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2c4105930 msgr2=0x7fd2c4198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:53.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2c359e700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2c4105930 0x7fd2c4198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:53.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2c359e700 1 -- 172.21.15.121:0/1815452522 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c41a0360 con 0x7fd2c4108260 2024-06-10T00:22:53.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2c2d9d700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2c4103000 0x7fd2c4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:22:53.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2c359e700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2c4108260 0x7fd2c419fcc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400e970 tx=0x7fd2b400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:53.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2bbfff700 1 -- 172.21.15.121:0/1815452522 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4004a30 con 0x7fd2c4108260 2024-06-10T00:22:53.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2bbfff700 1 -- 172.21.15.121:0/1815452522 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2b4005500 con 0x7fd2c4108260 2024-06-10T00:22:53.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2c90da700 1 -- 172.21.15.121:0/1815452522 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c41a0650 con 0x7fd2c4108260 2024-06-10T00:22:53.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2bbfff700 1 -- 172.21.15.121:0/1815452522 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4010d30 con 0x7fd2c4108260 2024-06-10T00:22:53.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.517+0000 7fd2c90da700 1 -- 172.21.15.121:0/1815452522 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c41a0b40 con 0x7fd2c4108260 2024-06-10T00:22:53.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.518+0000 7fd2c90da700 1 -- 172.21.15.121:0/1815452522 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c4069180 con 0x7fd2c4108260 2024-06-10T00:22:53.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.519+0000 7fd2bbfff700 1 -- 172.21.15.121:0/1815452522 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd2b4005020 con 0x7fd2c4108260 2024-06-10T00:22:53.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.519+0000 7fd2bbfff700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2b00a2c20 0x7fd2b00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:53.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.520+0000 7fd2c2d9d700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2b00a2c20 0x7fd2b00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:53.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.520+0000 7fd2bbfff700 1 -- 172.21.15.121:0/1815452522 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd2b40ceb90 con 0x7fd2c4108260 2024-06-10T00:22:53.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.520+0000 7fd2c2d9d700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2b00a2c20 0x7fd2b00a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009550 tx=0x7fd2bc0091f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:53.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.524+0000 7fd2bbfff700 1 -- 172.21.15.121:0/1815452522 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd2b4016330 con 0x7fd2c4108260 2024-06-10T00:22:53.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.720+0000 7fd2c90da700 1 -- 172.21.15.121:0/1815452522 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fd2c402dca0 con 0x7fd2c4108260 2024-06-10T00:22:53.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.721+0000 7fd2bbfff700 1 -- 172.21.15.121:0/1815452522 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7fd2b4084e20 con 0x7fd2c4108260 2024-06-10T00:22:53.720 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:22:53.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 -- 172.21.15.121:0/1815452522 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2b00a2c20 msgr2=0x7fd2b00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:53.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2b00a2c20 0x7fd2b00a50e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc009550 tx=0x7fd2bc0091f0 comp rx=0 tx=0).stop 2024-06-10T00:22:53.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 -- 172.21.15.121:0/1815452522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2c4108260 msgr2=0x7fd2c419fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:53.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2c4108260 0x7fd2c419fcc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd2b400e970 tx=0x7fd2b400ec80 comp rx=0 tx=0).stop 2024-06-10T00:22:53.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 -- 172.21.15.121:0/1815452522 shutdown_connections 2024-06-10T00:22:53.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2b00a2c20 0x7fd2b00a50e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:53.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2c4108260 0x7fd2c419fcc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:53.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2c4103000 0x7fd2c4198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:53.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 --2- 172.21.15.121:0/1815452522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2c4105930 0x7fd2c4198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:53.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 -- 172.21.15.121:0/1815452522 >> 172.21.15.121:0/1815452522 conn(0x7fd2c40fc990 msgr2=0x7fd2c4103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:53.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 -- 172.21.15.121:0/1815452522 shutdown_connections 2024-06-10T00:22:53.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:53.723+0000 7fd2c90da700 1 -- 172.21.15.121:0/1815452522 wait complete. 2024-06-10T00:22:54.375 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:54 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1815452522' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-10T00:22:54.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:54 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1815452522' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-10T00:22:54.401 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1717978970,"num_in_osds":12,"osd_in_since":1717978949,"num_remapped_pgs":0} 2024-06-10T00:22:54.401 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:22:54.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:54 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1815452522' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-10T00:22:54.639 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:22:55.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:55 smithi121 ceph-mon[32378]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:22:55.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:55 smithi135 ceph-mon[36424]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:22:55.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:55 smithi193 ceph-mon[36716]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:22:56.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.218+0000 7f15a7a41700 1 -- 172.21.15.121:0/84928976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a006a170 msgr2=0x7f15a006a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:56.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.218+0000 7f15a7a41700 1 --2- 172.21.15.121:0/84928976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a006a170 0x7f15a006a5f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1590009990 tx=0x7f1590009ca0 comp rx=0 tx=0).stop 2024-06-10T00:22:56.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.218+0000 7f15a7a41700 1 -- 172.21.15.121:0/84928976 shutdown_connections 2024-06-10T00:22:56.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.218+0000 7f15a7a41700 1 --2- 172.21.15.121:0/84928976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a006ab30 0x7f15a010f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:56.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.218+0000 7f15a7a41700 1 --2- 172.21.15.121:0/84928976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a0108500 0x7f15a01088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:56.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.218+0000 7f15a7a41700 1 --2- 172.21.15.121:0/84928976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a006a170 0x7f15a006a5f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:56.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.218+0000 7f15a7a41700 1 -- 172.21.15.121:0/84928976 >> 172.21.15.121:0/84928976 conn(0x7f15a0078340 msgr2=0x7f15a0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:56.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.218+0000 7f15a7a41700 1 -- 172.21.15.121:0/84928976 shutdown_connections 2024-06-10T00:22:56.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.218+0000 7f15a7a41700 1 -- 172.21.15.121:0/84928976 wait complete. 2024-06-10T00:22:56.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a7a41700 1 Processor -- start 2024-06-10T00:22:56.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a7a41700 1 -- start start 2024-06-10T00:22:56.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a7a41700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a006a170 0x7f15a019aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:56.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a7a41700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a006ab30 0x7f15a019b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:56.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a7a41700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a0108500 0x7f15a019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:56.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a7a41700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f15a01127e0 con 0x7f15a0108500 2024-06-10T00:22:56.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a7a41700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f15a0112660 con 0x7f15a006ab30 2024-06-10T00:22:56.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a7a41700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f15a0112960 con 0x7f15a006a170 2024-06-10T00:22:56.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a5fde700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a0108500 0x7f15a019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:56.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a5fde700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a0108500 0x7f15a019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59728/0 (socket says 172.21.15.121:59728) 2024-06-10T00:22:56.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a5fde700 1 -- 172.21.15.121:0/3261101938 learned_addr learned my addr 172.21.15.121:0/3261101938 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:22:56.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a4fdc700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a006ab30 0x7f15a019b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:56.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a57dd700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a006a170 0x7f15a019aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:56.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.219+0000 7f15a5fde700 1 -- 172.21.15.121:0/3261101938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a006a170 msgr2=0x7f15a019aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:56.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.220+0000 7f15a5fde700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a006a170 0x7f15a019aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:56.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.220+0000 7f15a5fde700 1 -- 172.21.15.121:0/3261101938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a006ab30 msgr2=0x7f15a019b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:56.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.220+0000 7f15a5fde700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a006ab30 0x7f15a019b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:56.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.220+0000 7f15a5fde700 1 -- 172.21.15.121:0/3261101938 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a019fed0 con 0x7f15a0108500 2024-06-10T00:22:56.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.220+0000 7f15a5fde700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a0108500 0x7f15a019f790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f159800b550 tx=0x7f159800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:56.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.220+0000 7f15967fc700 1 -- 172.21.15.121:0/3261101938 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598015020 con 0x7f15a0108500 2024-06-10T00:22:56.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.220+0000 7f15967fc700 1 -- 172.21.15.121:0/3261101938 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1598011d10 con 0x7f15a0108500 2024-06-10T00:22:56.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.220+0000 7f15a7a41700 1 -- 172.21.15.121:0/3261101938 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a01a01c0 con 0x7f15a0108500 2024-06-10T00:22:56.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.220+0000 7f15967fc700 1 -- 172.21.15.121:0/3261101938 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159800f2e0 con 0x7f15a0108500 2024-06-10T00:22:56.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.220+0000 7f15a7a41700 1 -- 172.21.15.121:0/3261101938 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a01a7a40 con 0x7f15a0108500 2024-06-10T00:22:56.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.221+0000 7f15a7a41700 1 -- 172.21.15.121:0/3261101938 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15a0069180 con 0x7f15a0108500 2024-06-10T00:22:56.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.222+0000 7f15967fc700 1 -- 172.21.15.121:0/3261101938 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f1598018020 con 0x7f15a0108500 2024-06-10T00:22:56.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.223+0000 7f15967fc700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f158c0a2c20 0x7f158c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:56.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.223+0000 7f15a57dd700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f158c0a2c20 0x7f158c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:56.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.223+0000 7f15967fc700 1 -- 172.21.15.121:0/3261101938 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f15980ce2e0 con 0x7f15a0108500 2024-06-10T00:22:56.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.223+0000 7f15a57dd700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f158c0a2c20 0x7f158c0a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f159c0060b0 tx=0x7f159c007b80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:56.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.227+0000 7f15967fc700 1 -- 172.21.15.121:0/3261101938 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f159801f020 con 0x7f15a0108500 2024-06-10T00:22:56.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.425+0000 7f15a7a41700 1 -- 172.21.15.121:0/3261101938 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15a01a7c70 con 0x7f15a0108500 2024-06-10T00:22:56.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.426+0000 7f15967fc700 1 -- 172.21.15.121:0/3261101938 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18524 (secure 0 0 0) 0x7f159809b6e0 con 0x7f15a0108500 2024-06-10T00:22:56.426 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:22:56.426 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":82,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:22:51.153317+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:22:56.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 -- 172.21.15.121:0/3261101938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f158c0a2c20 msgr2=0x7f158c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:56.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f158c0a2c20 0x7f158c0a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f159c0060b0 tx=0x7f159c007b80 comp rx=0 tx=0).stop 2024-06-10T00:22:56.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 -- 172.21.15.121:0/3261101938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a0108500 msgr2=0x7f15a019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:56.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a0108500 0x7f15a019f790 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f159800b550 tx=0x7f159800b860 comp rx=0 tx=0).stop 2024-06-10T00:22:56.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 -- 172.21.15.121:0/3261101938 shutdown_connections 2024-06-10T00:22:56.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f158c0a2c20 0x7f158c0a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:56.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a0108500 0x7f15a019f790 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:56.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a006a170 0x7f15a019aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:56.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 --2- 172.21.15.121:0/3261101938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a006ab30 0x7f15a019b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:56.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 -- 172.21.15.121:0/3261101938 >> 172.21.15.121:0/3261101938 conn(0x7f15a0078340 msgr2=0x7f15a010d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:56.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 -- 172.21.15.121:0/3261101938 shutdown_connections 2024-06-10T00:22:56.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:56.428+0000 7f15a7a41700 1 -- 172.21.15.121:0/3261101938 wait complete. 2024-06-10T00:22:57.100 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-10T00:19:04.337527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-06-10T00:22:57.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-10T00:22:57.335 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:22:57.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:57 smithi121 ceph-mon[32378]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:22:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:57 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3261101938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:22:57.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:57 smithi135 ceph-mon[36424]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:22:57.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:57 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3261101938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:22:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:57 smithi193 ceph-mon[36716]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:22:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:57 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3261101938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:22:58.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.997+0000 7fc1543bd700 1 -- 172.21.15.121:0/1089583662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc14c102550 msgr2=0x7fc14c1029d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:59.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.997+0000 7fc1543bd700 1 --2- 172.21.15.121:0/1089583662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc14c102550 0x7fc14c1029d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009990 tx=0x7fc13c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:22:59.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.997+0000 7fc1543bd700 1 -- 172.21.15.121:0/1089583662 shutdown_connections 2024-06-10T00:22:59.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.997+0000 7fc1543bd700 1 --2- 172.21.15.121:0/1089583662 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc14c102f10 0x7fc14c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:59.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.997+0000 7fc1543bd700 1 --2- 172.21.15.121:0/1089583662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc14c1084f0 0x7fc14c1088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:59.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.997+0000 7fc1543bd700 1 --2- 172.21.15.121:0/1089583662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc14c102550 0x7fc14c1029d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:59.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.997+0000 7fc1543bd700 1 -- 172.21.15.121:0/1089583662 >> 172.21.15.121:0/1089583662 conn(0x7fc14c079650 msgr2=0x7fc14c079a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:59.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.998+0000 7fc1543bd700 1 -- 172.21.15.121:0/1089583662 shutdown_connections 2024-06-10T00:22:59.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.998+0000 7fc1543bd700 1 -- 172.21.15.121:0/1089583662 wait complete. 2024-06-10T00:22:59.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.998+0000 7fc1543bd700 1 Processor -- start 2024-06-10T00:22:59.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.998+0000 7fc1543bd700 1 -- start start 2024-06-10T00:22:59.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.998+0000 7fc1543bd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc14c102550 0x7fc14c19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.998+0000 7fc1543bd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc14c102f10 0x7fc14c19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.998+0000 7fc1543bd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc14c1084f0 0x7fc14c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.998+0000 7fc1543bd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc14c112760 con 0x7fc14c102f10 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.998+0000 7fc1543bd700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc14c1125e0 con 0x7fc14c102550 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.998+0000 7fc1543bd700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc14c1128e0 con 0x7fc14c1084f0 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.998+0000 7fc151958700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc14c102f10 0x7fc14c19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc151958700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc14c102f10 0x7fc14c19b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35028/0 (socket says 172.21.15.121:35028) 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc151958700 1 -- 172.21.15.121:0/967059671 learned_addr learned my addr 172.21.15.121:0/967059671 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc15295a700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc14c1084f0 0x7fc14c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc152159700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc14c102550 0x7fc14c19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc151958700 1 -- 172.21.15.121:0/967059671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc14c1084f0 msgr2=0x7fc14c19f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:59.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc151958700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc14c1084f0 0x7fc14c19f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc151958700 1 -- 172.21.15.121:0/967059671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc14c102550 msgr2=0x7fc14c19ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc151958700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc14c102550 0x7fc14c19ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc151958700 1 -- 172.21.15.121:0/967059671 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14c19fdb0 con 0x7fc14c102f10 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc152159700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc14c102550 0x7fc14c19ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc15295a700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc14c1084f0 0x7fc14c19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc151958700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc14c102f10 0x7fc14c19b2e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc13c000c00 tx=0x7fc13c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc1437fe700 1 -- 172.21.15.121:0/967059671 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c027070 con 0x7fc14c102f10 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc1543bd700 1 -- 172.21.15.121:0/967059671 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc14c1a0040 con 0x7fc14c102f10 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc1437fe700 1 -- 172.21.15.121:0/967059671 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc13c004050 con 0x7fc14c102f10 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc1437fe700 1 -- 172.21.15.121:0/967059671 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c00f460 con 0x7fc14c102f10 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:58.999+0000 7fc1543bd700 1 -- 172.21.15.121:0/967059671 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc14c1a7920 con 0x7fc14c102f10 2024-06-10T00:22:59.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.000+0000 7fc1543bd700 1 -- 172.21.15.121:0/967059671 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc14c069180 con 0x7fc14c102f10 2024-06-10T00:22:59.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.001+0000 7fc1437fe700 1 -- 172.21.15.121:0/967059671 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc13c02a070 con 0x7fc14c102f10 2024-06-10T00:22:59.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.002+0000 7fc1437fe700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc1380a2c20 0x7fc1380a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:22:59.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.002+0000 7fc152159700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc1380a2c20 0x7fc1380a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:22:59.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.002+0000 7fc1437fe700 1 -- 172.21.15.121:0/967059671 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc13c0cfff0 con 0x7fc14c102f10 2024-06-10T00:22:59.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.003+0000 7fc152159700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc1380a2c20 0x7fc1380a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc148009b50 tx=0x7fc148009550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:22:59.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.006+0000 7fc1437fe700 1 -- 172.21.15.121:0/967059671 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc13c031070 con 0x7fc14c102f10 2024-06-10T00:22:59.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.203+0000 7fc1543bd700 1 -- 172.21.15.121:0/967059671 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc14c1a7af0 con 0x7fc14c102f10 2024-06-10T00:22:59.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.203+0000 7fc1437fe700 1 -- 172.21.15.121:0/967059671 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fc13c013070 con 0x7fc14c102f10 2024-06-10T00:22:59.203 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-06-10T00:22:59.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 -- 172.21.15.121:0/967059671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc1380a2c20 msgr2=0x7fc1380a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:59.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc1380a2c20 0x7fc1380a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc148009b50 tx=0x7fc148009550 comp rx=0 tx=0).stop 2024-06-10T00:22:59.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 -- 172.21.15.121:0/967059671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc14c102f10 msgr2=0x7fc14c19b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:22:59.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc14c102f10 0x7fc14c19b2e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc13c000c00 tx=0x7fc13c0047b0 comp rx=0 tx=0).stop 2024-06-10T00:22:59.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 -- 172.21.15.121:0/967059671 shutdown_connections 2024-06-10T00:22:59.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc1380a2c20 0x7fc1380a50e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:59.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc14c1084f0 0x7fc14c19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:59.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc14c102550 0x7fc14c19ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:59.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 --2- 172.21.15.121:0/967059671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc14c102f10 0x7fc14c19b2e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:22:59.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 -- 172.21.15.121:0/967059671 >> 172.21.15.121:0/967059671 conn(0x7fc14c079650 msgr2=0x7fc14c1003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:22:59.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 -- 172.21.15.121:0/967059671 shutdown_connections 2024-06-10T00:22:59.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:22:59.205+0000 7fc1543bd700 1 -- 172.21.15.121:0/967059671 wait complete. 2024-06-10T00:22:59.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:22:59 smithi121 ceph-mon[32378]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:22:59.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:22:59 smithi135 ceph-mon[36424]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:22:59.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:22:59 smithi193 ceph-mon[36716]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:00.018 INFO:tasks.cephadm:Adding mds.a on smithi121 2024-06-10T00:23:00.018 INFO:tasks.cephadm:Adding mds.d on smithi121 2024-06-10T00:23:00.018 INFO:tasks.cephadm:Adding mds.g on smithi121 2024-06-10T00:23:00.018 INFO:tasks.cephadm:Adding mds.j on smithi121 2024-06-10T00:23:00.019 INFO:tasks.cephadm:Adding mds.b on smithi135 2024-06-10T00:23:00.019 INFO:tasks.cephadm:Adding mds.e on smithi135 2024-06-10T00:23:00.019 INFO:tasks.cephadm:Adding mds.h on smithi135 2024-06-10T00:23:00.019 INFO:tasks.cephadm:Adding mds.k on smithi135 2024-06-10T00:23:00.019 INFO:tasks.cephadm:Adding mds.c on smithi193 2024-06-10T00:23:00.019 INFO:tasks.cephadm:Adding mds.f on smithi193 2024-06-10T00:23:00.019 INFO:tasks.cephadm:Adding mds.i on smithi193 2024-06-10T00:23:00.019 INFO:tasks.cephadm:Adding mds.l on smithi193 2024-06-10T00:23:00.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph orch apply mds all '12;smithi121=a;smithi121=d;smithi121=g;smithi121=j;smithi135=b;smithi135=e;smithi135=h;smithi135=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l' 2024-06-10T00:23:00.271 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:23:00.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:00 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/967059671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-10T00:23:00.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:00 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/967059671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-10T00:23:00.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:00 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/967059671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-10T00:23:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:01 smithi121 ceph-mon[32378]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:01.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:01 smithi135 ceph-mon[36424]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:01.531 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:01 smithi193 ceph-mon[36716]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:01.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.868+0000 7f76d695e700 1 -- 172.21.15.193:0/2007982755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d01025f0 msgr2=0x7f76d0102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:01.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.868+0000 7f76d695e700 1 --2- 172.21.15.193:0/2007982755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d01025f0 0x7f76d0102a50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009990 tx=0x7f76c4009ca0 comp rx=0 tx=0).stop 2024-06-10T00:23:01.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.868+0000 7f76d695e700 1 -- 172.21.15.193:0/2007982755 shutdown_connections 2024-06-10T00:23:01.870 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.868+0000 7f76d695e700 1 --2- 172.21.15.193:0/2007982755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76d0102f90 0x7f76d0109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.868+0000 7f76d695e700 1 --2- 172.21.15.193:0/2007982755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76d01013f0 0x7f76d0101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.868+0000 7f76d695e700 1 --2- 172.21.15.193:0/2007982755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d01025f0 0x7f76d0102a50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.868+0000 7f76d695e700 1 -- 172.21.15.193:0/2007982755 >> 172.21.15.193:0/2007982755 conn(0x7f76d00fc990 msgr2=0x7f76d00fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.868+0000 7f76d695e700 1 -- 172.21.15.193:0/2007982755 shutdown_connections 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.868+0000 7f76d695e700 1 -- 172.21.15.193:0/2007982755 wait complete. 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.868+0000 7f76d695e700 1 Processor -- start 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d695e700 1 -- start start 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d695e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76d01013f0 0x7f76d0198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d695e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76d01025f0 0x7f76d0198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d695e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d0102f90 0x7f76d019fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d695e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f76d010d080 con 0x7f76d01013f0 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d695e700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f76d010cf00 con 0x7f76d01025f0 2024-06-10T00:23:01.871 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d695e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76d010d200 con 0x7f76d0102f90 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d4efb700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d0102f90 0x7f76d019fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d4efb700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d0102f90 0x7f76d019fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:37738/0 (socket says 172.21.15.193:37738) 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d4efb700 1 -- 172.21.15.193:0/2667576114 learned_addr learned my addr 172.21.15.193:0/2667576114 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76cffff700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76d01013f0 0x7f76d0198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76cf7fe700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76d01025f0 0x7f76d0198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d4efb700 1 -- 172.21.15.193:0/2667576114 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76d01025f0 msgr2=0x7f76d0198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d4efb700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76d01025f0 0x7f76d0198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d4efb700 1 -- 172.21.15.193:0/2667576114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76d01013f0 msgr2=0x7f76d0198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d4efb700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76d01013f0 0x7f76d0198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76d4efb700 1 -- 172.21.15.193:0/2667576114 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d01a0450 con 0x7f76d0102f90 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.869+0000 7f76cf7fe700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76d01025f0 0x7f76d0198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.870+0000 7f76d4efb700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d0102f90 0x7f76d019fdb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76c800b9a0 tx=0x7f76c800bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.870+0000 7f76cd7fa700 1 -- 172.21.15.193:0/2667576114 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8019070 con 0x7f76d0102f90 2024-06-10T00:23:01.872 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.870+0000 7f76cd7fa700 1 -- 172.21.15.193:0/2667576114 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c800ce10 con 0x7f76d0102f90 2024-06-10T00:23:01.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.870+0000 7f76d695e700 1 -- 172.21.15.193:0/2667576114 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d01a0740 con 0x7f76d0102f90 2024-06-10T00:23:01.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.870+0000 7f76cd7fa700 1 -- 172.21.15.193:0/2667576114 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c80073d0 con 0x7f76d0102f90 2024-06-10T00:23:01.873 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.870+0000 7f76d695e700 1 -- 172.21.15.193:0/2667576114 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d01a0d50 con 0x7f76d0102f90 2024-06-10T00:23:01.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.871+0000 7f76d695e700 1 -- 172.21.15.193:0/2667576114 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d0069180 con 0x7f76d0102f90 2024-06-10T00:23:01.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.872+0000 7f76cd7fa700 1 -- 172.21.15.193:0/2667576114 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f76c801c070 con 0x7f76d0102f90 2024-06-10T00:23:01.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.873+0000 7f76cd7fa700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f76b80a2ca0 0x7f76b80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:01.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.873+0000 7f76cd7fa700 1 -- 172.21.15.193:0/2667576114 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f76c80ce330 con 0x7f76d0102f90 2024-06-10T00:23:01.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.873+0000 7f76cffff700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f76b80a2ca0 0x7f76b80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:01.874 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.874+0000 7f76cffff700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f76b80a2ca0 0x7f76b80a5160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f76d0102450 tx=0x7f76c000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:01.876 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:01.877+0000 7f76cd7fa700 1 -- 172.21.15.193:0/2667576114 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f76c809b730 con 0x7f76d0102f90 2024-06-10T00:23:02.082 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.081+0000 7f76d695e700 1 -- 172.21.15.193:0/2667576114 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi121=a;smithi121=d;smithi121=g;smithi121=j;smithi135=b;smithi135=e;smithi135=h;smithi135=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}) -- 0x7f76d00febb0 con 0x7f76b80a2ca0 2024-06-10T00:23:02.089 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.090+0000 7f76cd7fa700 1 -- 172.21.15.193:0/2667576114 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f76d00febb0 con 0x7f76b80a2ca0 2024-06-10T00:23:02.089 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mds.all update... 2024-06-10T00:23:02.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 -- 172.21.15.193:0/2667576114 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f76b80a2ca0 msgr2=0x7f76b80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:02.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f76b80a2ca0 0x7f76b80a5160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f76d0102450 tx=0x7f76c000a380 comp rx=0 tx=0).stop 2024-06-10T00:23:02.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 -- 172.21.15.193:0/2667576114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d0102f90 msgr2=0x7f76d019fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:02.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d0102f90 0x7f76d019fdb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76c800b9a0 tx=0x7f76c800bcb0 comp rx=0 tx=0).stop 2024-06-10T00:23:02.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 -- 172.21.15.193:0/2667576114 shutdown_connections 2024-06-10T00:23:02.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f76b80a2ca0 0x7f76b80a5160 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:02.092 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76d0102f90 0x7f76d019fdb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:02.093 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76d01013f0 0x7f76d0198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:02.093 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 --2- 172.21.15.193:0/2667576114 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76d01025f0 0x7f76d0198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:02.093 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 -- 172.21.15.193:0/2667576114 >> 172.21.15.193:0/2667576114 conn(0x7f76d00fc990 msgr2=0x7f76d00fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:02.093 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 -- 172.21.15.193:0/2667576114 shutdown_connections 2024-06-10T00:23:02.093 INFO:teuthology.orchestra.run.smithi193.stderr:2024-06-10T00:23:02.092+0000 7f76d695e700 1 -- 172.21.15.193:0/2667576114 wait complete. 2024-06-10T00:23:02.915 DEBUG:teuthology.orchestra.run.smithi121:mds.a> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.a.service 2024-06-10T00:23:02.918 DEBUG:teuthology.orchestra.run.smithi121:mds.d> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.d.service 2024-06-10T00:23:02.960 DEBUG:teuthology.orchestra.run.smithi121:mds.g> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.g.service 2024-06-10T00:23:03.004 DEBUG:teuthology.orchestra.run.smithi121:mds.j> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.j.service 2024-06-10T00:23:03.046 DEBUG:teuthology.orchestra.run.smithi135:mds.b> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.b.service 2024-06-10T00:23:03.049 DEBUG:teuthology.orchestra.run.smithi135:mds.e> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.e.service 2024-06-10T00:23:03.092 DEBUG:teuthology.orchestra.run.smithi135:mds.h> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.h.service 2024-06-10T00:23:03.135 DEBUG:teuthology.orchestra.run.smithi135:mds.k> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.k.service 2024-06-10T00:23:03.178 DEBUG:teuthology.orchestra.run.smithi193:mds.c> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.c.service 2024-06-10T00:23:03.184 DEBUG:teuthology.orchestra.run.smithi193:mds.f> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.f.service 2024-06-10T00:23:03.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:03 smithi121 ceph-mon[32378]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:03.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:03 smithi121 ceph-mon[32378]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi121=a;smithi121=d;smithi121=g;smithi121=j;smithi135=b;smithi135=e;smithi135=h;smithi135=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:23:03.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:03 smithi121 ceph-mon[32378]: Saving service mds.all spec with placement smithi121=a;smithi121=d;smithi121=g;smithi121=j;smithi135=b;smithi135=e;smithi135=h;smithi135=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-06-10T00:23:03.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:03.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:23:03.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:03.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:23:03.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:03.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:03.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:03.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:03 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:03.203 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:03 smithi193 ceph-mon[36716]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:03.203 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:03 smithi193 ceph-mon[36716]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi121=a;smithi121=d;smithi121=g;smithi121=j;smithi135=b;smithi135=e;smithi135=h;smithi135=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:23:03.204 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:03 smithi193 ceph-mon[36716]: Saving service mds.all spec with placement smithi121=a;smithi121=d;smithi121=g;smithi121=j;smithi135=b;smithi135=e;smithi135=h;smithi135=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-06-10T00:23:03.204 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:03.204 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:23:03.204 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:03.204 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:23:03.204 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:03.204 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:03.204 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:03.204 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:03 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:03.205 DEBUG:teuthology.orchestra.run.smithi193:mds.i> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.i.service 2024-06-10T00:23:03.247 DEBUG:teuthology.orchestra.run.smithi193:mds.l> sudo journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.l.service 2024-06-10T00:23:03.290 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-06-10T00:23:03.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:23:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:03 smithi135 ceph-mon[36424]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:03 smithi135 ceph-mon[36424]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi121=a;smithi121=d;smithi121=g;smithi121=j;smithi135=b;smithi135=e;smithi135=h;smithi135=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:23:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:03 smithi135 ceph-mon[36424]: Saving service mds.all spec with placement smithi121=a;smithi121=d;smithi121=g;smithi121=j;smithi135=b;smithi135=e;smithi135=h;smithi135=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-06-10T00:23:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:23:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:23:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:03 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:03.527 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:04.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:04 smithi193 ceph-mon[36716]: Deploying daemon mds.l on smithi193 2024-06-10T00:23:04.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:04 smithi121 ceph-mon[32378]: Deploying daemon mds.l on smithi193 2024-06-10T00:23:04.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:04 smithi135 ceph-mon[36424]: Deploying daemon mds.l on smithi193 2024-06-10T00:23:05.038 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:23:04 smithi193 systemd[1]: Starting Ceph mds.l for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:23:05.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.222+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3703744091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a8101680 msgr2=0x7fd2a8101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:05.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.222+0000 7fd2ae5d2700 1 --2- 172.21.15.121:0/3703744091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a8101680 0x7fd2a8101a60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd2980099c0 tx=0x7fd298009cd0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.222+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3703744091 shutdown_connections 2024-06-10T00:23:05.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.222+0000 7fd2ae5d2700 1 --2- 172.21.15.121:0/3703744091 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2a81066d0 0x7fd2a810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.222+0000 7fd2ae5d2700 1 --2- 172.21.15.121:0/3703744091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a8101680 0x7fd2a8101a60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.222+0000 7fd2ae5d2700 1 --2- 172.21.15.121:0/3703744091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2a8102030 0x7fd2a8105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.222+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3703744091 >> 172.21.15.121:0/3703744091 conn(0x7fd2a80773e0 msgr2=0x7fd2a80777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:05.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.222+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3703744091 shutdown_connections 2024-06-10T00:23:05.223 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:05 smithi121 ceph-mon[32378]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.222+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3703744091 wait complete. 2024-06-10T00:23:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.222+0000 7fd2ae5d2700 1 Processor -- start 2024-06-10T00:23:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.222+0000 7fd2ae5d2700 1 -- start start 2024-06-10T00:23:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2ae5d2700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2a8101680 0x7fd2a8199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2ae5d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a8102030 0x7fd2a8199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2ae5d2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2a81066d0 0x7fd2a81a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:05.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2ae5d2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a81116d0 con 0x7fd2a8102030 2024-06-10T00:23:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2ae5d2700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a8111550 con 0x7fd2a81066d0 2024-06-10T00:23:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2ae5d2700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a8111850 con 0x7fd2a8101680 2024-06-10T00:23:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2a77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a8102030 0x7fd2a8199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2a77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a8102030 0x7fd2a8199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35044/0 (socket says 172.21.15.121:35044) 2024-06-10T00:23:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2a77fe700 1 -- 172.21.15.121:0/3499391083 learned_addr learned my addr 172.21.15.121:0/3499391083 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2acb6f700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2a81066d0 0x7fd2a81a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2a77fe700 1 -- 172.21.15.121:0/3499391083 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2a8101680 msgr2=0x7fd2a8199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2a77fe700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2a8101680 0x7fd2a8199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2a77fe700 1 -- 172.21.15.121:0/3499391083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2a81066d0 msgr2=0x7fd2a81a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2a77fe700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2a81066d0 0x7fd2a81a4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.223+0000 7fd2a77fe700 1 -- 172.21.15.121:0/3499391083 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a81a5050 con 0x7fd2a8102030 2024-06-10T00:23:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.224+0000 7fd2a77fe700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a8102030 0x7fd2a8199fd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00e9f0 tx=0x7fd29c00edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.224+0000 7fd2a57fa700 1 -- 172.21.15.121:0/3499391083 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c00cbd0 con 0x7fd2a8102030 2024-06-10T00:23:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.224+0000 7fd2a57fa700 1 -- 172.21.15.121:0/3499391083 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd29c00cd70 con 0x7fd2a8102030 2024-06-10T00:23:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.224+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3499391083 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a81a5220 con 0x7fd2a8102030 2024-06-10T00:23:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.224+0000 7fd2a57fa700 1 -- 172.21.15.121:0/3499391083 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c010430 con 0x7fd2a8102030 2024-06-10T00:23:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.224+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3499391083 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a81a56e0 con 0x7fd2a8102030 2024-06-10T00:23:05.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.225+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3499391083 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a8102ad0 con 0x7fd2a8102030 2024-06-10T00:23:05.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.226+0000 7fd2a57fa700 1 -- 172.21.15.121:0/3499391083 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd29c004750 con 0x7fd2a8102030 2024-06-10T00:23:05.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.226+0000 7fd2a57fa700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2900a2c20 0x7fd2900a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:05.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.227+0000 7fd2a7fff700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2900a2c20 0x7fd2900a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:05.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.227+0000 7fd2a57fa700 1 -- 172.21.15.121:0/3499391083 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd29c0cef40 con 0x7fd2a8102030 2024-06-10T00:23:05.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.227+0000 7fd2a7fff700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2900a2c20 0x7fd2900a50e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd2980098e0 tx=0x7fd298000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:05.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.231+0000 7fd2a57fa700 1 -- 172.21.15.121:0/3499391083 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd29c09c340 con 0x7fd2a8102030 2024-06-10T00:23:05.310 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:05 smithi193 ceph-mon[36716]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:05.311 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:23:05 smithi193 podman[66278]: 2024-06-10 00:23:04.941421475 +0000 UTC m=+0.017717598 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:23:05.311 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:23:05 smithi193 podman[66278]: 2024-06-10 00:23:05.057917171 +0000 UTC m=+0.134213343 container create 3fe4e86aa09ffa3484cdf78deb1cdd4274de00adfb28e1b4671a19f893a87578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-l, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-06-10T00:23:05.311 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:23:05 smithi193 podman[66278]: 2024-06-10 00:23:05.311453628 +0000 UTC m=+0.387749752 container init 3fe4e86aa09ffa3484cdf78deb1cdd4274de00adfb28e1b4671a19f893a87578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-l, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-10T00:23:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.473+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3499391083 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2a806b1a0 con 0x7fd2a8102030 2024-06-10T00:23:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.474+0000 7fd2a57fa700 1 -- 172.21.15.121:0/3499391083 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7fd29c09bc40 con 0x7fd2a8102030 2024-06-10T00:23:05.474 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:23:05.474 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-06-10T00:23:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3499391083 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2900a2c20 msgr2=0x7fd2900a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2900a2c20 0x7fd2900a50e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd2980098e0 tx=0x7fd298000bc0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3499391083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a8102030 msgr2=0x7fd2a8199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a8102030 0x7fd2a8199fd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00e9f0 tx=0x7fd29c00edb0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3499391083 shutdown_connections 2024-06-10T00:23:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2900a2c20 0x7fd2900a50e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2a81066d0 0x7fd2a81a4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2a8101680 0x7fd2a8199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 --2- 172.21.15.121:0/3499391083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2a8102030 0x7fd2a8199fd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3499391083 >> 172.21.15.121:0/3499391083 conn(0x7fd2a80773e0 msgr2=0x7fd2a81005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3499391083 shutdown_connections 2024-06-10T00:23:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:05.476+0000 7fd2ae5d2700 1 -- 172.21.15.121:0/3499391083 wait complete. 2024-06-10T00:23:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 2 2024-06-10T00:23:05.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:05 smithi135 ceph-mon[36424]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:05.630 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:23:05 smithi193 podman[66278]: 2024-06-10 00:23:05.314177884 +0000 UTC m=+0.390474010 container start 3fe4e86aa09ffa3484cdf78deb1cdd4274de00adfb28e1b4671a19f893a87578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True) 2024-06-10T00:23:05.631 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:23:05 smithi193 ceph-mds[66297]: -- 172.21.15.193:0/1194250670 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc095ada40 con 0x55cc09581000 2024-06-10T00:23:05.631 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:23:05 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-l[66293]: starting mds.l at 2024-06-10T00:23:05.631 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:23:05 smithi193 bash[66278]: 3fe4e86aa09ffa3484cdf78deb1cdd4274de00adfb28e1b4671a19f893a87578 2024-06-10T00:23:05.631 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:23:05 smithi193 systemd[1]: Started Ceph mds.l for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:06.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.193:6832/360772282,v1:172.21.15.193:6833/360772282] up:boot 2024-06-10T00:23:06.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: fsmap 1 up:standby 2024-06-10T00:23:06.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-10T00:23:06.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:06.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:06.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3499391083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:06.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:06.440 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:06.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:06.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:06.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: Deploying daemon mds.k on smithi135 2024-06-10T00:23:06.441 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:06 smithi135 ceph-mon[36424]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:06.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.193:6832/360772282,v1:172.21.15.193:6833/360772282] up:boot 2024-06-10T00:23:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: fsmap 1 up:standby 2024-06-10T00:23:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-10T00:23:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3499391083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: Deploying daemon mds.k on smithi135 2024-06-10T00:23:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:06 smithi193 ceph-mon[36716]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:06.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.193:6832/360772282,v1:172.21.15.193:6833/360772282] up:boot 2024-06-10T00:23:06.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: fsmap 1 up:standby 2024-06-10T00:23:06.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-10T00:23:06.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:06.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:06.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3499391083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:06.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:06.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:06.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:06.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:06.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: Deploying daemon mds.k on smithi135 2024-06-10T00:23:06.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:06 smithi121 ceph-mon[32378]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:07.758 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:23:07 smithi135 systemd[1]: Starting Ceph mds.k for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:23:08.181 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:23:07 smithi135 podman[65233]: 2024-06-10 00:23:07.871042864 +0000 UTC m=+0.100279777 container create e4647ca801fdac7fc2d6c2be74158c6254d6781ddc004cd2d925350676838a02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0) 2024-06-10T00:23:08.181 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:23:07 smithi135 podman[65233]: 2024-06-10 00:23:07.78764191 +0000 UTC m=+0.016878797 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:23:08.181 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:23:08 smithi135 podman[65233]: 2024-06-10 00:23:08.090108757 +0000 UTC m=+0.319345638 container init e4647ca801fdac7fc2d6c2be74158c6254d6781ddc004cd2d925350676838a02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-06-10T00:23:08.181 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:23:08 smithi135 podman[65233]: 2024-06-10 00:23:08.092733605 +0000 UTC m=+0.321970490 container start e4647ca801fdac7fc2d6c2be74158c6254d6781ddc004cd2d925350676838a02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-k, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528) 2024-06-10T00:23:08.181 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:23:08 smithi135 ceph-mds[65252]: -- 172.21.15.135:0/2269618796 <== mon.2 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55761ef03a40 con 0x55761eed7000 2024-06-10T00:23:08.182 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:23:08 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-k[65248]: starting mds.k at 2024-06-10T00:23:08.255 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:23:08.507 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:23:08 smithi135 bash[65233]: e4647ca801fdac7fc2d6c2be74158c6254d6781ddc004cd2d925350676838a02 2024-06-10T00:23:08.508 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:23:08 smithi135 systemd[1]: Started Ceph mds.k for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:08.522 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:09.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:09 smithi121 ceph-mon[32378]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:09.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:09 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.135:6832/1842796932,v1:172.21.15.135:6833/1842796932] up:boot 2024-06-10T00:23:09.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:09 smithi121 ceph-mon[32378]: fsmap 2 up:standby 2024-06-10T00:23:09.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-10T00:23:09.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:09.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:09.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:09.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:09.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:09.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:09 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:09.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:09 smithi135 ceph-mon[36424]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:09.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:09 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.135:6832/1842796932,v1:172.21.15.135:6833/1842796932] up:boot 2024-06-10T00:23:09.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:09 smithi135 ceph-mon[36424]: fsmap 2 up:standby 2024-06-10T00:23:09.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-10T00:23:09.506 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:09 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:09.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:09 smithi193 ceph-mon[36716]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:09 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.135:6832/1842796932,v1:172.21.15.135:6833/1842796932] up:boot 2024-06-10T00:23:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:09 smithi193 ceph-mon[36716]: fsmap 2 up:standby 2024-06-10T00:23:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-10T00:23:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:09 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:10.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:10 smithi121 ceph-mon[32378]: Deploying daemon mds.j on smithi121 2024-06-10T00:23:10.431 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:10 smithi135 ceph-mon[36424]: Deploying daemon mds.j on smithi121 2024-06-10T00:23:10.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:10 smithi193 ceph-mon[36716]: Deploying daemon mds.j on smithi121 2024-06-10T00:23:10.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.754+0000 7fa87ae42700 1 -- 172.21.15.121:0/3915945440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa874104e90 msgr2=0x7fa874105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:10.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.754+0000 7fa87ae42700 1 --2- 172.21.15.121:0/3915945440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa874104e90 0x7fa874105320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00b0a0 tx=0x7fa86c00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:23:10.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.754+0000 7fa87ae42700 1 -- 172.21.15.121:0/3915945440 shutdown_connections 2024-06-10T00:23:10.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.754+0000 7fa87ae42700 1 --2- 172.21.15.121:0/3915945440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa874104e90 0x7fa874105320 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:10.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.754+0000 7fa87ae42700 1 --2- 172.21.15.121:0/3915945440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa874107380 0x7fa87406a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:10.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.754+0000 7fa87ae42700 1 --2- 172.21.15.121:0/3915945440 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa87406a6b0 0x7fa87406ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:10.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.754+0000 7fa87ae42700 1 -- 172.21.15.121:0/3915945440 >> 172.21.15.121:0/3915945440 conn(0x7fa8740782c0 msgr2=0x7fa8740786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:10.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.754+0000 7fa87ae42700 1 -- 172.21.15.121:0/3915945440 shutdown_connections 2024-06-10T00:23:10.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.754+0000 7fa87ae42700 1 -- 172.21.15.121:0/3915945440 wait complete. 2024-06-10T00:23:10.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa87ae42700 1 Processor -- start 2024-06-10T00:23:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa87ae42700 1 -- start start 2024-06-10T00:23:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa87ae42700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa87406a6b0 0x7fa87419f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa87ae42700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa874104e90 0x7fa87419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa87ae42700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa874107380 0x7fa8741a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa87ae42700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa874118d10 con 0x7fa874104e90 2024-06-10T00:23:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa87ae42700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fa874118b90 con 0x7fa874107380 2024-06-10T00:23:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa87ae42700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa874118e90 con 0x7fa87406a6b0 2024-06-10T00:23:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa873fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa874104e90 0x7fa87419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa873fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa874104e90 0x7fa87419f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55174/0 (socket says 172.21.15.121:55174) 2024-06-10T00:23:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa873fff700 1 -- 172.21.15.121:0/3436261388 learned_addr learned my addr 172.21.15.121:0/3436261388 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:10.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa8793df700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa874107380 0x7fa8741a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:10.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa878bde700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa87406a6b0 0x7fa87419f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:10.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa873fff700 1 -- 172.21.15.121:0/3436261388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa87406a6b0 msgr2=0x7fa87419f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:10.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa873fff700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa87406a6b0 0x7fa87419f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:10.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa873fff700 1 -- 172.21.15.121:0/3436261388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa874107380 msgr2=0x7fa8741a3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:10.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa873fff700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa874107380 0x7fa8741a3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:10.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.755+0000 7fa873fff700 1 -- 172.21.15.121:0/3436261388 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8741a4200 con 0x7fa874104e90 2024-06-10T00:23:10.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.756+0000 7fa873fff700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa874104e90 0x7fa87419f780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa86000d950 tx=0x7fa86000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:10.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.756+0000 7fa871ffb700 1 -- 172.21.15.121:0/3436261388 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa860016070 con 0x7fa874104e90 2024-06-10T00:23:10.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.756+0000 7fa871ffb700 1 -- 172.21.15.121:0/3436261388 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa860009e20 con 0x7fa874104e90 2024-06-10T00:23:10.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.756+0000 7fa87ae42700 1 -- 172.21.15.121:0/3436261388 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8741a44f0 con 0x7fa874104e90 2024-06-10T00:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.756+0000 7fa871ffb700 1 -- 172.21.15.121:0/3436261388 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86000f620 con 0x7fa874104e90 2024-06-10T00:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.756+0000 7fa87ae42700 1 -- 172.21.15.121:0/3436261388 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8741abd70 con 0x7fa874104e90 2024-06-10T00:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.757+0000 7fa87ae42700 1 -- 172.21.15.121:0/3436261388 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa874069180 con 0x7fa874104e90 2024-06-10T00:23:10.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.758+0000 7fa871ffb700 1 -- 172.21.15.121:0/3436261388 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa860019070 con 0x7fa874104e90 2024-06-10T00:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.758+0000 7fa871ffb700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa8640a2c20 0x7fa8640a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.759+0000 7fa878bde700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa8640a2c20 0x7fa8640a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.759+0000 7fa871ffb700 1 -- 172.21.15.121:0/3436261388 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa8600ce770 con 0x7fa874104e90 2024-06-10T00:23:10.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.759+0000 7fa878bde700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa8640a2c20 0x7fa8640a50e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa868005950 tx=0x7fa8680058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:10.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:10.763+0000 7fa871ffb700 1 -- 172.21.15.121:0/3436261388 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa86009bb70 con 0x7fa874104e90 2024-06-10T00:23:11.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.007+0000 7fa87ae42700 1 -- 172.21.15.121:0/3436261388 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8741abf40 con 0x7fa874104e90 2024-06-10T00:23:11.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.008+0000 7fa871ffb700 1 -- 172.21.15.121:0/3436261388 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fa86009b470 con 0x7fa874104e90 2024-06-10T00:23:11.008 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:23:11.008 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-06-10T00:23:11.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 -- 172.21.15.121:0/3436261388 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa8640a2c20 msgr2=0x7fa8640a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:11.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa8640a2c20 0x7fa8640a50e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa868005950 tx=0x7fa8680058e0 comp rx=0 tx=0).stop 2024-06-10T00:23:11.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 -- 172.21.15.121:0/3436261388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa874104e90 msgr2=0x7fa87419f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:11.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa874104e90 0x7fa87419f780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa86000d950 tx=0x7fa86000dc60 comp rx=0 tx=0).stop 2024-06-10T00:23:11.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 -- 172.21.15.121:0/3436261388 shutdown_connections 2024-06-10T00:23:11.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa8640a2c20 0x7fa8640a50e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:11.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa874107380 0x7fa8741a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:11.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa87406a6b0 0x7fa87419f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:11.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 --2- 172.21.15.121:0/3436261388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa874104e90 0x7fa87419f780 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:11.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 -- 172.21.15.121:0/3436261388 >> 172.21.15.121:0/3436261388 conn(0x7fa8740782c0 msgr2=0x7fa874103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:11.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 -- 172.21.15.121:0/3436261388 shutdown_connections 2024-06-10T00:23:11.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:11.010+0000 7fa87ae42700 1 -- 172.21.15.121:0/3436261388 wait complete. 2024-06-10T00:23:11.011 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 3 2024-06-10T00:23:11.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:11 smithi121 ceph-mon[32378]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:11.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:11 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3436261388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:11.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:11 smithi135 ceph-mon[36424]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:11.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:11 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3436261388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:11.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:11 smithi193 ceph-mon[36716]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:11 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3436261388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:12.070 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:23:11 smithi121 systemd[1]: Starting Ceph mds.j for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:23:12.379 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:23:12 smithi121 podman[72562]: 2024-06-10 00:23:12.272611691 +0000 UTC m=+0.106758244 container create 4bd66b89868871b209b2e5c6f3dbee7f01c1084bb6948d94128600777975e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-j, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:23:12.379 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:23:12 smithi121 podman[72562]: 2024-06-10 00:23:12.183050163 +0000 UTC m=+0.017196716 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:23:12.949 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:23:12 smithi121 podman[72562]: 2024-06-10 00:23:12.512186703 +0000 UTC m=+0.346333253 container init 4bd66b89868871b209b2e5c6f3dbee7f01c1084bb6948d94128600777975e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-10T00:23:12.949 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:23:12 smithi121 podman[72562]: 2024-06-10 00:23:12.515132207 +0000 UTC m=+0.349278754 container start 4bd66b89868871b209b2e5c6f3dbee7f01c1084bb6948d94128600777975e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-j, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-06-10T00:23:12.949 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:23:12 smithi121 ceph-mds[72581]: -- 172.21.15.121:0/2598651508 <== mon.0 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563eb75d3a40 con 0x563eb75a7000 2024-06-10T00:23:12.949 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:23:12 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-j[72577]: starting mds.j at 2024-06-10T00:23:12.949 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:23:12 smithi121 bash[72562]: 4bd66b89868871b209b2e5c6f3dbee7f01c1084bb6948d94128600777975e232 2024-06-10T00:23:12.949 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:23:12 smithi121 systemd[1]: Started Ceph mds.j for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:13.438 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:13 smithi121 ceph-mon[32378]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:13.438 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:13.439 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:13.439 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:13.439 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:13.439 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:13.439 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:13 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:13.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:13 smithi135 ceph-mon[36424]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:13.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:13 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:13.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:13 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:13.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:13 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:13.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:13 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:13.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:13 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:13.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:13 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:13.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:13 smithi193 ceph-mon[36716]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:13 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:13 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:13 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:13 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:13 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:13 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:14.099 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:23:14.435 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:14 smithi121 ceph-mon[32378]: Deploying daemon mds.d on smithi121 2024-06-10T00:23:14.436 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:14 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.121:6834/4144871914,v1:172.21.15.121:6835/4144871914] up:boot 2024-06-10T00:23:14.436 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:14 smithi121 ceph-mon[32378]: fsmap 3 up:standby 2024-06-10T00:23:14.436 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:14 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-10T00:23:14.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:14 smithi135 ceph-mon[36424]: Deploying daemon mds.d on smithi121 2024-06-10T00:23:14.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:14 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.121:6834/4144871914,v1:172.21.15.121:6835/4144871914] up:boot 2024-06-10T00:23:14.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:14 smithi135 ceph-mon[36424]: fsmap 3 up:standby 2024-06-10T00:23:14.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:14 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-10T00:23:14.533 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:14.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:14 smithi193 ceph-mon[36716]: Deploying daemon mds.d on smithi121 2024-06-10T00:23:14.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:14 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.121:6834/4144871914,v1:172.21.15.121:6835/4144871914] up:boot 2024-06-10T00:23:14.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:14 smithi193 ceph-mon[36716]: fsmap 3 up:standby 2024-06-10T00:23:14.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:14 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-10T00:23:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:15 smithi121 ceph-mon[32378]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:15.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:15 smithi135 ceph-mon[36424]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:15.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:15 smithi193 ceph-mon[36716]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:16.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.313+0000 7f8aaf758700 1 -- 172.21.15.121:0/2228227877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8aa8107870 msgr2=0x7f8aa810f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.313+0000 7f8aaf758700 1 --2- 172.21.15.121:0/2228227877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8aa8107870 0x7f8aa810f3f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000b0d0 tx=0x7f8aa000b3e0 comp rx=0 tx=0).stop 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.313+0000 7f8aaf758700 1 -- 172.21.15.121:0/2228227877 shutdown_connections 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.313+0000 7f8aaf758700 1 --2- 172.21.15.121:0/2228227877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8aa8107870 0x7f8aa810f3f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.313+0000 7f8aaf758700 1 --2- 172.21.15.121:0/2228227877 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8aa81027a0 0x7f8aa8102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.313+0000 7f8aaf758700 1 --2- 172.21.15.121:0/2228227877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8aa8103150 0x7f8aa8107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.313+0000 7f8aaf758700 1 -- 172.21.15.121:0/2228227877 >> 172.21.15.121:0/2228227877 conn(0x7f8aa8079570 msgr2=0x7f8aa8079980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.313+0000 7f8aaf758700 1 -- 172.21.15.121:0/2228227877 shutdown_connections 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.313+0000 7f8aaf758700 1 -- 172.21.15.121:0/2228227877 wait complete. 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.313+0000 7f8aaf758700 1 Processor -- start 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.313+0000 7f8aaf758700 1 -- start start 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aaf758700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8aa81027a0 0x7f8aa819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:16.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aaf758700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8aa8103150 0x7f8aa819ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aaf758700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8aa8107870 0x7f8aa81988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aaf758700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa81127b0 con 0x7f8aa81027a0 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aaf758700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa8112630 con 0x7f8aa8103150 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aaf758700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa8112930 con 0x7f8aa8107870 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aad4f4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8aa81027a0 0x7f8aa819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aad4f4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8aa81027a0 0x7f8aa819e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55210/0 (socket says 172.21.15.121:55210) 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aad4f4700 1 -- 172.21.15.121:0/778600767 learned_addr learned my addr 172.21.15.121:0/778600767 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aaccf3700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8aa8103150 0x7f8aa819ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aadcf5700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8aa8107870 0x7f8aa81988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aad4f4700 1 -- 172.21.15.121:0/778600767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8aa8107870 msgr2=0x7f8aa81988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aad4f4700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8aa8107870 0x7f8aa81988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:16.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aad4f4700 1 -- 172.21.15.121:0/778600767 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8aa8103150 msgr2=0x7f8aa819ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aad4f4700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8aa8103150 0x7f8aa819ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aad4f4700 1 -- 172.21.15.121:0/778600767 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa8199030 con 0x7f8aa81027a0 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aad4f4700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8aa81027a0 0x7f8aa819e710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400c9b0 tx=0x7f8aa400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8a9e7fc700 1 -- 172.21.15.121:0/778600767 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4007b00 con 0x7f8aa81027a0 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8a9e7fc700 1 -- 172.21.15.121:0/778600767 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aa4007ca0 con 0x7f8aa81027a0 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8aaf758700 1 -- 172.21.15.121:0/778600767 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa8199320 con 0x7f8aa81027a0 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.314+0000 7f8a9e7fc700 1 -- 172.21.15.121:0/778600767 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4018780 con 0x7f8aa81027a0 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.315+0000 7f8aaf758700 1 -- 172.21.15.121:0/778600767 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa81a55e0 con 0x7f8aa81027a0 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.315+0000 7f8aaf758700 1 -- 172.21.15.121:0/778600767 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aa8069180 con 0x7f8aa81027a0 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.316+0000 7f8a9e7fc700 1 -- 172.21.15.121:0/778600767 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8aa400f450 con 0x7f8aa81027a0 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.317+0000 7f8a9e7fc700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8a940a2c20 0x7f8a940a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.318+0000 7f8aaccf3700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8a940a2c20 0x7f8a940a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:16.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.318+0000 7f8a9e7fc700 1 -- 172.21.15.121:0/778600767 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8aa40cec10 con 0x7f8aa81027a0 2024-06-10T00:23:16.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.318+0000 7f8aaccf3700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8a940a2c20 0x7f8a940a50e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8108be0 tx=0x7f8a98005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:16.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.322+0000 7f8a9e7fc700 1 -- 172.21.15.121:0/778600767 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8aa4014030 con 0x7f8aa81027a0 2024-06-10T00:23:16.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.566+0000 7f8aaf758700 1 -- 172.21.15.121:0/778600767 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8aa81a5870 con 0x7f8aa81027a0 2024-06-10T00:23:16.566 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:23:16 smithi121 podman[72980]: 2024-06-10 00:23:16.469009911 +0000 UTC m=+0.387032642 container create cbb96db319935fb5de29038bb707c8ae8dcd1a8cae9f746e2c8b7ba3e50cae46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-06-10T00:23:16.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.566+0000 7f8a9e7fc700 1 -- 172.21.15.121:0/778600767 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f8aa409c010 con 0x7f8aa81027a0 2024-06-10T00:23:16.567 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:23:16.567 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-06-10T00:23:16.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.568+0000 7f8aaf758700 1 -- 172.21.15.121:0/778600767 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8a940a2c20 msgr2=0x7f8a940a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:16.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.568+0000 7f8aaf758700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8a940a2c20 0x7f8a940a50e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8108be0 tx=0x7f8a98005eb0 comp rx=0 tx=0).stop 2024-06-10T00:23:16.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.568+0000 7f8aaf758700 1 -- 172.21.15.121:0/778600767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8aa81027a0 msgr2=0x7f8aa819e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:16.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.568+0000 7f8aaf758700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8aa81027a0 0x7f8aa819e710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400c9b0 tx=0x7f8aa400cd70 comp rx=0 tx=0).stop 2024-06-10T00:23:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.568+0000 7f8aaf758700 1 -- 172.21.15.121:0/778600767 shutdown_connections 2024-06-10T00:23:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.568+0000 7f8aaf758700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8a940a2c20 0x7f8a940a50e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.569+0000 7f8aaf758700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8aa8107870 0x7f8aa81988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.569+0000 7f8aaf758700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8aa81027a0 0x7f8aa819e710 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.569+0000 7f8aaf758700 1 --2- 172.21.15.121:0/778600767 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8aa8103150 0x7f8aa819ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.569+0000 7f8aaf758700 1 -- 172.21.15.121:0/778600767 >> 172.21.15.121:0/778600767 conn(0x7f8aa8079570 msgr2=0x7f8aa81004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.569+0000 7f8aaf758700 1 -- 172.21.15.121:0/778600767 shutdown_connections 2024-06-10T00:23:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:16.569+0000 7f8aaf758700 1 -- 172.21.15.121:0/778600767 wait complete. 2024-06-10T00:23:16.569 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 4 2024-06-10T00:23:17.199 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:23:16 smithi121 podman[72980]: 2024-06-10 00:23:16.907027652 +0000 UTC m=+0.825050385 container init cbb96db319935fb5de29038bb707c8ae8dcd1a8cae9f746e2c8b7ba3e50cae46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:23:17.199 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:23:16 smithi121 podman[72980]: 2024-06-10 00:23:16.910310184 +0000 UTC m=+0.828332903 container start cbb96db319935fb5de29038bb707c8ae8dcd1a8cae9f746e2c8b7ba3e50cae46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-06-10T00:23:17.199 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:23:16 smithi121 ceph-mds[73037]: -- 172.21.15.121:0/2961639379 <== mon.0 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5592b6847a40 con 0x5592b681b000 2024-06-10T00:23:17.199 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:23:16 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-d[73020]: starting mds.d at 2024-06-10T00:23:17.199 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:23:17 smithi121 bash[72980]: cbb96db319935fb5de29038bb707c8ae8dcd1a8cae9f746e2c8b7ba3e50cae46 2024-06-10T00:23:17.199 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:23:17 smithi121 systemd[1]: Started Ceph mds.d for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:17.470 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:17 smithi121 ceph-mon[32378]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:17.471 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:17 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/778600767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:17.471 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:17 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:17.471 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:17 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:17.471 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:17 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:17.471 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:17 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:17.471 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:17 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:17.471 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:17 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:17 smithi135 ceph-mon[36424]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:17 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/778600767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:17 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:17 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:17 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:17 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:17 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:17 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:17.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:17 smithi193 ceph-mon[36716]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:17 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/778600767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:17 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:17 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:17 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:17 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:17 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:17 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:18.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:18 smithi135 ceph-mon[36424]: Deploying daemon mds.e on smithi135 2024-06-10T00:23:18.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:18 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.121:6836/64109840,v1:172.21.15.121:6837/64109840] up:boot 2024-06-10T00:23:18.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:18 smithi135 ceph-mon[36424]: fsmap 4 up:standby 2024-06-10T00:23:18.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:18 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-10T00:23:18.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:18 smithi193 ceph-mon[36716]: Deploying daemon mds.e on smithi135 2024-06-10T00:23:18.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:18 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.121:6836/64109840,v1:172.21.15.121:6837/64109840] up:boot 2024-06-10T00:23:18.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:18 smithi193 ceph-mon[36716]: fsmap 4 up:standby 2024-06-10T00:23:18.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:18 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-10T00:23:18.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:18 smithi121 ceph-mon[32378]: Deploying daemon mds.e on smithi135 2024-06-10T00:23:18.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:18 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.121:6836/64109840,v1:172.21.15.121:6837/64109840] up:boot 2024-06-10T00:23:18.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:18 smithi121 ceph-mon[32378]: fsmap 4 up:standby 2024-06-10T00:23:18.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:18 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-10T00:23:19.445 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:23:19.473 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:19 smithi121 ceph-mon[32378]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:19.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:19 smithi135 ceph-mon[36424]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:19.508 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:23:19 smithi135 systemd[1]: Starting Ceph mds.e for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:23:19.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:19 smithi193 ceph-mon[36716]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:19.695 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:19.764 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:23:19 smithi135 podman[65546]: 2024-06-10 00:23:19.414572762 +0000 UTC m=+0.016886720 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:23:19.764 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:23:19 smithi135 podman[65546]: 2024-06-10 00:23:19.538292711 +0000 UTC m=+0.140610938 container create d4a9ad535a9d0b1a0c343a6cb19dd9232a6130e382e401e2d4c8a097535bd45f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc) 2024-06-10T00:23:19.764 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:23:19 smithi135 podman[65546]: 2024-06-10 00:23:19.762295874 +0000 UTC m=+0.364609831 container init d4a9ad535a9d0b1a0c343a6cb19dd9232a6130e382e401e2d4c8a097535bd45f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:23:19.764 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:23:19 smithi135 podman[65546]: 2024-06-10 00:23:19.764898221 +0000 UTC m=+0.367212179 container start d4a9ad535a9d0b1a0c343a6cb19dd9232a6130e382e401e2d4c8a097535bd45f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True) 2024-06-10T00:23:20.257 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:23:19 smithi135 ceph-mds[65565]: -- 172.21.15.135:0/1590080118 <== mon.2 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b8da259a40 con 0x55b8da22cc00 2024-06-10T00:23:20.258 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:23:19 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e[65561]: starting mds.e at 2024-06-10T00:23:20.258 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:23:19 smithi135 bash[65546]: d4a9ad535a9d0b1a0c343a6cb19dd9232a6130e382e401e2d4c8a097535bd45f 2024-06-10T00:23:20.258 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:23:19 smithi135 systemd[1]: Started Ceph mds.e for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:21.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:20 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:21.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:20 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:21.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:20 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:21.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:20 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:21.144 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:20 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:21.145 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:20 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:21.145 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:20 smithi121 ceph-mon[32378]: Deploying daemon mds.f on smithi193 2024-06-10T00:23:21.145 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:20 smithi121 ceph-mon[32378]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:21.145 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:20 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.135:6834/284701222,v1:172.21.15.135:6835/284701222] up:boot 2024-06-10T00:23:21.145 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:20 smithi121 ceph-mon[32378]: fsmap 5 up:standby 2024-06-10T00:23:21.145 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:20 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-10T00:23:21.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:20 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:20 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:20 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:20 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:20 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:20 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:20 smithi135 ceph-mon[36424]: Deploying daemon mds.f on smithi193 2024-06-10T00:23:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:20 smithi135 ceph-mon[36424]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:20 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.135:6834/284701222,v1:172.21.15.135:6835/284701222] up:boot 2024-06-10T00:23:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:20 smithi135 ceph-mon[36424]: fsmap 5 up:standby 2024-06-10T00:23:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:20 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-10T00:23:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:20 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:20 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:20 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:20 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:20 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:20 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:20 smithi193 ceph-mon[36716]: Deploying daemon mds.f on smithi193 2024-06-10T00:23:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:20 smithi193 ceph-mon[36716]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:20 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.135:6834/284701222,v1:172.21.15.135:6835/284701222] up:boot 2024-06-10T00:23:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:20 smithi193 ceph-mon[36716]: fsmap 5 up:standby 2024-06-10T00:23:21.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:20 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-10T00:23:21.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.482+0000 7f575f32f700 1 -- 172.21.15.121:0/3626213025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5758102740 msgr2=0x7f5758102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:21.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.482+0000 7f575f32f700 1 --2- 172.21.15.121:0/3626213025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5758102740 0x7f5758102b20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f57540099c0 tx=0x7f5754009cd0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.482+0000 7f575f32f700 1 -- 172.21.15.121:0/3626213025 shutdown_connections 2024-06-10T00:23:21.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.482+0000 7f575f32f700 1 --2- 172.21.15.121:0/3626213025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5758107840 0x7f575810f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.482+0000 7f575f32f700 1 --2- 172.21.15.121:0/3626213025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5758102740 0x7f5758102b20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.482+0000 7f575f32f700 1 --2- 172.21.15.121:0/3626213025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57581030f0 0x7f5758107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.482+0000 7f575f32f700 1 -- 172.21.15.121:0/3626213025 >> 172.21.15.121:0/3626213025 conn(0x7f57580795f0 msgr2=0x7f5758079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:21.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575f32f700 1 -- 172.21.15.121:0/3626213025 shutdown_connections 2024-06-10T00:23:21.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575f32f700 1 -- 172.21.15.121:0/3626213025 wait complete. 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575f32f700 1 Processor -- start 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575f32f700 1 -- start start 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575f32f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5758102740 0x7f575819e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575f32f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57581030f0 0x7f575819edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575f32f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5758107840 0x7f5758198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575f32f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f57581127d0 con 0x7f5758107840 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575f32f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5758112650 con 0x7f57581030f0 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575f32f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5758112950 con 0x7f5758102740 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575d8cc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5758107840 0x7f5758198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575d8cc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5758107840 0x7f5758198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42886/0 (socket says 172.21.15.121:42886) 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.483+0000 7f575d8cc700 1 -- 172.21.15.121:0/2320420068 learned_addr learned my addr 172.21.15.121:0/2320420068 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f575c8ca700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57581030f0 0x7f575819edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f575d0cb700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5758102740 0x7f575819e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:21.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f575d8cc700 1 -- 172.21.15.121:0/2320420068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5758102740 msgr2=0x7f575819e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f575d8cc700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5758102740 0x7f575819e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f575d8cc700 1 -- 172.21.15.121:0/2320420068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57581030f0 msgr2=0x7f575819edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f575d8cc700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57581030f0 0x7f575819edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f575d8cc700 1 -- 172.21.15.121:0/2320420068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57581991f0 con 0x7f5758107840 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f575d8cc700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5758107840 0x7f5758198a90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5750007ee0 tx=0x7f575000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f574a7fc700 1 -- 172.21.15.121:0/2320420068 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575000dbd0 con 0x7f5758107840 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f575f32f700 1 -- 172.21.15.121:0/2320420068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57581994e0 con 0x7f5758107840 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f574a7fc700 1 -- 172.21.15.121:0/2320420068 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f575000dd70 con 0x7f5758107840 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f574a7fc700 1 -- 172.21.15.121:0/2320420068 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5750005a60 con 0x7f5758107840 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.484+0000 7f575f32f700 1 -- 172.21.15.121:0/2320420068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57581a5750 con 0x7f5758107840 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.485+0000 7f575f32f700 1 -- 172.21.15.121:0/2320420068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f575810aa80 con 0x7f5758107840 2024-06-10T00:23:21.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.486+0000 7f574a7fc700 1 -- 172.21.15.121:0/2320420068 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f575001a020 con 0x7f5758107840 2024-06-10T00:23:21.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.487+0000 7f574a7fc700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f57440a2c20 0x7f57440a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:21.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.487+0000 7f575d0cb700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f57440a2c20 0x7f57440a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:21.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.487+0000 7f574a7fc700 1 -- 172.21.15.121:0/2320420068 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f57500ce960 con 0x7f5758107840 2024-06-10T00:23:21.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.487+0000 7f575d0cb700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f57440a2c20 0x7f57440a50e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5754000c00 tx=0x7f5754005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:21.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.490+0000 7f574a7fc700 1 -- 172.21.15.121:0/2320420068 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5750010070 con 0x7f5758107840 2024-06-10T00:23:21.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.729+0000 7f575f32f700 1 -- 172.21.15.121:0/2320420068 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5758069180 con 0x7f5758107840 2024-06-10T00:23:21.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.730+0000 7f574a7fc700 1 -- 172.21.15.121:0/2320420068 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7f575009bd60 con 0x7f5758107840 2024-06-10T00:23:21.729 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:23:21.730 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-06-10T00:23:21.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 -- 172.21.15.121:0/2320420068 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f57440a2c20 msgr2=0x7f57440a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f57440a2c20 0x7f57440a50e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5754000c00 tx=0x7f5754005fb0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 -- 172.21.15.121:0/2320420068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5758107840 msgr2=0x7f5758198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5758107840 0x7f5758198a90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5750007ee0 tx=0x7f575000d3b0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 -- 172.21.15.121:0/2320420068 shutdown_connections 2024-06-10T00:23:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f57440a2c20 0x7f57440a50e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5758107840 0x7f5758198a90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5758102740 0x7f575819e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 --2- 172.21.15.121:0/2320420068 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f57581030f0 0x7f575819edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 -- 172.21.15.121:0/2320420068 >> 172.21.15.121:0/2320420068 conn(0x7f57580795f0 msgr2=0x7f5758108990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 -- 172.21.15.121:0/2320420068 shutdown_connections 2024-06-10T00:23:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:21.732+0000 7f575f32f700 1 -- 172.21.15.121:0/2320420068 wait complete. 2024-06-10T00:23:21.733 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 6 2024-06-10T00:23:22.039 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:21 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2320420068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:22.051 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:21 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2320420068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:22.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:21 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2320420068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:22.304 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:23:22 smithi193 systemd[1]: Starting Ceph mds.f for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:23:22.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:23:22 smithi193 podman[66584]: 2024-06-10 00:23:22.38293442 +0000 UTC m=+0.017550273 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:23:22.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:23:22 smithi193 podman[66584]: 2024-06-10 00:23:22.489391614 +0000 UTC m=+0.124007454 container create e76d9836ad77514ef787f398c68e405757b2a2c72ba321d742c97927f0c4edca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:23:22.982 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:23:22 smithi193 podman[66584]: 2024-06-10 00:23:22.726150247 +0000 UTC m=+0.360766083 container init e76d9836ad77514ef787f398c68e405757b2a2c72ba321d742c97927f0c4edca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-06-10T00:23:22.982 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:23:22 smithi193 podman[66584]: 2024-06-10 00:23:22.729088508 +0000 UTC m=+0.363704349 container start e76d9836ad77514ef787f398c68e405757b2a2c72ba321d742c97927f0c4edca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-10T00:23:22.982 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:23:22 smithi193 ceph-mds[66613]: -- 172.21.15.193:0/605492796 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560b868b3a40 con 0x560b86886c00 2024-06-10T00:23:22.983 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:23:22 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f[66607]: starting mds.f at 2024-06-10T00:23:22.983 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:23:22 smithi193 bash[66584]: e76d9836ad77514ef787f398c68e405757b2a2c72ba321d742c97927f0c4edca 2024-06-10T00:23:22.983 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:23:22 smithi193 systemd[1]: Started Ceph mds.f for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:23.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:22 smithi135 ceph-mon[36424]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:23.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:22 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:23.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:22 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:23.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:22 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:23.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:22 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:23.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:22 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:23.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:22 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:23.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:22 smithi193 ceph-mon[36716]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:23.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:22 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:23.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:22 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:23.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:22 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:23.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:22 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:23.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:22 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:23.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:22 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:22 smithi121 ceph-mon[32378]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:22 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:22 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:22 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:22 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:22 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:22 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:24.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:23 smithi193 ceph-mon[36716]: Deploying daemon mds.c on smithi193 2024-06-10T00:23:24.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:23 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.193:6834/3023450024,v1:172.21.15.193:6835/3023450024] up:boot 2024-06-10T00:23:24.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:23 smithi193 ceph-mon[36716]: fsmap 6 up:standby 2024-06-10T00:23:24.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:23 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-10T00:23:24.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:23 smithi135 ceph-mon[36424]: Deploying daemon mds.c on smithi193 2024-06-10T00:23:24.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:23 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.193:6834/3023450024,v1:172.21.15.193:6835/3023450024] up:boot 2024-06-10T00:23:24.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:23 smithi135 ceph-mon[36424]: fsmap 6 up:standby 2024-06-10T00:23:24.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:23 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-10T00:23:24.388 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:23:24.419 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:23 smithi121 ceph-mon[32378]: Deploying daemon mds.c on smithi193 2024-06-10T00:23:24.419 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:23 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.193:6834/3023450024,v1:172.21.15.193:6835/3023450024] up:boot 2024-06-10T00:23:24.419 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:23 smithi121 ceph-mon[32378]: fsmap 6 up:standby 2024-06-10T00:23:24.419 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:23 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-10T00:23:24.639 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:25.047 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:24 smithi121 ceph-mon[32378]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:24 smithi193 ceph-mon[36716]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:25.131 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:23:25 smithi193 systemd[1]: Starting Ceph mds.c for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:23:25.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:24 smithi135 ceph-mon[36424]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:25.506 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:23:25 smithi193 podman[66894]: 2024-06-10 00:23:25.118552301 +0000 UTC m=+0.017608304 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:23:25.507 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:23:25 smithi193 podman[66894]: 2024-06-10 00:23:25.24626688 +0000 UTC m=+0.145322858 container create 94260ed35f301a1e9c4aa79ed600c76667cb1e4b85adbd27a16868e35a9fddf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:23:25.507 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:23:25 smithi193 podman[66894]: 2024-06-10 00:23:25.507702619 +0000 UTC m=+0.406758605 container init 94260ed35f301a1e9c4aa79ed600c76667cb1e4b85adbd27a16868e35a9fddf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-10T00:23:25.881 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:23:25 smithi193 podman[66894]: 2024-06-10 00:23:25.510286479 +0000 UTC m=+0.409342468 container start 94260ed35f301a1e9c4aa79ed600c76667cb1e4b85adbd27a16868e35a9fddf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:23:25.881 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:23:25 smithi193 ceph-mds[66914]: -- 172.21.15.193:0/1447031377 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d43f943a40 con 0x55d43f916800 2024-06-10T00:23:25.881 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:23:25 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c[66910]: starting mds.c at 2024-06-10T00:23:25.881 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:23:25 smithi193 bash[66894]: 94260ed35f301a1e9c4aa79ed600c76667cb1e4b85adbd27a16868e35a9fddf9 2024-06-10T00:23:25.881 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:23:25 smithi193 systemd[1]: Started Ceph mds.c for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:26.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.315+0000 7f3bca073700 1 -- 172.21.15.121:0/973378383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bc41066b0 msgr2=0x7f3bc410e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:26.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.315+0000 7f3bca073700 1 --2- 172.21.15.121:0/973378383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bc41066b0 0x7f3bc410e280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00b0d0 tx=0x7f3bbc00b3e0 comp rx=0 tx=0).stop 2024-06-10T00:23:26.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.315+0000 7f3bca073700 1 -- 172.21.15.121:0/973378383 shutdown_connections 2024-06-10T00:23:26.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.316+0000 7f3bca073700 1 --2- 172.21.15.121:0/973378383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bc41066b0 0x7f3bc410e280 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:26.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.316+0000 7f3bca073700 1 --2- 172.21.15.121:0/973378383 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bc4101620 0x7f3bc4101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:26.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.316+0000 7f3bca073700 1 --2- 172.21.15.121:0/973378383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bc4101fd0 0x7f3bc4105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:26.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.316+0000 7f3bca073700 1 -- 172.21.15.121:0/973378383 >> 172.21.15.121:0/973378383 conn(0x7f3bc4078340 msgr2=0x7f3bc4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:26.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.316+0000 7f3bca073700 1 -- 172.21.15.121:0/973378383 shutdown_connections 2024-06-10T00:23:26.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.316+0000 7f3bca073700 1 -- 172.21.15.121:0/973378383 wait complete. 2024-06-10T00:23:26.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.316+0000 7f3bca073700 1 Processor -- start 2024-06-10T00:23:26.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bca073700 1 -- start start 2024-06-10T00:23:26.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bca073700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bc4101620 0x7f3bc419d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:26.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bca073700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bc4101fd0 0x7f3bc419dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bca073700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bc41066b0 0x7f3bc4197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bca073700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc41116c0 con 0x7f3bc41066b0 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bca073700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4111540 con 0x7f3bc4101fd0 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bca073700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc4111840 con 0x7f3bc4101620 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bc41066b0 0x7f3bc4197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc37fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bc4101620 0x7f3bc419d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc2ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bc4101fd0 0x7f3bc419dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc37fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bc4101620 0x7f3bc419d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:51210/0 (socket says 172.21.15.121:51210) 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bc41066b0 0x7f3bc4197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42902/0 (socket says 172.21.15.121:42902) 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc37fe700 1 -- 172.21.15.121:0/4271916053 learned_addr learned my addr 172.21.15.121:0/4271916053 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc37fe700 1 -- 172.21.15.121:0/4271916053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bc4101fd0 msgr2=0x7f3bc419dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc37fe700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bc4101fd0 0x7f3bc419dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:26.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc37fe700 1 -- 172.21.15.121:0/4271916053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bc41066b0 msgr2=0x7f3bc4197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:26.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc37fe700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bc41066b0 0x7f3bc4197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:26.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc37fe700 1 -- 172.21.15.121:0/4271916053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc41980e0 con 0x7f3bc4101620 2024-06-10T00:23:26.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.317+0000 7f3bc3fff700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bc41066b0 0x7f3bc4197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:23:26.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.318+0000 7f3bc37fe700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bc4101620 0x7f3bc419d770 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400c9b0 tx=0x7f3bb400ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:26.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.318+0000 7f3bc0ff9700 1 -- 172.21.15.121:0/4271916053 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4007b00 con 0x7f3bc4101620 2024-06-10T00:23:26.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.318+0000 7f3bca073700 1 -- 172.21.15.121:0/4271916053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc41983d0 con 0x7f3bc4101620 2024-06-10T00:23:26.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.318+0000 7f3bc0ff9700 1 -- 172.21.15.121:0/4271916053 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bb400ce50 con 0x7f3bc4101620 2024-06-10T00:23:26.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.318+0000 7f3bca073700 1 -- 172.21.15.121:0/4271916053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc41a4640 con 0x7f3bc4101620 2024-06-10T00:23:26.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.318+0000 7f3bc0ff9700 1 -- 172.21.15.121:0/4271916053 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb4005560 con 0x7f3bc4101620 2024-06-10T00:23:26.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.319+0000 7f3bca073700 1 -- 172.21.15.121:0/4271916053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc4069180 con 0x7f3bc4101620 2024-06-10T00:23:26.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.320+0000 7f3bc0ff9700 1 -- 172.21.15.121:0/4271916053 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3bb4004750 con 0x7f3bc4101620 2024-06-10T00:23:26.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.320+0000 7f3bc0ff9700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3bac0a2c20 0x7f3bac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:26.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.321+0000 7f3bc2ffd700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3bac0a2c20 0x7f3bac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:26.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.321+0000 7f3bc0ff9700 1 -- 172.21.15.121:0/4271916053 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f3bb40cee90 con 0x7f3bc4101620 2024-06-10T00:23:26.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.321+0000 7f3bc2ffd700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3bac0a2c20 0x7f3bac0a50e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4107ac0 tx=0x7f3bb8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:26.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.324+0000 7f3bc0ff9700 1 -- 172.21.15.121:0/4271916053 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3bb4014030 con 0x7f3bc4101620 2024-06-10T00:23:26.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.563+0000 7f3bca073700 1 -- 172.21.15.121:0/4271916053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bc4198e60 con 0x7f3bc4101620 2024-06-10T00:23:26.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.564+0000 7f3bc0ff9700 1 -- 172.21.15.121:0/4271916053 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f3bb409ce20 con 0x7f3bc4101620 2024-06-10T00:23:26.563 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:23:26.563 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-06-10T00:23:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 -- 172.21.15.121:0/4271916053 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3bac0a2c20 msgr2=0x7f3bac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3bac0a2c20 0x7f3bac0a50e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4107ac0 tx=0x7f3bb8009260 comp rx=0 tx=0).stop 2024-06-10T00:23:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 -- 172.21.15.121:0/4271916053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bc4101620 msgr2=0x7f3bc419d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bc4101620 0x7f3bc419d770 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3bb400c9b0 tx=0x7f3bb400ccc0 comp rx=0 tx=0).stop 2024-06-10T00:23:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 -- 172.21.15.121:0/4271916053 shutdown_connections 2024-06-10T00:23:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3bac0a2c20 0x7f3bac0a50e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bc41066b0 0x7f3bc4197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bc4101620 0x7f3bc419d770 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:26.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 --2- 172.21.15.121:0/4271916053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bc4101fd0 0x7f3bc419dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 -- 172.21.15.121:0/4271916053 >> 172.21.15.121:0/4271916053 conn(0x7f3bc4078340 msgr2=0x7f3bc40ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 -- 172.21.15.121:0/4271916053 shutdown_connections 2024-06-10T00:23:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:26.566+0000 7f3bca073700 1 -- 172.21.15.121:0/4271916053 wait complete. 2024-06-10T00:23:26.567 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 8 2024-06-10T00:23:26.868 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:26.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:26.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:26.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:26.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:26.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:26.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: Deploying daemon mds.h on smithi135 2024-06-10T00:23:26.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.193:6836/2863018656,v1:172.21.15.193:6837/2863018656] up:boot 2024-06-10T00:23:26.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: fsmap 7 up:standby 2024-06-10T00:23:26.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-10T00:23:26.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:26.869 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:26 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/4271916053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:27.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:27.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:27.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:27.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:27.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:27.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:27.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: Deploying daemon mds.h on smithi135 2024-06-10T00:23:27.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.193:6836/2863018656,v1:172.21.15.193:6837/2863018656] up:boot 2024-06-10T00:23:27.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: fsmap 7 up:standby 2024-06-10T00:23:27.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-10T00:23:27.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:27.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:26 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/4271916053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: Deploying daemon mds.h on smithi135 2024-06-10T00:23:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.193:6836/2863018656,v1:172.21.15.193:6837/2863018656] up:boot 2024-06-10T00:23:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: fsmap 7 up:standby 2024-06-10T00:23:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-10T00:23:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:27.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:26 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/4271916053' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:28.008 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:23:27 smithi135 systemd[1]: Starting Ceph mds.h for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:23:28.393 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:23:28 smithi135 podman[65862]: 2024-06-10 00:23:28.001876395 +0000 UTC m=+0.017703902 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:23:28.393 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:23:28 smithi135 podman[65862]: 2024-06-10 00:23:28.131709522 +0000 UTC m=+0.147537002 container create 6707dedba949b1781422c1782f7f480c30d9b47b48e2439a5eb9f658a17a4c86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-3d89acc, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-10T00:23:28.393 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:23:28 smithi135 podman[65862]: 2024-06-10 00:23:28.393232609 +0000 UTC m=+0.409060112 container init 6707dedba949b1781422c1782f7f480c30d9b47b48e2439a5eb9f658a17a4c86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, RELEASE=squid-3d89acc) 2024-06-10T00:23:28.757 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:23:28 smithi135 podman[65862]: 2024-06-10 00:23:28.396162658 +0000 UTC m=+0.411990148 container start 6707dedba949b1781422c1782f7f480c30d9b47b48e2439a5eb9f658a17a4c86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h, GIT_CLEAN=True, ceph=True, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-06-10T00:23:28.758 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:23:28 smithi135 ceph-mds[65882]: -- 172.21.15.135:0/703594711 <== mon.2 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558e1dbefa40 con 0x558e1dbc3000 2024-06-10T00:23:28.758 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:23:28 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: starting mds.h at 2024-06-10T00:23:28.758 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:23:28 smithi135 bash[65862]: 6707dedba949b1781422c1782f7f480c30d9b47b48e2439a5eb9f658a17a4c86 2024-06-10T00:23:28.758 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:23:28 smithi135 systemd[1]: Started Ceph mds.h for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:29.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:23:29.420 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:29.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:29 smithi193 ceph-mon[36716]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:29.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:29 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:29.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:29 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:29.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:29 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:29.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:29 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:29.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:29 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:29.436 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:29 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:29 smithi121 ceph-mon[32378]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:29 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:29 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:29 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:29 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:29 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:29 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:29.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:29 smithi135 ceph-mon[36424]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:29.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:29 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:29.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:29 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:29.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:29 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:29.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:29 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:29.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:29 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:29.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:29 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:30.207 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:30 smithi193 ceph-mon[36716]: Deploying daemon mds.i on smithi193 2024-06-10T00:23:30.208 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:30 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] up:boot 2024-06-10T00:23:30.208 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:30 smithi193 ceph-mon[36716]: fsmap 8 up:standby 2024-06-10T00:23:30.208 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:30 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-10T00:23:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:30 smithi121 ceph-mon[32378]: Deploying daemon mds.i on smithi193 2024-06-10T00:23:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:30 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] up:boot 2024-06-10T00:23:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:30 smithi121 ceph-mon[32378]: fsmap 8 up:standby 2024-06-10T00:23:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:30 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-10T00:23:30.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:30 smithi135 ceph-mon[36424]: Deploying daemon mds.i on smithi193 2024-06-10T00:23:30.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:30 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] up:boot 2024-06-10T00:23:30.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:30 smithi135 ceph-mon[36424]: fsmap 8 up:standby 2024-06-10T00:23:30.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:30 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-10T00:23:30.880 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:23:30 smithi193 systemd[1]: Starting Ceph mds.i for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:23:31.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.060+0000 7f679041b700 1 -- 172.21.15.121:0/2867245311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f678806abe0 msgr2=0x7f678806b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:31.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.060+0000 7f679041b700 1 --2- 172.21.15.121:0/2867245311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f678806abe0 0x7f678806b060 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6778009920 tx=0x7f6778009c30 comp rx=0 tx=0).stop 2024-06-10T00:23:31.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.060+0000 7f679041b700 1 -- 172.21.15.121:0/2867245311 shutdown_connections 2024-06-10T00:23:31.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.060+0000 7f679041b700 1 --2- 172.21.15.121:0/2867245311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f678806b5a0 0x7f678810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:31.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.060+0000 7f679041b700 1 --2- 172.21.15.121:0/2867245311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67881085a0 0x7f6788108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:31.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.060+0000 7f679041b700 1 --2- 172.21.15.121:0/2867245311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f678806abe0 0x7f678806b060 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:31.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.060+0000 7f679041b700 1 -- 172.21.15.121:0/2867245311 >> 172.21.15.121:0/2867245311 conn(0x7f6788078280 msgr2=0x7f6788078690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:31.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.060+0000 7f679041b700 1 -- 172.21.15.121:0/2867245311 shutdown_connections 2024-06-10T00:23:31.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.060+0000 7f679041b700 1 -- 172.21.15.121:0/2867245311 wait complete. 2024-06-10T00:23:31.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.060+0000 7f679041b700 1 Processor -- start 2024-06-10T00:23:31.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.060+0000 7f679041b700 1 -- start start 2024-06-10T00:23:31.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f679041b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f678806abe0 0x7f6788101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:31.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f679041b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f678806b5a0 0x7f6788101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:31.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f679041b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67881085a0 0x7f67881021c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:31.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f679041b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f67881127e0 con 0x7f67881085a0 2024-06-10T00:23:31.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f679041b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6788112660 con 0x7f678806b5a0 2024-06-10T00:23:31.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f679041b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6788112960 con 0x7f678806abe0 2024-06-10T00:23:31.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f678e9b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67881085a0 0x7f67881021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:31.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f678e9b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67881085a0 0x7f67881021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59132/0 (socket says 172.21.15.121:59132) 2024-06-10T00:23:31.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f678e9b8700 1 -- 172.21.15.121:0/3262333906 learned_addr learned my addr 172.21.15.121:0/3262333906 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:31.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f678e1b7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f678806abe0 0x7f6788101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:31.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f678d9b6700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f678806b5a0 0x7f6788101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:31.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f678e9b8700 1 -- 172.21.15.121:0/3262333906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f678806abe0 msgr2=0x7f6788101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:31.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f678e9b8700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f678806abe0 0x7f6788101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:31.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f678e9b8700 1 -- 172.21.15.121:0/3262333906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f678806b5a0 msgr2=0x7f6788101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:31.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f678e9b8700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f678806b5a0 0x7f6788101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:31.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f678e9b8700 1 -- 172.21.15.121:0/3262333906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67881a7170 con 0x7f67881085a0 2024-06-10T00:23:31.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f678e9b8700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67881085a0 0x7f67881021c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f678000c110 tx=0x7f678000c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:31.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f677f7fe700 1 -- 172.21.15.121:0/3262333906 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67800092e0 con 0x7f67881085a0 2024-06-10T00:23:31.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f679041b700 1 -- 172.21.15.121:0/3262333906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67881a7400 con 0x7f67881085a0 2024-06-10T00:23:31.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f677f7fe700 1 -- 172.21.15.121:0/3262333906 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6780004bb0 con 0x7f67881085a0 2024-06-10T00:23:31.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f677f7fe700 1 -- 172.21.15.121:0/3262333906 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6780007d10 con 0x7f67881085a0 2024-06-10T00:23:31.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.061+0000 7f679041b700 1 -- 172.21.15.121:0/3262333906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67881a7960 con 0x7f67881085a0 2024-06-10T00:23:31.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.062+0000 7f679041b700 1 -- 172.21.15.121:0/3262333906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6788103bd0 con 0x7f67881085a0 2024-06-10T00:23:31.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.064+0000 7f677f7fe700 1 -- 172.21.15.121:0/3262333906 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f678001a020 con 0x7f67881085a0 2024-06-10T00:23:31.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.065+0000 7f677f7fe700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67740a2c20 0x7f67740a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:31.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.065+0000 7f678e1b7700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67740a2c20 0x7f67740a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:31.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.065+0000 7f677f7fe700 1 -- 172.21.15.121:0/3262333906 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f67800cf180 con 0x7f67881085a0 2024-06-10T00:23:31.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.065+0000 7f678e1b7700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67740a2c20 0x7f67740a50e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6784009be0 tx=0x7f6784009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:31.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.069+0000 7f677f7fe700 1 -- 172.21.15.121:0/3262333906 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6780010040 con 0x7f67881085a0 2024-06-10T00:23:31.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:31 smithi193 ceph-mon[36716]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:31.288 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:23:30 smithi193 podman[67198]: 2024-06-10 00:23:30.825536132 +0000 UTC m=+0.017295434 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:23:31.288 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:23:30 smithi193 podman[67198]: 2024-06-10 00:23:30.943905965 +0000 UTC m=+0.135665258 container create e78cfb13592ac06d1be9ba102ce6b8819c52114f029cb15b6314d664494cbd0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, ceph=True, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=) 2024-06-10T00:23:31.288 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:23:31 smithi193 podman[67198]: 2024-06-10 00:23:31.289037882 +0000 UTC m=+0.480797176 container init e78cfb13592ac06d1be9ba102ce6b8819c52114f029cb15b6314d664494cbd0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528) 2024-06-10T00:23:31.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.309+0000 7f679041b700 1 -- 172.21.15.121:0/3262333906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f678802dca0 con 0x7f67881085a0 2024-06-10T00:23:31.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.309+0000 7f677f7fe700 1 -- 172.21.15.121:0/3262333906 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7180 (secure 0 0 0) 0x7f678009c4e0 con 0x7f67881085a0 2024-06-10T00:23:31.309 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:23:31.309 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-06-10T00:23:31.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.311+0000 7f679041b700 1 -- 172.21.15.121:0/3262333906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67740a2c20 msgr2=0x7f67740a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:31.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.311+0000 7f679041b700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67740a2c20 0x7f67740a50e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6784009be0 tx=0x7f6784009260 comp rx=0 tx=0).stop 2024-06-10T00:23:31.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.311+0000 7f679041b700 1 -- 172.21.15.121:0/3262333906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67881085a0 msgr2=0x7f67881021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:31.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.311+0000 7f679041b700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67881085a0 0x7f67881021c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f678000c110 tx=0x7f678000c420 comp rx=0 tx=0).stop 2024-06-10T00:23:31.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.311+0000 7f679041b700 1 -- 172.21.15.121:0/3262333906 shutdown_connections 2024-06-10T00:23:31.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.311+0000 7f679041b700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f67740a2c20 0x7f67740a50e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:31.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.311+0000 7f679041b700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67881085a0 0x7f67881021c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:31.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.311+0000 7f679041b700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f678806abe0 0x7f6788101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:31.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.311+0000 7f679041b700 1 --2- 172.21.15.121:0/3262333906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f678806b5a0 0x7f6788101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:31.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.311+0000 7f679041b700 1 -- 172.21.15.121:0/3262333906 >> 172.21.15.121:0/3262333906 conn(0x7f6788078280 msgr2=0x7f6788100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:31.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.311+0000 7f679041b700 1 -- 172.21.15.121:0/3262333906 shutdown_connections 2024-06-10T00:23:31.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:31.312+0000 7f679041b700 1 -- 172.21.15.121:0/3262333906 wait complete. 2024-06-10T00:23:31.312 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 9 2024-06-10T00:23:31.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:31 smithi121 ceph-mon[32378]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:31.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:31 smithi135 ceph-mon[36424]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:31.630 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:23:31 smithi193 podman[67198]: 2024-06-10 00:23:31.291977033 +0000 UTC m=+0.483736332 container start e78cfb13592ac06d1be9ba102ce6b8819c52114f029cb15b6314d664494cbd0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-10T00:23:31.631 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:23:31 smithi193 ceph-mds[67218]: -- 172.21.15.193:0/3916344247 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a48a2a3a40 con 0x55a48a276800 2024-06-10T00:23:31.631 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:23:31 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i[67214]: starting mds.i at 2024-06-10T00:23:31.631 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:23:31 smithi193 bash[67198]: e78cfb13592ac06d1be9ba102ce6b8819c52114f029cb15b6314d664494cbd0c 2024-06-10T00:23:31.631 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:23:31 smithi193 systemd[1]: Started Ceph mds.i for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:32.324 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:32 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3262333906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:32.324 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:32 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:32.324 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:32 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:32.324 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:32 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:32.324 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:32 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:32.324 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:32 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:32.324 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:32 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:32.445 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:32 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3262333906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:32.445 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:32 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:32.445 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:32 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:32.445 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:32 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:32.445 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:32 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:32.445 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:32 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:32.445 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:32 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:32.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:32 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3262333906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:32.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:32 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:32.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:32 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:32.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:32 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:32.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:32 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:32.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:32 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:32.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:32 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:33.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:33 smithi135 ceph-mon[36424]: Deploying daemon mds.g on smithi121 2024-06-10T00:23:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:33 smithi135 ceph-mon[36424]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:33 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] up:boot 2024-06-10T00:23:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:33 smithi135 ceph-mon[36424]: fsmap 9 up:standby 2024-06-10T00:23:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:33 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-10T00:23:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:33 smithi121 ceph-mon[32378]: Deploying daemon mds.g on smithi121 2024-06-10T00:23:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:33 smithi121 ceph-mon[32378]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:33 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] up:boot 2024-06-10T00:23:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:33 smithi121 ceph-mon[32378]: fsmap 9 up:standby 2024-06-10T00:23:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:33 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-10T00:23:33.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:33 smithi193 ceph-mon[36716]: Deploying daemon mds.g on smithi121 2024-06-10T00:23:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:33 smithi193 ceph-mon[36716]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:33 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] up:boot 2024-06-10T00:23:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:33 smithi193 ceph-mon[36716]: fsmap 9 up:standby 2024-06-10T00:23:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:33 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-10T00:23:33.865 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:23:33 smithi121 systemd[1]: Starting Ceph mds.g for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:23:33.998 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:23:34.176 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:23:33 smithi121 podman[73803]: 2024-06-10 00:23:33.76894144 +0000 UTC m=+0.017237980 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:23:34.176 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:23:33 smithi121 podman[73803]: 2024-06-10 00:23:33.890576057 +0000 UTC m=+0.138872603 container create 6b9283c82a030c1b568a063f5c4ddb16956ca84ff5431a68d0dec3a5423af40a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240528) 2024-06-10T00:23:34.176 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:23:34 smithi121 podman[73803]: 2024-06-10 00:23:34.08527312 +0000 UTC m=+0.333569637 container init 6b9283c82a030c1b568a063f5c4ddb16956ca84ff5431a68d0dec3a5423af40a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-g, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-06-10T00:23:34.176 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:23:34 smithi121 podman[73803]: 2024-06-10 00:23:34.089224908 +0000 UTC m=+0.337521413 container start 6b9283c82a030c1b568a063f5c4ddb16956ca84ff5431a68d0dec3a5423af40a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-g, RELEASE=squid-3d89acc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528) 2024-06-10T00:23:34.176 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:23:34 smithi121 ceph-mds[73840]: -- 172.21.15.121:0/4051986934 <== mon.0 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5601d67d1a40 con 0x5601d67a5000 2024-06-10T00:23:34.176 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:23:34 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-g[73818]: starting mds.g at 2024-06-10T00:23:34.176 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:23:34 smithi121 bash[73803]: 6b9283c82a030c1b568a063f5c4ddb16956ca84ff5431a68d0dec3a5423af40a 2024-06-10T00:23:34.279 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:34.449 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:23:34 smithi121 systemd[1]: Started Ceph mds.g for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:35.401 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:35 smithi121 ceph-mon[32378]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:35.401 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:35 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.121:6838/4288408975,v1:172.21.15.121:6839/4288408975] up:boot 2024-06-10T00:23:35.401 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:35 smithi121 ceph-mon[32378]: fsmap 10 up:standby 2024-06-10T00:23:35.401 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:35 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-10T00:23:35.401 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:35 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:35.401 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:35 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:35.401 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:35 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:35.401 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:35 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:35.401 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:35 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:35.402 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:35 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:35 smithi135 ceph-mon[36424]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:35 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.121:6838/4288408975,v1:172.21.15.121:6839/4288408975] up:boot 2024-06-10T00:23:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:35 smithi135 ceph-mon[36424]: fsmap 10 up:standby 2024-06-10T00:23:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:35 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-10T00:23:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:35 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:35 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:35 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:35 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:35 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:35 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:35.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:35 smithi193 ceph-mon[36716]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:35 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.121:6838/4288408975,v1:172.21.15.121:6839/4288408975] up:boot 2024-06-10T00:23:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:35 smithi193 ceph-mon[36716]: fsmap 10 up:standby 2024-06-10T00:23:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:35 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-10T00:23:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:35 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:35 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:35 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:35 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:35 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:35 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:36.435 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:36 smithi121 ceph-mon[32378]: Deploying daemon mds.a on smithi121 2024-06-10T00:23:36.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:36 smithi135 ceph-mon[36424]: Deploying daemon mds.a on smithi121 2024-06-10T00:23:36.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.535+0000 7f017a713700 1 -- 172.21.15.121:0/146844900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01741073d0 msgr2=0x7f01741077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:36.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.535+0000 7f017a713700 1 --2- 172.21.15.121:0/146844900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01741073d0 0x7f01741077b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0164009990 tx=0x7f0164009ca0 comp rx=0 tx=0).stop 2024-06-10T00:23:36.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.535+0000 7f017a713700 1 -- 172.21.15.121:0/146844900 shutdown_connections 2024-06-10T00:23:36.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.535+0000 7f017a713700 1 --2- 172.21.15.121:0/146844900 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0174101d30 0x7f017410e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:36.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.535+0000 7f017a713700 1 --2- 172.21.15.121:0/146844900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01741073d0 0x7f01741077b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:36.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.535+0000 7f017a713700 1 --2- 172.21.15.121:0/146844900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0174101390 0x7f01741017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:36.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.535+0000 7f017a713700 1 -- 172.21.15.121:0/146844900 >> 172.21.15.121:0/146844900 conn(0x7f01740782c0 msgr2=0x7f01740786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:36.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.535+0000 7f017a713700 1 -- 172.21.15.121:0/146844900 shutdown_connections 2024-06-10T00:23:36.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.535+0000 7f017a713700 1 -- 172.21.15.121:0/146844900 wait complete. 2024-06-10T00:23:36.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.536+0000 7f017a713700 1 Processor -- start 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.536+0000 7f017a713700 1 -- start start 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f017a713700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0174101390 0x7f017419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f017a713700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0174101d30 0x7f017419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f017a713700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01741073d0 0x7f01741979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f017a713700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f01741116b0 con 0x7f01741073d0 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f017a713700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f0174111530 con 0x7f0174101390 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f017a713700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0174111830 con 0x7f0174101d30 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f0178cb0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01741073d0 0x7f01741979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f0178cb0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01741073d0 0x7f01741979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59172/0 (socket says 172.21.15.121:59172) 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f0178cb0700 1 -- 172.21.15.121:0/904400312 learned_addr learned my addr 172.21.15.121:0/904400312 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f01737fe700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0174101d30 0x7f017419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:36.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f0173fff700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0174101390 0x7f017419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f0178cb0700 1 -- 172.21.15.121:0/904400312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0174101d30 msgr2=0x7f017419dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f0178cb0700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0174101d30 0x7f017419dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f0178cb0700 1 -- 172.21.15.121:0/904400312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0174101390 msgr2=0x7f017419d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f0178cb0700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0174101390 0x7f017419d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f0178cb0700 1 -- 172.21.15.121:0/904400312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0174198130 con 0x7f01741073d0 2024-06-10T00:23:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f0178cb0700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01741073d0 0x7f01741979d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f016c00b970 tx=0x7f016c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f01717fa700 1 -- 172.21.15.121:0/904400312 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c019070 con 0x7f01741073d0 2024-06-10T00:23:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f01717fa700 1 -- 172.21.15.121:0/904400312 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f016c00ce10 con 0x7f01741073d0 2024-06-10T00:23:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f017a713700 1 -- 172.21.15.121:0/904400312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0174198420 con 0x7f01741073d0 2024-06-10T00:23:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.537+0000 7f01717fa700 1 -- 172.21.15.121:0/904400312 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c007370 con 0x7f01741073d0 2024-06-10T00:23:36.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.538+0000 7f017a713700 1 -- 172.21.15.121:0/904400312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01741a4690 con 0x7f01741073d0 2024-06-10T00:23:36.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.538+0000 7f017a713700 1 -- 172.21.15.121:0/904400312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0174069180 con 0x7f01741073d0 2024-06-10T00:23:36.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.540+0000 7f01717fa700 1 -- 172.21.15.121:0/904400312 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f016c01c070 con 0x7f01741073d0 2024-06-10T00:23:36.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.540+0000 7f01717fa700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f01600a2ca0 0x7f01600a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:36.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.541+0000 7f0173fff700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f01600a2ca0 0x7f01600a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:36.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.541+0000 7f01717fa700 1 -- 172.21.15.121:0/904400312 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f016c0cdea0 con 0x7f01741073d0 2024-06-10T00:23:36.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.541+0000 7f0173fff700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f01600a2ca0 0x7f01600a5160 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0164006010 tx=0x7f016400b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:36.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.545+0000 7f01717fa700 1 -- 172.21.15.121:0/904400312 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f016c09b2a0 con 0x7f01741073d0 2024-06-10T00:23:36.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:36 smithi193 ceph-mon[36716]: Deploying daemon mds.a on smithi121 2024-06-10T00:23:36.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.789+0000 7f017a713700 1 -- 172.21.15.121:0/904400312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f017402dca0 con 0x7f01741073d0 2024-06-10T00:23:36.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.790+0000 7f01717fa700 1 -- 172.21.15.121:0/904400312 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8839 (secure 0 0 0) 0x7f016c09aba0 con 0x7f01741073d0 2024-06-10T00:23:36.791 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:23:36.791 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-06-10T00:23:36.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 -- 172.21.15.121:0/904400312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f01600a2ca0 msgr2=0x7f01600a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:36.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f01600a2ca0 0x7f01600a5160 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0164006010 tx=0x7f016400b540 comp rx=0 tx=0).stop 2024-06-10T00:23:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 -- 172.21.15.121:0/904400312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01741073d0 msgr2=0x7f01741979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01741073d0 0x7f01741979d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f016c00b970 tx=0x7f016c00bc80 comp rx=0 tx=0).stop 2024-06-10T00:23:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 -- 172.21.15.121:0/904400312 shutdown_connections 2024-06-10T00:23:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f01600a2ca0 0x7f01600a5160 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01741073d0 0x7f01741979d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0174101390 0x7f017419d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 --2- 172.21.15.121:0/904400312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0174101d30 0x7f017419dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 -- 172.21.15.121:0/904400312 >> 172.21.15.121:0/904400312 conn(0x7f01740782c0 msgr2=0x7f0174100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 -- 172.21.15.121:0/904400312 shutdown_connections 2024-06-10T00:23:36.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:36.792+0000 7f017a713700 1 -- 172.21.15.121:0/904400312 wait complete. 2024-06-10T00:23:36.794 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 11 2024-06-10T00:23:37.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:37 smithi121 ceph-mon[32378]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:37.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:37 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/904400312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:37.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:37 smithi135 ceph-mon[36424]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:37.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:37 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/904400312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:37.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:37 smithi193 ceph-mon[36716]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:37.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:37 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/904400312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:38.180 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:23:38 smithi121 podman[74260]: 2024-06-10 00:23:38.064302669 +0000 UTC m=+0.017283750 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:23:38.180 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:23:38 smithi121 podman[74260]: 2024-06-10 00:23:38.181160904 +0000 UTC m=+0.134141978 container create ab8ef27468e7502dfd515b707a0d165218b5587d540de7df7674fc067626acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-a, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-10T00:23:38.442 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:23:38 smithi121 podman[74260]: 2024-06-10 00:23:38.417505756 +0000 UTC m=+0.370486837 container init ab8ef27468e7502dfd515b707a0d165218b5587d540de7df7674fc067626acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:23:38.442 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:23:38 smithi121 podman[74260]: 2024-06-10 00:23:38.420134882 +0000 UTC m=+0.373115964 container start ab8ef27468e7502dfd515b707a0d165218b5587d540de7df7674fc067626acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-06-10T00:23:38.699 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:23:38 smithi121 ceph-mds[74279]: -- 172.21.15.121:0/1204300948 <== mon.0 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5569da729a40 con 0x5569da6fcc00 2024-06-10T00:23:38.699 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:23:38 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-a[74275]: starting mds.a at 2024-06-10T00:23:38.699 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:23:38 smithi121 bash[74260]: ab8ef27468e7502dfd515b707a0d165218b5587d540de7df7674fc067626acfb 2024-06-10T00:23:38.699 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:23:38 smithi121 systemd[1]: Started Ceph mds.a for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:39.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:39 smithi121 ceph-mon[32378]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:39.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:39.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:39.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:39.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:39.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:39.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:39 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:39.462 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:39 smithi135 ceph-mon[36424]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:39.462 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:39.462 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:39.462 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:39.462 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:39.462 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:39.462 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:39 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:39.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:39 smithi193 ceph-mon[36716]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-06-10T00:23:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-06-10T00:23:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:39 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:39.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:23:40.197 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:40.319 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:40 smithi121 ceph-mon[32378]: Deploying daemon mds.b on smithi135 2024-06-10T00:23:40.319 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:40 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.121:6840/691733071,v1:172.21.15.121:6841/691733071] up:boot 2024-06-10T00:23:40.319 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:40 smithi121 ceph-mon[32378]: fsmap 11 up:standby 2024-06-10T00:23:40.320 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:40 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-10T00:23:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:40 smithi135 ceph-mon[36424]: Deploying daemon mds.b on smithi135 2024-06-10T00:23:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:40 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.121:6840/691733071,v1:172.21.15.121:6841/691733071] up:boot 2024-06-10T00:23:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:40 smithi135 ceph-mon[36424]: fsmap 11 up:standby 2024-06-10T00:23:40.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:40 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-10T00:23:40.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:40 smithi193 ceph-mon[36716]: Deploying daemon mds.b on smithi135 2024-06-10T00:23:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:40 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.121:6840/691733071,v1:172.21.15.121:6841/691733071] up:boot 2024-06-10T00:23:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:40 smithi193 ceph-mon[36716]: fsmap 11 up:standby 2024-06-10T00:23:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:40 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-10T00:23:40.986 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:23:40 smithi135 systemd[1]: Starting Ceph mds.b for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:23:41.248 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:41 smithi135 ceph-mon[36424]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:41.249 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:23:40 smithi135 podman[66165]: 2024-06-10 00:23:40.987182265 +0000 UTC m=+0.107445856 container create 13bb6b83c0c3a1f53087af06776b31458ef1da2d5c0ad9740e3b4e890536c4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-10T00:23:41.249 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:23:40 smithi135 podman[66165]: 2024-06-10 00:23:40.897092794 +0000 UTC m=+0.017356390 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:23:41.249 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:23:41 smithi135 podman[66165]: 2024-06-10 00:23:41.223392717 +0000 UTC m=+0.343656307 container init 13bb6b83c0c3a1f53087af06776b31458ef1da2d5c0ad9740e3b4e890536c4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-06-10T00:23:41.249 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:23:41 smithi135 podman[66165]: 2024-06-10 00:23:41.226022285 +0000 UTC m=+0.346285880 container start 13bb6b83c0c3a1f53087af06776b31458ef1da2d5c0ad9740e3b4e890536c4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240528) 2024-06-10T00:23:41.249 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:23:41 smithi135 ceph-mds[66184]: -- 172.21.15.135:0/4054070654 <== mon.2 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55abd6681a40 con 0x55abd6655000 2024-06-10T00:23:41.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:41 smithi121 ceph-mon[32378]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:41.507 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:23:41 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b[66180]: starting mds.b at 2024-06-10T00:23:41.507 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:23:41 smithi135 bash[66165]: 13bb6b83c0c3a1f53087af06776b31458ef1da2d5c0ad9740e3b4e890536c4a1 2024-06-10T00:23:41.508 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:23:41 smithi135 systemd[1]: Started Ceph mds.b for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:23:41.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:41 smithi193 ceph-mon[36716]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:41.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.820+0000 7f91f8313700 1 -- 172.21.15.121:0/4022380100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f0107870 msgr2=0x7f91f010f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:41.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.820+0000 7f91f8313700 1 --2- 172.21.15.121:0/4022380100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f0107870 0x7f91f010f3f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f91e800b0d0 tx=0x7f91e800b3e0 comp rx=0 tx=0).stop 2024-06-10T00:23:41.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.820+0000 7f91f8313700 1 -- 172.21.15.121:0/4022380100 shutdown_connections 2024-06-10T00:23:41.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.820+0000 7f91f8313700 1 --2- 172.21.15.121:0/4022380100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f0107870 0x7f91f010f3f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:41.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.820+0000 7f91f8313700 1 --2- 172.21.15.121:0/4022380100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91f01027a0 0x7f91f0102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.820+0000 7f91f8313700 1 --2- 172.21.15.121:0/4022380100 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f0103150 0x7f91f0107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.820+0000 7f91f8313700 1 -- 172.21.15.121:0/4022380100 >> 172.21.15.121:0/4022380100 conn(0x7f91f0079520 msgr2=0x7f91f0079930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.820+0000 7f91f8313700 1 -- 172.21.15.121:0/4022380100 shutdown_connections 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.820+0000 7f91f8313700 1 -- 172.21.15.121:0/4022380100 wait complete. 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f8313700 1 Processor -- start 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f8313700 1 -- start start 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f8313700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f01027a0 0x7f91f019e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f8313700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91f0103150 0x7f91f019ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f8313700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f0107870 0x7f91f01989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f8313700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f91f01127b0 con 0x7f91f0107870 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f8313700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f91f0112630 con 0x7f91f01027a0 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f8313700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91f0112930 con 0x7f91f0103150 2024-06-10T00:23:41.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f68b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f0107870 0x7f91f01989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f60af700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f01027a0 0x7f91f019e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f58ae700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91f0103150 0x7f91f019ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f60af700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f01027a0 0x7f91f019e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:52180/0 (socket says 172.21.15.121:52180) 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f60af700 1 -- 172.21.15.121:0/300070961 learned_addr learned my addr 172.21.15.121:0/300070961 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.821+0000 7f91f68b0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f0107870 0x7f91f01989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38404/0 (socket says 172.21.15.121:38404) 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91f60af700 1 -- 172.21.15.121:0/300070961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91f0103150 msgr2=0x7f91f019ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91f60af700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91f0103150 0x7f91f019ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91f60af700 1 -- 172.21.15.121:0/300070961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f0107870 msgr2=0x7f91f01989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91f60af700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f0107870 0x7f91f01989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91f60af700 1 -- 172.21.15.121:0/300070961 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f0199110 con 0x7f91f01027a0 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91f58ae700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91f0103150 0x7f91f019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91f60af700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f01027a0 0x7f91f019e7a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00e9c0 tx=0x7f91ec00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:41.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91e77fe700 1 -- 172.21.15.121:0/300070961 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec00cbd0 con 0x7f91f01027a0 2024-06-10T00:23:41.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91f8313700 1 -- 172.21.15.121:0/300070961 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f0199400 con 0x7f91f01027a0 2024-06-10T00:23:41.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91e77fe700 1 -- 172.21.15.121:0/300070961 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91ec00ee60 con 0x7f91f01027a0 2024-06-10T00:23:41.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91f8313700 1 -- 172.21.15.121:0/300070961 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f01a5670 con 0x7f91f01027a0 2024-06-10T00:23:41.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.822+0000 7f91e77fe700 1 -- 172.21.15.121:0/300070961 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec005650 con 0x7f91f01027a0 2024-06-10T00:23:41.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.823+0000 7f91f8313700 1 -- 172.21.15.121:0/300070961 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91f0069180 con 0x7f91f01027a0 2024-06-10T00:23:41.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.824+0000 7f91e77fe700 1 -- 172.21.15.121:0/300070961 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f91ec045bd0 con 0x7f91f01027a0 2024-06-10T00:23:41.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.825+0000 7f91e77fe700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f91dc0a2c20 0x7f91dc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:41.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.825+0000 7f91f58ae700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f91dc0a2c20 0x7f91dc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:41.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.825+0000 7f91e77fe700 1 -- 172.21.15.121:0/300070961 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f91ec0cf110 con 0x7f91f01027a0 2024-06-10T00:23:41.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.825+0000 7f91f58ae700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f91dc0a2c20 0x7f91dc0a50e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f91f0108c30 tx=0x7f91e0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:41.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:41.829+0000 7f91e77fe700 1 -- 172.21.15.121:0/300070961 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f91ec09c4b0 con 0x7f91f01027a0 2024-06-10T00:23:42.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.082+0000 7f91f8313700 1 -- 172.21.15.121:0/300070961 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91f0199ec0 con 0x7f91f01027a0 2024-06-10T00:23:42.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.085+0000 7f91e77fe700 1 -- 172.21.15.121:0/300070961 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9665 (secure 0 0 0) 0x7f91f0199ec0 con 0x7f91f01027a0 2024-06-10T00:23:42.087 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:23:42.087 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-06-10T00:23:42.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 -- 172.21.15.121:0/300070961 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f91dc0a2c20 msgr2=0x7f91dc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:42.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f91dc0a2c20 0x7f91dc0a50e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f91f0108c30 tx=0x7f91e0009260 comp rx=0 tx=0).stop 2024-06-10T00:23:42.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 -- 172.21.15.121:0/300070961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f01027a0 msgr2=0x7f91f019e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:42.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f01027a0 0x7f91f019e7a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f91ec00e9c0 tx=0x7f91ec00ecd0 comp rx=0 tx=0).stop 2024-06-10T00:23:42.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 -- 172.21.15.121:0/300070961 shutdown_connections 2024-06-10T00:23:42.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f91dc0a2c20 0x7f91dc0a50e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:42.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91f0107870 0x7f91f01989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:42.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91f01027a0 0x7f91f019e7a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:42.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 --2- 172.21.15.121:0/300070961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91f0103150 0x7f91f019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:42.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 -- 172.21.15.121:0/300070961 >> 172.21.15.121:0/300070961 conn(0x7f91f0079520 msgr2=0x7f91f01004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:42.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 -- 172.21.15.121:0/300070961 shutdown_connections 2024-06-10T00:23:42.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:42.088+0000 7f91f8313700 1 -- 172.21.15.121:0/300070961 wait complete. 2024-06-10T00:23:42.090 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 12 2024-06-10T00:23:42.676 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.676 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.676 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.676 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.677 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.677 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:23:42.677 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.677 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/300070961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:42.677 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:42.677 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] up:boot 2024-06-10T00:23:42.677 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: fsmap 12 up:standby 2024-06-10T00:23:42.677 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-10T00:23:42.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:23:42.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/300070961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:42.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:42.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] up:boot 2024-06-10T00:23:42.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: fsmap 12 up:standby 2024-06-10T00:23:42.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-10T00:23:42.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:23:42.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:42.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/300070961' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:42.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:42.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] up:boot 2024-06-10T00:23:42.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: fsmap 12 up:standby 2024-06-10T00:23:42.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-10T00:23:44.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:44.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:23:44.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:23:44.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:23:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:23:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:23:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:23:44.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:23:45.444 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:45.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:45 smithi121 ceph-mon[32378]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:45 smithi135 ceph-mon[36424]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:45.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:45 smithi193 ceph-mon[36716]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:46.861 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:46.861 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:46.861 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:46.861 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:46.862 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:46 smithi121 ceph-mon[32378]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:47.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:47.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:47.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:47.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:47.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:46 smithi135 ceph-mon[36424]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:47.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.111+0000 7fd867e77700 1 -- 172.21.15.121:0/466476395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd860101d40 msgr2=0x7fd86010e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:47.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.111+0000 7fd867e77700 1 --2- 172.21.15.121:0/466476395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd860101d40 0x7fd86010e230 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd85800b0a0 tx=0x7fd85800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:23:47.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.111+0000 7fd867e77700 1 -- 172.21.15.121:0/466476395 shutdown_connections 2024-06-10T00:23:47.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.111+0000 7fd867e77700 1 --2- 172.21.15.121:0/466476395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd860101d40 0x7fd86010e230 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.111+0000 7fd867e77700 1 --2- 172.21.15.121:0/466476395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8601073f0 0x7fd8601077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.111+0000 7fd867e77700 1 --2- 172.21.15.121:0/466476395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd860101380 0x7fd860101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.111+0000 7fd867e77700 1 -- 172.21.15.121:0/466476395 >> 172.21.15.121:0/466476395 conn(0x7fd860078300 msgr2=0x7fd860078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.111+0000 7fd867e77700 1 -- 172.21.15.121:0/466476395 shutdown_connections 2024-06-10T00:23:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.111+0000 7fd867e77700 1 -- 172.21.15.121:0/466476395 wait complete. 2024-06-10T00:23:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd867e77700 1 Processor -- start 2024-06-10T00:23:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd867e77700 1 -- start start 2024-06-10T00:23:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd867e77700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd860101380 0x7fd860103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd867e77700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd860101d40 0x7fd8601035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd867e77700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8601073f0 0x7fd860103ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd867e77700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd860111660 con 0x7fd8601073f0 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd867e77700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd8601114e0 con 0x7fd860101380 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd867e77700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8601117e0 con 0x7fd860101d40 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd866414700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8601073f0 0x7fd860103ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd866414700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8601073f0 0x7fd860103ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38418/0 (socket says 172.21.15.121:38418) 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd866414700 1 -- 172.21.15.121:0/968762799 learned_addr learned my addr 172.21.15.121:0/968762799 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.112+0000 7fd865c13700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd860101380 0x7fd860103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd865412700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd860101d40 0x7fd8601035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd866414700 1 -- 172.21.15.121:0/968762799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd860101d40 msgr2=0x7fd8601035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd866414700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd860101d40 0x7fd8601035a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd866414700 1 -- 172.21.15.121:0/968762799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd860101380 msgr2=0x7fd860103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd866414700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd860101380 0x7fd860103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:47.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd866414700 1 -- 172.21.15.121:0/968762799 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8601a70d0 con 0x7fd8601073f0 2024-06-10T00:23:47.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd865412700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd860101d40 0x7fd8601035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:23:47.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd866414700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8601073f0 0x7fd860103ae0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd85800b780 tx=0x7fd858004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:47.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd856ffd700 1 -- 172.21.15.121:0/968762799 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd858021070 con 0x7fd8601073f0 2024-06-10T00:23:47.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd856ffd700 1 -- 172.21.15.121:0/968762799 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd858007b80 con 0x7fd8601073f0 2024-06-10T00:23:47.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd867e77700 1 -- 172.21.15.121:0/968762799 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8601a7360 con 0x7fd8601073f0 2024-06-10T00:23:47.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd856ffd700 1 -- 172.21.15.121:0/968762799 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8580052a0 con 0x7fd8601073f0 2024-06-10T00:23:47.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.113+0000 7fd867e77700 1 -- 172.21.15.121:0/968762799 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8601a7810 con 0x7fd8601073f0 2024-06-10T00:23:47.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.114+0000 7fd867e77700 1 -- 172.21.15.121:0/968762799 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd860069180 con 0x7fd8601073f0 2024-06-10T00:23:47.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.116+0000 7fd856ffd700 1 -- 172.21.15.121:0/968762799 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd858024070 con 0x7fd8601073f0 2024-06-10T00:23:47.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.116+0000 7fd856ffd700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd84c0a2c20 0x7fd84c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:47.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.116+0000 7fd865c13700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd84c0a2c20 0x7fd84c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:47.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.117+0000 7fd856ffd700 1 -- 172.21.15.121:0/968762799 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd85800f020 con 0x7fd8601073f0 2024-06-10T00:23:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.117+0000 7fd865c13700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd84c0a2c20 0x7fd84c0a50e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009be0 tx=0x7fd85c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:47.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.120+0000 7fd856ffd700 1 -- 172.21.15.121:0/968762799 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd85802b020 con 0x7fd8601073f0 2024-06-10T00:23:47.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:47.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:46 smithi193 ceph-mon[36716]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:47.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.386+0000 7fd867e77700 1 -- 172.21.15.121:0/968762799 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8601a7b30 con 0x7fd8601073f0 2024-06-10T00:23:47.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.387+0000 7fd856ffd700 1 -- 172.21.15.121:0/968762799 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10494 (secure 0 0 0) 0x7fd858007d30 con 0x7fd8601073f0 2024-06-10T00:23:47.386 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:23:47.387 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24479,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-06-10T00:23:47.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 -- 172.21.15.121:0/968762799 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd84c0a2c20 msgr2=0x7fd84c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:47.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd84c0a2c20 0x7fd84c0a50e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009be0 tx=0x7fd85c009260 comp rx=0 tx=0).stop 2024-06-10T00:23:47.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 -- 172.21.15.121:0/968762799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8601073f0 msgr2=0x7fd860103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:47.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8601073f0 0x7fd860103ae0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd85800b780 tx=0x7fd858004d00 comp rx=0 tx=0).stop 2024-06-10T00:23:47.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 -- 172.21.15.121:0/968762799 shutdown_connections 2024-06-10T00:23:47.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd84c0a2c20 0x7fd84c0a50e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:47.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8601073f0 0x7fd860103ae0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:47.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd860101380 0x7fd860103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:47.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 --2- 172.21.15.121:0/968762799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd860101d40 0x7fd8601035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:47.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 -- 172.21.15.121:0/968762799 >> 172.21.15.121:0/968762799 conn(0x7fd860078300 msgr2=0x7fd8601005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:47.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 -- 172.21.15.121:0/968762799 shutdown_connections 2024-06-10T00:23:47.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:47.389+0000 7fd867e77700 1 -- 172.21.15.121:0/968762799 wait complete. 2024-06-10T00:23:47.390 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 13 2024-06-10T00:23:48.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-06-10T00:23:48.357 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:48.612 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:48 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:48.612 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:48 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:48.612 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:48 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/968762799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:48.612 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:48 smithi121 ceph-mon[32378]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:48.612 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:48 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:48.612 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:48 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:23:48.612 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:48 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:48.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:48 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:48.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:48 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:48.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:48 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/968762799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:48.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:48 smithi193 ceph-mon[36716]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:48.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:48 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:48.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:48 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:23:48.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:48 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:48.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:48 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:48.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:48 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:48.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:48 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/968762799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:23:48.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:48 smithi135 ceph-mon[36424]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:48.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:48 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:23:48.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:48 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:23:48.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:48 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:23:49.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.993+0000 7fc08b3b1700 1 -- 172.21.15.121:0/249668640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc084101fd0 msgr2=0x7fc084105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:49.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.993+0000 7fc08b3b1700 1 --2- 172.21.15.121:0/249668640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc084101fd0 0x7fc084105fd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc0740099c0 tx=0x7fc074009cd0 comp rx=0 tx=0).stop 2024-06-10T00:23:49.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.993+0000 7fc08b3b1700 1 -- 172.21.15.121:0/249668640 shutdown_connections 2024-06-10T00:23:49.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.993+0000 7fc08b3b1700 1 --2- 172.21.15.121:0/249668640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0841066b0 0x7fc08410e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:49.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.993+0000 7fc08b3b1700 1 --2- 172.21.15.121:0/249668640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc084101620 0x7fc084101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:49.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.993+0000 7fc08b3b1700 1 --2- 172.21.15.121:0/249668640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc084101fd0 0x7fc084105fd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:49.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.993+0000 7fc08b3b1700 1 -- 172.21.15.121:0/249668640 >> 172.21.15.121:0/249668640 conn(0x7fc084078340 msgr2=0x7fc084078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:49.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.993+0000 7fc08b3b1700 1 -- 172.21.15.121:0/249668640 shutdown_connections 2024-06-10T00:23:49.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.993+0000 7fc08b3b1700 1 -- 172.21.15.121:0/249668640 wait complete. 2024-06-10T00:23:49.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08b3b1700 1 Processor -- start 2024-06-10T00:23:49.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08b3b1700 1 -- start start 2024-06-10T00:23:49.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08b3b1700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc084101620 0x7fc08419d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:49.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08b3b1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc084101fd0 0x7fc08419dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:49.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08b3b1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0841066b0 0x7fc084197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08b3b1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc0841116f0 con 0x7fc084101fd0 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08b3b1700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc084111570 con 0x7fc084101620 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08b3b1700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc084111870 con 0x7fc0841066b0 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08894c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc084101fd0 0x7fc08419dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08894c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc084101fd0 0x7fc08419dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35270/0 (socket says 172.21.15.121:35270) 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08894c700 1 -- 172.21.15.121:0/2891214312 learned_addr learned my addr 172.21.15.121:0/2891214312 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.994+0000 7fc08914d700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc084101620 0x7fc08419d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc08994e700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0841066b0 0x7fc084197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc08894c700 1 -- 172.21.15.121:0/2891214312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0841066b0 msgr2=0x7fc084197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc08894c700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0841066b0 0x7fc084197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc08894c700 1 -- 172.21.15.121:0/2891214312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc084101620 msgr2=0x7fc08419d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:49.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc08894c700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc084101620 0x7fc08419d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:49.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc08894c700 1 -- 172.21.15.121:0/2891214312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0841980f0 con 0x7fc084101fd0 2024-06-10T00:23:49.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc08994e700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0841066b0 0x7fc084197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:23:49.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc08914d700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc084101620 0x7fc08419d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:23:49.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc08894c700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc084101fd0 0x7fc08419dcc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc0740059d0 tx=0x7fc074005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:49.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc07a7fc700 1 -- 172.21.15.121:0/2891214312 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074027070 con 0x7fc084101fd0 2024-06-10T00:23:49.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc07a7fc700 1 -- 172.21.15.121:0/2891214312 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0740040d0 con 0x7fc084101fd0 2024-06-10T00:23:49.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc08b3b1700 1 -- 172.21.15.121:0/2891214312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc084198380 con 0x7fc084101fd0 2024-06-10T00:23:49.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc07a7fc700 1 -- 172.21.15.121:0/2891214312 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc07400f460 con 0x7fc084101fd0 2024-06-10T00:23:49.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.995+0000 7fc08b3b1700 1 -- 172.21.15.121:0/2891214312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc08406abc0 con 0x7fc084101fd0 2024-06-10T00:23:49.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.996+0000 7fc08b3b1700 1 -- 172.21.15.121:0/2891214312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc084102ab0 con 0x7fc084101fd0 2024-06-10T00:23:49.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.997+0000 7fc07a7fc700 1 -- 172.21.15.121:0/2891214312 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc07402a070 con 0x7fc084101fd0 2024-06-10T00:23:49.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.998+0000 7fc07a7fc700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc0700a2c20 0x7fc0700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:49.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.998+0000 7fc08914d700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc0700a2c20 0x7fc0700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:49.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.998+0000 7fc07a7fc700 1 -- 172.21.15.121:0/2891214312 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc0740d0220 con 0x7fc084101fd0 2024-06-10T00:23:50.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:49.998+0000 7fc08914d700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc0700a2c20 0x7fc0700a50e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc084078da0 tx=0x7fc08000a680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:50.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.001+0000 7fc07a7fc700 1 -- 172.21.15.121:0/2891214312 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc074031020 con 0x7fc084101fd0 2024-06-10T00:23:50.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.209+0000 7fc08b3b1700 1 -- 172.21.15.121:0/2891214312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc084069180 con 0x7fc084101fd0 2024-06-10T00:23:50.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.210+0000 7fc07a7fc700 1 -- 172.21.15.121:0/2891214312 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7fc084069180 con 0x7fc084101fd0 2024-06-10T00:23:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:23:50.210 INFO:teuthology.orchestra.run.smithi121.stdout:[ 2024-06-10T00:23:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:23:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: "poolnum": 1, 2024-06-10T00:23:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: "poolname": ".mgr" 2024-06-10T00:23:50.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:23:50.210 INFO:teuthology.orchestra.run.smithi121.stdout:] 2024-06-10T00:23:50.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 -- 172.21.15.121:0/2891214312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc0700a2c20 msgr2=0x7fc0700a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:50.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc0700a2c20 0x7fc0700a50e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc084078da0 tx=0x7fc08000a680 comp rx=0 tx=0).stop 2024-06-10T00:23:50.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 -- 172.21.15.121:0/2891214312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc084101fd0 msgr2=0x7fc08419dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:50.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc084101fd0 0x7fc08419dcc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc0740059d0 tx=0x7fc074005640 comp rx=0 tx=0).stop 2024-06-10T00:23:50.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 -- 172.21.15.121:0/2891214312 shutdown_connections 2024-06-10T00:23:50.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc0700a2c20 0x7fc0700a50e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:50.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0841066b0 0x7fc084197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:50.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc084101620 0x7fc08419d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:50.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 --2- 172.21.15.121:0/2891214312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc084101fd0 0x7fc08419dcc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:50.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 -- 172.21.15.121:0/2891214312 >> 172.21.15.121:0/2891214312 conn(0x7fc084078340 msgr2=0x7fc0840ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:50.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 -- 172.21.15.121:0/2891214312 shutdown_connections 2024-06-10T00:23:50.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:50.212+0000 7fc08b3b1700 1 -- 172.21.15.121:0/2891214312 wait complete. 2024-06-10T00:23:50.987 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-10T00:23:50.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-10T00:23:51.232 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:51.334 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:51 smithi193 ceph-mon[36716]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:51.334 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:51 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2891214312' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-06-10T00:23:51.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:51 smithi121 ceph-mon[32378]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:51.347 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:51 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2891214312' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-06-10T00:23:51.461 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:51 smithi135 ceph-mon[36424]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:51.461 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:51 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2891214312' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-06-10T00:23:52.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.915+0000 7f3a36478700 1 -- 172.21.15.121:0/1620466268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a30101440 msgr2=0x7f3a3010f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:52.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.915+0000 7f3a36478700 1 --2- 172.21.15.121:0/1620466268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a30101440 0x7f3a3010f330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800b0a0 tx=0x7f3a2800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:23:52.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.915+0000 7f3a36478700 1 -- 172.21.15.121:0/1620466268 shutdown_connections 2024-06-10T00:23:52.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.915+0000 7f3a36478700 1 --2- 172.21.15.121:0/1620466268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a30101440 0x7f3a3010f330 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:52.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.915+0000 7f3a36478700 1 --2- 172.21.15.121:0/1620466268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a30108510 0x7f3a301088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:52.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.915+0000 7f3a36478700 1 --2- 172.21.15.121:0/1620466268 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a30100a80 0x7f3a30100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:52.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.915+0000 7f3a36478700 1 -- 172.21.15.121:0/1620466268 >> 172.21.15.121:0/1620466268 conn(0x7f3a30077a00 msgr2=0x7f3a300fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:52.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.915+0000 7f3a36478700 1 -- 172.21.15.121:0/1620466268 shutdown_connections 2024-06-10T00:23:52.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.915+0000 7f3a36478700 1 -- 172.21.15.121:0/1620466268 wait complete. 2024-06-10T00:23:52.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a36478700 1 Processor -- start 2024-06-10T00:23:52.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a36478700 1 -- start start 2024-06-10T00:23:52.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a36478700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a30100a80 0x7f3a3019e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:52.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a36478700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a30101440 0x7f3a3019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:52.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a36478700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a30108510 0x7f3a30198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:52.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a36478700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3a30112790 con 0x7f3a30108510 2024-06-10T00:23:52.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a36478700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f3a30112610 con 0x7f3a30101440 2024-06-10T00:23:52.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a36478700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3a30112910 con 0x7f3a30100a80 2024-06-10T00:23:52.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a34a15700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a30108510 0x7f3a30198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:52.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a34a15700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a30108510 0x7f3a30198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35294/0 (socket says 172.21.15.121:35294) 2024-06-10T00:23:52.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a34a15700 1 -- 172.21.15.121:0/1557708220 learned_addr learned my addr 172.21.15.121:0/1557708220 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:52.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a27fff700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a30101440 0x7f3a3019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:52.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a2ffff700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a30100a80 0x7f3a3019e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:52.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a34a15700 1 -- 172.21.15.121:0/1557708220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a30100a80 msgr2=0x7f3a3019e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:52.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a34a15700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a30100a80 0x7f3a3019e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:52.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a34a15700 1 -- 172.21.15.121:0/1557708220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a30101440 msgr2=0x7f3a3019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:52.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a34a15700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a30101440 0x7f3a3019edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:52.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a34a15700 1 -- 172.21.15.121:0/1557708220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a301991d0 con 0x7f3a30108510 2024-06-10T00:23:52.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.916+0000 7f3a2ffff700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a30100a80 0x7f3a3019e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:23:52.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.917+0000 7f3a34a15700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a30108510 0x7f3a30198a70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800b780 tx=0x7f3a28004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:52.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.917+0000 7f3a2dffb700 1 -- 172.21.15.121:0/1557708220 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28021070 con 0x7f3a30108510 2024-06-10T00:23:52.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.917+0000 7f3a36478700 1 -- 172.21.15.121:0/1557708220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a30199460 con 0x7f3a30108510 2024-06-10T00:23:52.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.917+0000 7f3a2dffb700 1 -- 172.21.15.121:0/1557708220 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a28007c80 con 0x7f3a30108510 2024-06-10T00:23:52.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.917+0000 7f3a2dffb700 1 -- 172.21.15.121:0/1557708220 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28005400 con 0x7f3a30108510 2024-06-10T00:23:52.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.917+0000 7f3a36478700 1 -- 172.21.15.121:0/1557708220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a301a5780 con 0x7f3a30108510 2024-06-10T00:23:52.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.918+0000 7f3a36478700 1 -- 172.21.15.121:0/1557708220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3010a9f0 con 0x7f3a30108510 2024-06-10T00:23:52.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.919+0000 7f3a2dffb700 1 -- 172.21.15.121:0/1557708220 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3a28024070 con 0x7f3a30108510 2024-06-10T00:23:52.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.919+0000 7f3a2dffb700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a1c0a2c20 0x7f3a1c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:52.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.920+0000 7f3a2ffff700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a1c0a2c20 0x7f3a1c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:52.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.920+0000 7f3a2dffb700 1 -- 172.21.15.121:0/1557708220 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f3a2800f020 con 0x7f3a30108510 2024-06-10T00:23:52.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.920+0000 7f3a2ffff700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a1c0a2c20 0x7f3a1c0a50e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3a20005fd0 tx=0x7f3a20005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:52.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:52.924+0000 7f3a2dffb700 1 -- 172.21.15.121:0/1557708220 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3a2802b020 con 0x7f3a30108510 2024-06-10T00:23:53.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.122+0000 7f3a36478700 1 -- 172.21.15.121:0/1557708220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3a30199fa0 con 0x7f3a30108510 2024-06-10T00:23:53.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.408+0000 7f3a2dffb700 1 -- 172.21.15.121:0/1557708220 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f3a2809d640 con 0x7f3a30108510 2024-06-10T00:23:53.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 -- 172.21.15.121:0/1557708220 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a1c0a2c20 msgr2=0x7f3a1c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:53.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a1c0a2c20 0x7f3a1c0a50e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3a20005fd0 tx=0x7f3a20005d60 comp rx=0 tx=0).stop 2024-06-10T00:23:53.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 -- 172.21.15.121:0/1557708220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a30108510 msgr2=0x7f3a30198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:53.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a30108510 0x7f3a30198a70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800b780 tx=0x7f3a28004db0 comp rx=0 tx=0).stop 2024-06-10T00:23:53.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 -- 172.21.15.121:0/1557708220 shutdown_connections 2024-06-10T00:23:53.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a1c0a2c20 0x7f3a1c0a50e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:53.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a30108510 0x7f3a30198a70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:53.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a30100a80 0x7f3a3019e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:53.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 --2- 172.21.15.121:0/1557708220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a30101440 0x7f3a3019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:53.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 -- 172.21.15.121:0/1557708220 >> 172.21.15.121:0/1557708220 conn(0x7f3a30077a00 msgr2=0x7f3a300ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:53.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 -- 172.21.15.121:0/1557708220 shutdown_connections 2024-06-10T00:23:53.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:53.410+0000 7f3a36478700 1 -- 172.21.15.121:0/1557708220 wait complete. 2024-06-10T00:23:53.411 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'cephfs_metadata' created 2024-06-10T00:23:53.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:53 smithi193 ceph-mon[36716]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:53.698 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:53 smithi121 ceph-mon[32378]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:53.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:53 smithi135 ceph-mon[36424]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:54.091 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-06-10T00:23:54.335 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:54.611 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:54 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1557708220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-06-10T00:23:54.612 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:54 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1557708220' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-06-10T00:23:54.612 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:54 smithi121 ceph-mon[32378]: osdmap e83: 12 total, 12 up, 12 in 2024-06-10T00:23:54.612 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:54 smithi121 ceph-mon[32378]: pgmap v289: 33 pgs: 9 creating+peering, 1 active+clean, 23 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:54.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:54 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1557708220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-06-10T00:23:54.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:54 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1557708220' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-06-10T00:23:54.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:54 smithi193 ceph-mon[36716]: osdmap e83: 12 total, 12 up, 12 in 2024-06-10T00:23:54.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:54 smithi193 ceph-mon[36716]: pgmap v289: 33 pgs: 9 creating+peering, 1 active+clean, 23 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:54.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:54 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1557708220' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-06-10T00:23:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:54 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1557708220' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-06-10T00:23:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:54 smithi135 ceph-mon[36424]: osdmap e83: 12 total, 12 up, 12 in 2024-06-10T00:23:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:54 smithi135 ceph-mon[36424]: pgmap v289: 33 pgs: 9 creating+peering, 1 active+clean, 23 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:55.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:55 smithi121 ceph-mon[32378]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:23:55.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:55 smithi121 ceph-mon[32378]: osdmap e84: 12 total, 12 up, 12 in 2024-06-10T00:23:55.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:55 smithi193 ceph-mon[36716]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:23:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:55 smithi193 ceph-mon[36716]: osdmap e84: 12 total, 12 up, 12 in 2024-06-10T00:23:55.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:55 smithi135 ceph-mon[36424]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:23:55.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:55 smithi135 ceph-mon[36424]: osdmap e84: 12 total, 12 up, 12 in 2024-06-10T00:23:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.035+0000 7fa39f059700 1 -- 172.21.15.121:0/3322567981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa398102ee0 msgr2=0x7fa39810f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.035+0000 7fa39f059700 1 --2- 172.21.15.121:0/3322567981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa398102ee0 0x7fa39810f3c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa390009990 tx=0x7fa390009ca0 comp rx=0 tx=0).stop 2024-06-10T00:23:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.035+0000 7fa39f059700 1 -- 172.21.15.121:0/3322567981 shutdown_connections 2024-06-10T00:23:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.035+0000 7fa39f059700 1 --2- 172.21.15.121:0/3322567981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa398102ee0 0x7fa39810f3c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.035+0000 7fa39f059700 1 --2- 172.21.15.121:0/3322567981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa398107450 0x7fa398107830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:56.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.035+0000 7fa39f059700 1 --2- 172.21.15.121:0/3322567981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa398102540 0x7fa3981029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.035+0000 7fa39f059700 1 -- 172.21.15.121:0/3322567981 >> 172.21.15.121:0/3322567981 conn(0x7fa3980795f0 msgr2=0x7fa398079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.035+0000 7fa39f059700 1 -- 172.21.15.121:0/3322567981 shutdown_connections 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.035+0000 7fa39f059700 1 -- 172.21.15.121:0/3322567981 wait complete. 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa39f059700 1 Processor -- start 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa39f059700 1 -- start start 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa39f059700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa398102540 0x7fa39819e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa39f059700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa398102ee0 0x7fa39819edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa39f059700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa398107450 0x7fa398198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa39f059700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa3981127e0 con 0x7fa398102ee0 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa39f059700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fa398112660 con 0x7fa398102540 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa39f059700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa398112960 con 0x7fa398107450 2024-06-10T00:23:56.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa38ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa398102ee0 0x7fa39819edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa38ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa398102ee0 0x7fa39819edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35312/0 (socket says 172.21.15.121:35312) 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa38ffff700 1 -- 172.21.15.121:0/1141580999 learned_addr learned my addr 172.21.15.121:0/1141580999 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.036+0000 7fa39cdf5700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa398102540 0x7fa39819e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa39d5f6700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa398107450 0x7fa398198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa38ffff700 1 -- 172.21.15.121:0/1141580999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa398107450 msgr2=0x7fa398198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa38ffff700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa398107450 0x7fa398198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa38ffff700 1 -- 172.21.15.121:0/1141580999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa398102540 msgr2=0x7fa39819e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa38ffff700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa398102540 0x7fa39819e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa38ffff700 1 -- 172.21.15.121:0/1141580999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3981991a0 con 0x7fa398102ee0 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa38ffff700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa398102ee0 0x7fa39819edc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa38800c920 tx=0x7fa38800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa38dffb700 1 -- 172.21.15.121:0/1141580999 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa388007b00 con 0x7fa398102ee0 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa39f059700 1 -- 172.21.15.121:0/1141580999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa398199490 con 0x7fa398102ee0 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa38dffb700 1 -- 172.21.15.121:0/1141580999 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa388004d10 con 0x7fa398102ee0 2024-06-10T00:23:56.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa38dffb700 1 -- 172.21.15.121:0/1141580999 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa388005680 con 0x7fa398102ee0 2024-06-10T00:23:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.037+0000 7fa39f059700 1 -- 172.21.15.121:0/1141580999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3981a5750 con 0x7fa398102ee0 2024-06-10T00:23:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.038+0000 7fa39f059700 1 -- 172.21.15.121:0/1141580999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa398069180 con 0x7fa398102ee0 2024-06-10T00:23:56.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.040+0000 7fa38dffb700 1 -- 172.21.15.121:0/1141580999 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa388004750 con 0x7fa398102ee0 2024-06-10T00:23:56.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.041+0000 7fa38dffb700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa3800a2c20 0x7fa3800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:56.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.041+0000 7fa38dffb700 1 -- 172.21.15.121:0/1141580999 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7497+0+0 (secure 0 0 0) 0x7fa3880cecb0 con 0x7fa398102ee0 2024-06-10T00:23:56.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.041+0000 7fa39cdf5700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa3800a2c20 0x7fa3800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:56.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.042+0000 7fa39cdf5700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa3800a2c20 0x7fa3800a50e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa394009550 tx=0x7fa3940091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:56.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.045+0000 7fa38dffb700 1 -- 172.21.15.121:0/1141580999 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa388014030 con 0x7fa398102ee0 2024-06-10T00:23:56.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.247+0000 7fa39f059700 1 -- 172.21.15.121:0/1141580999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fa398199f70 con 0x7fa398102ee0 2024-06-10T00:23:56.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.534+0000 7fa38dffb700 1 -- 172.21.15.121:0/1141580999 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v86) ==== 171+0+0 (secure 0 0 0) 0x7fa38809bf40 con 0x7fa398102ee0 2024-06-10T00:23:56.534 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:56 smithi121 ceph-mon[32378]: osdmap e85: 12 total, 12 up, 12 in 2024-06-10T00:23:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 -- 172.21.15.121:0/1141580999 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa3800a2c20 msgr2=0x7fa3800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa3800a2c20 0x7fa3800a50e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa394009550 tx=0x7fa3940091b0 comp rx=0 tx=0).stop 2024-06-10T00:23:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 -- 172.21.15.121:0/1141580999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa398102ee0 msgr2=0x7fa39819edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa398102ee0 0x7fa39819edc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa38800c920 tx=0x7fa38800cc30 comp rx=0 tx=0).stop 2024-06-10T00:23:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 -- 172.21.15.121:0/1141580999 shutdown_connections 2024-06-10T00:23:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa3800a2c20 0x7fa3800a50e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa398107450 0x7fa398198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa398102540 0x7fa39819e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 --2- 172.21.15.121:0/1141580999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa398102ee0 0x7fa39819edc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:56.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 -- 172.21.15.121:0/1141580999 >> 172.21.15.121:0/1141580999 conn(0x7fa3980795f0 msgr2=0x7fa398100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:56.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 -- 172.21.15.121:0/1141580999 shutdown_connections 2024-06-10T00:23:56.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:56.537+0000 7fa39f059700 1 -- 172.21.15.121:0/1141580999 wait complete. 2024-06-10T00:23:56.539 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'cephfs_data' created 2024-06-10T00:23:56.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:56 smithi135 ceph-mon[36424]: osdmap e85: 12 total, 12 up, 12 in 2024-06-10T00:23:56.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:56 smithi193 ceph-mon[36716]: osdmap e85: 12 total, 12 up, 12 in 2024-06-10T00:23:57.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-10T00:23:57.403 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:23:57.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:57 smithi135 ceph-mon[36424]: pgmap v292: 33 pgs: 15 creating+peering, 2 active+clean, 16 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:57.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:57 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1141580999' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-06-10T00:23:57.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:57 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1141580999' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-06-10T00:23:57.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:57 smithi135 ceph-mon[36424]: osdmap e86: 12 total, 12 up, 12 in 2024-06-10T00:23:57.777 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:57 smithi121 ceph-mon[32378]: pgmap v292: 33 pgs: 15 creating+peering, 2 active+clean, 16 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:57.777 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:57 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1141580999' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-06-10T00:23:57.777 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:57 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1141580999' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-06-10T00:23:57.777 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:57 smithi121 ceph-mon[32378]: osdmap e86: 12 total, 12 up, 12 in 2024-06-10T00:23:57.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:57 smithi193 ceph-mon[36716]: pgmap v292: 33 pgs: 15 creating+peering, 2 active+clean, 16 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:57.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:57 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1141580999' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-06-10T00:23:57.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:57 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1141580999' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-06-10T00:23:57.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:57 smithi193 ceph-mon[36716]: osdmap e86: 12 total, 12 up, 12 in 2024-06-10T00:23:58.590 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:58 smithi121 ceph-mon[32378]: osdmap e87: 12 total, 12 up, 12 in 2024-06-10T00:23:58.590 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:58 smithi121 ceph-mon[32378]: pgmap v295: 97 pgs: 33 creating+peering, 23 active+clean, 41 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:58.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:58 smithi193 ceph-mon[36716]: osdmap e87: 12 total, 12 up, 12 in 2024-06-10T00:23:58.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:58 smithi193 ceph-mon[36716]: pgmap v295: 97 pgs: 33 creating+peering, 23 active+clean, 41 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:59.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:58 smithi135 ceph-mon[36424]: osdmap e87: 12 total, 12 up, 12 in 2024-06-10T00:23:59.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:58 smithi135 ceph-mon[36424]: pgmap v295: 97 pgs: 33 creating+peering, 23 active+clean, 41 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:23:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.140+0000 7fd60cf5d700 1 -- 172.21.15.121:0/850011693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608107400 msgr2=0x7fd6081077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.140+0000 7fd60cf5d700 1 --2- 172.21.15.121:0/850011693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608107400 0x7fd6081077e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009990 tx=0x7fd5f8009ca0 comp rx=0 tx=0).stop 2024-06-10T00:23:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.140+0000 7fd60cf5d700 1 -- 172.21.15.121:0/850011693 shutdown_connections 2024-06-10T00:23:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.140+0000 7fd60cf5d700 1 --2- 172.21.15.121:0/850011693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608101d50 0x7fd60810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.140+0000 7fd60cf5d700 1 --2- 172.21.15.121:0/850011693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608107400 0x7fd6081077e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.140+0000 7fd60cf5d700 1 --2- 172.21.15.121:0/850011693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd608101390 0x7fd608101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.140+0000 7fd60cf5d700 1 -- 172.21.15.121:0/850011693 >> 172.21.15.121:0/850011693 conn(0x7fd608078340 msgr2=0x7fd608078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.140+0000 7fd60cf5d700 1 -- 172.21.15.121:0/850011693 shutdown_connections 2024-06-10T00:23:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.140+0000 7fd60cf5d700 1 -- 172.21.15.121:0/850011693 wait complete. 2024-06-10T00:23:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd60cf5d700 1 Processor -- start 2024-06-10T00:23:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd60cf5d700 1 -- start start 2024-06-10T00:23:59.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd60cf5d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd608101390 0x7fd60819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:59.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd60cf5d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101d50 0x7fd60819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:59.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd60cf5d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608107400 0x7fd6081989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:59.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd60cf5d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd608111670 con 0x7fd608101d50 2024-06-10T00:23:59.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd60cf5d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd6081114f0 con 0x7fd608107400 2024-06-10T00:23:59.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd60cf5d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6081117f0 con 0x7fd608101390 2024-06-10T00:23:59.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd605d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101d50 0x7fd60819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd605d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101d50 0x7fd60819ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35332/0 (socket says 172.21.15.121:35332) 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd605d9b700 1 -- 172.21.15.121:0/2917061846 learned_addr learned my addr 172.21.15.121:0/2917061846 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.141+0000 7fd60659c700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd608101390 0x7fd60819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd606d9d700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608107400 0x7fd6081989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd605d9b700 1 -- 172.21.15.121:0/2917061846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd608101390 msgr2=0x7fd60819e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd605d9b700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd608101390 0x7fd60819e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd605d9b700 1 -- 172.21.15.121:0/2917061846 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608107400 msgr2=0x7fd6081989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd605d9b700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608107400 0x7fd6081989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd605d9b700 1 -- 172.21.15.121:0/2917061846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd608199120 con 0x7fd608101d50 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd60659c700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd608101390 0x7fd60819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd605d9b700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101d50 0x7fd60819ecf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00c920 tx=0x7fd5fc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd5f77fe700 1 -- 172.21.15.121:0/2917061846 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc007b00 con 0x7fd608101d50 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd60cf5d700 1 -- 172.21.15.121:0/2917061846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd608199410 con 0x7fd608101d50 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd5f77fe700 1 -- 172.21.15.121:0/2917061846 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5fc004d10 con 0x7fd608101d50 2024-06-10T00:23:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd5f77fe700 1 -- 172.21.15.121:0/2917061846 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0056e0 con 0x7fd608101d50 2024-06-10T00:23:59.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.142+0000 7fd60cf5d700 1 -- 172.21.15.121:0/2917061846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6081a5680 con 0x7fd608101d50 2024-06-10T00:23:59.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.143+0000 7fd60cf5d700 1 -- 172.21.15.121:0/2917061846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd608069180 con 0x7fd608101d50 2024-06-10T00:23:59.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.144+0000 7fd5f77fe700 1 -- 172.21.15.121:0/2917061846 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd5fc007ca0 con 0x7fd608101d50 2024-06-10T00:23:59.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.145+0000 7fd5f77fe700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2c20 0x7fd5f00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:23:59.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.145+0000 7fd60659c700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2c20 0x7fd5f00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:23:59.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.145+0000 7fd5f77fe700 1 -- 172.21.15.121:0/2917061846 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7879+0+0 (secure 0 0 0) 0x7fd5fc0cef70 con 0x7fd608101d50 2024-06-10T00:23:59.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.146+0000 7fd60659c700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2c20 0x7fd5f00a50e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8006010 tx=0x7fd5f800b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:23:59.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.149+0000 7fd5f77fe700 1 -- 172.21.15.121:0/2917061846 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd5fc014030 con 0x7fd608101d50 2024-06-10T00:23:59.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.390+0000 7fd60cf5d700 1 -- 172.21.15.121:0/2917061846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd608199ea0 con 0x7fd608101d50 2024-06-10T00:23:59.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.543+0000 7fd5f77fe700 1 -- 172.21.15.121:0/2917061846 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fd608199ea0 con 0x7fd608101d50 2024-06-10T00:23:59.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.545+0000 7fd60cf5d700 1 -- 172.21.15.121:0/2917061846 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2c20 msgr2=0x7fd5f00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:59.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.545+0000 7fd60cf5d700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2c20 0x7fd5f00a50e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8006010 tx=0x7fd5f800b540 comp rx=0 tx=0).stop 2024-06-10T00:23:59.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.546+0000 7fd60cf5d700 1 -- 172.21.15.121:0/2917061846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101d50 msgr2=0x7fd60819ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:23:59.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.546+0000 7fd60cf5d700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101d50 0x7fd60819ecf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00c920 tx=0x7fd5fc00cce0 comp rx=0 tx=0).stop 2024-06-10T00:23:59.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.546+0000 7fd60cf5d700 1 -- 172.21.15.121:0/2917061846 shutdown_connections 2024-06-10T00:23:59.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.546+0000 7fd60cf5d700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2c20 0x7fd5f00a50e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:59.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.546+0000 7fd60cf5d700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608107400 0x7fd6081989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:59.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.546+0000 7fd60cf5d700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd608101390 0x7fd60819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:59.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.546+0000 7fd60cf5d700 1 --2- 172.21.15.121:0/2917061846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101d50 0x7fd60819ecf0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:23:59.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.546+0000 7fd60cf5d700 1 -- 172.21.15.121:0/2917061846 >> 172.21.15.121:0/2917061846 conn(0x7fd608078340 msgr2=0x7fd6081005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:23:59.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.546+0000 7fd60cf5d700 1 -- 172.21.15.121:0/2917061846 shutdown_connections 2024-06-10T00:23:59.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:23:59.546+0000 7fd60cf5d700 1 -- 172.21.15.121:0/2917061846 wait complete. 2024-06-10T00:23:59.547 INFO:teuthology.orchestra.run.smithi121.stderr:new fs with metadata pool 2 and data pool 3 2024-06-10T00:23:59.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:59 smithi121 ceph-mon[32378]: osdmap e88: 12 total, 12 up, 12 in 2024-06-10T00:23:59.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:59 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2917061846' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-06-10T00:23:59.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:23:59 smithi121 ceph-mon[32378]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-06-10T00:23:59.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:59 smithi193 ceph-mon[36716]: osdmap e88: 12 total, 12 up, 12 in 2024-06-10T00:23:59.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:59 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2917061846' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-06-10T00:23:59.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:23:59 smithi193 ceph-mon[36716]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-06-10T00:24:00.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:59 smithi135 ceph-mon[36424]: osdmap e88: 12 total, 12 up, 12 in 2024-06-10T00:24:00.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:59 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2917061846' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-06-10T00:24:00.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:23:59 smithi135 ceph-mon[36424]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-06-10T00:24:00.258 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-06-10T00:24:00.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-06-10T00:24:00.499 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:00.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:00 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2917061846' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-06-10T00:24:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:00 smithi193 ceph-mon[36716]: osdmap e89: 12 total, 12 up, 12 in 2024-06-10T00:24:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:00 smithi193 ceph-mon[36716]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-06-10T00:24:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:00 smithi193 ceph-mon[36716]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-06-10T00:24:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:00 smithi193 ceph-mon[36716]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:24:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:00 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] up:active 2024-06-10T00:24:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:00 smithi193 ceph-mon[36716]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:24:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:00 smithi193 ceph-mon[36716]: pgmap v298: 97 pgs: 18 creating+peering, 61 active+clean, 18 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:00.894 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:00 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2917061846' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-06-10T00:24:00.894 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:00 smithi121 ceph-mon[32378]: osdmap e89: 12 total, 12 up, 12 in 2024-06-10T00:24:00.894 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:00 smithi121 ceph-mon[32378]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-06-10T00:24:00.894 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:00 smithi121 ceph-mon[32378]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-06-10T00:24:00.894 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:00 smithi121 ceph-mon[32378]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:24:00.894 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:00 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] up:active 2024-06-10T00:24:00.895 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:00 smithi121 ceph-mon[32378]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:24:00.895 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:00 smithi121 ceph-mon[32378]: pgmap v298: 97 pgs: 18 creating+peering, 61 active+clean, 18 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:01.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:00 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2917061846' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-06-10T00:24:01.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:00 smithi135 ceph-mon[36424]: osdmap e89: 12 total, 12 up, 12 in 2024-06-10T00:24:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:00 smithi135 ceph-mon[36424]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-06-10T00:24:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:00 smithi135 ceph-mon[36424]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-06-10T00:24:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:00 smithi135 ceph-mon[36424]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:24:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:00 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] up:active 2024-06-10T00:24:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:00 smithi135 ceph-mon[36424]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:24:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:00 smithi135 ceph-mon[36424]: pgmap v298: 97 pgs: 18 creating+peering, 61 active+clean, 18 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:01.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:01 smithi121 ceph-mon[32378]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-06-10T00:24:01.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:01 smithi121 ceph-mon[32378]: Cluster is now healthy 2024-06-10T00:24:01.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:01 smithi193 ceph-mon[36716]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-06-10T00:24:01.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:01 smithi193 ceph-mon[36716]: Cluster is now healthy 2024-06-10T00:24:02.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:01 smithi135 ceph-mon[36424]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-06-10T00:24:02.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:01 smithi135 ceph-mon[36424]: Cluster is now healthy 2024-06-10T00:24:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.362+0000 7f4d8d72d700 1 -- 172.21.15.121:0/795497491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d881027a0 msgr2=0x7f4d88102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.362+0000 7f4d8d72d700 1 --2- 172.21.15.121:0/795497491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d881027a0 0x7f4d88102b80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4d780099c0 tx=0x7f4d78009cd0 comp rx=0 tx=0).stop 2024-06-10T00:24:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.362+0000 7f4d8d72d700 1 -- 172.21.15.121:0/795497491 shutdown_connections 2024-06-10T00:24:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.362+0000 7f4d8d72d700 1 --2- 172.21.15.121:0/795497491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88107870 0x7f4d8810f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.362+0000 7f4d8d72d700 1 --2- 172.21.15.121:0/795497491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d881027a0 0x7f4d88102b80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.362+0000 7f4d8d72d700 1 --2- 172.21.15.121:0/795497491 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d88103150 0x7f4d88107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.362+0000 7f4d8d72d700 1 -- 172.21.15.121:0/795497491 >> 172.21.15.121:0/795497491 conn(0x7f4d88079570 msgr2=0x7f4d88079980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:02.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.362+0000 7f4d8d72d700 1 -- 172.21.15.121:0/795497491 shutdown_connections 2024-06-10T00:24:02.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.362+0000 7f4d8d72d700 1 -- 172.21.15.121:0/795497491 wait complete. 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.363+0000 7f4d8d72d700 1 Processor -- start 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.363+0000 7f4d8d72d700 1 -- start start 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.363+0000 7f4d8d72d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d881027a0 0x7f4d8819e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.363+0000 7f4d8d72d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88103150 0x7f4d8819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.363+0000 7f4d8d72d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d88107870 0x7f4d881989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.363+0000 7f4d8d72d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88112790 con 0x7f4d881027a0 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.363+0000 7f4d8d72d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88112610 con 0x7f4d88107870 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.363+0000 7f4d8d72d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88112910 con 0x7f4d88103150 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.363+0000 7f4d86ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d881027a0 0x7f4d8819e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.363+0000 7f4d86ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d881027a0 0x7f4d8819e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35338/0 (socket says 172.21.15.121:35338) 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.363+0000 7f4d86ffd700 1 -- 172.21.15.121:0/3180820362 learned_addr learned my addr 172.21.15.121:0/3180820362 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:02.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d877fe700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d88107870 0x7f4d881989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:02.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d7ffff700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88103150 0x7f4d8819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:02.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d86ffd700 1 -- 172.21.15.121:0/3180820362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88103150 msgr2=0x7f4d8819ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:02.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d86ffd700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88103150 0x7f4d8819ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:02.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d86ffd700 1 -- 172.21.15.121:0/3180820362 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d88107870 msgr2=0x7f4d881989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:02.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d86ffd700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d88107870 0x7f4d881989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:02.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d86ffd700 1 -- 172.21.15.121:0/3180820362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d88199110 con 0x7f4d881027a0 2024-06-10T00:24:02.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d86ffd700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d881027a0 0x7f4d8819e7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4d780059d0 tx=0x7f4d780038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:02.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d84ff9700 1 -- 172.21.15.121:0/3180820362 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78027070 con 0x7f4d881027a0 2024-06-10T00:24:02.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d8d72d700 1 -- 172.21.15.121:0/3180820362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d881993a0 con 0x7f4d881027a0 2024-06-10T00:24:02.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d84ff9700 1 -- 172.21.15.121:0/3180820362 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d78004050 con 0x7f4d881027a0 2024-06-10T00:24:02.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d8d72d700 1 -- 172.21.15.121:0/3180820362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d881a5670 con 0x7f4d881027a0 2024-06-10T00:24:02.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.364+0000 7f4d84ff9700 1 -- 172.21.15.121:0/3180820362 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78022db0 con 0x7f4d881027a0 2024-06-10T00:24:02.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.365+0000 7f4d8d72d700 1 -- 172.21.15.121:0/3180820362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d88069180 con 0x7f4d881027a0 2024-06-10T00:24:02.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.366+0000 7f4d84ff9700 1 -- 172.21.15.121:0/3180820362 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4d7802a070 con 0x7f4d881027a0 2024-06-10T00:24:02.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.367+0000 7f4d84ff9700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4d740a2c20 0x7f4d740a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:02.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.367+0000 7f4d7ffff700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4d740a2c20 0x7f4d740a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:02.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.367+0000 7f4d84ff9700 1 -- 172.21.15.121:0/3180820362 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7979+0+0 (secure 0 0 0) 0x7f4d780cfb50 con 0x7f4d881027a0 2024-06-10T00:24:02.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.367+0000 7f4d7ffff700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4d740a2c20 0x7f4d740a50e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4d70005950 tx=0x7f4d700058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:02.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.371+0000 7f4d84ff9700 1 -- 172.21.15.121:0/3180820362 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4d78013070 con 0x7f4d881027a0 2024-06-10T00:24:02.508 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:24:02 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b[66180]: 2024-06-10T00:24:02.198+0000 7fca91c22700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-06-10T00:24:02.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:02.582+0000 7f4d8d72d700 1 -- 172.21.15.121:0/3180820362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f4d8802dca0 con 0x7f4d881027a0 2024-06-10T00:24:02.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:02 smithi121 ceph-mon[32378]: pgmap v299: 97 pgs: 8 creating+peering, 89 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 368 B/s wr, 1 op/s 2024-06-10T00:24:02.882 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:02 smithi193 ceph-mon[36716]: pgmap v299: 97 pgs: 8 creating+peering, 89 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 368 B/s wr, 1 op/s 2024-06-10T00:24:03.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:02 smithi135 ceph-mon[36424]: pgmap v299: 97 pgs: 8 creating+peering, 89 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 368 B/s wr, 1 op/s 2024-06-10T00:24:03.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.569+0000 7f4d84ff9700 1 -- 172.21.15.121:0/3180820362 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v90) ==== 157+0+0 (secure 0 0 0) 0x7f4d7809cc90 con 0x7f4d881027a0 2024-06-10T00:24:03.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.572+0000 7f4d8d72d700 1 -- 172.21.15.121:0/3180820362 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4d740a2c20 msgr2=0x7f4d740a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:03.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.572+0000 7f4d8d72d700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4d740a2c20 0x7f4d740a50e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4d70005950 tx=0x7f4d700058e0 comp rx=0 tx=0).stop 2024-06-10T00:24:03.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.572+0000 7f4d8d72d700 1 -- 172.21.15.121:0/3180820362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d881027a0 msgr2=0x7f4d8819e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:03.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.572+0000 7f4d8d72d700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d881027a0 0x7f4d8819e7a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4d780059d0 tx=0x7f4d780038b0 comp rx=0 tx=0).stop 2024-06-10T00:24:03.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.572+0000 7f4d8d72d700 1 -- 172.21.15.121:0/3180820362 shutdown_connections 2024-06-10T00:24:03.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.572+0000 7f4d8d72d700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4d740a2c20 0x7f4d740a50e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:03.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.572+0000 7f4d8d72d700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4d88107870 0x7f4d881989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:03.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.572+0000 7f4d8d72d700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d881027a0 0x7f4d8819e7a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:03.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.572+0000 7f4d8d72d700 1 --2- 172.21.15.121:0/3180820362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88103150 0x7f4d8819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:03.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.572+0000 7f4d8d72d700 1 -- 172.21.15.121:0/3180820362 >> 172.21.15.121:0/3180820362 conn(0x7f4d88079570 msgr2=0x7f4d88100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:03.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.573+0000 7f4d8d72d700 1 -- 172.21.15.121:0/3180820362 shutdown_connections 2024-06-10T00:24:03.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:03.573+0000 7f4d8d72d700 1 -- 172.21.15.121:0/3180820362 wait complete. 2024-06-10T00:24:03.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:03 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3180820362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-06-10T00:24:03.927 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:03 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3180820362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-06-10T00:24:04.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:03 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3180820362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-06-10T00:24:04.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-06-10T00:24:04.619 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:04.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:04 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3180820362' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-06-10T00:24:04.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:04 smithi121 ceph-mon[32378]: osdmap e90: 12 total, 12 up, 12 in 2024-06-10T00:24:04.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:04 smithi121 ceph-mon[32378]: pgmap v301: 97 pgs: 97 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-06-10T00:24:04.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:04 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3180820362' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-06-10T00:24:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:04 smithi193 ceph-mon[36716]: osdmap e90: 12 total, 12 up, 12 in 2024-06-10T00:24:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:04 smithi193 ceph-mon[36716]: pgmap v301: 97 pgs: 97 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-06-10T00:24:05.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:04 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3180820362' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-06-10T00:24:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:04 smithi135 ceph-mon[36424]: osdmap e90: 12 total, 12 up, 12 in 2024-06-10T00:24:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:04 smithi135 ceph-mon[36424]: pgmap v301: 97 pgs: 97 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-06-10T00:24:06.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.272+0000 7f44e6fdf700 1 -- 172.21.15.121:0/2166823102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e0101fa0 msgr2=0x7f44e0105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:06.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.272+0000 7f44e6fdf700 1 --2- 172.21.15.121:0/2166823102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e0101fa0 0x7f44e0105fe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f44d00099c0 tx=0x7f44d0009cd0 comp rx=0 tx=0).stop 2024-06-10T00:24:06.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.272+0000 7f44e6fdf700 1 -- 172.21.15.121:0/2166823102 shutdown_connections 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.272+0000 7f44e6fdf700 1 --2- 172.21.15.121:0/2166823102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44e01066c0 0x7f44e010e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.272+0000 7f44e6fdf700 1 --2- 172.21.15.121:0/2166823102 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44e01015f0 0x7f44e01019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.272+0000 7f44e6fdf700 1 --2- 172.21.15.121:0/2166823102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e0101fa0 0x7f44e0105fe0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.272+0000 7f44e6fdf700 1 -- 172.21.15.121:0/2166823102 >> 172.21.15.121:0/2166823102 conn(0x7f44e0078330 msgr2=0x7f44e0078740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.272+0000 7f44e6fdf700 1 -- 172.21.15.121:0/2166823102 shutdown_connections 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.272+0000 7f44e6fdf700 1 -- 172.21.15.121:0/2166823102 wait complete. 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44e6fdf700 1 Processor -- start 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44e6fdf700 1 -- start start 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44e6fdf700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44e01015f0 0x7f44e0199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44e6fdf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e0101fa0 0x7f44e0199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44e6fdf700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44e01066c0 0x7f44e01a4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44e6fdf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f44e0111700 con 0x7f44e0101fa0 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44e6fdf700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f44e0111580 con 0x7f44e01015f0 2024-06-10T00:24:06.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44e6fdf700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f44e0111880 con 0x7f44e01066c0 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e0101fa0 0x7f44e0199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44d7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e0101fa0 0x7f44e0199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35356/0 (socket says 172.21.15.121:35356) 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44d7fff700 1 -- 172.21.15.121:0/3848873782 learned_addr learned my addr 172.21.15.121:0/3848873782 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44e557c700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44e01066c0 0x7f44e01a4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44e4d7b700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44e01015f0 0x7f44e0199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44d7fff700 1 -- 172.21.15.121:0/3848873782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44e01066c0 msgr2=0x7f44e01a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44d7fff700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44e01066c0 0x7f44e01a4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44d7fff700 1 -- 172.21.15.121:0/3848873782 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44e01015f0 msgr2=0x7f44e0199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44d7fff700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44e01015f0 0x7f44e0199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44d7fff700 1 -- 172.21.15.121:0/3848873782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e01a5000 con 0x7f44e0101fa0 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44e4d7b700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44e01015f0 0x7f44e0199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.273+0000 7f44d7fff700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e0101fa0 0x7f44e0199f80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f44d0000c00 tx=0x7f44d0004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:06.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.274+0000 7f44d5ffb700 1 -- 172.21.15.121:0/3848873782 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d0027070 con 0x7f44e0101fa0 2024-06-10T00:24:06.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.274+0000 7f44e6fdf700 1 -- 172.21.15.121:0/3848873782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e01a51d0 con 0x7f44e0101fa0 2024-06-10T00:24:06.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.274+0000 7f44d5ffb700 1 -- 172.21.15.121:0/3848873782 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44d0004130 con 0x7f44e0101fa0 2024-06-10T00:24:06.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.274+0000 7f44d5ffb700 1 -- 172.21.15.121:0/3848873782 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d000f460 con 0x7f44e0101fa0 2024-06-10T00:24:06.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.274+0000 7f44e6fdf700 1 -- 172.21.15.121:0/3848873782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e01a5680 con 0x7f44e0101fa0 2024-06-10T00:24:06.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.275+0000 7f44e6fdf700 1 -- 172.21.15.121:0/3848873782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44e0050a00 con 0x7f44e0101fa0 2024-06-10T00:24:06.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.276+0000 7f44d5ffb700 1 -- 172.21.15.121:0/3848873782 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f44d002a070 con 0x7f44e0101fa0 2024-06-10T00:24:06.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.276+0000 7f44d5ffb700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44c80a2cc0 0x7f44c80a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:06.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.277+0000 7f44e4d7b700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44c80a2cc0 0x7f44c80a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:06.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.277+0000 7f44d5ffb700 1 -- 172.21.15.121:0/3848873782 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8277+0+0 (secure 0 0 0) 0x7f44d00d04e0 con 0x7f44e0101fa0 2024-06-10T00:24:06.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.277+0000 7f44e4d7b700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44c80a2cc0 0x7f44c80a5180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f44dc005fd0 tx=0x7f44dc005dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:06.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.280+0000 7f44d5ffb700 1 -- 172.21.15.121:0/3848873782 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f44d0013070 con 0x7f44e0101fa0 2024-06-10T00:24:06.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:06.477+0000 7f44e6fdf700 1 -- 172.21.15.121:0/3848873782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f44e01a59a0 con 0x7f44e0101fa0 2024-06-10T00:24:07.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:07 smithi121 ceph-mon[32378]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-06-10T00:24:07.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:07 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3848873782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-06-10T00:24:07.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:07 smithi135 ceph-mon[36424]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-06-10T00:24:07.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:07 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3848873782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-06-10T00:24:07.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:07 smithi193 ceph-mon[36716]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-06-10T00:24:07.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:07 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3848873782' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-06-10T00:24:08.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.150+0000 7f44d5ffb700 1 -- 172.21.15.121:0/3848873782 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v92) ==== 228+0+0 (secure 0 0 0) 0x7f44d009d4a0 con 0x7f44e0101fa0 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 -- 172.21.15.121:0/3848873782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44c80a2cc0 msgr2=0x7f44c80a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44c80a2cc0 0x7f44c80a5180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f44dc005fd0 tx=0x7f44dc005dc0 comp rx=0 tx=0).stop 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 -- 172.21.15.121:0/3848873782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e0101fa0 msgr2=0x7f44e0199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e0101fa0 0x7f44e0199f80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f44d0000c00 tx=0x7f44d0004630 comp rx=0 tx=0).stop 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 -- 172.21.15.121:0/3848873782 shutdown_connections 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44c80a2cc0 0x7f44c80a5180 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44e01066c0 0x7f44e01a4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44e01015f0 0x7f44e0199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 --2- 172.21.15.121:0/3848873782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e0101fa0 0x7f44e0199f80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 -- 172.21.15.121:0/3848873782 >> 172.21.15.121:0/3848873782 conn(0x7f44e0078330 msgr2=0x7f44e00ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 -- 172.21.15.121:0/3848873782 shutdown_connections 2024-06-10T00:24:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:08.154+0000 7f44e6fdf700 1 -- 172.21.15.121:0/3848873782 wait complete. 2024-06-10T00:24:08.155 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'cephfs_data_ec' created 2024-06-10T00:24:08.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:08 smithi121 ceph-mon[32378]: osdmap e91: 12 total, 12 up, 12 in 2024-06-10T00:24:08.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:08 smithi121 ceph-mon[32378]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-06-10T00:24:08.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:08 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3848873782' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-06-10T00:24:08.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:08 smithi121 ceph-mon[32378]: osdmap e92: 12 total, 12 up, 12 in 2024-06-10T00:24:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:08 smithi135 ceph-mon[36424]: osdmap e91: 12 total, 12 up, 12 in 2024-06-10T00:24:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:08 smithi135 ceph-mon[36424]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-06-10T00:24:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:08 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3848873782' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-06-10T00:24:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:08 smithi135 ceph-mon[36424]: osdmap e92: 12 total, 12 up, 12 in 2024-06-10T00:24:08.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:08 smithi193 ceph-mon[36716]: osdmap e91: 12 total, 12 up, 12 in 2024-06-10T00:24:08.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:08 smithi193 ceph-mon[36716]: pgmap v304: 97 pgs: 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-06-10T00:24:08.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:08 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3848873782' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-06-10T00:24:08.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:08 smithi193 ceph-mon[36716]: osdmap e92: 12 total, 12 up, 12 in 2024-06-10T00:24:08.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-06-10T00:24:09.183 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:10.352 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:10 smithi121 ceph-mon[32378]: osdmap e93: 12 total, 12 up, 12 in 2024-06-10T00:24:10.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:10 smithi135 ceph-mon[36424]: osdmap e93: 12 total, 12 up, 12 in 2024-06-10T00:24:10.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:10 smithi193 ceph-mon[36716]: osdmap e93: 12 total, 12 up, 12 in 2024-06-10T00:24:10.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.878+0000 7f4141f8f700 1 -- 172.21.15.121:0/1692935935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c101620 msgr2=0x7f413c101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:10.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.878+0000 7f4141f8f700 1 --2- 172.21.15.121:0/1692935935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c101620 0x7f413c101a00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f412c0099c0 tx=0x7f412c009cd0 comp rx=0 tx=0).stop 2024-06-10T00:24:10.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.878+0000 7f4141f8f700 1 -- 172.21.15.121:0/1692935935 shutdown_connections 2024-06-10T00:24:10.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.878+0000 7f4141f8f700 1 --2- 172.21.15.121:0/1692935935 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f413c1066b0 0x7f413c10e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:10.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.878+0000 7f4141f8f700 1 --2- 172.21.15.121:0/1692935935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c101620 0x7f413c101a00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.878+0000 7f4141f8f700 1 --2- 172.21.15.121:0/1692935935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c101fd0 0x7f413c105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.878+0000 7f4141f8f700 1 -- 172.21.15.121:0/1692935935 >> 172.21.15.121:0/1692935935 conn(0x7f413c078340 msgr2=0x7f413c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.879+0000 7f4141f8f700 1 -- 172.21.15.121:0/1692935935 shutdown_connections 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.879+0000 7f4141f8f700 1 -- 172.21.15.121:0/1692935935 wait complete. 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.879+0000 7f4141f8f700 1 Processor -- start 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.879+0000 7f4141f8f700 1 -- start start 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.879+0000 7f4141f8f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f413c101620 0x7f413c19d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.879+0000 7f4141f8f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c101fd0 0x7f413c19dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.879+0000 7f4141f8f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c1066b0 0x7f413c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.879+0000 7f4141f8f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f413c111710 con 0x7f413c1066b0 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.879+0000 7f4141f8f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f413c111590 con 0x7f413c101620 2024-06-10T00:24:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.879+0000 7f4141f8f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f413c111890 con 0x7f413c101fd0 2024-06-10T00:24:10.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c1066b0 0x7f413c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:10.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c1066b0 0x7f413c1979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50164/0 (socket says 172.21.15.121:50164) 2024-06-10T00:24:10.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413bfff700 1 -- 172.21.15.121:0/664788548 learned_addr learned my addr 172.21.15.121:0/664788548 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:10.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413b7fe700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f413c101620 0x7f413c19d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:10.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413affd700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c101fd0 0x7f413c19dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:10.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413b7fe700 1 -- 172.21.15.121:0/664788548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c101fd0 msgr2=0x7f413c19dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:10.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413b7fe700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c101fd0 0x7f413c19dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:10.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413b7fe700 1 -- 172.21.15.121:0/664788548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c1066b0 msgr2=0x7f413c1979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:10.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413b7fe700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c1066b0 0x7f413c1979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:10.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413b7fe700 1 -- 172.21.15.121:0/664788548 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c198110 con 0x7f413c101620 2024-06-10T00:24:10.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413bfff700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c1066b0 0x7f413c1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.880+0000 7f413b7fe700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f413c101620 0x7f413c19d7f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f412c009fd0 tx=0x7f412c005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.881+0000 7f4138ff9700 1 -- 172.21.15.121:0/664788548 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c019810 con 0x7f413c101620 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.881+0000 7f4141f8f700 1 -- 172.21.15.121:0/664788548 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f413c1983a0 con 0x7f413c101620 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.881+0000 7f4138ff9700 1 -- 172.21.15.121:0/664788548 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f412c0047f0 con 0x7f413c101620 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.881+0000 7f4138ff9700 1 -- 172.21.15.121:0/664788548 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c0053a0 con 0x7f413c101620 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.881+0000 7f4141f8f700 1 -- 172.21.15.121:0/664788548 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f413c06abc0 con 0x7f413c101620 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.882+0000 7f4141f8f700 1 -- 172.21.15.121:0/664788548 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f413c102ab0 con 0x7f413c101620 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.883+0000 7f4138ff9700 1 -- 172.21.15.121:0/664788548 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f412c02a060 con 0x7f413c101620 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.884+0000 7f4138ff9700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f41280a2c20 0x7f41280a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.884+0000 7f413affd700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f41280a2c20 0x7f41280a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.884+0000 7f4138ff9700 1 -- 172.21.15.121:0/664788548 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8770+0+0 (secure 0 0 0) 0x7f412c0d1d50 con 0x7f413c101620 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.884+0000 7f413affd700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f41280a2c20 0x7f41280a50e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f413c100ee0 tx=0x7f4124009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:10.888+0000 7f4138ff9700 1 -- 172.21.15.121:0/664788548 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f412c05e850 con 0x7f413c101620 2024-06-10T00:24:11.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.085+0000 7f4141f8f700 1 -- 172.21.15.121:0/664788548 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f413c06b650 con 0x7f413c101620 2024-06-10T00:24:11.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.169+0000 7f4138ff9700 1 -- 172.21.15.121:0/664788548 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v95) ==== 169+0+0 (secure 0 0 0) 0x7f412c013070 con 0x7f413c101620 2024-06-10T00:24:11.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.171+0000 7f4141f8f700 1 -- 172.21.15.121:0/664788548 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f41280a2c20 msgr2=0x7f41280a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:11.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.171+0000 7f4141f8f700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f41280a2c20 0x7f41280a50e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f413c100ee0 tx=0x7f4124009190 comp rx=0 tx=0).stop 2024-06-10T00:24:11.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.171+0000 7f4141f8f700 1 -- 172.21.15.121:0/664788548 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f413c101620 msgr2=0x7f413c19d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:11.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.171+0000 7f4141f8f700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f413c101620 0x7f413c19d7f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f412c009fd0 tx=0x7f412c005f10 comp rx=0 tx=0).stop 2024-06-10T00:24:11.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.171+0000 7f4141f8f700 1 -- 172.21.15.121:0/664788548 shutdown_connections 2024-06-10T00:24:11.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.171+0000 7f4141f8f700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f41280a2c20 0x7f41280a50e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:11.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.171+0000 7f4141f8f700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c1066b0 0x7f413c1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:11.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.171+0000 7f4141f8f700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f413c101620 0x7f413c19d7f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:11.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.171+0000 7f4141f8f700 1 --2- 172.21.15.121:0/664788548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f413c101fd0 0x7f413c19dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:11.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.171+0000 7f4141f8f700 1 -- 172.21.15.121:0/664788548 >> 172.21.15.121:0/664788548 conn(0x7f413c078340 msgr2=0x7f413c105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:11.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.172+0000 7f4141f8f700 1 -- 172.21.15.121:0/664788548 shutdown_connections 2024-06-10T00:24:11.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:11.172+0000 7f4141f8f700 1 -- 172.21.15.121:0/664788548 wait complete. 2024-06-10T00:24:11.172 INFO:teuthology.orchestra.run.smithi121.stderr:set pool 4 allow_ec_overwrites to true 2024-06-10T00:24:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:11 smithi121 ceph-mon[32378]: pgmap v307: 129 pgs: 9 creating+peering, 23 unknown, 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2024-06-10T00:24:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:11 smithi121 ceph-mon[32378]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:24:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:11 smithi121 ceph-mon[32378]: osdmap e94: 12 total, 12 up, 12 in 2024-06-10T00:24:11.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:11 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/664788548' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-06-10T00:24:11.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:11 smithi121 ceph-mon[32378]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-06-10T00:24:11.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:11 smithi135 ceph-mon[36424]: pgmap v307: 129 pgs: 9 creating+peering, 23 unknown, 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2024-06-10T00:24:11.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:11 smithi135 ceph-mon[36424]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:24:11.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:11 smithi135 ceph-mon[36424]: osdmap e94: 12 total, 12 up, 12 in 2024-06-10T00:24:11.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:11 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/664788548' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-06-10T00:24:11.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:11 smithi135 ceph-mon[36424]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-06-10T00:24:11.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:11 smithi193 ceph-mon[36716]: pgmap v307: 129 pgs: 9 creating+peering, 23 unknown, 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s wr, 1 op/s 2024-06-10T00:24:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:11 smithi193 ceph-mon[36716]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-10T00:24:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:11 smithi193 ceph-mon[36716]: osdmap e94: 12 total, 12 up, 12 in 2024-06-10T00:24:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:11 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/664788548' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-06-10T00:24:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:11 smithi193 ceph-mon[36716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-06-10T00:24:11.958 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-06-10T00:24:12.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:12 smithi121 ceph-mon[32378]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-06-10T00:24:12.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:12 smithi121 ceph-mon[32378]: osdmap e95: 12 total, 12 up, 12 in 2024-06-10T00:24:12.199 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:12.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:12 smithi135 ceph-mon[36424]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-06-10T00:24:12.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:12 smithi135 ceph-mon[36424]: osdmap e95: 12 total, 12 up, 12 in 2024-06-10T00:24:12.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:12 smithi193 ceph-mon[36716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-06-10T00:24:12.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:12 smithi193 ceph-mon[36716]: osdmap e95: 12 total, 12 up, 12 in 2024-06-10T00:24:13.420 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:13 smithi121 ceph-mon[32378]: pgmap v310: 129 pgs: 9 creating+peering, 6 unknown, 114 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:13.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:13 smithi135 ceph-mon[36424]: pgmap v310: 129 pgs: 9 creating+peering, 6 unknown, 114 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:13.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:13 smithi193 ceph-mon[36716]: pgmap v310: 129 pgs: 9 creating+peering, 6 unknown, 114 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:13.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.945+0000 7fb06612e700 1 -- 172.21.15.121:0/360596774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb06006a170 msgr2=0x7fb06006a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:13.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.945+0000 7fb06612e700 1 --2- 172.21.15.121:0/360596774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb06006a170 0x7fb06006a5f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb054009990 tx=0x7fb054009ca0 comp rx=0 tx=0).stop 2024-06-10T00:24:13.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.945+0000 7fb06612e700 1 -- 172.21.15.121:0/360596774 shutdown_connections 2024-06-10T00:24:13.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.945+0000 7fb06612e700 1 --2- 172.21.15.121:0/360596774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb06006ab30 0x7fb06010f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:13.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.945+0000 7fb06612e700 1 --2- 172.21.15.121:0/360596774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb060108500 0x7fb0601088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:13.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.945+0000 7fb06612e700 1 --2- 172.21.15.121:0/360596774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb06006a170 0x7fb06006a5f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:13.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.945+0000 7fb06612e700 1 -- 172.21.15.121:0/360596774 >> 172.21.15.121:0/360596774 conn(0x7fb060078340 msgr2=0x7fb060078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:13.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.945+0000 7fb06612e700 1 -- 172.21.15.121:0/360596774 shutdown_connections 2024-06-10T00:24:13.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.945+0000 7fb06612e700 1 -- 172.21.15.121:0/360596774 wait complete. 2024-06-10T00:24:13.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.945+0000 7fb06612e700 1 Processor -- start 2024-06-10T00:24:13.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb06612e700 1 -- start start 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb06612e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb06006a170 0x7fb060101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb06612e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb06006ab30 0x7fb0601019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb06612e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb060108500 0x7fb060101ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb06612e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb060112860 con 0x7fb060108500 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb06612e700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fb0601126e0 con 0x7fb06006ab30 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb06612e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb0601129e0 con 0x7fb06006a170 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb05ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb060108500 0x7fb060101ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb05ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb060108500 0x7fb060101ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50184/0 (socket says 172.21.15.121:50184) 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb05ffff700 1 -- 172.21.15.121:0/3866139906 learned_addr learned my addr 172.21.15.121:0/3866139906 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb05f7fe700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb06006a170 0x7fb060101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb05effd700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb06006ab30 0x7fb0601019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:13.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb05ffff700 1 -- 172.21.15.121:0/3866139906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb06006a170 msgr2=0x7fb060101460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:13.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb05ffff700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb06006a170 0x7fb060101460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:13.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb05ffff700 1 -- 172.21.15.121:0/3866139906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb06006ab30 msgr2=0x7fb0601019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:13.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb05ffff700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb06006ab30 0x7fb0601019a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:13.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.946+0000 7fb05ffff700 1 -- 172.21.15.121:0/3866139906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0601a72b0 con 0x7fb060108500 2024-06-10T00:24:13.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.947+0000 7fb05ffff700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb060108500 0x7fb060101ee0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb05800b970 tx=0x7fb05800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:13.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.947+0000 7fb05cff9700 1 -- 172.21.15.121:0/3866139906 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb058019070 con 0x7fb060108500 2024-06-10T00:24:13.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.947+0000 7fb05cff9700 1 -- 172.21.15.121:0/3866139906 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb05800ce10 con 0x7fb060108500 2024-06-10T00:24:13.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.947+0000 7fb06612e700 1 -- 172.21.15.121:0/3866139906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0601a75a0 con 0x7fb060108500 2024-06-10T00:24:13.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.947+0000 7fb05cff9700 1 -- 172.21.15.121:0/3866139906 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb058007370 con 0x7fb060108500 2024-06-10T00:24:13.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.947+0000 7fb06612e700 1 -- 172.21.15.121:0/3866139906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0601a7a90 con 0x7fb060108500 2024-06-10T00:24:13.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.948+0000 7fb06612e700 1 -- 172.21.15.121:0/3866139906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb06010a9e0 con 0x7fb060108500 2024-06-10T00:24:13.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.949+0000 7fb05cff9700 1 -- 172.21.15.121:0/3866139906 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb05801c070 con 0x7fb060108500 2024-06-10T00:24:13.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.949+0000 7fb05cff9700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0480a2c20 0x7fb0480a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:13.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.950+0000 7fb05f7fe700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0480a2c20 0x7fb0480a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:13.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.950+0000 7fb05cff9700 1 -- 172.21.15.121:0/3866139906 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8770+0+0 (secure 0 0 0) 0x7fb0580ceae0 con 0x7fb060108500 2024-06-10T00:24:13.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.950+0000 7fb05f7fe700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0480a2c20 0x7fb0480a50e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb050009a10 tx=0x7fb050009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:13.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:13.953+0000 7fb05cff9700 1 -- 172.21.15.121:0/3866139906 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb05809b820 con 0x7fb060108500 2024-06-10T00:24:14.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:14.199+0000 7fb06612e700 1 -- 172.21.15.121:0/3866139906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fb06010a3c0 con 0x7fb060108500 2024-06-10T00:24:15.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.185+0000 7fb05cff9700 1 -- 172.21.15.121:0/3866139906 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7fb05809b120 con 0x7fb060108500 2024-06-10T00:24:15.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.188+0000 7fb06612e700 1 -- 172.21.15.121:0/3866139906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0480a2c20 msgr2=0x7fb0480a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:15.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.188+0000 7fb06612e700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0480a2c20 0x7fb0480a50e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb050009a10 tx=0x7fb050009190 comp rx=0 tx=0).stop 2024-06-10T00:24:15.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.188+0000 7fb06612e700 1 -- 172.21.15.121:0/3866139906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb060108500 msgr2=0x7fb060101ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:15.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.188+0000 7fb06612e700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb060108500 0x7fb060101ee0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb05800b970 tx=0x7fb05800bc80 comp rx=0 tx=0).stop 2024-06-10T00:24:15.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.189+0000 7fb06612e700 1 -- 172.21.15.121:0/3866139906 shutdown_connections 2024-06-10T00:24:15.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.189+0000 7fb06612e700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb0480a2c20 0x7fb0480a50e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:15.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.189+0000 7fb06612e700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb060108500 0x7fb060101ee0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:15.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.189+0000 7fb06612e700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb06006a170 0x7fb060101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:15.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.189+0000 7fb06612e700 1 --2- 172.21.15.121:0/3866139906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb06006ab30 0x7fb0601019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:15.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.189+0000 7fb06612e700 1 -- 172.21.15.121:0/3866139906 >> 172.21.15.121:0/3866139906 conn(0x7fb060078340 msgr2=0x7fb0601000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:15.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.190+0000 7fb06612e700 1 -- 172.21.15.121:0/3866139906 shutdown_connections 2024-06-10T00:24:15.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:15.190+0000 7fb06612e700 1 -- 172.21.15.121:0/3866139906 wait complete. 2024-06-10T00:24:15.190 INFO:teuthology.orchestra.run.smithi121.stderr:added data pool 4 to fsmap 2024-06-10T00:24:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:15 smithi121 ceph-mon[32378]: pgmap v311: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:15 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3866139906' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-06-10T00:24:15.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:15 smithi135 ceph-mon[36424]: pgmap v311: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:15.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:15 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3866139906' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-06-10T00:24:15.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:15 smithi193 ceph-mon[36716]: pgmap v311: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:15.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:15 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3866139906' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-06-10T00:24:15.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:24:16.170 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:16.430 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:16 smithi121 ceph-mon[32378]: osdmap e96: 12 total, 12 up, 12 in 2024-06-10T00:24:16.430 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:16 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3866139906' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-06-10T00:24:16.430 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:16 smithi121 ceph-mon[32378]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:24:16.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:16 smithi135 ceph-mon[36424]: osdmap e96: 12 total, 12 up, 12 in 2024-06-10T00:24:16.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:16 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3866139906' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-06-10T00:24:16.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:16 smithi135 ceph-mon[36424]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:24:16.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:16 smithi193 ceph-mon[36716]: osdmap e96: 12 total, 12 up, 12 in 2024-06-10T00:24:16.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:16 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3866139906' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-06-10T00:24:16.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:16 smithi193 ceph-mon[36716]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:24:17.427 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:17 smithi121 ceph-mon[32378]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:17.428 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:17 smithi121 ceph-mon[32378]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-10T00:24:17.428 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:17 smithi121 ceph-mon[32378]: Cluster is now healthy 2024-06-10T00:24:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:17 smithi135 ceph-mon[36424]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:17 smithi135 ceph-mon[36424]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-10T00:24:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:17 smithi135 ceph-mon[36424]: Cluster is now healthy 2024-06-10T00:24:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:17 smithi193 ceph-mon[36716]: pgmap v313: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:17 smithi193 ceph-mon[36716]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-06-10T00:24:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:17 smithi193 ceph-mon[36716]: Cluster is now healthy 2024-06-10T00:24:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.911+0000 7ffb68d86700 1 -- 172.21.15.121:0/3554004720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb641066b0 msgr2=0x7ffb6410e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.911+0000 7ffb68d86700 1 --2- 172.21.15.121:0/3554004720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb641066b0 0x7ffb6410e280 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00b0d0 tx=0x7ffb5c00b3e0 comp rx=0 tx=0).stop 2024-06-10T00:24:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.911+0000 7ffb68d86700 1 -- 172.21.15.121:0/3554004720 shutdown_connections 2024-06-10T00:24:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.911+0000 7ffb68d86700 1 --2- 172.21.15.121:0/3554004720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb641066b0 0x7ffb6410e280 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.911+0000 7ffb68d86700 1 --2- 172.21.15.121:0/3554004720 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb64101620 0x7ffb64101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.911+0000 7ffb68d86700 1 --2- 172.21.15.121:0/3554004720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb64101fd0 0x7ffb64105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.911+0000 7ffb68d86700 1 -- 172.21.15.121:0/3554004720 >> 172.21.15.121:0/3554004720 conn(0x7ffb64078340 msgr2=0x7ffb64078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.911+0000 7ffb68d86700 1 -- 172.21.15.121:0/3554004720 shutdown_connections 2024-06-10T00:24:17.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.911+0000 7ffb68d86700 1 -- 172.21.15.121:0/3554004720 wait complete. 2024-06-10T00:24:17.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb68d86700 1 Processor -- start 2024-06-10T00:24:17.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb68d86700 1 -- start start 2024-06-10T00:24:17.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb68d86700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb64101620 0x7ffb64103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:17.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb68d86700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb64101fd0 0x7ffb641035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:17.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb68d86700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb641066b0 0x7ffb64103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:17.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb68d86700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffb64111790 con 0x7ffb64101620 2024-06-10T00:24:17.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb68d86700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ffb64111610 con 0x7ffb641066b0 2024-06-10T00:24:17.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb68d86700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb64111910 con 0x7ffb64101fd0 2024-06-10T00:24:17.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb6259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb64101620 0x7ffb64103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:17.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb6259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb64101620 0x7ffb64103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50212/0 (socket says 172.21.15.121:50212) 2024-06-10T00:24:17.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb6259c700 1 -- 172.21.15.121:0/534791290 learned_addr learned my addr 172.21.15.121:0/534791290 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:17.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb61d9b700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb64101fd0 0x7ffb641035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:17.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb62d9d700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb641066b0 0x7ffb64103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:17.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb6259c700 1 -- 172.21.15.121:0/534791290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb64101fd0 msgr2=0x7ffb641035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:17.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb6259c700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb64101fd0 0x7ffb641035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb6259c700 1 -- 172.21.15.121:0/534791290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb641066b0 msgr2=0x7ffb64103af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb6259c700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb641066b0 0x7ffb64103af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb6259c700 1 -- 172.21.15.121:0/534791290 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb641a7290 con 0x7ffb64101620 2024-06-10T00:24:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb61d9b700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb64101fd0 0x7ffb641035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:24:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.912+0000 7ffb62d9d700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb641066b0 0x7ffb64103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:24:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.913+0000 7ffb6259c700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb64101620 0x7ffb64103070 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400b530 tx=0x7ffb5400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.913+0000 7ffb537fe700 1 -- 172.21.15.121:0/534791290 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54015020 con 0x7ffb64101620 2024-06-10T00:24:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.913+0000 7ffb537fe700 1 -- 172.21.15.121:0/534791290 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb54011d10 con 0x7ffb64101620 2024-06-10T00:24:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.913+0000 7ffb68d86700 1 -- 172.21.15.121:0/534791290 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb641a7580 con 0x7ffb64101620 2024-06-10T00:24:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.913+0000 7ffb537fe700 1 -- 172.21.15.121:0/534791290 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb54010e50 con 0x7ffb64101620 2024-06-10T00:24:17.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.913+0000 7ffb68d86700 1 -- 172.21.15.121:0/534791290 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb641a7ac0 con 0x7ffb64101620 2024-06-10T00:24:17.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.914+0000 7ffb68d86700 1 -- 172.21.15.121:0/534791290 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb64069180 con 0x7ffb64101620 2024-06-10T00:24:17.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.915+0000 7ffb537fe700 1 -- 172.21.15.121:0/534791290 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ffb54018020 con 0x7ffb64101620 2024-06-10T00:24:17.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.915+0000 7ffb537fe700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ffb4c0a2dc0 0x7ffb4c0a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:17.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.915+0000 7ffb61d9b700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ffb4c0a2dc0 0x7ffb4c0a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:17.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.916+0000 7ffb537fe700 1 -- 172.21.15.121:0/534791290 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8802+0+0 (secure 0 0 0) 0x7ffb540cf020 con 0x7ffb64101620 2024-06-10T00:24:17.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.916+0000 7ffb61d9b700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ffb4c0a2dc0 0x7ffb4c0a5280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffb641060f0 tx=0x7ffb58008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:17.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:17.919+0000 7ffb537fe700 1 -- 172.21.15.121:0/534791290 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffb5401f020 con 0x7ffb64101620 2024-06-10T00:24:18.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.159+0000 7ffb68d86700 1 -- 172.21.15.121:0/534791290 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb641045a0 con 0x7ffb64101620 2024-06-10T00:24:18.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.160+0000 7ffb537fe700 1 -- 172.21.15.121:0/534791290 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11812 (secure 0 0 0) 0x7ffb54097070 con 0x7ffb64101620 2024-06-10T00:24:18.160 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:24:18.160 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:24:14.200723+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24479,"qdb_cluster":[24479]},"id":1}]} 2024-06-10T00:24:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 -- 172.21.15.121:0/534791290 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ffb4c0a2dc0 msgr2=0x7ffb4c0a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ffb4c0a2dc0 0x7ffb4c0a5280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffb641060f0 tx=0x7ffb58008040 comp rx=0 tx=0).stop 2024-06-10T00:24:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 -- 172.21.15.121:0/534791290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb64101620 msgr2=0x7ffb64103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb64101620 0x7ffb64103070 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ffb5400b530 tx=0x7ffb5400b8f0 comp rx=0 tx=0).stop 2024-06-10T00:24:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 -- 172.21.15.121:0/534791290 shutdown_connections 2024-06-10T00:24:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ffb4c0a2dc0 0x7ffb4c0a5280 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb641066b0 0x7ffb64103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb64101620 0x7ffb64103070 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:18.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 --2- 172.21.15.121:0/534791290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb64101fd0 0x7ffb641035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:18.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 -- 172.21.15.121:0/534791290 >> 172.21.15.121:0/534791290 conn(0x7ffb64078340 msgr2=0x7ffb6406aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:18.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 -- 172.21.15.121:0/534791290 shutdown_connections 2024-06-10T00:24:18.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:18.162+0000 7ffb68d86700 1 -- 172.21.15.121:0/534791290 wait complete. 2024-06-10T00:24:18.163 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 16 2024-06-10T00:24:18.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:18 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/534791290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:24:18.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:18 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/534791290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:24:18.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:18 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/534791290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:24:18.857 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:24:19.095 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:19.442 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:19 smithi121 ceph-mon[32378]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:19.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:19 smithi135 ceph-mon[36424]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:19.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:19 smithi193 ceph-mon[36716]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.835+0000 7f3a55e72700 1 -- 172.21.15.121:0/1513278927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a50105f50 msgr2=0x7f3a5010da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.835+0000 7f3a55e72700 1 --2- 172.21.15.121:0/1513278927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a50105f50 0x7f3a5010da90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800b0a0 tx=0x7f3a4800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:24:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.836+0000 7f3a55e72700 1 -- 172.21.15.121:0/1513278927 shutdown_connections 2024-06-10T00:24:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.836+0000 7f3a55e72700 1 --2- 172.21.15.121:0/1513278927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a50105f50 0x7f3a5010da90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.836+0000 7f3a55e72700 1 --2- 172.21.15.121:0/1513278927 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a501010b0 0x7f3a50101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.836+0000 7f3a55e72700 1 --2- 172.21.15.121:0/1513278927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a50101a60 0x7f3a50105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.836+0000 7f3a55e72700 1 -- 172.21.15.121:0/1513278927 >> 172.21.15.121:0/1513278927 conn(0x7f3a500766b0 msgr2=0x7f3a50076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:20.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.836+0000 7f3a55e72700 1 -- 172.21.15.121:0/1513278927 shutdown_connections 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.836+0000 7f3a55e72700 1 -- 172.21.15.121:0/1513278927 wait complete. 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.836+0000 7f3a55e72700 1 Processor -- start 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.836+0000 7f3a55e72700 1 -- start start 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a55e72700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a501010b0 0x7f3a5019d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a55e72700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a50101a60 0x7f3a5019dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a55e72700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a50105f50 0x7f3a50197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a55e72700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3a50110ee0 con 0x7f3a50101a60 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a55e72700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f3a50110d60 con 0x7f3a501010b0 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a55e72700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3a50111060 con 0x7f3a50105f50 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a50101a60 0x7f3a5019dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a50101a60 0x7f3a5019dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59152/0 (socket says 172.21.15.121:59152) 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4effd700 1 -- 172.21.15.121:0/1874129024 learned_addr learned my addr 172.21.15.121:0/1874129024 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4f7fe700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a501010b0 0x7f3a5019d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:20.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4ffff700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a50105f50 0x7f3a50197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4effd700 1 -- 172.21.15.121:0/1874129024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a50105f50 msgr2=0x7f3a50197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4effd700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a50105f50 0x7f3a50197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4effd700 1 -- 172.21.15.121:0/1874129024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a501010b0 msgr2=0x7f3a5019d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4effd700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a501010b0 0x7f3a5019d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4effd700 1 -- 172.21.15.121:0/1874129024 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a50198020 con 0x7f3a50101a60 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4f7fe700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a501010b0 0x7f3a5019d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4effd700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a50101a60 0x7f3a5019dc00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c8c0 tx=0x7f3a4400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4cff9700 1 -- 172.21.15.121:0/1874129024 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44004220 con 0x7f3a50101a60 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a55e72700 1 -- 172.21.15.121:0/1874129024 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a50198310 con 0x7f3a50101a60 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4cff9700 1 -- 172.21.15.121:0/1874129024 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a44007de0 con 0x7f3a50101a60 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a4cff9700 1 -- 172.21.15.121:0/1874129024 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a44005020 con 0x7f3a50101a60 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.837+0000 7f3a55e72700 1 -- 172.21.15.121:0/1874129024 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a501a4520 con 0x7f3a50101a60 2024-06-10T00:24:20.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.838+0000 7f3a55e72700 1 -- 172.21.15.121:0/1874129024 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a50102290 con 0x7f3a50101a60 2024-06-10T00:24:20.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.839+0000 7f3a4cff9700 1 -- 172.21.15.121:0/1874129024 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3a44007900 con 0x7f3a50101a60 2024-06-10T00:24:20.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.840+0000 7f3a4cff9700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a380a2c20 0x7f3a380a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:20.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.840+0000 7f3a4f7fe700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a380a2c20 0x7f3a380a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:20.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.840+0000 7f3a4cff9700 1 -- 172.21.15.121:0/1874129024 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8802+0+0 (secure 0 0 0) 0x7f3a440cee50 con 0x7f3a50101a60 2024-06-10T00:24:20.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.840+0000 7f3a4f7fe700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a380a2c20 0x7f3a380a50e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a40005fd0 tx=0x7f3a40005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:20.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:20.845+0000 7f3a4cff9700 1 -- 172.21.15.121:0/1874129024 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3a44014030 con 0x7f3a50101a60 2024-06-10T00:24:21.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.043+0000 7f3a55e72700 1 -- 172.21.15.121:0/1874129024 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a501a47b0 con 0x7f3a50101a60 2024-06-10T00:24:21.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.045+0000 7f3a4cff9700 1 -- 172.21.15.121:0/1874129024 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24196 (secure 0 0 0) 0x7f3a4409bb70 con 0x7f3a50101a60 2024-06-10T00:24:21.045 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:24:21.045 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":96,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:24:14.200563+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:24:21.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 -- 172.21.15.121:0/1874129024 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a380a2c20 msgr2=0x7f3a380a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:21.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a380a2c20 0x7f3a380a50e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a40005fd0 tx=0x7f3a40005e30 comp rx=0 tx=0).stop 2024-06-10T00:24:21.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 -- 172.21.15.121:0/1874129024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a50101a60 msgr2=0x7f3a5019dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:21.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a50101a60 0x7f3a5019dc00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c8c0 tx=0x7f3a4400cc80 comp rx=0 tx=0).stop 2024-06-10T00:24:21.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 -- 172.21.15.121:0/1874129024 shutdown_connections 2024-06-10T00:24:21.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3a380a2c20 0x7f3a380a50e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:21.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a50105f50 0x7f3a50197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:21.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3a501010b0 0x7f3a5019d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:21.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 --2- 172.21.15.121:0/1874129024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a50101a60 0x7f3a5019dc00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:21.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 -- 172.21.15.121:0/1874129024 >> 172.21.15.121:0/1874129024 conn(0x7f3a500766b0 msgr2=0x7f3a50104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:21.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 -- 172.21.15.121:0/1874129024 shutdown_connections 2024-06-10T00:24:21.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:21.047+0000 7f3a55e72700 1 -- 172.21.15.121:0/1874129024 wait complete. 2024-06-10T00:24:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:21 smithi121 ceph-mon[32378]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:21 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1874129024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:21.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:21 smithi135 ceph-mon[36424]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:21 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1874129024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:21.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:21 smithi193 ceph-mon[36716]: pgmap v315: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:21.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:21 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1874129024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:21.906 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:24:22.148 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:23.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:23 smithi121 ceph-mon[32378]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:23.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:23 smithi135 ceph-mon[36424]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:23.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:23 smithi193 ceph-mon[36716]: pgmap v316: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:23.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.922+0000 7f4c39b90700 1 -- 172.21.15.121:0/1439896941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c3410ef40 msgr2=0x7f4c34111330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:23.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.922+0000 7f4c39b90700 1 --2- 172.21.15.121:0/1439896941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c3410ef40 0x7f4c34111330 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00b0a0 tx=0x7f4c2c00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:24:23.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.923+0000 7f4c39b90700 1 -- 172.21.15.121:0/1439896941 shutdown_connections 2024-06-10T00:24:23.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.923+0000 7f4c39b90700 1 --2- 172.21.15.121:0/1439896941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c3410ef40 0x7f4c34111330 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.923+0000 7f4c39b90700 1 --2- 172.21.15.121:0/1439896941 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c3406a170 0x7f4c3406a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.923+0000 7f4c39b90700 1 --2- 172.21.15.121:0/1439896941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c3406aa90 0x7f4c3410ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.923+0000 7f4c39b90700 1 -- 172.21.15.121:0/1439896941 >> 172.21.15.121:0/1439896941 conn(0x7f4c340782c0 msgr2=0x7f4c340786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.923+0000 7f4c39b90700 1 -- 172.21.15.121:0/1439896941 shutdown_connections 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.923+0000 7f4c39b90700 1 -- 172.21.15.121:0/1439896941 wait complete. 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.923+0000 7f4c39b90700 1 Processor -- start 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.923+0000 7f4c39b90700 1 -- start start 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c39b90700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c3406a170 0x7f4c3419afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c39b90700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c3406aa90 0x7f4c3419b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c39b90700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c3410ef40 0x7f4c3419f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c39b90700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4c34114a30 con 0x7f4c3406a170 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c39b90700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4c341148b0 con 0x7f4c3410ef40 2024-06-10T00:24:23.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c39b90700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4c34114bb0 con 0x7f4c3406aa90 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c337fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c3406a170 0x7f4c3419afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c337fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c3406a170 0x7f4c3419afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59174/0 (socket says 172.21.15.121:59174) 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c337fe700 1 -- 172.21.15.121:0/3217608727 learned_addr learned my addr 172.21.15.121:0/3217608727 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c32ffd700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c3406aa90 0x7f4c3419b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c33fff700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c3410ef40 0x7f4c3419f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c337fe700 1 -- 172.21.15.121:0/3217608727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c3406aa90 msgr2=0x7f4c3419b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c337fe700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c3406aa90 0x7f4c3419b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c337fe700 1 -- 172.21.15.121:0/3217608727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c3410ef40 msgr2=0x7f4c3419f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c337fe700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c3410ef40 0x7f4c3419f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c337fe700 1 -- 172.21.15.121:0/3217608727 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c3419ffb0 con 0x7f4c3406a170 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c33fff700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c3410ef40 0x7f4c3419f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:24:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c337fe700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c3406a170 0x7f4c3419afa0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400b530 tx=0x7f4c2400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.924+0000 7f4c30ff9700 1 -- 172.21.15.121:0/3217608727 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c24015020 con 0x7f4c3406a170 2024-06-10T00:24:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.925+0000 7f4c30ff9700 1 -- 172.21.15.121:0/3217608727 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c24011d10 con 0x7f4c3406a170 2024-06-10T00:24:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.925+0000 7f4c39b90700 1 -- 172.21.15.121:0/3217608727 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c341a02a0 con 0x7f4c3406a170 2024-06-10T00:24:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.925+0000 7f4c30ff9700 1 -- 172.21.15.121:0/3217608727 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c2400f2e0 con 0x7f4c3406a170 2024-06-10T00:24:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.925+0000 7f4c39b90700 1 -- 172.21.15.121:0/3217608727 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c341a7b20 con 0x7f4c3406a170 2024-06-10T00:24:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.925+0000 7f4c39b90700 1 -- 172.21.15.121:0/3217608727 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c34069180 con 0x7f4c3406a170 2024-06-10T00:24:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.927+0000 7f4c30ff9700 1 -- 172.21.15.121:0/3217608727 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f4c24018020 con 0x7f4c3406a170 2024-06-10T00:24:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.928+0000 7f4c30ff9700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4c1c0a2c20 0x7f4c1c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.928+0000 7f4c32ffd700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4c1c0a2c20 0x7f4c1c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.928+0000 7f4c30ff9700 1 -- 172.21.15.121:0/3217608727 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8802+0+0 (secure 0 0 0) 0x7f4c240cef00 con 0x7f4c3406a170 2024-06-10T00:24:23.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.928+0000 7f4c32ffd700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4c1c0a2c20 0x7f4c1c0a50e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4c34102990 tx=0x7f4c28009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:23.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:23.931+0000 7f4c30ff9700 1 -- 172.21.15.121:0/3217608727 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4c2401f020 con 0x7f4c3406a170 2024-06-10T00:24:24.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.129+0000 7f4c39b90700 1 -- 172.21.15.121:0/3217608727 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c3419c300 con 0x7f4c3406a170 2024-06-10T00:24:24.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.131+0000 7f4c30ff9700 1 -- 172.21.15.121:0/3217608727 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24196 (secure 0 0 0) 0x7f4c2409bc20 con 0x7f4c3406a170 2024-06-10T00:24:24.131 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:24:24.131 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":96,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:24:14.200563+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 -- 172.21.15.121:0/3217608727 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4c1c0a2c20 msgr2=0x7f4c1c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4c1c0a2c20 0x7f4c1c0a50e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4c34102990 tx=0x7f4c28009190 comp rx=0 tx=0).stop 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 -- 172.21.15.121:0/3217608727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c3406a170 msgr2=0x7f4c3419afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c3406a170 0x7f4c3419afa0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400b530 tx=0x7f4c2400b840 comp rx=0 tx=0).stop 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 -- 172.21.15.121:0/3217608727 shutdown_connections 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4c1c0a2c20 0x7f4c1c0a50e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c3410ef40 0x7f4c3419f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c3406a170 0x7f4c3419afa0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 --2- 172.21.15.121:0/3217608727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c3406aa90 0x7f4c3419b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 -- 172.21.15.121:0/3217608727 >> 172.21.15.121:0/3217608727 conn(0x7f4c340782c0 msgr2=0x7f4c341103f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 -- 172.21.15.121:0/3217608727 shutdown_connections 2024-06-10T00:24:24.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:24.133+0000 7f4c39b90700 1 -- 172.21.15.121:0/3217608727 wait complete. 2024-06-10T00:24:24.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:24 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3217608727' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:24.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:24 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3217608727' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:24.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:24 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3217608727' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:24.862 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:24:25.105 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:25 smithi121 ceph-mon[32378]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:25.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:25 smithi135 ceph-mon[36424]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:25.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:25 smithi193 ceph-mon[36716]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.932+0000 7f494e58a700 1 -- 172.21.15.121:0/73189300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4948100a80 msgr2=0x7f4948100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.932+0000 7f494e58a700 1 --2- 172.21.15.121:0/73189300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4948100a80 0x7f4948100f00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4930009990 tx=0x7f4930009ca0 comp rx=0 tx=0).stop 2024-06-10T00:24:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.932+0000 7f494e58a700 1 -- 172.21.15.121:0/73189300 shutdown_connections 2024-06-10T00:24:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.932+0000 7f494e58a700 1 --2- 172.21.15.121:0/73189300 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4948101440 0x7f494810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.932+0000 7f494e58a700 1 --2- 172.21.15.121:0/73189300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4948108510 0x7f49481088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.932+0000 7f494e58a700 1 --2- 172.21.15.121:0/73189300 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4948100a80 0x7f4948100f00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:26.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.932+0000 7f494e58a700 1 -- 172.21.15.121:0/73189300 >> 172.21.15.121:0/73189300 conn(0x7f4948077a00 msgr2=0x7f49480fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.932+0000 7f494e58a700 1 -- 172.21.15.121:0/73189300 shutdown_connections 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.932+0000 7f494e58a700 1 -- 172.21.15.121:0/73189300 wait complete. 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f494e58a700 1 Processor -- start 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f494e58a700 1 -- start start 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f494e58a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4948100a80 0x7f4948101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f494e58a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4948101440 0x7f49481021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f494e58a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4948108510 0x7f49481026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f494e58a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4948112790 con 0x7f4948101440 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f494e58a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4948112610 con 0x7f4948108510 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f494e58a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4948112910 con 0x7f4948100a80 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f493ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4948101440 0x7f49481021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f493ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4948101440 0x7f49481021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59184/0 (socket says 172.21.15.121:59184) 2024-06-10T00:24:26.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f493ffff700 1 -- 172.21.15.121:0/1685473499 learned_addr learned my addr 172.21.15.121:0/1685473499 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f4947fff700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4948100a80 0x7f4948101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f494cb27700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4948108510 0x7f49481026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f493ffff700 1 -- 172.21.15.121:0/1685473499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4948100a80 msgr2=0x7f4948101c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f493ffff700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4948100a80 0x7f4948101c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f493ffff700 1 -- 172.21.15.121:0/1685473499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4948108510 msgr2=0x7f49481026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f493ffff700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4948108510 0x7f49481026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.933+0000 7f493ffff700 1 -- 172.21.15.121:0/1685473499 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49481a7120 con 0x7f4948101440 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.934+0000 7f4947fff700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4948100a80 0x7f4948101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.934+0000 7f494cb27700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4948108510 0x7f49481026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.934+0000 7f493ffff700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4948101440 0x7f49481021a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4930000c00 tx=0x7f493000bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.934+0000 7f4945ffb700 1 -- 172.21.15.121:0/1685473499 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930027070 con 0x7f4948101440 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.934+0000 7f494e58a700 1 -- 172.21.15.121:0/1685473499 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49481a73b0 con 0x7f4948101440 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.934+0000 7f4945ffb700 1 -- 172.21.15.121:0/1685473499 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4930022470 con 0x7f4948101440 2024-06-10T00:24:26.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.934+0000 7f4945ffb700 1 -- 172.21.15.121:0/1685473499 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493000f6a0 con 0x7f4948101440 2024-06-10T00:24:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.934+0000 7f494e58a700 1 -- 172.21.15.121:0/1685473499 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49481a7860 con 0x7f4948101440 2024-06-10T00:24:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.935+0000 7f494e58a700 1 -- 172.21.15.121:0/1685473499 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f494810a9f0 con 0x7f4948101440 2024-06-10T00:24:26.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.936+0000 7f4945ffb700 1 -- 172.21.15.121:0/1685473499 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f493002a070 con 0x7f4948101440 2024-06-10T00:24:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.937+0000 7f4945ffb700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49340a2c20 0x7f49340a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.937+0000 7f4947fff700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49340a2c20 0x7f49340a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.937+0000 7f4945ffb700 1 -- 172.21.15.121:0/1685473499 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8802+0+0 (secure 0 0 0) 0x7f49300d03f0 con 0x7f4948101440 2024-06-10T00:24:26.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.937+0000 7f4947fff700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49340a2c20 0x7f49340a50e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4938009b50 tx=0x7f4938009550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:26.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:26.941+0000 7f4945ffb700 1 -- 172.21.15.121:0/1685473499 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4930013070 con 0x7f4948101440 2024-06-10T00:24:27.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.178+0000 7f494e58a700 1 -- 172.21.15.121:0/1685473499 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4948103140 con 0x7f4948101440 2024-06-10T00:24:27.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.179+0000 7f4945ffb700 1 -- 172.21.15.121:0/1685473499 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11812 (secure 0 0 0) 0x7f493009d140 con 0x7f4948101440 2024-06-10T00:24:27.179 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:24:27.179 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24448,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24472,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:24:14.200723+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24479,"qdb_cluster":[24479]},"id":1}]} 2024-06-10T00:24:27.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.181+0000 7f494e58a700 1 -- 172.21.15.121:0/1685473499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49340a2c20 msgr2=0x7f49340a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:27.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.181+0000 7f494e58a700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49340a2c20 0x7f49340a50e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4938009b50 tx=0x7f4938009550 comp rx=0 tx=0).stop 2024-06-10T00:24:27.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.181+0000 7f494e58a700 1 -- 172.21.15.121:0/1685473499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4948101440 msgr2=0x7f49481021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:27.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.181+0000 7f494e58a700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4948101440 0x7f49481021a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4930000c00 tx=0x7f493000bb60 comp rx=0 tx=0).stop 2024-06-10T00:24:27.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.181+0000 7f494e58a700 1 -- 172.21.15.121:0/1685473499 shutdown_connections 2024-06-10T00:24:27.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.181+0000 7f494e58a700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49340a2c20 0x7f49340a50e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:27.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.181+0000 7f494e58a700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4948108510 0x7f49481026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:27.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.181+0000 7f494e58a700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4948100a80 0x7f4948101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:27.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.181+0000 7f494e58a700 1 --2- 172.21.15.121:0/1685473499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4948101440 0x7f49481021a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:27.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.181+0000 7f494e58a700 1 -- 172.21.15.121:0/1685473499 >> 172.21.15.121:0/1685473499 conn(0x7f4948077a00 msgr2=0x7f49480ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:27.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.182+0000 7f494e58a700 1 -- 172.21.15.121:0/1685473499 shutdown_connections 2024-06-10T00:24:27.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:27.182+0000 7f494e58a700 1 -- 172.21.15.121:0/1685473499 wait complete. 2024-06-10T00:24:27.182 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 16 2024-06-10T00:24:27.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:27 smithi121 ceph-mon[32378]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:27.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:27 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1685473499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:24:27.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:27 smithi135 ceph-mon[36424]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:27.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:27 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1685473499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:24:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:27 smithi193 ceph-mon[36716]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:27 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1685473499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:24:27.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:24:28.174 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:29.297 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:29 smithi121 ceph-mon[32378]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:29.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:29 smithi135 ceph-mon[36424]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:29.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:29 smithi193 ceph-mon[36716]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:29.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.852+0000 7fefec528700 1 -- 172.21.15.121:0/1847717965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefe4101d50 msgr2=0x7fefe410e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:29.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.852+0000 7fefec528700 1 --2- 172.21.15.121:0/1847717965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefe4101d50 0x7fefe410e240 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00b0a0 tx=0x7fefdc00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:24:29.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.852+0000 7fefec528700 1 -- 172.21.15.121:0/1847717965 shutdown_connections 2024-06-10T00:24:29.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.852+0000 7fefec528700 1 --2- 172.21.15.121:0/1847717965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefe4101d50 0x7fefe410e240 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:29.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.852+0000 7fefec528700 1 --2- 172.21.15.121:0/1847717965 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fefe4107400 0x7fefe41077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:29.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.852+0000 7fefec528700 1 --2- 172.21.15.121:0/1847717965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe4101390 0x7fefe4101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:29.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.852+0000 7fefec528700 1 -- 172.21.15.121:0/1847717965 >> 172.21.15.121:0/1847717965 conn(0x7fefe4078340 msgr2=0x7fefe4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:29.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.852+0000 7fefec528700 1 -- 172.21.15.121:0/1847717965 shutdown_connections 2024-06-10T00:24:29.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.852+0000 7fefec528700 1 -- 172.21.15.121:0/1847717965 wait complete. 2024-06-10T00:24:29.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.853+0000 7fefec528700 1 Processor -- start 2024-06-10T00:24:29.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.853+0000 7fefec528700 1 -- start start 2024-06-10T00:24:29.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.853+0000 7fefec528700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe4101390 0x7fefe4199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:29.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.853+0000 7fefec528700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefe4101d50 0x7fefe4199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:29.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.853+0000 7fefec528700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fefe4107400 0x7fefe41a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:29.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.853+0000 7fefec528700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41116a0 con 0x7fefe4101d50 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.853+0000 7fefec528700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fefe4111520 con 0x7fefe4107400 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.853+0000 7fefec528700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fefe4111820 con 0x7fefe4101390 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.853+0000 7fefe9ac3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefe4101d50 0x7fefe4199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.853+0000 7fefe9ac3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefe4101d50 0x7fefe4199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60872/0 (socket says 172.21.15.121:60872) 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.853+0000 7fefe9ac3700 1 -- 172.21.15.121:0/4156017153 learned_addr learned my addr 172.21.15.121:0/4156017153 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefea2c4700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe4101390 0x7fefe4199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefe9ac3700 1 -- 172.21.15.121:0/4156017153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe4101390 msgr2=0x7fefe4199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefe9ac3700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe4101390 0x7fefe4199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefeaac5700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fefe4107400 0x7fefe41a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefe9ac3700 1 -- 172.21.15.121:0/4156017153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fefe4107400 msgr2=0x7fefe41a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefe9ac3700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fefe4107400 0x7fefe41a4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefe9ac3700 1 -- 172.21.15.121:0/4156017153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe41a5060 con 0x7fefe4101d50 2024-06-10T00:24:29.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefea2c4700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe4101390 0x7fefe4199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:24:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefeaac5700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fefe4107400 0x7fefe41a4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:24:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefe9ac3700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefe4101d50 0x7fefe4199fe0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fefe000c970 tx=0x7fefe000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefdb7fe700 1 -- 172.21.15.121:0/4156017153 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe0007aa0 con 0x7fefe4101d50 2024-06-10T00:24:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefdb7fe700 1 -- 172.21.15.121:0/4156017153 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefe0004d10 con 0x7fefe4101d50 2024-06-10T00:24:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefec528700 1 -- 172.21.15.121:0/4156017153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe41a5290 con 0x7fefe4101d50 2024-06-10T00:24:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefdb7fe700 1 -- 172.21.15.121:0/4156017153 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe0005690 con 0x7fefe4101d50 2024-06-10T00:24:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.854+0000 7fefec528700 1 -- 172.21.15.121:0/4156017153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe41a5780 con 0x7fefe4101d50 2024-06-10T00:24:29.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.855+0000 7fefec528700 1 -- 172.21.15.121:0/4156017153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe4069180 con 0x7fefe4101d50 2024-06-10T00:24:29.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.856+0000 7fefdb7fe700 1 -- 172.21.15.121:0/4156017153 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fefe0004750 con 0x7fefe4101d50 2024-06-10T00:24:29.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.857+0000 7fefdb7fe700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fefd00a2c20 0x7fefd00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:29.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.857+0000 7fefea2c4700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fefd00a2c20 0x7fefd00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:29.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.857+0000 7fefdb7fe700 1 -- 172.21.15.121:0/4156017153 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8802+0+0 (secure 0 0 0) 0x7fefe00cf490 con 0x7fefe4101d50 2024-06-10T00:24:29.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.857+0000 7fefea2c4700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fefd00a2c20 0x7fefd00a50e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fefd4005e10 tx=0x7fefd4005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:29.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:29.860+0000 7fefdb7fe700 1 -- 172.21.15.121:0/4156017153 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fefe009c240 con 0x7fefe4101d50 2024-06-10T00:24:30.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.059+0000 7fefec528700 1 -- 172.21.15.121:0/4156017153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fefe402dca0 con 0x7fefe4101d50 2024-06-10T00:24:30.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.060+0000 7fefdb7fe700 1 -- 172.21.15.121:0/4156017153 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+24196 (secure 0 0 0) 0x7fefe009bb40 con 0x7fefe4101d50 2024-06-10T00:24:30.060 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:24:30.060 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":96,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:24:14.200563+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:24:30.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.062+0000 7fefec528700 1 -- 172.21.15.121:0/4156017153 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fefd00a2c20 msgr2=0x7fefd00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:30.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.062+0000 7fefec528700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fefd00a2c20 0x7fefd00a50e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fefd4005e10 tx=0x7fefd4005d60 comp rx=0 tx=0).stop 2024-06-10T00:24:30.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.062+0000 7fefec528700 1 -- 172.21.15.121:0/4156017153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefe4101d50 msgr2=0x7fefe4199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:30.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.062+0000 7fefec528700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefe4101d50 0x7fefe4199fe0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fefe000c970 tx=0x7fefe000cd30 comp rx=0 tx=0).stop 2024-06-10T00:24:30.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.062+0000 7fefec528700 1 -- 172.21.15.121:0/4156017153 shutdown_connections 2024-06-10T00:24:30.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.062+0000 7fefec528700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fefd00a2c20 0x7fefd00a50e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:30.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.062+0000 7fefec528700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fefe4107400 0x7fefe41a4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:30.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.062+0000 7fefec528700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe4101390 0x7fefe4199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:30.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.062+0000 7fefec528700 1 --2- 172.21.15.121:0/4156017153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fefe4101d50 0x7fefe4199fe0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:30.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.062+0000 7fefec528700 1 -- 172.21.15.121:0/4156017153 >> 172.21.15.121:0/4156017153 conn(0x7fefe4078340 msgr2=0x7fefe40ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:30.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.063+0000 7fefec528700 1 -- 172.21.15.121:0/4156017153 shutdown_connections 2024-06-10T00:24:30.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:30.063+0000 7fefec528700 1 -- 172.21.15.121:0/4156017153 wait complete. 2024-06-10T00:24:30.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:30 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/4156017153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:30.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:30 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/4156017153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:30.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:30 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/4156017153' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:30.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-06-10T00:23:59.392036+0000', 'modified': '2024-06-10T00:24:14.200723+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24479}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24479': {'gid': 24479, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.135:6839/1599631648', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6838', 'nonce': 1599631648}, {'type': 'v1', 'addr': '172.21.15.135:6839', 'nonce': 1599631648}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24479, 'qdb_cluster': [24479]} 2024-06-10T00:24:30.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-06-10T00:24:30.771 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-06-10T00:24:31.013 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:31.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:31 smithi121 ceph-mon[32378]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:31.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:31 smithi135 ceph-mon[36424]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:31.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:31 smithi193 ceph-mon[36716]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:32.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.832+0000 7fbaf20ef700 1 -- 172.21.15.121:0/3665647114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaec107870 msgr2=0x7fbaec10f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:32.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.832+0000 7fbaf20ef700 1 --2- 172.21.15.121:0/3665647114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaec107870 0x7fbaec10f3f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbae40099c0 tx=0x7fbae4009cd0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.832+0000 7fbaf20ef700 1 -- 172.21.15.121:0/3665647114 shutdown_connections 2024-06-10T00:24:32.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.832+0000 7fbaf20ef700 1 --2- 172.21.15.121:0/3665647114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaec107870 0x7fbaec10f3f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.832+0000 7fbaf20ef700 1 --2- 172.21.15.121:0/3665647114 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbaec1027a0 0x7fbaec102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.832+0000 7fbaf20ef700 1 --2- 172.21.15.121:0/3665647114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaec103150 0x7fbaec107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.832+0000 7fbaf20ef700 1 -- 172.21.15.121:0/3665647114 >> 172.21.15.121:0/3665647114 conn(0x7fbaec079570 msgr2=0x7fbaec079980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:32.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.832+0000 7fbaf20ef700 1 -- 172.21.15.121:0/3665647114 shutdown_connections 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.832+0000 7fbaf20ef700 1 -- 172.21.15.121:0/3665647114 wait complete. 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.833+0000 7fbaf20ef700 1 Processor -- start 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.833+0000 7fbaf20ef700 1 -- start start 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.833+0000 7fbaf20ef700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaec1027a0 0x7fbaec19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.833+0000 7fbaf20ef700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbaec103150 0x7fbaec19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.833+0000 7fbaf20ef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaec107870 0x7fbaec198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.833+0000 7fbaf20ef700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec112880 con 0x7fbaec107870 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.833+0000 7fbaf20ef700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec112700 con 0x7fbaec103150 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.833+0000 7fbaf20ef700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec112a00 con 0x7fbaec1027a0 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.833+0000 7fbaebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaec107870 0x7fbaec198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.833+0000 7fbaebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaec107870 0x7fbaec198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60892/0 (socket says 172.21.15.121:60892) 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbaebfff700 1 -- 172.21.15.121:0/725906260 learned_addr learned my addr 172.21.15.121:0/725906260 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:32.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaec1027a0 0x7fbaec19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbaebfff700 1 -- 172.21.15.121:0/725906260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaec1027a0 msgr2=0x7fbaec19e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbaebfff700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaec1027a0 0x7fbaec19e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbaebfff700 1 -- 172.21.15.121:0/725906260 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbaec103150 msgr2=0x7fbaec19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbaebfff700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbaec103150 0x7fbaec19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbaebfff700 1 -- 172.21.15.121:0/725906260 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaec199200 con 0x7fbaec107870 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbaebfff700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaec107870 0x7fbaec198aa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbae40059d0 tx=0x7fbae4003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbae8ff9700 1 -- 172.21.15.121:0/725906260 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae4027070 con 0x7fbaec107870 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbae8ff9700 1 -- 172.21.15.121:0/725906260 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbae4004150 con 0x7fbaec107870 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbaf20ef700 1 -- 172.21.15.121:0/725906260 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaec199490 con 0x7fbaec107870 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbae8ff9700 1 -- 172.21.15.121:0/725906260 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae400f460 con 0x7fbaec107870 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.834+0000 7fbaf20ef700 1 -- 172.21.15.121:0/725906260 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaec1a57b0 con 0x7fbaec107870 2024-06-10T00:24:32.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.836+0000 7fbae8ff9700 1 -- 172.21.15.121:0/725906260 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbae402a070 con 0x7fbaec107870 2024-06-10T00:24:32.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.837+0000 7fbae8ff9700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbad40a2c20 0x7fbad40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.837+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbad40a2c20 0x7fbad40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.837+0000 7fbae8ff9700 1 -- 172.21.15.121:0/725906260 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8802+0+0 (secure 0 0 0) 0x7fbae40d00b0 con 0x7fbaec107870 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.837+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbad40a2c20 0x7fbad40a50e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbaec199be0 tx=0x7fbadc0094f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbafc972b80 1 Processor -- start 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbafc972b80 1 -- start start 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 0x562ef9967e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9968340 0x562ef9961da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef99622e0 0x562ef9962700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbafc972b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562ef98af7c0 con 0x562ef9968340 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbafc972b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x562ef98576d0 con 0x562ef9967a20 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbafc972b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562ef98ab780 con 0x562ef99622e0 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbaeaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9968340 0x562ef9961da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbaeaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9968340 0x562ef9961da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60894/0 (socket says 172.21.15.121:60894) 2024-06-10T00:24:32.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbaeaffd700 1 -- 172.21.15.121:0/3416432559 learned_addr learned my addr 172.21.15.121:0/3416432559 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbaebfff700 1 --2- 172.21.15.121:0/3416432559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef99622e0 0x562ef9962700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.838+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/3416432559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 0x562ef9967e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.839+0000 7fbaeaffd700 1 -- 172.21.15.121:0/3416432559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef99622e0 msgr2=0x562ef9962700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.839+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/3416432559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef99622e0 0x562ef9962700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.839+0000 7fbaeaffd700 1 -- 172.21.15.121:0/3416432559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 msgr2=0x562ef9967e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.839+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/3416432559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 0x562ef9967e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.839+0000 7fbaeaffd700 1 -- 172.21.15.121:0/3416432559 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ef984df00 con 0x562ef9968340 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.839+0000 7fbaebfff700 1 --2- 172.21.15.121:0/3416432559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef99622e0 0x562ef9962700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.839+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/3416432559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 0x562ef9967e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.839+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/3416432559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9968340 0x562ef9961da0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009870 tx=0x7fbae0009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e737b46857768d3 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.839+0000 7fbad1ffb700 1 -- 172.21.15.121:0/3416432559 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0015020 con 0x562ef9968340 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.839+0000 7fbad1ffb700 1 -- 172.21.15.121:0/3416432559 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbae0011d10 con 0x562ef9968340 2024-06-10T00:24:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.839+0000 7fbad1ffb700 1 -- 172.21.15.121:0/3416432559 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0010380 con 0x562ef9968340 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.840+0000 7fbafc972b80 1 -- 172.21.15.121:0/3416432559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9968340 msgr2=0x562ef9961da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.840+0000 7fbafc972b80 1 --2- 172.21.15.121:0/3416432559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9968340 0x562ef9961da0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009870 tx=0x7fbae0009b80 comp rx=0 tx=0).stop 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.840+0000 7fbafc972b80 1 -- 172.21.15.121:0/3416432559 shutdown_connections 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.840+0000 7fbafc972b80 1 --2- 172.21.15.121:0/3416432559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef99622e0 0x562ef9962700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.840+0000 7fbafc972b80 1 --2- 172.21.15.121:0/3416432559 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 0x562ef9967e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.840+0000 7fbafc972b80 1 --2- 172.21.15.121:0/3416432559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9968340 0x562ef9961da0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.840+0000 7fbafc972b80 1 -- 172.21.15.121:0/3416432559 >> 172.21.15.121:0/3416432559 conn(0x562ef9888c70 msgr2=0x562ef9958800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.840+0000 7fbafc972b80 1 -- 172.21.15.121:0/3416432559 shutdown_connections 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.840+0000 7fbafc972b80 1 -- 172.21.15.121:0/3416432559 wait complete. 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.840+0000 7fbafc972b80 1 -- start start 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.840+0000 7fbafc972b80 1 Processor -- start 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef99622e0 0x562ef996e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9967a20 0x562ef996b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9968340 0x562ef996b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbafc972b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562ef996ed80 con 0x562ef9967a20 2024-06-10T00:24:32.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbafc972b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x562ef984fba0 con 0x562ef99622e0 2024-06-10T00:24:32.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbafc972b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562ef996ef00 con 0x562ef9968340 2024-06-10T00:24:32.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbafc972b80 10 client.0 _ll_register_callbacks cb 0x562ef98498d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:24:32.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbaeaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9967a20 0x562ef996b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbaeaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9967a20 0x562ef996b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60906/0 (socket says 172.21.15.121:60906) 2024-06-10T00:24:32.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbaeaffd700 1 -- 172.21.15.121:0/3875513284 learned_addr learned my addr 172.21.15.121:0/3875513284 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:32.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef99622e0 0x562ef996e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbaebfff700 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9968340 0x562ef996b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbaeaffd700 1 -- 172.21.15.121:0/3875513284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9968340 msgr2=0x562ef996b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:24:32.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9968340 0x562ef996b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbaeaffd700 1 -- 172.21.15.121:0/3875513284 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef99622e0 msgr2=0x562ef996e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:24:32.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef99622e0 0x562ef996e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbaeaffd700 1 -- 172.21.15.121:0/3875513284 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ef97fe880 con 0x562ef9967a20 2024-06-10T00:24:32.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9967a20 0x562ef996b3d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbae000baa0 tx=0x7fbae000b420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610bf29faa9d6c65 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:32.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbacaffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:24:32.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbacaffd700 1 -- 172.21.15.121:0/3875513284 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0015030 con 0x562ef9967a20 2024-06-10T00:24:32.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbacaffd700 1 -- 172.21.15.121:0/3875513284 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8802+0+0 (secure 0 0 0) 0x7fbae0011630 con 0x562ef9967a20 2024-06-10T00:24:32.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.841+0000 7fbafc972b80 1 -- 172.21.15.121:0/3875513284 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbae0011d10 con 0x562ef9967a20 2024-06-10T00:24:32.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.842+0000 7fbacaffd700 1 -- 172.21.15.121:0/3875513284 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fbae0011830 con 0x562ef9967a20 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.842+0000 7fbafc972b80 10 client.14691 fetch_fsmap learned FSMap version 16 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.842+0000 7fbafc972b80 1 -- 172.21.15.121:0/3875513284 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x562ef99796b0 con 0x562ef9967a20 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.842+0000 7fbacaffd700 1 -- 172.21.15.121:0/3875513284 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbae0011d10 con 0x562ef9967a20 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.842+0000 7fbacaffd700 1 -- 172.21.15.121:0/3875513284 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7fbae001a430 con 0x562ef9967a20 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.843+0000 7fbafc972b80 10 client.14691 fetch_fsmap finished waiting for FSMap version 16 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.843+0000 7fbafc972b80 10 client.14691 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.843+0000 7fbafc972b80 20 client.14691 populate_metadata read hostname 'smithi121' 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.843+0000 7fbafc972b80 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562ef997b9e0 0x562ef997ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.843+0000 7fbafc972b80 4 client.14691 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.843+0000 7fbafc972b80 1 -- 172.21.15.121:0/3875513284 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ef98bccc0 con 0x562ef997b9e0 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.843+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562ef997b9e0 0x562ef997ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.843+0000 7fbacaffd700 10 client.14691 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:24:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.843+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562ef997b9e0 0x562ef997ce50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff93dd848b1cefbe server_cookie=9b2a1f8ce854b076 in_seq=0 out_seq=0 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.844+0000 7fbacaffd700 1 -- 172.21.15.121:0/3875513284 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x562ef98bccc0 con 0x562ef997b9e0 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.844+0000 7fbacaffd700 10 client.14691 handle_command_reply: tid=0 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.844+0000 7fbafc972b80 1 client.14691 shutdown 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.844+0000 7fbafc972b80 1 -- 172.21.15.121:0/3875513284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9967a20 msgr2=0x562ef996b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.844+0000 7fbafc972b80 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9967a20 0x562ef996b3d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbae000baa0 tx=0x7fbae000b420 comp rx=0 tx=0).stop 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.844+0000 7fbafc972b80 1 -- 172.21.15.121:0/3875513284 shutdown_connections 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.844+0000 7fbafc972b80 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562ef997b9e0 0x562ef997ce50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.844+0000 7fbafc972b80 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9968340 0x562ef996b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.844+0000 7fbafc972b80 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef99622e0 0x562ef996e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.844+0000 7fbafc972b80 1 --2- 172.21.15.121:0/3875513284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef9967a20 0x562ef996b3d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.845+0000 7fbafc972b80 1 -- 172.21.15.121:0/3875513284 >> 172.21.15.121:0/3875513284 conn(0x562ef9888c70 msgr2=0x562ef995c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.845+0000 7fbacaffd700 0 client.14691 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.845+0000 7fbafc972b80 1 -- 172.21.15.121:0/3875513284 shutdown_connections 2024-06-10T00:24:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.845+0000 7fbafc972b80 1 -- 172.21.15.121:0/3875513284 wait complete. 2024-06-10T00:24:32.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.845+0000 7fbafc972b80 20 client.14691 trim_cache size 0 max 16384 2024-06-10T00:24:32.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.865+0000 7fbafc972b80 1 Processor -- start 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.865+0000 7fbafc972b80 1 -- start start 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef996e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9967a20 0x562ef997aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9968340 0x562ef995c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbafc972b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562ef997b190 con 0x562ef99622e0 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbafc972b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x562ef997b860 con 0x562ef9968340 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbafc972b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562ef997bf30 con 0x562ef9967a20 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbaeb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef996e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbaeb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef996e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60916/0 (socket says 172.21.15.121:60916) 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbaeb7fe700 1 -- 172.21.15.121:0/775532249 learned_addr learned my addr 172.21.15.121:0/775532249 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/775532249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9967a20 0x562ef997aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbaeb7fe700 1 -- 172.21.15.121:0/775532249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9967a20 msgr2=0x562ef997aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/775532249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9967a20 0x562ef997aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbaeb7fe700 1 -- 172.21.15.121:0/775532249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9968340 msgr2=0x562ef995c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/775532249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9968340 0x562ef995c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbaeb7fe700 1 -- 172.21.15.121:0/775532249 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ef9969940 con 0x562ef99622e0 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/775532249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9967a20 0x562ef997aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/775532249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef996e0d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbadc009d00 tx=0x7fbadc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e4d941ed544053 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbad17fa700 1 -- 172.21.15.121:0/775532249 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc033070 con 0x562ef99622e0 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbad17fa700 1 -- 172.21.15.121:0/775532249 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbadc00cdd0 con 0x562ef99622e0 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.866+0000 7fbad17fa700 1 -- 172.21.15.121:0/775532249 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc024650 con 0x562ef99622e0 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.867+0000 7fbafc972b80 1 -- 172.21.15.121:0/775532249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 msgr2=0x562ef996e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.867+0000 7fbafc972b80 1 --2- 172.21.15.121:0/775532249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef996e0d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbadc009d00 tx=0x7fbadc015000 comp rx=0 tx=0).stop 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.867+0000 7fbafc972b80 1 -- 172.21.15.121:0/775532249 shutdown_connections 2024-06-10T00:24:32.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.867+0000 7fbafc972b80 1 --2- 172.21.15.121:0/775532249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9968340 0x562ef995c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.867+0000 7fbafc972b80 1 --2- 172.21.15.121:0/775532249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef996e0d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.867+0000 7fbafc972b80 1 --2- 172.21.15.121:0/775532249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9967a20 0x562ef997aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.867+0000 7fbafc972b80 1 -- 172.21.15.121:0/775532249 >> 172.21.15.121:0/775532249 conn(0x562ef9888c70 msgr2=0x562ef9969cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:32.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.867+0000 7fbafc972b80 1 -- 172.21.15.121:0/775532249 shutdown_connections 2024-06-10T00:24:32.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.867+0000 7fbafc972b80 1 -- 172.21.15.121:0/775532249 wait complete. 2024-06-10T00:24:32.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.867+0000 7fbafc972b80 1 -- start start 2024-06-10T00:24:32.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.867+0000 7fbafc972b80 1 Processor -- start 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef9986520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 0x562ef9986a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbafc972b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9968340 0x562ef99771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbafc972b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562ef997ccd0 con 0x562ef99622e0 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbafc972b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x562ef997c600 con 0x562ef9967a20 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbafc972b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562ef997d3a0 con 0x562ef9968340 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbafc972b80 10 client.0 _ll_register_callbacks cb 0x562ef98498d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaeb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef9986520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaeb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef9986520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60928/0 (socket says 172.21.15.121:60928) 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaeb7fe700 1 -- 172.21.15.121:0/1313117036 learned_addr learned my addr 172.21.15.121:0/1313117036 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaebfff700 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9968340 0x562ef99771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 0x562ef9986a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaeb7fe700 1 -- 172.21.15.121:0/1313117036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9968340 msgr2=0x562ef99771f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9968340 0x562ef99771f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaeb7fe700 1 -- 172.21.15.121:0/1313117036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 msgr2=0x562ef9986a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 0x562ef9986a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaeb7fe700 1 -- 172.21.15.121:0/1313117036 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ef9977a60 con 0x562ef99622e0 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaebfff700 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9968340 0x562ef99771f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 0x562ef9986a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbaeb7fe700 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef9986520 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbadc019010 tx=0x7fbadc02eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48530a7be65fcca1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbacaffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbacaffd700 1 -- 172.21.15.121:0/1313117036 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc03e060 con 0x562ef99622e0 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.868+0000 7fbacaffd700 1 -- 172.21.15.121:0/1313117036 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 8802+0+0 (secure 0 0 0) 0x7fbadc02e6d0 con 0x562ef99622e0 2024-06-10T00:24:32.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbafc972b80 1 -- 172.21.15.121:0/1313117036 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbadc00cdd0 con 0x562ef99622e0 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbacaffd700 1 -- 172.21.15.121:0/1313117036 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fbadc045050 con 0x562ef99622e0 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbafc972b80 10 client.14697 fetch_fsmap learned FSMap version 16 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbafc972b80 1 -- 172.21.15.121:0/1313117036 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x562ef9985dd0 con 0x562ef99622e0 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbacaffd700 1 -- 172.21.15.121:0/1313117036 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbadc00cdd0 con 0x562ef99622e0 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbacaffd700 1 -- 172.21.15.121:0/1313117036 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7fbadc02d500 con 0x562ef99622e0 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbafc972b80 10 client.14697 fetch_fsmap finished waiting for FSMap version 16 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbafc972b80 10 client.14697 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbafc972b80 20 client.14697 populate_metadata read hostname 'smithi121' 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbafc972b80 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562ef99eb700 0x562ef99ea410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbafc972b80 4 client.14697 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbafc972b80 1 -- 172.21.15.121:0/1313117036 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "status"}) -- 0x562ef98b7240 con 0x562ef99eb700 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.869+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562ef99eb700 0x562ef99ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbacaffd700 10 client.14697 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:24:32.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbaeaffd700 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562ef99eb700 0x562ef99ea410 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4547bb6b3ba53b97 server_cookie=897396e5e1a3a905 in_seq=0 out_seq=0 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbacaffd700 1 -- 172.21.15.121:0/1313117036 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562ef98b7240 con 0x562ef99eb700 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbacaffd700 10 client.14697 handle_command_reply: tid=0 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 1 client.14697 shutdown 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 1 -- 172.21.15.121:0/1313117036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 msgr2=0x562ef9986520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef9986520 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbadc019010 tx=0x7fbadc02eae0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 1 -- 172.21.15.121:0/1313117036 shutdown_connections 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562ef99eb700 0x562ef99ea410 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562ef9968340 0x562ef99771f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562ef99622e0 0x562ef9986520 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbacaffd700 0 client.14697 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 1 --2- 172.21.15.121:0/1313117036 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562ef9967a20 0x562ef9986a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 1 -- 172.21.15.121:0/1313117036 >> 172.21.15.121:0/1313117036 conn(0x562ef9888c70 msgr2=0x562ef9951db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 1 -- 172.21.15.121:0/1313117036 shutdown_connections 2024-06-10T00:24:32.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 1 -- 172.21.15.121:0/1313117036 wait complete. 2024-06-10T00:24:32.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.870+0000 7fbafc972b80 20 client.14697 trim_cache size 0 max 16384 2024-06-10T00:24:32.877 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:24:32.877 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:24:32.877 INFO:teuthology.orchestra.run.smithi121.stdout: "whoami": 0, 2024-06-10T00:24:32.877 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 24479, 2024-06-10T00:24:32.877 INFO:teuthology.orchestra.run.smithi121.stdout: "want_state": "up:active", 2024-06-10T00:24:32.877 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "up:active", 2024-06-10T00:24:32.877 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs", 2024-06-10T00:24:32.877 INFO:teuthology.orchestra.run.smithi121.stdout: "rank_uptime": 33.324734284000002, 2024-06-10T00:24:32.877 INFO:teuthology.orchestra.run.smithi121.stdout: "mdsmap_epoch": 16, 2024-06-10T00:24:32.878 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch": 96, 2024-06-10T00:24:32.878 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch_barrier": 96, 2024-06-10T00:24:32.878 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 51.618726662999997 2024-06-10T00:24:32.878 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:24:32.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 -- 172.21.15.121:0/725906260 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbad40a2c20 msgr2=0x7fbad40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:32.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbad40a2c20 0x7fbad40a50e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbaec199be0 tx=0x7fbadc0094f0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 -- 172.21.15.121:0/725906260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaec107870 msgr2=0x7fbaec198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:32.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaec107870 0x7fbaec198aa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbae40059d0 tx=0x7fbae4003730 comp rx=0 tx=0).stop 2024-06-10T00:24:32.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 -- 172.21.15.121:0/725906260 shutdown_connections 2024-06-10T00:24:32.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbad40a2c20 0x7fbad40a50e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaec107870 0x7fbaec198aa0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbaec1027a0 0x7fbaec19e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 --2- 172.21.15.121:0/725906260 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbaec103150 0x7fbaec19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:32.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 -- 172.21.15.121:0/725906260 >> 172.21.15.121:0/725906260 conn(0x7fbaec079570 msgr2=0x7fbaec100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:32.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 -- 172.21.15.121:0/725906260 shutdown_connections 2024-06-10T00:24:32.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:32.871+0000 7fbaf20ef700 1 -- 172.21.15.121:0/725906260 wait complete. 2024-06-10T00:24:33.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:33 smithi135 ceph-mon[36424]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:33.508 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:33 smithi121 ceph-mon[32378]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:33.537 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-10T00:24:33.537 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-06-10T00:24:33.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-06-10T00:24:33.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:33 smithi193 ceph-mon[36716]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:33.779 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:35.505 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:35 smithi121 ceph-mon[32378]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:35.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:35 smithi135 ceph-mon[36424]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:35.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.514+0000 7fd60d8e7700 1 -- 172.21.15.121:0/4284964640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101380 msgr2=0x7fd608101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:35.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.514+0000 7fd60d8e7700 1 --2- 172.21.15.121:0/4284964640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101380 0x7fd608101800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009990 tx=0x7fd5fc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:24:35.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.514+0000 7fd60d8e7700 1 -- 172.21.15.121:0/4284964640 shutdown_connections 2024-06-10T00:24:35.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.514+0000 7fd60d8e7700 1 --2- 172.21.15.121:0/4284964640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd608101d40 0x7fd60810e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:35.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.514+0000 7fd60d8e7700 1 --2- 172.21.15.121:0/4284964640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6081073f0 0x7fd6081077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:35.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.514+0000 7fd60d8e7700 1 --2- 172.21.15.121:0/4284964640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101380 0x7fd608101800 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:35.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.514+0000 7fd60d8e7700 1 -- 172.21.15.121:0/4284964640 >> 172.21.15.121:0/4284964640 conn(0x7fd608078300 msgr2=0x7fd608078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:35.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.514+0000 7fd60d8e7700 1 -- 172.21.15.121:0/4284964640 shutdown_connections 2024-06-10T00:24:35.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.514+0000 7fd60d8e7700 1 -- 172.21.15.121:0/4284964640 wait complete. 2024-06-10T00:24:35.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.514+0000 7fd60d8e7700 1 Processor -- start 2024-06-10T00:24:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd60d8e7700 1 -- start start 2024-06-10T00:24:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd60d8e7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101380 0x7fd60819e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd60d8e7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608101d40 0x7fd60819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd60d8e7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6081073f0 0x7fd608198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd60d8e7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd6081116e0 con 0x7fd608101380 2024-06-10T00:24:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd60d8e7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd608111560 con 0x7fd608101d40 2024-06-10T00:24:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd60d8e7700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd608111860 con 0x7fd6081073f0 2024-06-10T00:24:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd606ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101380 0x7fd60819e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:35.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd606ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101380 0x7fd60819e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60942/0 (socket says 172.21.15.121:60942) 2024-06-10T00:24:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd606ffd700 1 -- 172.21.15.121:0/2002012162 learned_addr learned my addr 172.21.15.121:0/2002012162 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd6077fe700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6081073f0 0x7fd608198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd6067fc700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608101d40 0x7fd60819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd606ffd700 1 -- 172.21.15.121:0/2002012162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6081073f0 msgr2=0x7fd608198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd606ffd700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6081073f0 0x7fd608198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd606ffd700 1 -- 172.21.15.121:0/2002012162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608101d40 msgr2=0x7fd60819edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd606ffd700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608101d40 0x7fd60819edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd606ffd700 1 -- 172.21.15.121:0/2002012162 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6081991d0 con 0x7fd608101380 2024-06-10T00:24:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd6067fc700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608101d40 0x7fd60819edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:24:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.515+0000 7fd606ffd700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101380 0x7fd60819e8b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800d950 tx=0x7fd5f800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:35.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.516+0000 7fd60c8e5700 1 -- 172.21.15.121:0/2002012162 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8016070 con 0x7fd608101380 2024-06-10T00:24:35.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.516+0000 7fd60d8e7700 1 -- 172.21.15.121:0/2002012162 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6081994c0 con 0x7fd608101380 2024-06-10T00:24:35.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.516+0000 7fd60c8e5700 1 -- 172.21.15.121:0/2002012162 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5f8009b40 con 0x7fd608101380 2024-06-10T00:24:35.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.516+0000 7fd60d8e7700 1 -- 172.21.15.121:0/2002012162 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6081a5780 con 0x7fd608101380 2024-06-10T00:24:35.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.516+0000 7fd60c8e5700 1 -- 172.21.15.121:0/2002012162 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f800f620 con 0x7fd608101380 2024-06-10T00:24:35.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.517+0000 7fd60d8e7700 1 -- 172.21.15.121:0/2002012162 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f4005330 con 0x7fd608101380 2024-06-10T00:24:35.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.518+0000 7fd60c8e5700 1 -- 172.21.15.121:0/2002012162 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd5f8019070 con 0x7fd608101380 2024-06-10T00:24:35.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.518+0000 7fd60c8e5700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2ca0 0x7fd5f00a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:35.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.518+0000 7fd6067fc700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2ca0 0x7fd5f00a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:35.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.519+0000 7fd60c8e5700 1 -- 172.21.15.121:0/2002012162 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8802+0+0 (secure 0 0 0) 0x7fd5f80cf0e0 con 0x7fd608101380 2024-06-10T00:24:35.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.519+0000 7fd6067fc700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2ca0 0x7fd5f00a5160 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00b5c0 tx=0x7fd5fc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:35.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.522+0000 7fd60c8e5700 1 -- 172.21.15.121:0/2002012162 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd5f809be00 con 0x7fd608101380 2024-06-10T00:24:35.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:35 smithi193 ceph-mon[36716]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:35.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.766+0000 7fd60d8e7700 1 -- 172.21.15.121:0/2002012162 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd5f40058f0 con 0x7fd608101380 2024-06-10T00:24:35.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.766+0000 7fd60c8e5700 1 -- 172.21.15.121:0/2002012162 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7fd5f809b700 con 0x7fd608101380 2024-06-10T00:24:35.766 INFO:teuthology.orchestra.run.smithi121.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 -- 172.21.15.121:0/2002012162 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2ca0 msgr2=0x7fd5f00a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2ca0 0x7fd5f00a5160 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00b5c0 tx=0x7fd5fc005fb0 comp rx=0 tx=0).stop 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 -- 172.21.15.121:0/2002012162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101380 msgr2=0x7fd60819e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101380 0x7fd60819e8b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800d950 tx=0x7fd5f800dd10 comp rx=0 tx=0).stop 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 -- 172.21.15.121:0/2002012162 shutdown_connections 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd5f00a2ca0 0x7fd5f00a5160 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6081073f0 0x7fd608198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd608101380 0x7fd60819e8b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 --2- 172.21.15.121:0/2002012162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd608101d40 0x7fd60819edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 -- 172.21.15.121:0/2002012162 >> 172.21.15.121:0/2002012162 conn(0x7fd608078300 msgr2=0x7fd608100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 -- 172.21.15.121:0/2002012162 shutdown_connections 2024-06-10T00:24:35.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:35.769+0000 7fd60d8e7700 1 -- 172.21.15.121:0/2002012162 wait complete. 2024-06-10T00:24:36.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:36 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2002012162' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-10T00:24:36.422 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-06-10T00:24:36.422 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-10T00:24:36.422 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi121.front.sepia.ceph.com 2024-06-10T00:24:36.422 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-06-10T00:24:36.422 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-06-10T00:24:36.422 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-06-10T00:24:36.422 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-10T00:24:36.422 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-10T00:24:36.422 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-06-10T00:24:36.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:36.423 DEBUG:teuthology.orchestra.run.smithi121:> ip addr 2024-06-10T00:24:36.439 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-10T00:24:36.439 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-10T00:24:36.439 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:82 brd ff:ff:ff:ff:ff:ff 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f0 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:83 brd ff:ff:ff:ff:ff:ff 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f1 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a2 brd ff:ff:ff:ff:ff:ff 2024-06-10T00:24:36.440 INFO:teuthology.orchestra.run.smithi121.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-10T00:24:36.441 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a3 brd ff:ff:ff:ff:ff:ff 2024-06-10T00:24:36.441 INFO:teuthology.orchestra.run.smithi121.stdout: inet 172.21.15.121/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-10T00:24:36.441 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft 42195sec preferred_lft 42195sec 2024-06-10T00:24:36.441 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2024-06-10T00:24:36.441 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-06-10T00:24:36.441 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-06-10T00:24:36.442 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-10T00:24:36.442 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-06-10T00:24:36.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add name ceph-brx type bridge 2024-06-10T00:24:36.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr flush dev ceph-brx 2024-06-10T00:24:36.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set ceph-brx up 2024-06-10T00:24:36.442 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-06-10T00:24:36.442 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-06-10T00:24:36.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:36 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2002012162' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-10T00:24:36.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-10T00:24:36.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:36 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2002012162' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-10T00:24:36.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:24:36.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:36.643 DEBUG:teuthology.orchestra.run.smithi121:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-10T00:24:36.667 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-06-10T00:24:36.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:36.669 DEBUG:teuthology.orchestra.run.smithi121:> route 2024-06-10T00:24:36.751 INFO:teuthology.orchestra.run.smithi121.stdout:Kernel IP routing table 2024-06-10T00:24:36.752 INFO:teuthology.orchestra.run.smithi121.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-10T00:24:36.752 INFO:teuthology.orchestra.run.smithi121.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-10T00:24:36.752 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-10T00:24:36.752 INFO:teuthology.orchestra.run.smithi121.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-10T00:24:36.752 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-10T00:24:36.752 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-06-10T00:24:36.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-10T00:24:36.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-10T00:24:36.753 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-10T00:24:36.753 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-06-10T00:24:36.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-10T00:24:37.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:24:37.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:37.059 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-06-10T00:24:37.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:37.075 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list-id 2024-06-10T00:24:37.131 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-10T00:24:37.131 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-06-10T00:24:37.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-10T00:24:37.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-06-10T00:24:37.132 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-06-10T00:24:37.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-10T00:24:37.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:24:37.257 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-06-10T00:24:37.258 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-10T00:24:37.258 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-06-10T00:24:37.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-06-10T00:24:37.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-06-10T00:24:37.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-06-10T00:24:37.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-06-10T00:24:37.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-06-10T00:24:37.259 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-06-10T00:24:37.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-10T00:24:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:24:37.383 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-10T00:24:37.383 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-06-10T00:24:37.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set brx.0 up 2024-06-10T00:24:37.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set dev brx.0 master ceph-brx 2024-06-10T00:24:37.384 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-06-10T00:24:37.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-10T00:24:37.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:24:37.462 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-06-10T00:24:37.462 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-10T00:24:37.462 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-06-10T00:24:37.477 INFO:teuthology.orchestra.run.smithi121.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-06-10T00:24:37.477 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-10T00:24:37.477 DEBUG:teuthology.orchestra.run.smithi121:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-06-10T00:24:37.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:37 smithi135 ceph-mon[36424]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:37.532 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2024-06-10T00:24:37.594 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:37 smithi121 ceph-mon[32378]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:37.598 DEBUG:teuthology.orchestra.run.smithi121:> cat /proc/self/mounts | awk '{print $2}' 2024-06-10T00:24:37.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:37 smithi193 ceph-mon[36716]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:37.655 INFO:teuthology.orchestra.run.smithi121.stdout:/proc 2024-06-10T00:24:37.655 INFO:teuthology.orchestra.run.smithi121.stdout:/sys 2024-06-10T00:24:37.655 INFO:teuthology.orchestra.run.smithi121.stdout:/dev 2024-06-10T00:24:37.655 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/security 2024-06-10T00:24:37.655 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/shm 2024-06-10T00:24:37.655 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/pts 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/run 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/cgroup 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/pstore 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/bpf 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/ 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/selinux 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/mqueue 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/debug 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/hugepages 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/tracing 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/fuse/connections 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/config 2024-06-10T00:24:37.656 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-sysctl.service 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/nfs/rpc_pipefs 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/1000 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/622c4f1a93dd4f2b22b329835cd2812f4c1f15f21c9656191ffee99295e6eb26/merged 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/73cbe159337f7bf270e1771a1749def1a68efc61c27eb4584cf8eed2cd1ab747/merged 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/0 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/7b329ac753f76c08018cf4f0475ebe8d9f94e8ce61922451957619e948e75ee0/merged 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/274b75b7f792331c5a30be972722840c37b90da2b7b6eeb5eaea68c66ed9df0f/merged 2024-06-10T00:24:37.657 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/0944eb3314744e1698af30d3956b82fcb82bdde6f0b96b440b6662297baa2a4c/merged 2024-06-10T00:24:37.658 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/cfaac1049b32142faf0e69d9e9cbdbf2195d8e0cbeb7f52028c91e51dfb6ebb3/merged 2024-06-10T00:24:37.658 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/48ecfac32414f44f1bd2b4cbb4094efa4e3e0d007507efd559be7e5af8208355/merged 2024-06-10T00:24:37.658 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/e4cea2e006a182f0554fe8a3487a8bcc0f4c7bd4730c70908bf0a84220b3d418/merged 2024-06-10T00:24:37.658 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/4c3daf5b457f24d95e12b2edff0728cb4e8d6f7d7f2f4857f944125a4496ad39/merged 2024-06-10T00:24:37.658 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/568b3a748a1ad52793cf29a7530f00ca68cfe0d3cceb2b90c60571f448a18757/merged 2024-06-10T00:24:37.658 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns 2024-06-10T00:24:37.658 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-10T00:24:37.658 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-10T00:24:37.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:37.658 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2024-06-10T00:24:37.711 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-06-10T00:24:37.711 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-06-10T00:24:37.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2024-06-10T00:24:37.778 DEBUG:teuthology.orchestra.run.smithi121:> cat /proc/self/mounts | awk '{print $2}' 2024-06-10T00:24:37.834 INFO:teuthology.orchestra.run.smithi121.stdout:/proc 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/sys 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/dev 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/security 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/shm 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/pts 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/run 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/cgroup 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/pstore 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/bpf 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/ 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/selinux 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2024-06-10T00:24:37.835 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/mqueue 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/debug 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/hugepages 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/tracing 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/fuse/connections 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/config 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-sysctl.service 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/nfs/rpc_pipefs 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/1000 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/622c4f1a93dd4f2b22b329835cd2812f4c1f15f21c9656191ffee99295e6eb26/merged 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/73cbe159337f7bf270e1771a1749def1a68efc61c27eb4584cf8eed2cd1ab747/merged 2024-06-10T00:24:37.836 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/0 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/7b329ac753f76c08018cf4f0475ebe8d9f94e8ce61922451957619e948e75ee0/merged 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/274b75b7f792331c5a30be972722840c37b90da2b7b6eeb5eaea68c66ed9df0f/merged 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/0944eb3314744e1698af30d3956b82fcb82bdde6f0b96b440b6662297baa2a4c/merged 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/cfaac1049b32142faf0e69d9e9cbdbf2195d8e0cbeb7f52028c91e51dfb6ebb3/merged 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/48ecfac32414f44f1bd2b4cbb4094efa4e3e0d007507efd559be7e5af8208355/merged 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/e4cea2e006a182f0554fe8a3487a8bcc0f4c7bd4730c70908bf0a84220b3d418/merged 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/4c3daf5b457f24d95e12b2edff0728cb4e8d6f7d7f2f4857f944125a4496ad39/merged 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/containers/storage/overlay/568b3a748a1ad52793cf29a7530f00ca68cfe0d3cceb2b90c60571f448a18757/merged 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-10T00:24:37.837 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-10T00:24:37.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:37.838 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2024-06-10T00:24:37.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi121.stderr:2024-06-10T00:24:37.864+0000 7f10fd309480 -1 init, newargv = 0x55b38baab7c0 newargc=15 2024-06-10T00:24:37.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi121.stderr:ceph-fuse[78296]: starting ceph client 2024-06-10T00:24:37.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi121.stderr:ceph-fuse[78296]: starting fuse 2024-06-10T00:24:37.880 INFO:teuthology.orchestra.run.smithi121.stdout:119 2024-06-10T00:24:37.880 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-06-10T00:24:37.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo stdin-killer -- python3 -c ' 2024-06-10T00:24:37.880 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2024-06-10T00:24:37.880 DEBUG:teuthology.orchestra.run.smithi121:> import re 2024-06-10T00:24:37.880 DEBUG:teuthology.orchestra.run.smithi121:> import os 2024-06-10T00:24:37.880 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2024-06-10T00:24:37.880 DEBUG:teuthology.orchestra.run.smithi121:> 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> def _find_admin_socket(client_name): 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2024-06-10T00:24:37.881 DEBUG:teuthology.orchestra.run.smithi121:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-06-10T00:24:37.882 DEBUG:teuthology.orchestra.run.smithi121:> contents = proc_f.read() 2024-06-10T00:24:37.882 DEBUG:teuthology.orchestra.run.smithi121:> if mountpoint in contents: 2024-06-10T00:24:37.882 DEBUG:teuthology.orchestra.run.smithi121:> return f 2024-06-10T00:24:37.882 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-06-10T00:24:37.882 DEBUG:teuthology.orchestra.run.smithi121:> 2024-06-10T00:24:37.882 DEBUG:teuthology.orchestra.run.smithi121:> print(_find_admin_socket("client.admin")) 2024-06-10T00:24:37.882 DEBUG:teuthology.orchestra.run.smithi121:> ' 2024-06-10T00:24:37.999 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.admin.78296.asok 2024-06-10T00:24:38.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:24:38.010 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.78296.asok 2024-06-10T00:24:38.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:38.013 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.78296.asok status 2024-06-10T00:24:38.147 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:24:38.147 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2024-06-10T00:24:38.147 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-10T00:24:38.147 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)", 2024-06-10T00:24:38.147 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "admin", 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "78296", 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 14709, 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 14709 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:24:38.148 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.1:0", 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2645963340 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.1:0", 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2645963340 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.14709 192.168.0.1:0/2645963340", 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "192.168.0.1:0/2645963340", 2024-06-10T00:24:38.149 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2024-06-10T00:24:38.150 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 16, 2024-06-10T00:24:38.150 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 96, 2024-06-10T00:24:38.150 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2024-06-10T00:24:38.150 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklisted": false, 2024-06-10T00:24:38.150 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2024-06-10T00:24:38.150 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:24:38.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:38.156 DEBUG:teuthology.orchestra.run.smithi121:> stat --file-system '--printf=%T 2024-06-10T00:24:38.156 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-06-10T00:24:38.172 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2024-06-10T00:24:38.172 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-06-10T00:24:38.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:38.172 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-06-10T00:24:38.239 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-06-10T00:24:38.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-10T00:24:38.350 INFO:teuthology.orchestra.run.smithi121.stdout:# file: . 2024-06-10T00:24:38.350 INFO:teuthology.orchestra.run.smithi121.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-06-10T00:24:38.350 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:24:38.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:24:38.356 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:24:38.356 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-10T00:24:38.373 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-06-10T00:24:38.651 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:39.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:39 smithi121 ceph-mon[32378]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:39.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:39 smithi135 ceph-mon[36424]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:39.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:39 smithi193 ceph-mon[36716]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:40.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.411+0000 7fc3eda40700 1 -- 172.21.15.121:0/3593453573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e8102030 msgr2=0x7fc3e8105ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:40.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.411+0000 7fc3eda40700 1 --2- 172.21.15.121:0/3593453573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e8102030 0x7fc3e8105ff0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0099c0 tx=0x7fc3dc009cd0 comp rx=0 tx=0).stop 2024-06-10T00:24:40.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.411+0000 7fc3eda40700 1 -- 172.21.15.121:0/3593453573 shutdown_connections 2024-06-10T00:24:40.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.411+0000 7fc3eda40700 1 --2- 172.21.15.121:0/3593453573 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3e81066d0 0x7fc3e810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:40.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.411+0000 7fc3eda40700 1 --2- 172.21.15.121:0/3593453573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3e8101680 0x7fc3e8101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:40.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.411+0000 7fc3eda40700 1 --2- 172.21.15.121:0/3593453573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e8102030 0x7fc3e8105ff0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:40.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.411+0000 7fc3eda40700 1 -- 172.21.15.121:0/3593453573 >> 172.21.15.121:0/3593453573 conn(0x7fc3e80773e0 msgr2=0x7fc3e80777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:40.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.411+0000 7fc3eda40700 1 -- 172.21.15.121:0/3593453573 shutdown_connections 2024-06-10T00:24:40.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.412+0000 7fc3eda40700 1 -- 172.21.15.121:0/3593453573 wait complete. 2024-06-10T00:24:40.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.412+0000 7fc3eda40700 1 Processor -- start 2024-06-10T00:24:40.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.412+0000 7fc3eda40700 1 -- start start 2024-06-10T00:24:40.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3eda40700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3e8101680 0x7fc3e819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:40.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3eda40700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3e8102030 0x7fc3e819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:40.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3eda40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e81066d0 0x7fc3e8198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:40.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3eda40700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e8111690 con 0x7fc3e81066d0 2024-06-10T00:24:40.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3eda40700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e8111510 con 0x7fc3e8102030 2024-06-10T00:24:40.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3eda40700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e8111810 con 0x7fc3e8101680 2024-06-10T00:24:40.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e81066d0 0x7fc3e8198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:40.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e81066d0 0x7fc3e8198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36024/0 (socket says 172.21.15.121:36024) 2024-06-10T00:24:40.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e77fe700 1 -- 172.21.15.121:0/1042557197 learned_addr learned my addr 172.21.15.121:0/1042557197 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:40.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e6ffd700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3e8101680 0x7fc3e819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:40.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e67fc700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3e8102030 0x7fc3e819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:40.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e67fc700 1 -- 172.21.15.121:0/1042557197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3e8101680 msgr2=0x7fc3e819e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:40.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e67fc700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3e8101680 0x7fc3e819e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:40.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e67fc700 1 -- 172.21.15.121:0/1042557197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e81066d0 msgr2=0x7fc3e8198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:40.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e67fc700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e81066d0 0x7fc3e8198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:40.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e67fc700 1 -- 172.21.15.121:0/1042557197 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e8199170 con 0x7fc3e8102030 2024-06-10T00:24:40.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e77fe700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e81066d0 0x7fc3e8198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:24:40.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e6ffd700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3e8101680 0x7fc3e819e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:24:40.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.413+0000 7fc3e67fc700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3e8102030 0x7fc3e819ed90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0098e0 tx=0x7fc3dc0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:40.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.414+0000 7fc3eca3e700 1 -- 172.21.15.121:0/1042557197 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc027070 con 0x7fc3e8102030 2024-06-10T00:24:40.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.414+0000 7fc3eca3e700 1 -- 172.21.15.121:0/1042557197 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3dc003e40 con 0x7fc3e8102030 2024-06-10T00:24:40.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.414+0000 7fc3eda40700 1 -- 172.21.15.121:0/1042557197 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e8199400 con 0x7fc3e8102030 2024-06-10T00:24:40.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.414+0000 7fc3eca3e700 1 -- 172.21.15.121:0/1042557197 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc022ba0 con 0x7fc3e8102030 2024-06-10T00:24:40.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.414+0000 7fc3eda40700 1 -- 172.21.15.121:0/1042557197 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e81a5720 con 0x7fc3e8102030 2024-06-10T00:24:40.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.415+0000 7fc3eda40700 1 -- 172.21.15.121:0/1042557197 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3e8102ad0 con 0x7fc3e8102030 2024-06-10T00:24:40.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.417+0000 7fc3eca3e700 1 -- 172.21.15.121:0/1042557197 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc3dc02a070 con 0x7fc3e8102030 2024-06-10T00:24:40.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.417+0000 7fc3eca3e700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc3d00a2ca0 0x7fc3d00a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:40.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.417+0000 7fc3e6ffd700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc3d00a2ca0 0x7fc3d00a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:40.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.417+0000 7fc3eca3e700 1 -- 172.21.15.121:0/1042557197 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8802+0+0 (secure 0 0 0) 0x7fc3dc0d0860 con 0x7fc3e8102030 2024-06-10T00:24:40.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.418+0000 7fc3e6ffd700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc3d00a2ca0 0x7fc3d00a5160 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8005950 tx=0x7fc3d800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:40.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.420+0000 7fc3eca3e700 1 -- 172.21.15.121:0/1042557197 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc3dc031070 con 0x7fc3e8102030 2024-06-10T00:24:40.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.630+0000 7fc3eda40700 1 -- 172.21.15.121:0/1042557197 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc3e802dca0 con 0x7fc3e8102030 2024-06-10T00:24:40.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.630+0000 7fc3eca3e700 1 -- 172.21.15.121:0/1042557197 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v96) ==== 81+0+598 (secure 0 0 0) 0x7fc3dc013070 con 0x7fc3e8102030 2024-06-10T00:24:40.630 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1710032094 2024-06-11T00:16:42.011037+0000 2024-06-10T00:24:40.631 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/704666019 2024-06-11T00:16:42.011037+0000 2024-06-10T00:24:40.631 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/986214985 2024-06-11T00:16:42.011037+0000 2024-06-10T00:24:40.631 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/3588432900 2024-06-11T00:16:15.442852+0000 2024-06-10T00:24:40.631 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/3588432900 2024-06-11T00:16:15.442852+0000 2024-06-10T00:24:40.631 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/704666019 2024-06-11T00:16:42.011037+0000 2024-06-10T00:24:40.631 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/743919928 2024-06-11T00:16:42.011037+0000 2024-06-10T00:24:40.631 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/2834833132 2024-06-11T00:16:15.442852+0000 2024-06-10T00:24:40.632 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1390094598 2024-06-11T00:16:15.442852+0000 2024-06-10T00:24:40.632 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1651500994 2024-06-11T00:16:15.442852+0000 2024-06-10T00:24:40.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 -- 172.21.15.121:0/1042557197 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc3d00a2ca0 msgr2=0x7fc3d00a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:40.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc3d00a2ca0 0x7fc3d00a5160 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8005950 tx=0x7fc3d800a300 comp rx=0 tx=0).stop 2024-06-10T00:24:40.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 -- 172.21.15.121:0/1042557197 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3e8102030 msgr2=0x7fc3e819ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:40.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3e8102030 0x7fc3e819ed90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0098e0 tx=0x7fc3dc0047b0 comp rx=0 tx=0).stop 2024-06-10T00:24:40.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 -- 172.21.15.121:0/1042557197 shutdown_connections 2024-06-10T00:24:40.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc3d00a2ca0 0x7fc3d00a5160 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:40.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3e81066d0 0x7fc3e8198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:40.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3e8101680 0x7fc3e819e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:40.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 --2- 172.21.15.121:0/1042557197 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3e8102030 0x7fc3e819ed90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:40.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 -- 172.21.15.121:0/1042557197 >> 172.21.15.121:0/1042557197 conn(0x7fc3e80773e0 msgr2=0x7fc3e806aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:40.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 -- 172.21.15.121:0/1042557197 shutdown_connections 2024-06-10T00:24:40.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:40.633+0000 7fc3eda40700 1 -- 172.21.15.121:0/1042557197 wait complete. 2024-06-10T00:24:40.634 INFO:teuthology.orchestra.run.smithi121.stderr:listed 10 entries 2024-06-10T00:24:41.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:41 smithi121 ceph-mon[32378]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-06-10T00:24:41.338 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:41 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1042557197' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-10T00:24:41.366 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:24:41.366 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-10T00:24:41.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-06-10T00:24:41.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:41 smithi135 ceph-mon[36424]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-06-10T00:24:41.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:41 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1042557197' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-10T00:24:41.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:41 smithi193 ceph-mon[36716]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-06-10T00:24:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:41 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1042557197' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-10T00:24:41.665 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-06-10T00:24:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-06-10T00:24:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-06-10T00:24:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-06-10T00:24:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-06-10T00:24:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2024-06-10T00:24:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-06-10T00:24:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-06-10T00:24:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]: dispatch 2024-06-10T00:24:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:42 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-06-10T00:24:42.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-06-10T00:24:42.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-06-10T00:24:42.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-06-10T00:24:42.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-06-10T00:24:42.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-06-10T00:24:42.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2024-06-10T00:24:42.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-06-10T00:24:42.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-06-10T00:24:42.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]: dispatch 2024-06-10T00:24:42.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:42 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-06-10T00:24:42.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-06-10T00:24:42.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-06-10T00:24:42.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-06-10T00:24:42.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-06-10T00:24:42.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-06-10T00:24:42.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2024-06-10T00:24:42.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-06-10T00:24:42.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-06-10T00:24:42.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]: dispatch 2024-06-10T00:24:42.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:42 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-06-10T00:24:43.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-06-10T00:24:43.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-06-10T00:24:43.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-06-10T00:24:43.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-06-10T00:24:43.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-06-10T00:24:43.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-06-10T00:24:43.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-06-10T00:24:43.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-06-10T00:24:43.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-06-10T00:24:43.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]': finished 2024-06-10T00:24:43.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-06-10T00:24:43.359 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:43 smithi121 ceph-mon[32378]: osdmap e97: 12 total, 12 up, 12 in 2024-06-10T00:24:43.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.357+0000 7f368450c700 1 -- 172.21.15.121:0/359477800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f367c108500 msgr2=0x7f367c1088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:43.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.357+0000 7f368450c700 1 --2- 172.21.15.121:0/359477800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f367c108500 0x7f367c1088e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3678009990 tx=0x7f3678009ca0 comp rx=0 tx=0).stop 2024-06-10T00:24:43.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.357+0000 7f368450c700 1 -- 172.21.15.121:0/359477800 shutdown_connections 2024-06-10T00:24:43.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.357+0000 7f368450c700 1 --2- 172.21.15.121:0/359477800 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f367c06ab30 0x7f367c10f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:43.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.357+0000 7f368450c700 1 --2- 172.21.15.121:0/359477800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f367c108500 0x7f367c1088e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.357+0000 7f368450c700 1 --2- 172.21.15.121:0/359477800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f367c06a170 0x7f367c06a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.357+0000 7f368450c700 1 -- 172.21.15.121:0/359477800 >> 172.21.15.121:0/359477800 conn(0x7f367c078340 msgr2=0x7f367c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.357+0000 7f368450c700 1 -- 172.21.15.121:0/359477800 shutdown_connections 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.357+0000 7f368450c700 1 -- 172.21.15.121:0/359477800 wait complete. 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.358+0000 7f368450c700 1 Processor -- start 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.358+0000 7f368450c700 1 -- start start 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.358+0000 7f368450c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f367c06a170 0x7f367c19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f368450c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f367c06ab30 0x7f367c19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f368450c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f367c108500 0x7f367c198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f368450c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f367c112780 con 0x7f367c06a170 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f368450c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f367c112600 con 0x7f367c06ab30 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f368450c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f367c112900 con 0x7f367c108500 2024-06-10T00:24:43.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36822a8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f367c06a170 0x7f367c19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36822a8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f367c06a170 0x7f367c19e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36042/0 (socket says 172.21.15.121:36042) 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36822a8700 1 -- 172.21.15.121:0/2390009944 learned_addr learned my addr 172.21.15.121:0/2390009944 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f3681aa7700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f367c06ab30 0x7f367c19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f3682aa9700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f367c108500 0x7f367c198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36822a8700 1 -- 172.21.15.121:0/2390009944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f367c108500 msgr2=0x7f367c198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36822a8700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f367c108500 0x7f367c198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36822a8700 1 -- 172.21.15.121:0/2390009944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f367c06ab30 msgr2=0x7f367c19ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36822a8700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f367c06ab30 0x7f367c19ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36822a8700 1 -- 172.21.15.121:0/2390009944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f367c1991c0 con 0x7f367c06a170 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f3682aa9700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f367c108500 0x7f367c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f3681aa7700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f367c06ab30 0x7f367c19ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:24:43.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36822a8700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f367c06a170 0x7f367c19e8a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3678000c00 tx=0x7f367800bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36737fe700 1 -- 172.21.15.121:0/2390009944 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3678027070 con 0x7f367c06a170 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f368450c700 1 -- 172.21.15.121:0/2390009944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f367c199450 con 0x7f367c06a170 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36737fe700 1 -- 172.21.15.121:0/2390009944 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3678022470 con 0x7f367c06a170 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f36737fe700 1 -- 172.21.15.121:0/2390009944 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367800f6a0 con 0x7f367c06a170 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.359+0000 7f368450c700 1 -- 172.21.15.121:0/2390009944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f367c1a5770 con 0x7f367c06a170 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.360+0000 7f368450c700 1 -- 172.21.15.121:0/2390009944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f367c10a9e0 con 0x7f367c06a170 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.362+0000 7f36737fe700 1 -- 172.21.15.121:0/2390009944 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f367802a070 con 0x7f367c06a170 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.362+0000 7f36737fe700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f36680a2c20 0x7f36680a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.362+0000 7f3681aa7700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f36680a2c20 0x7f36680a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.362+0000 7f36737fe700 1 -- 172.21.15.121:0/2390009944 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f36780d06f0 con 0x7f367c06a170 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.363+0000 7f3681aa7700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f36680a2c20 0x7f36680a50e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f367c078a20 tx=0x7f366c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.366+0000 7f36737fe700 1 -- 172.21.15.121:0/2390009944 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3678013070 con 0x7f367c06a170 2024-06-10T00:24:43.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.575+0000 7f368450c700 1 -- 172.21.15.121:0/2390009944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f367c199f90 con 0x7f367c06a170 2024-06-10T00:24:43.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.575+0000 7f36737fe700 1 -- 172.21.15.121:0/2390009944 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v98) ==== 81+0+598 (secure 0 0 0) 0x7f367809d380 con 0x7f367c06a170 2024-06-10T00:24:43.575 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1710032094 2024-06-11T00:16:42.011037+0000 2024-06-10T00:24:43.576 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/704666019 2024-06-11T00:16:42.011037+0000 2024-06-10T00:24:43.576 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/986214985 2024-06-11T00:16:42.011037+0000 2024-06-10T00:24:43.576 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/3588432900 2024-06-11T00:16:15.442852+0000 2024-06-10T00:24:43.576 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/3588432900 2024-06-11T00:16:15.442852+0000 2024-06-10T00:24:43.576 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/704666019 2024-06-11T00:16:42.011037+0000 2024-06-10T00:24:43.576 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/743919928 2024-06-11T00:16:42.011037+0000 2024-06-10T00:24:43.576 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/2834833132 2024-06-11T00:16:15.442852+0000 2024-06-10T00:24:43.576 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1390094598 2024-06-11T00:16:15.442852+0000 2024-06-10T00:24:43.576 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1651500994 2024-06-11T00:16:15.442852+0000 2024-06-10T00:24:43.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 -- 172.21.15.121:0/2390009944 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f36680a2c20 msgr2=0x7f36680a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:43.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f36680a2c20 0x7f36680a50e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f367c078a20 tx=0x7f366c009260 comp rx=0 tx=0).stop 2024-06-10T00:24:43.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 -- 172.21.15.121:0/2390009944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f367c06a170 msgr2=0x7f367c19e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:43.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f367c06a170 0x7f367c19e8a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3678000c00 tx=0x7f367800bb60 comp rx=0 tx=0).stop 2024-06-10T00:24:43.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 -- 172.21.15.121:0/2390009944 shutdown_connections 2024-06-10T00:24:43.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f36680a2c20 0x7f36680a50e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:43.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f367c108500 0x7f367c198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:43.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f367c06a170 0x7f367c19e8a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:43.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 --2- 172.21.15.121:0/2390009944 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f367c06ab30 0x7f367c19ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:43.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 -- 172.21.15.121:0/2390009944 >> 172.21.15.121:0/2390009944 conn(0x7f367c078340 msgr2=0x7f367c100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:43.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 -- 172.21.15.121:0/2390009944 shutdown_connections 2024-06-10T00:24:43.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:43.578+0000 7f368450c700 1 -- 172.21.15.121:0/2390009944 wait complete. 2024-06-10T00:24:43.578 INFO:teuthology.orchestra.run.smithi121.stderr:listed 10 entries 2024-06-10T00:24:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-06-10T00:24:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-06-10T00:24:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-06-10T00:24:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-06-10T00:24:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-06-10T00:24:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-06-10T00:24:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-06-10T00:24:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-06-10T00:24:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-06-10T00:24:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]': finished 2024-06-10T00:24:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-06-10T00:24:43.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:43 smithi193 ceph-mon[36716]: osdmap e97: 12 total, 12 up, 12 in 2024-06-10T00:24:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-06-10T00:24:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-06-10T00:24:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-06-10T00:24:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-06-10T00:24:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-06-10T00:24:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-06-10T00:24:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2024-06-10T00:24:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-06-10T00:24:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-06-10T00:24:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]': finished 2024-06-10T00:24:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-06-10T00:24:43.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:43 smithi135 ceph-mon[36424]: osdmap e97: 12 total, 12 up, 12 in 2024-06-10T00:24:44.288 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2024-06-10T00:24:44.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:44.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-06-10T00:24:44.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi121.stderr:ceph-fuse[78296]: fuse finished with error 0 and tester_r 0 2024-06-10T00:24:44.316 INFO:teuthology.orchestra.run:waiting for 300 2024-06-10T00:24:44.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:44 smithi121 ceph-mon[32378]: osdmap e98: 12 total, 12 up, 12 in 2024-06-10T00:24:44.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:44 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2390009944' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-10T00:24:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:44 smithi193 ceph-mon[36716]: osdmap e98: 12 total, 12 up, 12 in 2024-06-10T00:24:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:44 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2390009944' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-10T00:24:44.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:44 smithi135 ceph-mon[36424]: osdmap e98: 12 total, 12 up, 12 in 2024-06-10T00:24:44.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:44 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2390009944' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-10T00:24:45.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:45 smithi193 ceph-mon[36716]: pgmap v329: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-06-10T00:24:45.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:45 smithi121 ceph-mon[32378]: pgmap v329: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-06-10T00:24:45.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:45 smithi135 ceph-mon[36424]: pgmap v329: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-06-10T00:24:46.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:46 smithi193 ceph-mon[36716]: pgmap v330: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-06-10T00:24:46.682 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:46 smithi121 ceph-mon[32378]: pgmap v330: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-06-10T00:24:46.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:46 smithi135 ceph-mon[36424]: pgmap v330: 129 pgs: 3 peering, 126 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2024-06-10T00:24:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:49 smithi121 ceph-mon[32378]: pgmap v331: 129 pgs: 2 peering, 127 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:49 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:24:49.455 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:49 smithi135 ceph-mon[36424]: pgmap v331: 129 pgs: 2 peering, 127 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:49.455 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:49 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:24:49.588 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:49 smithi193 ceph-mon[36716]: pgmap v331: 129 pgs: 2 peering, 127 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:24:49.589 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:49 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:24:50.418 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-06-10T00:24:50.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:24:50.418 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-06-10T00:24:50.451 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:24:50.736 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:51 smithi121 ceph-mon[32378]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-06-10T00:24:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:24:51.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:24:51.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:51 smithi135 ceph-mon[36424]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-06-10T00:24:51.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:24:51.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:24:51.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:51 smithi193 ceph-mon[36716]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-06-10T00:24:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:24:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:24:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:24:52.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.459+0000 7f14d69a1700 1 -- 172.21.15.121:0/3659747542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14d0107400 msgr2=0x7f14d01077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:52.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.459+0000 7f14d69a1700 1 --2- 172.21.15.121:0/3659747542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14d0107400 0x7f14d01077e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f14c0009990 tx=0x7f14c0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:24:52.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.459+0000 7f14d69a1700 1 -- 172.21.15.121:0/3659747542 shutdown_connections 2024-06-10T00:24:52.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.459+0000 7f14d69a1700 1 --2- 172.21.15.121:0/3659747542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d0101d50 0x7f14d010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:52.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.459+0000 7f14d69a1700 1 --2- 172.21.15.121:0/3659747542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14d0107400 0x7f14d01077e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:52.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.459+0000 7f14d69a1700 1 --2- 172.21.15.121:0/3659747542 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14d0101390 0x7f14d0101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:52.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.459+0000 7f14d69a1700 1 -- 172.21.15.121:0/3659747542 >> 172.21.15.121:0/3659747542 conn(0x7f14d0078340 msgr2=0x7f14d0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.459+0000 7f14d69a1700 1 -- 172.21.15.121:0/3659747542 shutdown_connections 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.459+0000 7f14d69a1700 1 -- 172.21.15.121:0/3659747542 wait complete. 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.460+0000 7f14d69a1700 1 Processor -- start 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.460+0000 7f14d69a1700 1 -- start start 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.460+0000 7f14d69a1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d0101390 0x7f14d019e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.460+0000 7f14d69a1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14d0101d50 0x7f14d019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.460+0000 7f14d69a1700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14d0107400 0x7f14d0198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.460+0000 7f14d69a1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f14d01116a0 con 0x7f14d0101d50 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.460+0000 7f14d69a1700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f14d0111520 con 0x7f14d0107400 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.460+0000 7f14d69a1700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14d0111820 con 0x7f14d0101390 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.460+0000 7f14c7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14d0101d50 0x7f14d019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:52.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.460+0000 7f14c7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14d0101d50 0x7f14d019edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35372/0 (socket says 172.21.15.121:35372) 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.460+0000 7f14c7fff700 1 -- 172.21.15.121:0/673735108 learned_addr learned my addr 172.21.15.121:0/673735108 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14d4f3e700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14d0107400 0x7f14d0198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14cffff700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d0101390 0x7f14d019e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14c7fff700 1 -- 172.21.15.121:0/673735108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d0101390 msgr2=0x7f14d019e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14c7fff700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d0101390 0x7f14d019e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14c7fff700 1 -- 172.21.15.121:0/673735108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14d0107400 msgr2=0x7f14d0198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14c7fff700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14d0107400 0x7f14d0198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14c7fff700 1 -- 172.21.15.121:0/673735108 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d0199200 con 0x7f14d0101d50 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14cffff700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d0101390 0x7f14d019e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14c7fff700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14d0101d50 0x7f14d019edd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f14b800d950 tx=0x7f14b800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14cdffb700 1 -- 172.21.15.121:0/673735108 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b8016070 con 0x7f14d0101d50 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14cdffb700 1 -- 172.21.15.121:0/673735108 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14b8009b40 con 0x7f14d0101d50 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14d69a1700 1 -- 172.21.15.121:0/673735108 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d01994f0 con 0x7f14d0101d50 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14cdffb700 1 -- 172.21.15.121:0/673735108 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b800f620 con 0x7f14d0101d50 2024-06-10T00:24:52.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.461+0000 7f14d69a1700 1 -- 172.21.15.121:0/673735108 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d01a5760 con 0x7f14d0101d50 2024-06-10T00:24:52.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.462+0000 7f14d69a1700 1 -- 172.21.15.121:0/673735108 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14d0069180 con 0x7f14d0101d50 2024-06-10T00:24:52.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.463+0000 7f14cdffb700 1 -- 172.21.15.121:0/673735108 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f14b8019070 con 0x7f14d0101d50 2024-06-10T00:24:52.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.464+0000 7f14cdffb700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f14bc0a2c20 0x7f14bc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.464+0000 7f14cffff700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f14bc0a2c20 0x7f14bc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.464+0000 7f14cdffb700 1 -- 172.21.15.121:0/673735108 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f14b80cf340 con 0x7f14d0101d50 2024-06-10T00:24:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.464+0000 7f14cffff700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f14bc0a2c20 0x7f14bc0a50e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f14c0000c00 tx=0x7f14c0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:52.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.468+0000 7f14cdffb700 1 -- 172.21.15.121:0/673735108 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f14b809bfe0 con 0x7f14d0101d50 2024-06-10T00:24:52.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.668+0000 7f14d69a1700 1 -- 172.21.15.121:0/673735108 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14d01a5990 con 0x7f14d0101d50 2024-06-10T00:24:52.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.670+0000 7f14cdffb700 1 -- 172.21.15.121:0/673735108 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+24780 (secure 0 0 0) 0x7f14b809b8e0 con 0x7f14d0101d50 2024-06-10T00:24:52.671 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:24:52.671 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":98,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:24:43.250643+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:24:52.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 -- 172.21.15.121:0/673735108 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f14bc0a2c20 msgr2=0x7f14bc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:52.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f14bc0a2c20 0x7f14bc0a50e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f14c0000c00 tx=0x7f14c0005fb0 comp rx=0 tx=0).stop 2024-06-10T00:24:52.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 -- 172.21.15.121:0/673735108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14d0101d50 msgr2=0x7f14d019edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:52.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14d0101d50 0x7f14d019edd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f14b800d950 tx=0x7f14b800dd10 comp rx=0 tx=0).stop 2024-06-10T00:24:52.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 -- 172.21.15.121:0/673735108 shutdown_connections 2024-06-10T00:24:52.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f14bc0a2c20 0x7f14bc0a50e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:52.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14d0107400 0x7f14d0198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:52.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14d0101390 0x7f14d019e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:52.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 --2- 172.21.15.121:0/673735108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14d0101d50 0x7f14d019edd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:52.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 -- 172.21.15.121:0/673735108 >> 172.21.15.121:0/673735108 conn(0x7f14d0078340 msgr2=0x7f14d01006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:52.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 -- 172.21.15.121:0/673735108 shutdown_connections 2024-06-10T00:24:52.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:52.672+0000 7f14d69a1700 1 -- 172.21.15.121:0/673735108 wait complete. 2024-06-10T00:24:53.361 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:24:53.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:53 smithi135 ceph-mon[36424]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s wr, 0 op/s 2024-06-10T00:24:53.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:53 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/673735108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:53.605 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:53.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:53 smithi193 ceph-mon[36716]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s wr, 0 op/s 2024-06-10T00:24:53.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:53 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/673735108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:53.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:53 smithi121 ceph-mon[32378]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 103 B/s wr, 0 op/s 2024-06-10T00:24:53.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:53 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/673735108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:55.320 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:55 smithi121 ceph-mon[32378]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s 2024-06-10T00:24:55.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.421+0000 7f8795510700 1 -- 172.21.15.121:0/2107962230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8790101d40 msgr2=0x7f879010e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:55.422 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.421+0000 7f8795510700 1 --2- 172.21.15.121:0/2107962230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8790101d40 0x7f879010e230 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f878800b0a0 tx=0x7f878800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:24:55.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.421+0000 7f8795510700 1 -- 172.21.15.121:0/2107962230 shutdown_connections 2024-06-10T00:24:55.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.421+0000 7f8795510700 1 --2- 172.21.15.121:0/2107962230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8790101d40 0x7f879010e230 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:55.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.421+0000 7f8795510700 1 --2- 172.21.15.121:0/2107962230 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87901073f0 0x7f87901077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:55.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.421+0000 7f8795510700 1 --2- 172.21.15.121:0/2107962230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8790101380 0x7f8790101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:55.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.421+0000 7f8795510700 1 -- 172.21.15.121:0/2107962230 >> 172.21.15.121:0/2107962230 conn(0x7f8790078300 msgr2=0x7f8790078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:55.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.421+0000 7f8795510700 1 -- 172.21.15.121:0/2107962230 shutdown_connections 2024-06-10T00:24:55.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.421+0000 7f8795510700 1 -- 172.21.15.121:0/2107962230 wait complete. 2024-06-10T00:24:55.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f8795510700 1 Processor -- start 2024-06-10T00:24:55.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f8795510700 1 -- start start 2024-06-10T00:24:55.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f8795510700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8790101380 0x7f879019e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:55.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f8795510700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8790101d40 0x7f879019ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:55.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f8795510700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87901073f0 0x7f8790198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:55.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f8795510700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8790111760 con 0x7f8790101380 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f8795510700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f87901115e0 con 0x7f8790101d40 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f8795510700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f87901118e0 con 0x7f87901073f0 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f878effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8790101380 0x7f879019e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f878effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8790101380 0x7f879019e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35404/0 (socket says 172.21.15.121:35404) 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f878effd700 1 -- 172.21.15.121:0/1470754611 learned_addr learned my addr 172.21.15.121:0/1470754611 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.422+0000 7f87867fc700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8790101d40 0x7f879019ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f878f7fe700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87901073f0 0x7f8790198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f878effd700 1 -- 172.21.15.121:0/1470754611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87901073f0 msgr2=0x7f8790198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f878effd700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87901073f0 0x7f8790198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f878effd700 1 -- 172.21.15.121:0/1470754611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8790101d40 msgr2=0x7f879019ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f878effd700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8790101d40 0x7f879019ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f878effd700 1 -- 172.21.15.121:0/1470754611 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8790199230 con 0x7f8790101380 2024-06-10T00:24:55.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f878effd700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8790101380 0x7f879019e910 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f878000b530 tx=0x7f878000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:55.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f878cff9700 1 -- 172.21.15.121:0/1470754611 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8780015020 con 0x7f8790101380 2024-06-10T00:24:55.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f878cff9700 1 -- 172.21.15.121:0/1470754611 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8780011d10 con 0x7f8790101380 2024-06-10T00:24:55.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f8795510700 1 -- 172.21.15.121:0/1470754611 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8790199520 con 0x7f8790101380 2024-06-10T00:24:55.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f878cff9700 1 -- 172.21.15.121:0/1470754611 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878000f2e0 con 0x7f8790101380 2024-06-10T00:24:55.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.423+0000 7f8795510700 1 -- 172.21.15.121:0/1470754611 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87901a57e0 con 0x7f8790101380 2024-06-10T00:24:55.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.424+0000 7f8795510700 1 -- 172.21.15.121:0/1470754611 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8790069180 con 0x7f8790101380 2024-06-10T00:24:55.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.426+0000 7f878cff9700 1 -- 172.21.15.121:0/1470754611 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8780018020 con 0x7f8790101380 2024-06-10T00:24:55.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.427+0000 7f878cff9700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f877c0a2c20 0x7f877c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:55.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.427+0000 7f87867fc700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f877c0a2c20 0x7f877c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:55.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.427+0000 7f878cff9700 1 -- 172.21.15.121:0/1470754611 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f87800cf020 con 0x7f8790101380 2024-06-10T00:24:55.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.427+0000 7f87867fc700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f877c0a2c20 0x7f877c0a50e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f879006bae0 tx=0x7f8778009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:55.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.431+0000 7f878cff9700 1 -- 172.21.15.121:0/1470754611 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f878001f020 con 0x7f8790101380 2024-06-10T00:24:55.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:55 smithi135 ceph-mon[36424]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s 2024-06-10T00:24:55.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.629+0000 7f8795510700 1 -- 172.21.15.121:0/1470754611 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8790199fc0 con 0x7f8790101380 2024-06-10T00:24:55.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:55 smithi193 ceph-mon[36716]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s 2024-06-10T00:24:55.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.631+0000 7f878cff9700 1 -- 172.21.15.121:0/1470754611 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+24780 (secure 0 0 0) 0x7f878009bc20 con 0x7f8790101380 2024-06-10T00:24:55.632 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:24:55.632 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":98,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:24:43.250643+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:24:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 -- 172.21.15.121:0/1470754611 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f877c0a2c20 msgr2=0x7f877c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f877c0a2c20 0x7f877c0a50e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f879006bae0 tx=0x7f8778009190 comp rx=0 tx=0).stop 2024-06-10T00:24:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 -- 172.21.15.121:0/1470754611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8790101380 msgr2=0x7f879019e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8790101380 0x7f879019e910 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f878000b530 tx=0x7f878000b840 comp rx=0 tx=0).stop 2024-06-10T00:24:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 -- 172.21.15.121:0/1470754611 shutdown_connections 2024-06-10T00:24:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f877c0a2c20 0x7f877c0a50e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87901073f0 0x7f8790198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:55.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8790101380 0x7f879019e910 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:55.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 --2- 172.21.15.121:0/1470754611 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8790101d40 0x7f879019ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:55.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 -- 172.21.15.121:0/1470754611 >> 172.21.15.121:0/1470754611 conn(0x7f8790078300 msgr2=0x7f87901053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:55.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 -- 172.21.15.121:0/1470754611 shutdown_connections 2024-06-10T00:24:55.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:55.633+0000 7f8795510700 1 -- 172.21.15.121:0/1470754611 wait complete. 2024-06-10T00:24:56.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-06-10T00:24:56.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:56 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1470754611' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:56.599 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:24:56.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:56 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1470754611' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:56.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:56 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1470754611' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:24:57.446 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:57 smithi121 ceph-mon[32378]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-10T00:24:57.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:57 smithi135 ceph-mon[36424]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-10T00:24:57.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:57 smithi193 ceph-mon[36716]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-10T00:24:58.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.322+0000 7f3fd1dac700 1 -- 172.21.15.121:0/4276556768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fcc101d50 msgr2=0x7f3fcc10e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.322+0000 7f3fd1dac700 1 --2- 172.21.15.121:0/4276556768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fcc101d50 0x7f3fcc10e240 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400b0a0 tx=0x7f3fc400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:24:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.322+0000 7f3fd1dac700 1 -- 172.21.15.121:0/4276556768 shutdown_connections 2024-06-10T00:24:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.322+0000 7f3fd1dac700 1 --2- 172.21.15.121:0/4276556768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fcc101d50 0x7f3fcc10e240 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.322+0000 7f3fd1dac700 1 --2- 172.21.15.121:0/4276556768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fcc107400 0x7f3fcc1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.322+0000 7f3fd1dac700 1 --2- 172.21.15.121:0/4276556768 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc101390 0x7f3fcc101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.322+0000 7f3fd1dac700 1 -- 172.21.15.121:0/4276556768 >> 172.21.15.121:0/4276556768 conn(0x7f3fcc078340 msgr2=0x7f3fcc078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.322+0000 7f3fd1dac700 1 -- 172.21.15.121:0/4276556768 shutdown_connections 2024-06-10T00:24:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.322+0000 7f3fd1dac700 1 -- 172.21.15.121:0/4276556768 wait complete. 2024-06-10T00:24:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.323+0000 7f3fd1dac700 1 Processor -- start 2024-06-10T00:24:58.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.323+0000 7f3fd1dac700 1 -- start start 2024-06-10T00:24:58.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.323+0000 7f3fd1dac700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fcc101390 0x7f3fcc19e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:58.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.323+0000 7f3fd1dac700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fcc101d50 0x7f3fcc19edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:58.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.323+0000 7f3fd1dac700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc107400 0x7f3fcc198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:58.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.323+0000 7f3fd1dac700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc1116f0 con 0x7f3fcc101390 2024-06-10T00:24:58.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.323+0000 7f3fd1dac700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc111570 con 0x7f3fcc107400 2024-06-10T00:24:58.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.323+0000 7f3fd1dac700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc111870 con 0x7f3fcc101d50 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.323+0000 7f3fcb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fcc101390 0x7f3fcc19e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.323+0000 7f3fcb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fcc101390 0x7f3fcc19e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:35420/0 (socket says 172.21.15.121:35420) 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.323+0000 7f3fcb7fe700 1 -- 172.21.15.121:0/506613149 learned_addr learned my addr 172.21.15.121:0/506613149 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fcaffd700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fcc101d50 0x7f3fcc19edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fcb7fe700 1 -- 172.21.15.121:0/506613149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fcc101d50 msgr2=0x7f3fcc19edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fcb7fe700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fcc101d50 0x7f3fcc19edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fcb7fe700 1 -- 172.21.15.121:0/506613149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc107400 msgr2=0x7f3fcc198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fcb7fe700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc107400 0x7f3fcc198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fcb7fe700 1 -- 172.21.15.121:0/506613149 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fcc1991a0 con 0x7f3fcc101390 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fcaffd700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fcc101d50 0x7f3fcc19edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fcb7fe700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fcc101390 0x7f3fcc19e880 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400c9b0 tx=0x7f3fb400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:58.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fc8ff9700 1 -- 172.21.15.121:0/506613149 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4007b00 con 0x7f3fcc101390 2024-06-10T00:24:58.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fd1dac700 1 -- 172.21.15.121:0/506613149 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fcc199490 con 0x7f3fcc101390 2024-06-10T00:24:58.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fc8ff9700 1 -- 172.21.15.121:0/506613149 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fb4007ca0 con 0x7f3fcc101390 2024-06-10T00:24:58.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fc8ff9700 1 -- 172.21.15.121:0/506613149 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4018780 con 0x7f3fcc101390 2024-06-10T00:24:58.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.324+0000 7f3fd1dac700 1 -- 172.21.15.121:0/506613149 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fcc1a5750 con 0x7f3fcc101390 2024-06-10T00:24:58.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.325+0000 7f3fd1dac700 1 -- 172.21.15.121:0/506613149 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fcc069180 con 0x7f3fcc101390 2024-06-10T00:24:58.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.327+0000 7f3fc8ff9700 1 -- 172.21.15.121:0/506613149 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3fb400f450 con 0x7f3fcc101390 2024-06-10T00:24:58.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.327+0000 7f3fc8ff9700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3fb80a2c20 0x7f3fb80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:24:58.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.327+0000 7f3fcaffd700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3fb80a2c20 0x7f3fb80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:24:58.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.327+0000 7f3fc8ff9700 1 -- 172.21.15.121:0/506613149 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f3fb40cf6c0 con 0x7f3fcc101390 2024-06-10T00:24:58.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.328+0000 7f3fcaffd700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3fb80a2c20 0x7f3fb80a50e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc06b250 tx=0x7f3fbc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:24:58.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.331+0000 7f3fc8ff9700 1 -- 172.21.15.121:0/506613149 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3fb4014030 con 0x7f3fcc101390 2024-06-10T00:24:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:58.572+0000 7f3fd1dac700 1 -- 172.21.15.121:0/506613149 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3fcc1a5aa0 con 0x7f3fcc101390 2024-06-10T00:24:59.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.215+0000 7f3fc8ff9700 1 -- 172.21.15.121:0/506613149 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f3fb400de20 con 0x7f3fcc101390 2024-06-10T00:24:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 -- 172.21.15.121:0/506613149 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3fb80a2c20 msgr2=0x7f3fb80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3fb80a2c20 0x7f3fb80a50e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc06b250 tx=0x7f3fbc008040 comp rx=0 tx=0).stop 2024-06-10T00:24:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 -- 172.21.15.121:0/506613149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fcc101390 msgr2=0x7f3fcc19e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:24:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fcc101390 0x7f3fcc19e880 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400c9b0 tx=0x7f3fb400cd70 comp rx=0 tx=0).stop 2024-06-10T00:24:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 -- 172.21.15.121:0/506613149 shutdown_connections 2024-06-10T00:24:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3fb80a2c20 0x7f3fb80a50e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:59.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3fcc107400 0x7f3fcc198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fcc101390 0x7f3fcc19e880 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 --2- 172.21.15.121:0/506613149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fcc101d50 0x7f3fcc19edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:24:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 -- 172.21.15.121:0/506613149 >> 172.21.15.121:0/506613149 conn(0x7f3fcc078340 msgr2=0x7f3fcc105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:24:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 -- 172.21.15.121:0/506613149 shutdown_connections 2024-06-10T00:24:59.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:24:59.218+0000 7f3fd1dac700 1 -- 172.21.15.121:0/506613149 wait complete. 2024-06-10T00:24:59.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:59 smithi135 ceph-mon[36424]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-10T00:24:59.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:24:59 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/506613149' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-06-10T00:24:59.605 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:59 smithi121 ceph-mon[32378]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-10T00:24:59.605 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:24:59 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/506613149' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-06-10T00:24:59.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:59 smithi193 ceph-mon[36716]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-10T00:24:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:24:59 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/506613149' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-06-10T00:24:59.950 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-06-10T00:24:59.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-06-10T00:25:00.199 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:00.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:00 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/506613149' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-06-10T00:25:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:00 smithi135 ceph-mon[36424]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:25:00.560 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:00 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/506613149' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-06-10T00:25:00.560 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:00 smithi121 ceph-mon[32378]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:25:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:00 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/506613149' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-06-10T00:25:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:00 smithi193 ceph-mon[36716]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:25:01.298 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:01 smithi121 ceph-mon[32378]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-10T00:25:01.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:01 smithi135 ceph-mon[36424]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-10T00:25:01.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:01 smithi193 ceph-mon[36716]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-06-10T00:25:01.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.879+0000 7f7552f51700 1 -- 172.21.15.121:0/2187383605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f754c1013c0 msgr2=0x7f754c101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:01.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.879+0000 7f7552f51700 1 --2- 172.21.15.121:0/2187383605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f754c1013c0 0x7f754c101840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7540009990 tx=0x7f7540009ca0 comp rx=0 tx=0).stop 2024-06-10T00:25:01.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.879+0000 7f7552f51700 1 -- 172.21.15.121:0/2187383605 shutdown_connections 2024-06-10T00:25:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.879+0000 7f7552f51700 1 --2- 172.21.15.121:0/2187383605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f754c101d80 0x7f754c10e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.879+0000 7f7552f51700 1 --2- 172.21.15.121:0/2187383605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f754c1032e0 0x7f754c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.879+0000 7f7552f51700 1 --2- 172.21.15.121:0/2187383605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f754c1013c0 0x7f754c101840 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.879+0000 7f7552f51700 1 -- 172.21.15.121:0/2187383605 >> 172.21.15.121:0/2187383605 conn(0x7f754c077400 msgr2=0x7f754c077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.879+0000 7f7552f51700 1 -- 172.21.15.121:0/2187383605 shutdown_connections 2024-06-10T00:25:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.879+0000 7f7552f51700 1 -- 172.21.15.121:0/2187383605 wait complete. 2024-06-10T00:25:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f7552f51700 1 Processor -- start 2024-06-10T00:25:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f7552f51700 1 -- start start 2024-06-10T00:25:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f7552f51700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f754c1013c0 0x7f754c1a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f7552f51700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f754c101d80 0x7f754c1a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f7552f51700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f754c1032e0 0x7f754c19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f7552f51700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f754c111790 con 0x7f754c1013c0 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f7552f51700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f754c111610 con 0x7f754c1032e0 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f7552f51700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f754c111910 con 0x7f754c101d80 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f7550ced700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f754c1013c0 0x7f754c1a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f7550ced700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f754c1013c0 0x7f754c1a5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51350/0 (socket says 172.21.15.121:51350) 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f7550ced700 1 -- 172.21.15.121:0/147852551 learned_addr learned my addr 172.21.15.121:0/147852551 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f754bfff700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f754c101d80 0x7f754c1a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.880+0000 7f75514ee700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f754c1032e0 0x7f754c19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.881+0000 7f7550ced700 1 -- 172.21.15.121:0/147852551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f754c101d80 msgr2=0x7f754c1a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.881+0000 7f7550ced700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f754c101d80 0x7f754c1a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.881+0000 7f7550ced700 1 -- 172.21.15.121:0/147852551 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f754c1032e0 msgr2=0x7f754c19f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.881+0000 7f7550ced700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f754c1032e0 0x7f754c19f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:01.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.881+0000 7f7550ced700 1 -- 172.21.15.121:0/147852551 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754c19fc40 con 0x7f754c1013c0 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.881+0000 7f7550ced700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f754c1013c0 0x7f754c1a5030 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f753c00d980 tx=0x7f753c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.881+0000 7f7549ffb700 1 -- 172.21.15.121:0/147852551 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c017070 con 0x7f754c1013c0 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.881+0000 7f7549ffb700 1 -- 172.21.15.121:0/147852551 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f753c005020 con 0x7f754c1013c0 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.881+0000 7f7552f51700 1 -- 172.21.15.121:0/147852551 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f754c19ff30 con 0x7f754c1013c0 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.881+0000 7f7549ffb700 1 -- 172.21.15.121:0/147852551 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c00fbf0 con 0x7f754c1013c0 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.881+0000 7f7552f51700 1 -- 172.21.15.121:0/147852551 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f754c1abe90 con 0x7f754c1013c0 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.882+0000 7f7552f51700 1 -- 172.21.15.121:0/147852551 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f754c069180 con 0x7f754c1013c0 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.883+0000 7f7549ffb700 1 -- 172.21.15.121:0/147852551 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f753c0051d0 con 0x7f754c1013c0 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.884+0000 7f7549ffb700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f75340a2ca0 0x7f75340a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.884+0000 7f754bfff700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f75340a2ca0 0x7f75340a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.884+0000 7f7549ffb700 1 -- 172.21.15.121:0/147852551 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f753c0cf110 con 0x7f754c1013c0 2024-06-10T00:25:01.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.884+0000 7f754bfff700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f75340a2ca0 0x7f75340a5160 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f754000b5c0 tx=0x7f7540005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:01.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:01.888+0000 7f7549ffb700 1 -- 172.21.15.121:0/147852551 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f753c020020 con 0x7f754c1013c0 2024-06-10T00:25:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.133+0000 7f7552f51700 1 -- 172.21.15.121:0/147852551 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f754c1ac060 con 0x7f754c1013c0 2024-06-10T00:25:02.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.238+0000 7f7549ffb700 1 -- 172.21.15.121:0/147852551 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f753c09bd10 con 0x7f754c1013c0 2024-06-10T00:25:02.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 -- 172.21.15.121:0/147852551 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f75340a2ca0 msgr2=0x7f75340a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:02.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f75340a2ca0 0x7f75340a5160 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f754000b5c0 tx=0x7f7540005960 comp rx=0 tx=0).stop 2024-06-10T00:25:02.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 -- 172.21.15.121:0/147852551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f754c1013c0 msgr2=0x7f754c1a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:02.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f754c1013c0 0x7f754c1a5030 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f753c00d980 tx=0x7f753c00dc90 comp rx=0 tx=0).stop 2024-06-10T00:25:02.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 -- 172.21.15.121:0/147852551 shutdown_connections 2024-06-10T00:25:02.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f75340a2ca0 0x7f75340a5160 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:02.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f754c1032e0 0x7f754c19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:02.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f754c1013c0 0x7f754c1a5030 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:02.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 --2- 172.21.15.121:0/147852551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f754c101d80 0x7f754c1a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:02.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 -- 172.21.15.121:0/147852551 >> 172.21.15.121:0/147852551 conn(0x7f754c077400 msgr2=0x7f754c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:02.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 -- 172.21.15.121:0/147852551 shutdown_connections 2024-06-10T00:25:02.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:02.240+0000 7f7552f51700 1 -- 172.21.15.121:0/147852551 wait complete. 2024-06-10T00:25:02.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:02 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/147852551' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-06-10T00:25:02.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:02 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/147852551' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-06-10T00:25:02.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:02 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/147852551' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-06-10T00:25:02.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-06-10T00:25:03.170 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:03.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:03 smithi135 ceph-mon[36424]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:25:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:03 smithi135 ceph-mon[36424]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-10T00:25:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:03 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/147852551' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-06-10T00:25:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:03 smithi135 ceph-mon[36424]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-06-10T00:25:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:03 smithi135 ceph-mon[36424]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:25:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:03 smithi135 ceph-mon[36424]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-06-10T00:25:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:03 smithi135 ceph-mon[36424]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-06-10T00:25:03.602 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:03 smithi121 ceph-mon[32378]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:25:03.603 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:03 smithi121 ceph-mon[32378]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-10T00:25:03.603 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:03 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/147852551' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-06-10T00:25:03.603 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:03 smithi121 ceph-mon[32378]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-06-10T00:25:03.603 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:03 smithi121 ceph-mon[32378]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:25:03.603 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:03 smithi121 ceph-mon[32378]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-06-10T00:25:03.603 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:03 smithi121 ceph-mon[32378]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-06-10T00:25:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:03 smithi193 ceph-mon[36716]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:25:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:03 smithi193 ceph-mon[36716]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-06-10T00:25:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:03 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/147852551' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-06-10T00:25:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:03 smithi193 ceph-mon[36716]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-06-10T00:25:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:03 smithi193 ceph-mon[36716]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-06-10T00:25:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:03 smithi193 ceph-mon[36716]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-06-10T00:25:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:03 smithi193 ceph-mon[36716]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-06-10T00:25:04.341 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:04 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] up:active 2024-06-10T00:25:04.341 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:04 smithi121 ceph-mon[32378]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-06-10T00:25:04.341 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:04 smithi121 ceph-mon[32378]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-10T00:25:04.342 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:04 smithi121 ceph-mon[32378]: Cluster is now healthy 2024-06-10T00:25:04.342 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:04 smithi121 ceph-mon[32378]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-06-10T00:25:04.342 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:04 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-06-10T00:25:04.342 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:04 smithi121 ceph-mon[32378]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-06-10T00:25:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:04 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] up:active 2024-06-10T00:25:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:04 smithi135 ceph-mon[36424]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-06-10T00:25:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:04 smithi135 ceph-mon[36424]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-10T00:25:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:04 smithi135 ceph-mon[36424]: Cluster is now healthy 2024-06-10T00:25:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:04 smithi135 ceph-mon[36424]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-06-10T00:25:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:04 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-06-10T00:25:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:04 smithi135 ceph-mon[36424]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-06-10T00:25:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:04 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] up:active 2024-06-10T00:25:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:04 smithi193 ceph-mon[36716]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-06-10T00:25:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:04 smithi193 ceph-mon[36716]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-06-10T00:25:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:04 smithi193 ceph-mon[36716]: Cluster is now healthy 2024-06-10T00:25:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:04 smithi193 ceph-mon[36716]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-06-10T00:25:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:04 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-06-10T00:25:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:04 smithi193 ceph-mon[36716]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-06-10T00:25:04.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.924+0000 7f629439e700 1 -- 172.21.15.121:0/664081607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f628c101390 msgr2=0x7f628c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:04.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.924+0000 7f629439e700 1 --2- 172.21.15.121:0/664081607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f628c101390 0x7f628c101810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f627c009990 tx=0x7f627c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:25:04.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.924+0000 7f629439e700 1 -- 172.21.15.121:0/664081607 shutdown_connections 2024-06-10T00:25:04.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.924+0000 7f629439e700 1 --2- 172.21.15.121:0/664081607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f628c101d50 0x7f628c10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:04.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.924+0000 7f629439e700 1 --2- 172.21.15.121:0/664081607 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f628c107400 0x7f628c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:04.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.924+0000 7f629439e700 1 --2- 172.21.15.121:0/664081607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f628c101390 0x7f628c101810 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.924+0000 7f629439e700 1 -- 172.21.15.121:0/664081607 >> 172.21.15.121:0/664081607 conn(0x7f628c078340 msgr2=0x7f628c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.924+0000 7f629439e700 1 -- 172.21.15.121:0/664081607 shutdown_connections 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.924+0000 7f629439e700 1 -- 172.21.15.121:0/664081607 wait complete. 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f629439e700 1 Processor -- start 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f629439e700 1 -- start start 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f629439e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f628c101390 0x7f628c19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f629439e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f628c101d50 0x7f628c19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f629439e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f628c107400 0x7f628c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f629439e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f628c111670 con 0x7f628c101390 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f629439e700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f628c1114f0 con 0x7f628c107400 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f629439e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f628c1117f0 con 0x7f628c101d50 2024-06-10T00:25:04.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f629213a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f628c101390 0x7f628c19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:04.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f629213a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f628c101390 0x7f628c19ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51374/0 (socket says 172.21.15.121:51374) 2024-06-10T00:25:04.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f629213a700 1 -- 172.21.15.121:0/2383319862 learned_addr learned my addr 172.21.15.121:0/2383319862 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:04.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.925+0000 7f6291939700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f628c101d50 0x7f628c19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:04.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.926+0000 7f629213a700 1 -- 172.21.15.121:0/2383319862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f628c101d50 msgr2=0x7f628c19b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:04.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.926+0000 7f629213a700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f628c101d50 0x7f628c19b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:04.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.926+0000 7f629213a700 1 -- 172.21.15.121:0/2383319862 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f628c107400 msgr2=0x7f628c19f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:04.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.926+0000 7f629213a700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f628c107400 0x7f628c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:04.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.926+0000 7f629213a700 1 -- 172.21.15.121:0/2383319862 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628c19fdb0 con 0x7f628c101390 2024-06-10T00:25:04.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.926+0000 7f629213a700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f628c101390 0x7f628c19ada0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f628800d950 tx=0x7f628800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:04.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.926+0000 7f62837fe700 1 -- 172.21.15.121:0/2383319862 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6288016070 con 0x7f628c101390 2024-06-10T00:25:04.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.926+0000 7f62837fe700 1 -- 172.21.15.121:0/2383319862 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6288009da0 con 0x7f628c101390 2024-06-10T00:25:04.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.926+0000 7f629439e700 1 -- 172.21.15.121:0/2383319862 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f628c1a00a0 con 0x7f628c101390 2024-06-10T00:25:04.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.926+0000 7f62837fe700 1 -- 172.21.15.121:0/2383319862 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628800f620 con 0x7f628c101390 2024-06-10T00:25:04.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.926+0000 7f629439e700 1 -- 172.21.15.121:0/2383319862 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f628c1a7920 con 0x7f628c101390 2024-06-10T00:25:04.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.927+0000 7f629439e700 1 -- 172.21.15.121:0/2383319862 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f628c069180 con 0x7f628c101390 2024-06-10T00:25:04.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.929+0000 7f62837fe700 1 -- 172.21.15.121:0/2383319862 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6288019070 con 0x7f628c101390 2024-06-10T00:25:04.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.929+0000 7f62837fe700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62780a2c20 0x7f62780a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:04.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.930+0000 7f6291939700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62780a2c20 0x7f62780a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:04.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.930+0000 7f62837fe700 1 -- 172.21.15.121:0/2383319862 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f62880cee60 con 0x7f628c101390 2024-06-10T00:25:04.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.930+0000 7f6291939700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62780a2c20 0x7f62780a50e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f627c00b2e0 tx=0x7f627c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:04.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:04.933+0000 7f62837fe700 1 -- 172.21.15.121:0/2383319862 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f628809ba60 con 0x7f628c101390 2024-06-10T00:25:05.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.178+0000 7f629439e700 1 -- 172.21.15.121:0/2383319862 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f628c1a7b90 con 0x7f628c101390 2024-06-10T00:25:05.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.270+0000 7f62837fe700 1 -- 172.21.15.121:0/2383319862 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f628809b360 con 0x7f628c101390 2024-06-10T00:25:05.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 -- 172.21.15.121:0/2383319862 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62780a2c20 msgr2=0x7f62780a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:05.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62780a2c20 0x7f62780a50e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f627c00b2e0 tx=0x7f627c005960 comp rx=0 tx=0).stop 2024-06-10T00:25:05.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 -- 172.21.15.121:0/2383319862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f628c101390 msgr2=0x7f628c19ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:05.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f628c101390 0x7f628c19ada0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f628800d950 tx=0x7f628800dc60 comp rx=0 tx=0).stop 2024-06-10T00:25:05.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 -- 172.21.15.121:0/2383319862 shutdown_connections 2024-06-10T00:25:05.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f62780a2c20 0x7f62780a50e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:05.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f628c107400 0x7f628c19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:05.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f628c101390 0x7f628c19ada0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:05.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 --2- 172.21.15.121:0/2383319862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f628c101d50 0x7f628c19b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:05.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 -- 172.21.15.121:0/2383319862 >> 172.21.15.121:0/2383319862 conn(0x7f628c078340 msgr2=0x7f628c06a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:05.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 -- 172.21.15.121:0/2383319862 shutdown_connections 2024-06-10T00:25:05.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:05.272+0000 7f629439e700 1 -- 172.21.15.121:0/2383319862 wait complete. 2024-06-10T00:25:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:05 smithi121 ceph-mon[32378]: pgmap v339: 129 pgs: 129 active+clean; 582 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-06-10T00:25:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:05 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] up:active 2024-06-10T00:25:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:05 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-06-10T00:25:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:05 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2383319862' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-06-10T00:25:05.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:05 smithi193 ceph-mon[36716]: pgmap v339: 129 pgs: 129 active+clean; 582 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-06-10T00:25:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:05 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] up:active 2024-06-10T00:25:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:05 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-06-10T00:25:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:05 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2383319862' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-06-10T00:25:05.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:05 smithi135 ceph-mon[36424]: pgmap v339: 129 pgs: 129 active+clean; 582 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-06-10T00:25:05.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:05 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] up:active 2024-06-10T00:25:05.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:05 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-06-10T00:25:05.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:05 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2383319862' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-06-10T00:25:05.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-06-10T00:25:06.207 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:06.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:06 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2383319862' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-06-10T00:25:06.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:06 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-06-10T00:25:06.595 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:06 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:25:06.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:06 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2383319862' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-06-10T00:25:06.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:06 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-06-10T00:25:06.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:06 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:25:06.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:06 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2383319862' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-06-10T00:25:06.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:06 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-06-10T00:25:06.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:06 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:25:07.350 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:07 smithi121 ceph-mon[32378]: pgmap v340: 129 pgs: 129 active+clean; 582 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 4 op/s 2024-06-10T00:25:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:07 smithi193 ceph-mon[36716]: pgmap v340: 129 pgs: 129 active+clean; 582 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 4 op/s 2024-06-10T00:25:07.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:07 smithi135 ceph-mon[36424]: pgmap v340: 129 pgs: 129 active+clean; 582 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 4 op/s 2024-06-10T00:25:07.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.902+0000 7f385835b700 1 -- 172.21.15.121:0/1653185103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38501066d0 msgr2=0x7f385010e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:07.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.902+0000 7f385835b700 1 --2- 172.21.15.121:0/1653185103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38501066d0 0x7f385010e2a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f384800b0d0 tx=0x7f384800b3e0 comp rx=0 tx=0).stop 2024-06-10T00:25:07.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.902+0000 7f385835b700 1 -- 172.21.15.121:0/1653185103 shutdown_connections 2024-06-10T00:25:07.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.902+0000 7f385835b700 1 --2- 172.21.15.121:0/1653185103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38501066d0 0x7f385010e2a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:07.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.902+0000 7f385835b700 1 --2- 172.21.15.121:0/1653185103 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3850101680 0x7f3850101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:07.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.902+0000 7f385835b700 1 --2- 172.21.15.121:0/1653185103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850102030 0x7f3850105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:07.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.902+0000 7f385835b700 1 -- 172.21.15.121:0/1653185103 >> 172.21.15.121:0/1653185103 conn(0x7f38500773e0 msgr2=0x7f38500777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:07.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.902+0000 7f385835b700 1 -- 172.21.15.121:0/1653185103 shutdown_connections 2024-06-10T00:25:07.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.902+0000 7f385835b700 1 -- 172.21.15.121:0/1653185103 wait complete. 2024-06-10T00:25:07.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.902+0000 7f385835b700 1 Processor -- start 2024-06-10T00:25:07.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.902+0000 7f385835b700 1 -- start start 2024-06-10T00:25:07.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f385835b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850101680 0x7f385019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f385835b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3850102030 0x7f385019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f385835b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38501066d0 0x7f3850198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f385835b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f38501116e0 con 0x7f38501066d0 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f385835b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f3850111560 con 0x7f3850102030 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f385835b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3850111860 con 0x7f3850101680 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38568f8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38501066d0 0x7f3850198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38568f8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38501066d0 0x7f3850198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:51400/0 (socket says 172.21.15.121:51400) 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38568f8700 1 -- 172.21.15.121:0/1813101054 learned_addr learned my addr 172.21.15.121:0/1813101054 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38558f6700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3850102030 0x7f385019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38560f7700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850101680 0x7f385019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38568f8700 1 -- 172.21.15.121:0/1813101054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850101680 msgr2=0x7f385019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38568f8700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850101680 0x7f385019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:07.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38568f8700 1 -- 172.21.15.121:0/1813101054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3850102030 msgr2=0x7f385019ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:07.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38568f8700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3850102030 0x7f385019ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:07.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38568f8700 1 -- 172.21.15.121:0/1813101054 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38501991c0 con 0x7f38501066d0 2024-06-10T00:25:07.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38568f8700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38501066d0 0x7f3850198a60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3848006070 tx=0x7f3848004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:07.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f38477fe700 1 -- 172.21.15.121:0/1813101054 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848021070 con 0x7f38501066d0 2024-06-10T00:25:07.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.903+0000 7f385835b700 1 -- 172.21.15.121:0/1813101054 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3850199450 con 0x7f38501066d0 2024-06-10T00:25:07.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.904+0000 7f38477fe700 1 -- 172.21.15.121:0/1813101054 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3848007b80 con 0x7f38501066d0 2024-06-10T00:25:07.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.904+0000 7f38477fe700 1 -- 172.21.15.121:0/1813101054 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38480052a0 con 0x7f38501066d0 2024-06-10T00:25:07.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.904+0000 7f385835b700 1 -- 172.21.15.121:0/1813101054 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38501a5720 con 0x7f38501066d0 2024-06-10T00:25:07.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.904+0000 7f385835b700 1 -- 172.21.15.121:0/1813101054 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3850069180 con 0x7f38501066d0 2024-06-10T00:25:07.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.906+0000 7f38477fe700 1 -- 172.21.15.121:0/1813101054 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3848024070 con 0x7f38501066d0 2024-06-10T00:25:07.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.906+0000 7f38477fe700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f383c0a2c20 0x7f383c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:07.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.906+0000 7f38560f7700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f383c0a2c20 0x7f383c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:07.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.906+0000 7f38477fe700 1 -- 172.21.15.121:0/1813101054 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f384800f020 con 0x7f38501066d0 2024-06-10T00:25:07.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.907+0000 7f38560f7700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f383c0a2c20 0x7f383c0a50e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3850077ba0 tx=0x7f384c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:07.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:07.910+0000 7f38477fe700 1 -- 172.21.15.121:0/1813101054 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f384802b070 con 0x7f38501066d0 2024-06-10T00:25:08.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.155+0000 7f385835b700 1 -- 172.21.15.121:0/1813101054 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f3850199f40 con 0x7f38501066d0 2024-06-10T00:25:08.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.285+0000 7f38477fe700 1 -- 172.21.15.121:0/1813101054 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7f384809d770 con 0x7f38501066d0 2024-06-10T00:25:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 -- 172.21.15.121:0/1813101054 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f383c0a2c20 msgr2=0x7f383c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f383c0a2c20 0x7f383c0a50e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3850077ba0 tx=0x7f384c00b410 comp rx=0 tx=0).stop 2024-06-10T00:25:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 -- 172.21.15.121:0/1813101054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38501066d0 msgr2=0x7f3850198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38501066d0 0x7f3850198a60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3848006070 tx=0x7f3848004d00 comp rx=0 tx=0).stop 2024-06-10T00:25:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 -- 172.21.15.121:0/1813101054 shutdown_connections 2024-06-10T00:25:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f383c0a2c20 0x7f383c0a50e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38501066d0 0x7f3850198a60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3850101680 0x7f385019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 --2- 172.21.15.121:0/1813101054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3850102030 0x7f385019ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:08.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 -- 172.21.15.121:0/1813101054 >> 172.21.15.121:0/1813101054 conn(0x7f38500773e0 msgr2=0x7f385006ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:08.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 -- 172.21.15.121:0/1813101054 shutdown_connections 2024-06-10T00:25:08.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:08.287+0000 7f385835b700 1 -- 172.21.15.121:0/1813101054 wait complete. 2024-06-10T00:25:08.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:08 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1813101054' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-06-10T00:25:08.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:08 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1813101054' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-06-10T00:25:08.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:08 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1813101054' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-06-10T00:25:08.933 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-06-10T00:25:08.933 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-06-10T00:25:09.176 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:09.546 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:09 smithi121 ceph-mon[32378]: pgmap v341: 129 pgs: 129 active+clean; 584 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.9 KiB/s wr, 7 op/s 2024-06-10T00:25:09.546 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:09 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1813101054' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-06-10T00:25:09.546 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:09 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:25:09.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:09 smithi193 ceph-mon[36716]: pgmap v341: 129 pgs: 129 active+clean; 584 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.9 KiB/s wr, 7 op/s 2024-06-10T00:25:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:09 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1813101054' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-06-10T00:25:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:09 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:25:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:09 smithi135 ceph-mon[36424]: pgmap v341: 129 pgs: 129 active+clean; 584 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.9 KiB/s wr, 7 op/s 2024-06-10T00:25:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:09 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1813101054' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-06-10T00:25:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:09 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:25:10.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.893+0000 7f1b17ad3700 1 -- 172.21.15.121:0/3140137469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b10113190 msgr2=0x7f1b10115580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:10.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.893+0000 7f1b17ad3700 1 --2- 172.21.15.121:0/3140137469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b10113190 0x7f1b10115580 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1b0800b0a0 tx=0x7f1b0800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:25:10.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.893+0000 7f1b17ad3700 1 -- 172.21.15.121:0/3140137469 shutdown_connections 2024-06-10T00:25:10.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.893+0000 7f1b17ad3700 1 --2- 172.21.15.121:0/3140137469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b10113190 0x7f1b10115580 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:10.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.893+0000 7f1b17ad3700 1 --2- 172.21.15.121:0/3140137469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b10076470 0x7f1b100746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:10.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.893+0000 7f1b17ad3700 1 --2- 172.21.15.121:0/3140137469 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b10074c00 0x7f1b10075080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:10.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.893+0000 7f1b17ad3700 1 -- 172.21.15.121:0/3140137469 >> 172.21.15.121:0/3140137469 conn(0x7f1b10079650 msgr2=0x7f1b10079a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:10.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.893+0000 7f1b17ad3700 1 -- 172.21.15.121:0/3140137469 shutdown_connections 2024-06-10T00:25:10.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.893+0000 7f1b17ad3700 1 -- 172.21.15.121:0/3140137469 wait complete. 2024-06-10T00:25:10.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.893+0000 7f1b17ad3700 1 Processor -- start 2024-06-10T00:25:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b17ad3700 1 -- start start 2024-06-10T00:25:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b17ad3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b10074c00 0x7f1b1019f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b17ad3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b10076470 0x7f1b1019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b17ad3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b10113190 0x7f1b101a3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b17ad3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1b10118e30 con 0x7f1b10076470 2024-06-10T00:25:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b17ad3700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f1b10118cb0 con 0x7f1b10074c00 2024-06-10T00:25:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b17ad3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1b10118fb0 con 0x7f1b10113190 2024-06-10T00:25:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b1506e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b10076470 0x7f1b1019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b1506e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b10076470 0x7f1b1019f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42826/0 (socket says 172.21.15.121:42826) 2024-06-10T00:25:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b1506e700 1 -- 172.21.15.121:0/2052132344 learned_addr learned my addr 172.21.15.121:0/2052132344 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:10.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b1586f700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b10074c00 0x7f1b1019f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b16070700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b10113190 0x7f1b101a3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b1506e700 1 -- 172.21.15.121:0/2052132344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b10113190 msgr2=0x7f1b101a3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b1506e700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b10113190 0x7f1b101a3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b1506e700 1 -- 172.21.15.121:0/2052132344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b10074c00 msgr2=0x7f1b1019f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b1506e700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b10074c00 0x7f1b1019f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b1506e700 1 -- 172.21.15.121:0/2052132344 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b101a4280 con 0x7f1b10076470 2024-06-10T00:25:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b1506e700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b10076470 0x7f1b1019f800 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000b530 tx=0x7f1b0000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.894+0000 7f1b06ffd700 1 -- 172.21.15.121:0/2052132344 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b00015020 con 0x7f1b10076470 2024-06-10T00:25:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.895+0000 7f1b17ad3700 1 -- 172.21.15.121:0/2052132344 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b101a4570 con 0x7f1b10076470 2024-06-10T00:25:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.895+0000 7f1b06ffd700 1 -- 172.21.15.121:0/2052132344 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b00011d10 con 0x7f1b10076470 2024-06-10T00:25:10.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.895+0000 7f1b06ffd700 1 -- 172.21.15.121:0/2052132344 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0000f2e0 con 0x7f1b10076470 2024-06-10T00:25:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.895+0000 7f1b17ad3700 1 -- 172.21.15.121:0/2052132344 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b101abdf0 con 0x7f1b10076470 2024-06-10T00:25:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.895+0000 7f1b17ad3700 1 -- 172.21.15.121:0/2052132344 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b10069180 con 0x7f1b10076470 2024-06-10T00:25:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.897+0000 7f1b06ffd700 1 -- 172.21.15.121:0/2052132344 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f1b00018020 con 0x7f1b10076470 2024-06-10T00:25:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.897+0000 7f1b06ffd700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1afc0a2c20 0x7f1afc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.897+0000 7f1b1586f700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1afc0a2c20 0x7f1afc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.898+0000 7f1b06ffd700 1 -- 172.21.15.121:0/2052132344 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f1b000cf000 con 0x7f1b10076470 2024-06-10T00:25:10.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.898+0000 7f1b1586f700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1afc0a2c20 0x7f1afc0a50e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c004630 tx=0x7f1b0c00b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:10.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:10.901+0000 7f1b06ffd700 1 -- 172.21.15.121:0/2052132344 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1b0001f020 con 0x7f1b10076470 2024-06-10T00:25:11.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.125+0000 7f1b17ad3700 1 -- 172.21.15.121:0/2052132344 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1b101125a0 con 0x7f1afc0a2c20 2024-06-10T00:25:11.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.151+0000 7f1b06ffd700 1 -- 172.21.15.121:0/2052132344 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1b101125a0 con 0x7f1afc0a2c20 2024-06-10T00:25:11.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.153+0000 7f1b17ad3700 1 -- 172.21.15.121:0/2052132344 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1afc0a2c20 msgr2=0x7f1afc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:11.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.153+0000 7f1b17ad3700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1afc0a2c20 0x7f1afc0a50e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c004630 tx=0x7f1b0c00b3c0 comp rx=0 tx=0).stop 2024-06-10T00:25:11.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.153+0000 7f1b17ad3700 1 -- 172.21.15.121:0/2052132344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b10076470 msgr2=0x7f1b1019f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:11.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.153+0000 7f1b17ad3700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b10076470 0x7f1b1019f800 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1b0000b530 tx=0x7f1b0000b840 comp rx=0 tx=0).stop 2024-06-10T00:25:11.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.153+0000 7f1b17ad3700 1 -- 172.21.15.121:0/2052132344 shutdown_connections 2024-06-10T00:25:11.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.153+0000 7f1b17ad3700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1afc0a2c20 0x7f1afc0a50e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:11.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.153+0000 7f1b17ad3700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b10113190 0x7f1b101a3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:11.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.153+0000 7f1b17ad3700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b10074c00 0x7f1b1019f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:11.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.153+0000 7f1b17ad3700 1 --2- 172.21.15.121:0/2052132344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b10076470 0x7f1b1019f800 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:11.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.153+0000 7f1b17ad3700 1 -- 172.21.15.121:0/2052132344 >> 172.21.15.121:0/2052132344 conn(0x7f1b10079650 msgr2=0x7f1b10111e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:11.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.154+0000 7f1b17ad3700 1 -- 172.21.15.121:0/2052132344 shutdown_connections 2024-06-10T00:25:11.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:11.154+0000 7f1b17ad3700 1 -- 172.21.15.121:0/2052132344 wait complete. 2024-06-10T00:25:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:11 smithi121 ceph-mon[32378]: pgmap v342: 129 pgs: 129 active+clean; 584 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-06-10T00:25:11.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:25:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:25:11.130+0000 7f6aba056700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:25:11.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:25:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:25:11.130+0000 7f6aba056700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:25:11.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:25:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:25:11.130+0000 7f6aba056700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:25:11.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:25:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:25:11.130+0000 7f6aba056700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:25:11.450 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:25:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:25:11.130+0000 7f6aba056700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:25:11.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:11 smithi193 ceph-mon[36716]: pgmap v342: 129 pgs: 129 active+clean; 584 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-06-10T00:25:11.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:11 smithi135 ceph-mon[36424]: pgmap v342: 129 pgs: 129 active+clean; 584 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-06-10T00:25:11.910 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-06-10T00:25:12.152 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:12.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:12 smithi121 ceph-mon[32378]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:12.540 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:12 smithi121 ceph-mon[32378]: pgmap v343: 129 pgs: 129 active+clean; 584 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-06-10T00:25:12.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:12 smithi193 ceph-mon[36716]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:12.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:12 smithi193 ceph-mon[36716]: pgmap v343: 129 pgs: 129 active+clean; 584 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-06-10T00:25:12.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:12 smithi135 ceph-mon[36424]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:12.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:12 smithi135 ceph-mon[36424]: pgmap v343: 129 pgs: 129 active+clean; 584 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-06-10T00:25:13.605 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:13 smithi121 ceph-mon[32378]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-06-10T00:25:13.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:13 smithi193 ceph-mon[36716]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-06-10T00:25:13.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:13 smithi135 ceph-mon[36424]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.847+0000 7f0e02787700 1 -- 172.21.15.121:0/1965819885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dfc101440 msgr2=0x7f0dfc10f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.847+0000 7f0e02787700 1 --2- 172.21.15.121:0/1965819885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dfc101440 0x7f0dfc10f330 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0df400b0a0 tx=0x7f0df400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.847+0000 7f0e02787700 1 -- 172.21.15.121:0/1965819885 shutdown_connections 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.847+0000 7f0e02787700 1 --2- 172.21.15.121:0/1965819885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dfc101440 0x7f0dfc10f330 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.847+0000 7f0e02787700 1 --2- 172.21.15.121:0/1965819885 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0dfc108510 0x7f0dfc1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.847+0000 7f0e02787700 1 --2- 172.21.15.121:0/1965819885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dfc100a80 0x7f0dfc100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.847+0000 7f0e02787700 1 -- 172.21.15.121:0/1965819885 >> 172.21.15.121:0/1965819885 conn(0x7f0dfc077a00 msgr2=0x7f0dfc0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.847+0000 7f0e02787700 1 -- 172.21.15.121:0/1965819885 shutdown_connections 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.847+0000 7f0e02787700 1 -- 172.21.15.121:0/1965819885 wait complete. 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0e02787700 1 Processor -- start 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0e02787700 1 -- start start 2024-06-10T00:25:13.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0e02787700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dfc100a80 0x7f0dfc19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0e02787700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dfc101440 0x7f0dfc19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0e02787700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0dfc108510 0x7f0dfc1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0e02787700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc112790 con 0x7f0dfc101440 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0e02787700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc112610 con 0x7f0dfc108510 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0e02787700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc112910 con 0x7f0dfc100a80 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0dfb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dfc101440 0x7f0dfc19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0dfb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dfc101440 0x7f0dfc19ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42864/0 (socket says 172.21.15.121:42864) 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0dfb7fe700 1 -- 172.21.15.121:0/1681600993 learned_addr learned my addr 172.21.15.121:0/1681600993 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0dfbfff700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dfc100a80 0x7f0dfc19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0e00d24700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0dfc108510 0x7f0dfc1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0dfb7fe700 1 -- 172.21.15.121:0/1681600993 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dfc100a80 msgr2=0x7f0dfc19e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.848+0000 7f0dfb7fe700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dfc100a80 0x7f0dfc19e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:13.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.849+0000 7f0dfb7fe700 1 -- 172.21.15.121:0/1681600993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0dfc108510 msgr2=0x7f0dfc1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.849+0000 7f0dfb7fe700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0dfc108510 0x7f0dfc1989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.849+0000 7f0dfb7fe700 1 -- 172.21.15.121:0/1681600993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc199110 con 0x7f0dfc101440 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.849+0000 7f0e00d24700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0dfc108510 0x7f0dfc1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.849+0000 7f0dfb7fe700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dfc101440 0x7f0dfc19ed30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c970 tx=0x7f0df000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.849+0000 7f0df97fa700 1 -- 172.21.15.121:0/1681600993 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df0007aa0 con 0x7f0dfc101440 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.849+0000 7f0df97fa700 1 -- 172.21.15.121:0/1681600993 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0df0004d10 con 0x7f0dfc101440 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.849+0000 7f0e02787700 1 -- 172.21.15.121:0/1681600993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc199400 con 0x7f0dfc101440 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.849+0000 7f0df97fa700 1 -- 172.21.15.121:0/1681600993 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df000f450 con 0x7f0dfc101440 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.849+0000 7f0e02787700 1 -- 172.21.15.121:0/1681600993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc1a56c0 con 0x7f0dfc101440 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.850+0000 7f0e02787700 1 -- 172.21.15.121:0/1681600993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dfc069180 con 0x7f0dfc101440 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.851+0000 7f0df97fa700 1 -- 172.21.15.121:0/1681600993 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f0df0004750 con 0x7f0dfc101440 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.852+0000 7f0df97fa700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0de40a2d30 0x7f0de40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:13.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.852+0000 7f0dfbfff700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0de40a2d30 0x7f0de40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:13.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.852+0000 7f0df97fa700 1 -- 172.21.15.121:0/1681600993 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f0df00cf550 con 0x7f0dfc101440 2024-06-10T00:25:13.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.852+0000 7f0dfbfff700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0de40a2d30 0x7f0de40a51f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0dec005950 tx=0x7f0dec00a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:13.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:13.855+0000 7f0df97fa700 1 -- 172.21.15.121:0/1681600993 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0df009c110 con 0x7f0dfc101440 2024-06-10T00:25:14.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.084+0000 7f0e02787700 1 -- 172.21.15.121:0/1681600993 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0dfc0628d0 con 0x7f0de40a2d30 2024-06-10T00:25:14.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.111+0000 7f0df97fa700 1 -- 172.21.15.121:0/1681600993 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0dfc0628d0 con 0x7f0de40a2d30 2024-06-10T00:25:14.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 -- 172.21.15.121:0/1681600993 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0de40a2d30 msgr2=0x7f0de40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0de40a2d30 0x7f0de40a51f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0dec005950 tx=0x7f0dec00a300 comp rx=0 tx=0).stop 2024-06-10T00:25:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 -- 172.21.15.121:0/1681600993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dfc101440 msgr2=0x7f0dfc19ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dfc101440 0x7f0dfc19ed30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0df000c970 tx=0x7f0df000cc80 comp rx=0 tx=0).stop 2024-06-10T00:25:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 -- 172.21.15.121:0/1681600993 shutdown_connections 2024-06-10T00:25:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0de40a2d30 0x7f0de40a51f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0dfc108510 0x7f0dfc1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dfc100a80 0x7f0dfc19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 --2- 172.21.15.121:0/1681600993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0dfc101440 0x7f0dfc19ed30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 -- 172.21.15.121:0/1681600993 >> 172.21.15.121:0/1681600993 conn(0x7f0dfc077a00 msgr2=0x7f0dfc0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 -- 172.21.15.121:0/1681600993 shutdown_connections 2024-06-10T00:25:14.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:14.114+0000 7f0e02787700 1 -- 172.21.15.121:0/1681600993 wait complete. 2024-06-10T00:25:14.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:14 smithi121 ceph-mon[32378]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:14.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:14 smithi121 ceph-mon[32378]: pgmap v344: 129 pgs: 129 active+clean; 598 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-06-10T00:25:14.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:14 smithi193 ceph-mon[36716]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:14.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:14 smithi193 ceph-mon[36716]: pgmap v344: 129 pgs: 129 active+clean; 598 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-06-10T00:25:14.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:14 smithi135 ceph-mon[36424]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:14.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:14 smithi135 ceph-mon[36424]: pgmap v344: 129 pgs: 129 active+clean; 598 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-06-10T00:25:14.867 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-06-10T00:25:15.106 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:16.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.892+0000 7f807459d700 1 -- 172.21.15.121:0/3513334516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c102ea0 msgr2=0x7f806c10f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:16.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.892+0000 7f807459d700 1 --2- 172.21.15.121:0/3513334516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c102ea0 0x7f806c10f380 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8064009990 tx=0x7f8064009ca0 comp rx=0 tx=0).stop 2024-06-10T00:25:16.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.892+0000 7f807459d700 1 -- 172.21.15.121:0/3513334516 shutdown_connections 2024-06-10T00:25:16.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.892+0000 7f807459d700 1 --2- 172.21.15.121:0/3513334516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c102ea0 0x7f806c10f380 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:16.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.892+0000 7f807459d700 1 --2- 172.21.15.121:0/3513334516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f806c1085f0 0x7f806c1089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:16.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.892+0000 7f807459d700 1 --2- 172.21.15.121:0/3513334516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f806c1024e0 0x7f806c102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.892+0000 7f807459d700 1 -- 172.21.15.121:0/3513334516 >> 172.21.15.121:0/3513334516 conn(0x7f806c079640 msgr2=0x7f806c079a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.892+0000 7f807459d700 1 -- 172.21.15.121:0/3513334516 shutdown_connections 2024-06-10T00:25:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.892+0000 7f807459d700 1 -- 172.21.15.121:0/3513334516 wait complete. 2024-06-10T00:25:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f807459d700 1 Processor -- start 2024-06-10T00:25:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f807459d700 1 -- start start 2024-06-10T00:25:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f807459d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f806c1024e0 0x7f806c19e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f807459d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c102ea0 0x7f806c19ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f807459d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f806c1085f0 0x7f806c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f807459d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f806c112810 con 0x7f806c102ea0 2024-06-10T00:25:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f807459d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f806c112690 con 0x7f806c1024e0 2024-06-10T00:25:16.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f807459d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f806c112990 con 0x7f806c1085f0 2024-06-10T00:25:16.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f8071b38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c102ea0 0x7f806c19ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:16.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f8071b38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c102ea0 0x7f806c19ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42880/0 (socket says 172.21.15.121:42880) 2024-06-10T00:25:16.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f8071b38700 1 -- 172.21.15.121:0/1399888473 learned_addr learned my addr 172.21.15.121:0/1399888473 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:16.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f8072339700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f806c1024e0 0x7f806c19e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:16.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f8072b3a700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f806c1085f0 0x7f806c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:16.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f8071b38700 1 -- 172.21.15.121:0/1399888473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f806c1085f0 msgr2=0x7f806c1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:16.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f8071b38700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f806c1085f0 0x7f806c1989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:16.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f8071b38700 1 -- 172.21.15.121:0/1399888473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f806c1024e0 msgr2=0x7f806c19e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:16.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f8071b38700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f806c1024e0 0x7f806c19e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:16.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.893+0000 7f8071b38700 1 -- 172.21.15.121:0/1399888473 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c199110 con 0x7f806c102ea0 2024-06-10T00:25:16.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.894+0000 7f8071b38700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c102ea0 0x7f806c19ece0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f805c00b530 tx=0x7f805c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:16.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.894+0000 7f80637fe700 1 -- 172.21.15.121:0/1399888473 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c015020 con 0x7f806c102ea0 2024-06-10T00:25:16.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.894+0000 7f80637fe700 1 -- 172.21.15.121:0/1399888473 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f805c011d10 con 0x7f806c102ea0 2024-06-10T00:25:16.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.894+0000 7f807459d700 1 -- 172.21.15.121:0/1399888473 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806c199400 con 0x7f806c102ea0 2024-06-10T00:25:16.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.894+0000 7f80637fe700 1 -- 172.21.15.121:0/1399888473 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c010e50 con 0x7f806c102ea0 2024-06-10T00:25:16.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.894+0000 7f807459d700 1 -- 172.21.15.121:0/1399888473 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f806c1a5670 con 0x7f806c102ea0 2024-06-10T00:25:16.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.895+0000 7f807459d700 1 -- 172.21.15.121:0/1399888473 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f806c069180 con 0x7f806c102ea0 2024-06-10T00:25:16.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.896+0000 7f80637fe700 1 -- 172.21.15.121:0/1399888473 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f805c018020 con 0x7f806c102ea0 2024-06-10T00:25:16.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.896+0000 7f80637fe700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f80580a2e50 0x7f80580a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:16.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.897+0000 7f8072339700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f80580a2e50 0x7f80580a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:16.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.897+0000 7f80637fe700 1 -- 172.21.15.121:0/1399888473 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f805c0cecb0 con 0x7f806c102ea0 2024-06-10T00:25:16.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.897+0000 7f8072339700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f80580a2e50 0x7f80580a5310 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f806c108dc0 tx=0x7f8068009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:16.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:16.901+0000 7f80637fe700 1 -- 172.21.15.121:0/1399888473 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f805c01f020 con 0x7f806c102ea0 2024-06-10T00:25:17.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.132+0000 7f807459d700 1 -- 172.21.15.121:0/1399888473 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f806c1014e0 con 0x7f80580a2e50 2024-06-10T00:25:17.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.137+0000 7f80637fe700 1 -- 172.21.15.121:0/1399888473 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f806c1014e0 con 0x7f80580a2e50 2024-06-10T00:25:17.137 INFO:teuthology.orchestra.run.smithi121.stdout:/volumes/qa/sv_0/4e3126f6-c737-4056-af99-c6da77680a6d 2024-06-10T00:25:17.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.139+0000 7f807459d700 1 -- 172.21.15.121:0/1399888473 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f80580a2e50 msgr2=0x7f80580a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:17.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.139+0000 7f807459d700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f80580a2e50 0x7f80580a5310 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f806c108dc0 tx=0x7f8068009040 comp rx=0 tx=0).stop 2024-06-10T00:25:17.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.139+0000 7f807459d700 1 -- 172.21.15.121:0/1399888473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c102ea0 msgr2=0x7f806c19ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:17.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.139+0000 7f807459d700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c102ea0 0x7f806c19ece0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f805c00b530 tx=0x7f805c00b8f0 comp rx=0 tx=0).stop 2024-06-10T00:25:17.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.140+0000 7f807459d700 1 -- 172.21.15.121:0/1399888473 shutdown_connections 2024-06-10T00:25:17.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.140+0000 7f807459d700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f80580a2e50 0x7f80580a5310 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:17.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.140+0000 7f807459d700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f806c1085f0 0x7f806c1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:17.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.140+0000 7f807459d700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f806c1024e0 0x7f806c19e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:17.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.140+0000 7f807459d700 1 --2- 172.21.15.121:0/1399888473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f806c102ea0 0x7f806c19ece0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:17.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.140+0000 7f807459d700 1 -- 172.21.15.121:0/1399888473 >> 172.21.15.121:0/1399888473 conn(0x7f806c079640 msgr2=0x7f806c100dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:17.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.140+0000 7f807459d700 1 -- 172.21.15.121:0/1399888473 shutdown_connections 2024-06-10T00:25:17.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:17.140+0000 7f807459d700 1 -- 172.21.15.121:0/1399888473 wait complete. 2024-06-10T00:25:17.432 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:17 smithi121 ceph-mon[32378]: pgmap v345: 129 pgs: 129 active+clean; 598 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-06-10T00:25:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:17 smithi135 ceph-mon[36424]: pgmap v345: 129 pgs: 129 active+clean; 598 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-06-10T00:25:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:17 smithi193 ceph-mon[36716]: pgmap v345: 129 pgs: 129 active+clean; 598 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-06-10T00:25:17.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-06-10T00:25:18.116 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:18.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:18 smithi121 ceph-mon[32378]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:18.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:18 smithi135 ceph-mon[36424]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:18.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:18 smithi193 ceph-mon[36716]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:19.251 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:19 smithi121 ceph-mon[32378]: pgmap v346: 129 pgs: 129 active+clean; 599 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-06-10T00:25:19.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:19 smithi135 ceph-mon[36424]: pgmap v346: 129 pgs: 129 active+clean; 599 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-06-10T00:25:19.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:19 smithi193 ceph-mon[36716]: pgmap v346: 129 pgs: 129 active+clean; 599 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2024-06-10T00:25:19.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.804+0000 7ff6d2d70700 1 -- 172.21.15.121:0/3892163943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6cc107840 msgr2=0x7ff6cc10f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.804+0000 7ff6d2d70700 1 --2- 172.21.15.121:0/3892163943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6cc107840 0x7ff6cc10f3c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400b0d0 tx=0x7ff6c400b3e0 comp rx=0 tx=0).stop 2024-06-10T00:25:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 -- 172.21.15.121:0/3892163943 shutdown_connections 2024-06-10T00:25:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 --2- 172.21.15.121:0/3892163943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6cc107840 0x7ff6cc10f3c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 --2- 172.21.15.121:0/3892163943 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6cc102740 0x7ff6cc102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 --2- 172.21.15.121:0/3892163943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6cc1030f0 0x7ff6cc107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 -- 172.21.15.121:0/3892163943 >> 172.21.15.121:0/3892163943 conn(0x7ff6cc0795f0 msgr2=0x7ff6cc079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 -- 172.21.15.121:0/3892163943 shutdown_connections 2024-06-10T00:25:19.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 -- 172.21.15.121:0/3892163943 wait complete. 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 Processor -- start 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 -- start start 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6cc102740 0x7ff6cc19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6cc1030f0 0x7ff6cc19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6cc107840 0x7ff6cc198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc112760 con 0x7ff6cc1030f0 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1125e0 con 0x7ff6cc102740 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.805+0000 7ff6d2d70700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1128e0 con 0x7ff6cc107840 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6cbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6cc1030f0 0x7ff6cc19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6cbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6cc1030f0 0x7ff6cc19ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58456/0 (socket says 172.21.15.121:58456) 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6cbfff700 1 -- 172.21.15.121:0/2403339306 learned_addr learned my addr 172.21.15.121:0/2403339306 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6d0b0c700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6cc102740 0x7ff6cc19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:19.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6d130d700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6cc107840 0x7ff6cc198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6cbfff700 1 -- 172.21.15.121:0/2403339306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6cc107840 msgr2=0x7ff6cc198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6cbfff700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6cc107840 0x7ff6cc198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6cbfff700 1 -- 172.21.15.121:0/2403339306 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6cc102740 msgr2=0x7ff6cc19e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6cbfff700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6cc102740 0x7ff6cc19e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6cbfff700 1 -- 172.21.15.121:0/2403339306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6cc199060 con 0x7ff6cc1030f0 2024-06-10T00:25:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6cbfff700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6cc1030f0 0x7ff6cc19ec30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff6b800d950 tx=0x7ff6b800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6c9ffb700 1 -- 172.21.15.121:0/2403339306 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8016070 con 0x7ff6cc1030f0 2024-06-10T00:25:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6c9ffb700 1 -- 172.21.15.121:0/2403339306 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b800fd70 con 0x7ff6cc1030f0 2024-06-10T00:25:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6d2d70700 1 -- 172.21.15.121:0/2403339306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6cc199350 con 0x7ff6cc1030f0 2024-06-10T00:25:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.806+0000 7ff6c9ffb700 1 -- 172.21.15.121:0/2403339306 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b800e610 con 0x7ff6cc1030f0 2024-06-10T00:25:19.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.807+0000 7ff6d2d70700 1 -- 172.21.15.121:0/2403339306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6cc1a55c0 con 0x7ff6cc1030f0 2024-06-10T00:25:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.807+0000 7ff6d2d70700 1 -- 172.21.15.121:0/2403339306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6cc069180 con 0x7ff6cc1030f0 2024-06-10T00:25:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.808+0000 7ff6c9ffb700 1 -- 172.21.15.121:0/2403339306 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff6b8019070 con 0x7ff6cc1030f0 2024-06-10T00:25:19.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.809+0000 7ff6c9ffb700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff6bc0a2cb0 0x7ff6bc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.809+0000 7ff6d0b0c700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff6bc0a2cb0 0x7ff6bc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.809+0000 7ff6c9ffb700 1 -- 172.21.15.121:0/2403339306 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7ff6b80cefe0 con 0x7ff6cc1030f0 2024-06-10T00:25:19.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.809+0000 7ff6d0b0c700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff6bc0a2cb0 0x7ff6bc0a5170 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc07a090 tx=0x7ff6c000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:19.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:19.814+0000 7ff6c9ffb700 1 -- 172.21.15.121:0/2403339306 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff6b809ba80 con 0x7ff6cc1030f0 2024-06-10T00:25:20.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.042+0000 7ff6d2d70700 1 -- 172.21.15.121:0/2403339306 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff6cc0628d0 con 0x7ff6bc0a2cb0 2024-06-10T00:25:20.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.072+0000 7ff6c9ffb700 1 -- 172.21.15.121:0/2403339306 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff6cc0628d0 con 0x7ff6bc0a2cb0 2024-06-10T00:25:20.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.074+0000 7ff6d2d70700 1 -- 172.21.15.121:0/2403339306 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff6bc0a2cb0 msgr2=0x7ff6bc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.074+0000 7ff6d2d70700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff6bc0a2cb0 0x7ff6bc0a5170 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc07a090 tx=0x7ff6c000a380 comp rx=0 tx=0).stop 2024-06-10T00:25:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.074+0000 7ff6d2d70700 1 -- 172.21.15.121:0/2403339306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6cc1030f0 msgr2=0x7ff6cc19ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.074+0000 7ff6d2d70700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6cc1030f0 0x7ff6cc19ec30 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff6b800d950 tx=0x7ff6b800dc60 comp rx=0 tx=0).stop 2024-06-10T00:25:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.074+0000 7ff6d2d70700 1 -- 172.21.15.121:0/2403339306 shutdown_connections 2024-06-10T00:25:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.074+0000 7ff6d2d70700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff6bc0a2cb0 0x7ff6bc0a5170 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.074+0000 7ff6d2d70700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6cc107840 0x7ff6cc198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.074+0000 7ff6d2d70700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6cc102740 0x7ff6cc19e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.074+0000 7ff6d2d70700 1 --2- 172.21.15.121:0/2403339306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6cc1030f0 0x7ff6cc19ec30 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.074+0000 7ff6d2d70700 1 -- 172.21.15.121:0/2403339306 >> 172.21.15.121:0/2403339306 conn(0x7ff6cc0795f0 msgr2=0x7ff6cc100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.075+0000 7ff6d2d70700 1 -- 172.21.15.121:0/2403339306 shutdown_connections 2024-06-10T00:25:20.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:20.075+0000 7ff6d2d70700 1 -- 172.21.15.121:0/2403339306 wait complete. 2024-06-10T00:25:20.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-06-10T00:25:21.094 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:21.433 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:21 smithi193 ceph-mon[36716]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:21.433 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:21 smithi193 ceph-mon[36716]: pgmap v347: 129 pgs: 129 active+clean; 610 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 9 op/s 2024-06-10T00:25:21.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:21 smithi121 ceph-mon[32378]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:21 smithi121 ceph-mon[32378]: pgmap v347: 129 pgs: 129 active+clean; 610 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 9 op/s 2024-06-10T00:25:21.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:21 smithi135 ceph-mon[36424]: from='client.14787 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:21 smithi135 ceph-mon[36424]: pgmap v347: 129 pgs: 129 active+clean; 610 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.1 KiB/s wr, 9 op/s 2024-06-10T00:25:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.765+0000 7f2ea9613700 1 -- 172.21.15.121:0/2630759551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ea4101d50 msgr2=0x7f2ea410e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.765+0000 7f2ea9613700 1 --2- 172.21.15.121:0/2630759551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ea4101d50 0x7f2ea410e240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00b0a0 tx=0x7f2e9c00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:25:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.765+0000 7f2ea9613700 1 -- 172.21.15.121:0/2630759551 shutdown_connections 2024-06-10T00:25:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.765+0000 7f2ea9613700 1 --2- 172.21.15.121:0/2630759551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ea4101d50 0x7f2ea410e240 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.765+0000 7f2ea9613700 1 --2- 172.21.15.121:0/2630759551 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea4107400 0x7f2ea41077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.765+0000 7f2ea9613700 1 --2- 172.21.15.121:0/2630759551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ea4101390 0x7f2ea4101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.765+0000 7f2ea9613700 1 -- 172.21.15.121:0/2630759551 >> 172.21.15.121:0/2630759551 conn(0x7f2ea4078340 msgr2=0x7f2ea4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:22.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.765+0000 7f2ea9613700 1 -- 172.21.15.121:0/2630759551 shutdown_connections 2024-06-10T00:25:22.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.765+0000 7f2ea9613700 1 -- 172.21.15.121:0/2630759551 wait complete. 2024-06-10T00:25:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea9613700 1 Processor -- start 2024-06-10T00:25:22.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea9613700 1 -- start start 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea9613700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ea4101390 0x7f2ea419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea9613700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea4101d50 0x7f2ea419ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea9613700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ea4107400 0x7f2ea4198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea9613700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea41116a0 con 0x7f2ea4101390 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea9613700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea4111520 con 0x7f2ea4101d50 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea9613700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea4111820 con 0x7f2ea4107400 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ea4101390 0x7f2ea419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ea4101390 0x7f2ea419e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58482/0 (socket says 172.21.15.121:58482) 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea2ffd700 1 -- 172.21.15.121:0/4196776460 learned_addr learned my addr 172.21.15.121:0/4196776460 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2ea37fe700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ea4107400 0x7f2ea4198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.766+0000 7f2e9a7fc700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea4101d50 0x7f2ea419ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.767+0000 7f2ea2ffd700 1 -- 172.21.15.121:0/4196776460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ea4107400 msgr2=0x7f2ea4198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.767+0000 7f2ea2ffd700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ea4107400 0x7f2ea4198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:22.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.767+0000 7f2ea2ffd700 1 -- 172.21.15.121:0/4196776460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea4101d50 msgr2=0x7f2ea419ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.767+0000 7f2ea2ffd700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea4101d50 0x7f2ea419ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.767+0000 7f2ea2ffd700 1 -- 172.21.15.121:0/4196776460 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea41990f0 con 0x7f2ea4101390 2024-06-10T00:25:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.767+0000 7f2ea2ffd700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ea4101390 0x7f2ea419e7d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400c9b0 tx=0x7f2e9400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.767+0000 7f2ea0ff9700 1 -- 172.21.15.121:0/4196776460 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94007b00 con 0x7f2ea4101390 2024-06-10T00:25:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.767+0000 7f2ea0ff9700 1 -- 172.21.15.121:0/4196776460 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e9400ce50 con 0x7f2ea4101390 2024-06-10T00:25:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.767+0000 7f2ea9613700 1 -- 172.21.15.121:0/4196776460 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea41993e0 con 0x7f2ea4101390 2024-06-10T00:25:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.767+0000 7f2ea0ff9700 1 -- 172.21.15.121:0/4196776460 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94018660 con 0x7f2ea4101390 2024-06-10T00:25:22.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.767+0000 7f2ea9613700 1 -- 172.21.15.121:0/4196776460 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea41a56a0 con 0x7f2ea4101390 2024-06-10T00:25:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.768+0000 7f2ea9613700 1 -- 172.21.15.121:0/4196776460 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea4069180 con 0x7f2ea4101390 2024-06-10T00:25:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.769+0000 7f2ea0ff9700 1 -- 172.21.15.121:0/4196776460 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2e9400f450 con 0x7f2ea4101390 2024-06-10T00:25:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.770+0000 7f2ea0ff9700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e900a2cb0 0x7f2e900a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.770+0000 7f2e9a7fc700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e900a2cb0 0x7f2e900a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:22.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.770+0000 7f2ea0ff9700 1 -- 172.21.15.121:0/4196776460 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f2e940cf340 con 0x7f2ea4101390 2024-06-10T00:25:22.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.770+0000 7f2e9a7fc700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e900a2cb0 0x7f2e900a5170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40788b0 tx=0x7f2e8c00b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:22.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:22.775+0000 7f2ea0ff9700 1 -- 172.21.15.121:0/4196776460 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2e94014030 con 0x7f2ea4101390 2024-06-10T00:25:23.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.005+0000 7f2ea9613700 1 -- 172.21.15.121:0/4196776460 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f2ea41051b0 con 0x7f2e900a2cb0 2024-06-10T00:25:23.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.010+0000 7f2ea0ff9700 1 -- 172.21.15.121:0/4196776460 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f2ea41051b0 con 0x7f2e900a2cb0 2024-06-10T00:25:23.010 INFO:teuthology.orchestra.run.smithi121.stdout:/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca 2024-06-10T00:25:23.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.012+0000 7f2ea9613700 1 -- 172.21.15.121:0/4196776460 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e900a2cb0 msgr2=0x7f2e900a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:23.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.012+0000 7f2ea9613700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e900a2cb0 0x7f2e900a5170 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40788b0 tx=0x7f2e8c00b3c0 comp rx=0 tx=0).stop 2024-06-10T00:25:23.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.012+0000 7f2ea9613700 1 -- 172.21.15.121:0/4196776460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ea4101390 msgr2=0x7f2ea419e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:23.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.012+0000 7f2ea9613700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ea4101390 0x7f2ea419e7d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400c9b0 tx=0x7f2e9400ccc0 comp rx=0 tx=0).stop 2024-06-10T00:25:23.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.012+0000 7f2ea9613700 1 -- 172.21.15.121:0/4196776460 shutdown_connections 2024-06-10T00:25:23.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.012+0000 7f2ea9613700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2e900a2cb0 0x7f2e900a5170 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:23.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.012+0000 7f2ea9613700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ea4107400 0x7f2ea4198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:23.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.012+0000 7f2ea9613700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ea4101390 0x7f2ea419e7d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:23.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.012+0000 7f2ea9613700 1 --2- 172.21.15.121:0/4196776460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ea4101d50 0x7f2ea419ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:23.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.012+0000 7f2ea9613700 1 -- 172.21.15.121:0/4196776460 >> 172.21.15.121:0/4196776460 conn(0x7f2ea4078340 msgr2=0x7f2ea4100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:23.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.013+0000 7f2ea9613700 1 -- 172.21.15.121:0/4196776460 shutdown_connections 2024-06-10T00:25:23.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:23.013+0000 7f2ea9613700 1 -- 172.21.15.121:0/4196776460 wait complete. 2024-06-10T00:25:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:23 smithi121 ceph-mon[32378]: pgmap v348: 129 pgs: 129 active+clean; 610 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-06-10T00:25:23.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:23 smithi135 ceph-mon[36424]: pgmap v348: 129 pgs: 129 active+clean; 610 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-06-10T00:25:23.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:23 smithi193 ceph-mon[36716]: pgmap v348: 129 pgs: 129 active+clean; 610 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 3.1 KiB/s wr, 7 op/s 2024-06-10T00:25:23.690 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:25:23.930 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:24.306 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:24 smithi121 ceph-mon[32378]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:24.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:24 smithi135 ceph-mon[36424]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:24.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:24 smithi193 ceph-mon[36716]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:25:25.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:25 smithi121 ceph-mon[32378]: pgmap v349: 129 pgs: 129 active+clean; 632 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.1 KiB/s wr, 9 op/s 2024-06-10T00:25:25.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:25 smithi135 ceph-mon[36424]: pgmap v349: 129 pgs: 129 active+clean; 632 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.1 KiB/s wr, 9 op/s 2024-06-10T00:25:25.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.603+0000 7f670e60f700 1 -- 172.21.15.121:0/3116845475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6708108570 msgr2=0x7f6708108950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:25.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.603+0000 7f670e60f700 1 --2- 172.21.15.121:0/3116845475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6708108570 0x7f6708108950 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f66f8009990 tx=0x7f66f8009ca0 comp rx=0 tx=0).stop 2024-06-10T00:25:25.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.603+0000 7f670e60f700 1 -- 172.21.15.121:0/3116845475 shutdown_connections 2024-06-10T00:25:25.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.603+0000 7f670e60f700 1 --2- 172.21.15.121:0/3116845475 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f670806b5c0 0x7f670810f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:25.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.603+0000 7f670e60f700 1 --2- 172.21.15.121:0/3116845475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6708108570 0x7f6708108950 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:25.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.603+0000 7f670e60f700 1 --2- 172.21.15.121:0/3116845475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f670806ac00 0x7f670806b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:25.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.603+0000 7f670e60f700 1 -- 172.21.15.121:0/3116845475 >> 172.21.15.121:0/3116845475 conn(0x7f67080773e0 msgr2=0x7f67080777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:25.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.603+0000 7f670e60f700 1 -- 172.21.15.121:0/3116845475 shutdown_connections 2024-06-10T00:25:25.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.603+0000 7f670e60f700 1 -- 172.21.15.121:0/3116845475 wait complete. 2024-06-10T00:25:25.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f670e60f700 1 Processor -- start 2024-06-10T00:25:25.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f670e60f700 1 -- start start 2024-06-10T00:25:25.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f670e60f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670806ac00 0x7f670819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:25.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f670e60f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f670806b5c0 0x7f670819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:25.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f670e60f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6708108570 0x7f6708198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:25.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f670e60f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6708112820 con 0x7f670806ac00 2024-06-10T00:25:25.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f670e60f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f67081126a0 con 0x7f6708108570 2024-06-10T00:25:25.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f670e60f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f67081129a0 con 0x7f670806b5c0 2024-06-10T00:25:25.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f6707fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670806ac00 0x7f670819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:25.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f6707fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670806ac00 0x7f670819e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58500/0 (socket says 172.21.15.121:58500) 2024-06-10T00:25:25.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f6707fff700 1 -- 172.21.15.121:0/2368009637 learned_addr learned my addr 172.21.15.121:0/2368009637 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:25.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f670cbac700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6708108570 0x7f6708198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:25.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f66fffff700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f670806b5c0 0x7f670819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:25.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f6707fff700 1 -- 172.21.15.121:0/2368009637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f670806b5c0 msgr2=0x7f670819eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:25.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.604+0000 7f6707fff700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f670806b5c0 0x7f670819eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:25.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.605+0000 7f6707fff700 1 -- 172.21.15.121:0/2368009637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6708108570 msgr2=0x7f6708198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:25.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.605+0000 7f6707fff700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6708108570 0x7f6708198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:25.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.605+0000 7f6707fff700 1 -- 172.21.15.121:0/2368009637 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6708199180 con 0x7f670806ac00 2024-06-10T00:25:25.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.605+0000 7f670cbac700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6708108570 0x7f6708198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:25:25.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.605+0000 7f6707fff700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670806ac00 0x7f670819e860 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f66f8000c00 tx=0x7f66f80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:25.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.605+0000 7f6705ffb700 1 -- 172.21.15.121:0/2368009637 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f8027070 con 0x7f670806ac00 2024-06-10T00:25:25.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.605+0000 7f670e60f700 1 -- 172.21.15.121:0/2368009637 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6708199410 con 0x7f670806ac00 2024-06-10T00:25:25.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.605+0000 7f6705ffb700 1 -- 172.21.15.121:0/2368009637 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66f8004050 con 0x7f670806ac00 2024-06-10T00:25:25.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.605+0000 7f6705ffb700 1 -- 172.21.15.121:0/2368009637 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f800f460 con 0x7f670806ac00 2024-06-10T00:25:25.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.605+0000 7f670e60f700 1 -- 172.21.15.121:0/2368009637 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67081a5730 con 0x7f670806ac00 2024-06-10T00:25:25.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.606+0000 7f670e60f700 1 -- 172.21.15.121:0/2368009637 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f670810aa20 con 0x7f670806ac00 2024-06-10T00:25:25.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.607+0000 7f6705ffb700 1 -- 172.21.15.121:0/2368009637 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f66f802a070 con 0x7f670806ac00 2024-06-10T00:25:25.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.608+0000 7f6705ffb700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f66f40a2cb0 0x7f66f40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:25.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.608+0000 7f66fffff700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f66f40a2cb0 0x7f66f40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:25.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.608+0000 7f6705ffb700 1 -- 172.21.15.121:0/2368009637 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f66f80d08e0 con 0x7f670806ac00 2024-06-10T00:25:25.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.608+0000 7f66fffff700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f66f40a2cb0 0x7f66f40a5170 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f66f0006fd0 tx=0x7f66f0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:25.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.612+0000 7f6705ffb700 1 -- 172.21.15.121:0/2368009637 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f66f8031070 con 0x7f670806ac00 2024-06-10T00:25:25.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:25 smithi193 ceph-mon[36716]: pgmap v349: 129 pgs: 129 active+clean; 632 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.1 KiB/s wr, 9 op/s 2024-06-10T00:25:25.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.852+0000 7f670e60f700 1 -- 172.21.15.121:0/2368009637 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6708069180 con 0x7f670806ac00 2024-06-10T00:25:25.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.853+0000 7f6705ffb700 1 -- 172.21.15.121:0/2368009637 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11885 (secure 0 0 0) 0x7f66f8013070 con 0x7f670806ac00 2024-06-10T00:25:25.853 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:25:25.853 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:25:08.157143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:25:25.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.855+0000 7f670e60f700 1 -- 172.21.15.121:0/2368009637 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f66f40a2cb0 msgr2=0x7f66f40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:25.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.855+0000 7f670e60f700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f66f40a2cb0 0x7f66f40a5170 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f66f0006fd0 tx=0x7f66f0008040 comp rx=0 tx=0).stop 2024-06-10T00:25:25.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.855+0000 7f670e60f700 1 -- 172.21.15.121:0/2368009637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670806ac00 msgr2=0x7f670819e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:25.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.855+0000 7f670e60f700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670806ac00 0x7f670819e860 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f66f8000c00 tx=0x7f66f80047b0 comp rx=0 tx=0).stop 2024-06-10T00:25:25.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.855+0000 7f670e60f700 1 -- 172.21.15.121:0/2368009637 shutdown_connections 2024-06-10T00:25:25.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.856+0000 7f670e60f700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f66f40a2cb0 0x7f66f40a5170 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:25.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.856+0000 7f670e60f700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6708108570 0x7f6708198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:25.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.856+0000 7f670e60f700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670806ac00 0x7f670819e860 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:25.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.856+0000 7f670e60f700 1 --2- 172.21.15.121:0/2368009637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f670806b5c0 0x7f670819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:25.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.856+0000 7f670e60f700 1 -- 172.21.15.121:0/2368009637 >> 172.21.15.121:0/2368009637 conn(0x7f67080773e0 msgr2=0x7f67080fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:25.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.856+0000 7f670e60f700 1 -- 172.21.15.121:0/2368009637 shutdown_connections 2024-06-10T00:25:25.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:25.856+0000 7f670e60f700 1 -- 172.21.15.121:0/2368009637 wait complete. 2024-06-10T00:25:25.856 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 25 2024-06-10T00:25:26.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:26 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2368009637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:25:26.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:26 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2368009637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:25:26.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:26 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2368009637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:25:26.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:25:26.881 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:27.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:27 smithi121 ceph-mon[32378]: pgmap v350: 129 pgs: 129 active+clean; 632 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.8 KiB/s wr, 7 op/s 2024-06-10T00:25:27.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:27 smithi135 ceph-mon[36424]: pgmap v350: 129 pgs: 129 active+clean; 632 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.8 KiB/s wr, 7 op/s 2024-06-10T00:25:27.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:27 smithi193 ceph-mon[36716]: pgmap v350: 129 pgs: 129 active+clean; 632 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.8 KiB/s wr, 7 op/s 2024-06-10T00:25:28.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.728+0000 7f0828423700 1 -- 172.21.15.121:0/2653739643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0820101d50 msgr2=0x7f082010e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:28.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.728+0000 7f0828423700 1 --2- 172.21.15.121:0/2653739643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0820101d50 0x7f082010e240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f081800b0a0 tx=0x7f081800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:25:28.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.728+0000 7f0828423700 1 -- 172.21.15.121:0/2653739643 shutdown_connections 2024-06-10T00:25:28.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.728+0000 7f0828423700 1 --2- 172.21.15.121:0/2653739643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0820101d50 0x7f082010e240 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:28.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.728+0000 7f0828423700 1 --2- 172.21.15.121:0/2653739643 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0820107400 0x7f08201077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:28.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.728+0000 7f0828423700 1 --2- 172.21.15.121:0/2653739643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0820101390 0x7f0820101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.728+0000 7f0828423700 1 -- 172.21.15.121:0/2653739643 >> 172.21.15.121:0/2653739643 conn(0x7f0820078340 msgr2=0x7f0820078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.728+0000 7f0828423700 1 -- 172.21.15.121:0/2653739643 shutdown_connections 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.728+0000 7f0828423700 1 -- 172.21.15.121:0/2653739643 wait complete. 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f0828423700 1 Processor -- start 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f0828423700 1 -- start start 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f0828423700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0820101390 0x7f082019e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f0828423700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0820101d50 0x7f082019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f0828423700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0820107400 0x7f0820198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f0828423700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0820111770 con 0x7f0820107400 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f0828423700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f08201115f0 con 0x7f0820101390 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f0828423700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f08201118f0 con 0x7f0820101d50 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f08269c0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0820107400 0x7f0820198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:28.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f08269c0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0820107400 0x7f0820198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58522/0 (socket says 172.21.15.121:58522) 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f08269c0700 1 -- 172.21.15.121:0/2329147493 learned_addr learned my addr 172.21.15.121:0/2329147493 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f08259be700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0820101d50 0x7f082019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f08261bf700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0820101390 0x7f082019e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f08269c0700 1 -- 172.21.15.121:0/2329147493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0820101d50 msgr2=0x7f082019ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f08269c0700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0820101d50 0x7f082019ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f08269c0700 1 -- 172.21.15.121:0/2329147493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0820101390 msgr2=0x7f082019e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.729+0000 7f08269c0700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0820101390 0x7f082019e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.730+0000 7f08269c0700 1 -- 172.21.15.121:0/2329147493 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0820199250 con 0x7f0820107400 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.730+0000 7f08269c0700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0820107400 0x7f0820198af0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f08180062a0 tx=0x7f081800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.730+0000 7f08177fe700 1 -- 172.21.15.121:0/2329147493 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818021070 con 0x7f0820107400 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.730+0000 7f08177fe700 1 -- 172.21.15.121:0/2329147493 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0818007db0 con 0x7f0820107400 2024-06-10T00:25:28.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.730+0000 7f08177fe700 1 -- 172.21.15.121:0/2329147493 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818005460 con 0x7f0820107400 2024-06-10T00:25:28.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.730+0000 7f0828423700 1 -- 172.21.15.121:0/2329147493 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08201994e0 con 0x7f0820107400 2024-06-10T00:25:28.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.730+0000 7f0828423700 1 -- 172.21.15.121:0/2329147493 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08201a57b0 con 0x7f0820107400 2024-06-10T00:25:28.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.731+0000 7f0828423700 1 -- 172.21.15.121:0/2329147493 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0820102ab0 con 0x7f0820107400 2024-06-10T00:25:28.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.732+0000 7f08177fe700 1 -- 172.21.15.121:0/2329147493 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f0818024070 con 0x7f0820107400 2024-06-10T00:25:28.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.733+0000 7f08177fe700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f080c0a2cb0 0x7f080c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:28.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.733+0000 7f08261bf700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f080c0a2cb0 0x7f080c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:28.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.733+0000 7f08177fe700 1 -- 172.21.15.121:0/2329147493 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f08180d0850 con 0x7f0820107400 2024-06-10T00:25:28.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.733+0000 7f08261bf700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f080c0a2cb0 0x7f080c0a5170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0820106c80 tx=0x7f081c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:28.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.736+0000 7f08177fe700 1 -- 172.21.15.121:0/2329147493 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f081809d380 con 0x7f0820107400 2024-06-10T00:25:28.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.932+0000 7f0828423700 1 -- 172.21.15.121:0/2329147493 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08201a5980 con 0x7f0820107400 2024-06-10T00:25:28.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.934+0000 7f08177fe700 1 -- 172.21.15.121:0/2329147493 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+24780 (secure 0 0 0) 0x7f081809cc80 con 0x7f0820107400 2024-06-10T00:25:28.934 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:25:28.934 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":98,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:24:43.250643+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:25:28.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 -- 172.21.15.121:0/2329147493 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f080c0a2cb0 msgr2=0x7f080c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:28.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f080c0a2cb0 0x7f080c0a5170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0820106c80 tx=0x7f081c008040 comp rx=0 tx=0).stop 2024-06-10T00:25:28.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 -- 172.21.15.121:0/2329147493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0820107400 msgr2=0x7f0820198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:28.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0820107400 0x7f0820198af0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f08180062a0 tx=0x7f081800bf10 comp rx=0 tx=0).stop 2024-06-10T00:25:28.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 -- 172.21.15.121:0/2329147493 shutdown_connections 2024-06-10T00:25:28.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f080c0a2cb0 0x7f080c0a5170 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:28.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0820107400 0x7f0820198af0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:28.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0820101390 0x7f082019e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:28.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 --2- 172.21.15.121:0/2329147493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0820101d50 0x7f082019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:28.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 -- 172.21.15.121:0/2329147493 >> 172.21.15.121:0/2329147493 conn(0x7f0820078340 msgr2=0x7f08201005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:28.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 -- 172.21.15.121:0/2329147493 shutdown_connections 2024-06-10T00:25:28.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:28.936+0000 7f0828423700 1 -- 172.21.15.121:0/2329147493 wait complete. 2024-06-10T00:25:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:29 smithi121 ceph-mon[32378]: pgmap v351: 129 pgs: 129 active+clean; 632 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-06-10T00:25:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:29 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2329147493' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:25:29.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:29 smithi135 ceph-mon[36424]: pgmap v351: 129 pgs: 129 active+clean; 632 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-06-10T00:25:29.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:29 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2329147493' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:25:29.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:29 smithi193 ceph-mon[36716]: pgmap v351: 129 pgs: 129 active+clean; 632 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-06-10T00:25:29.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:29 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2329147493' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:25:29.718 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-10T00:25:29.718 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-10T00:25:29.959 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:31.437 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:31 smithi121 ceph-mon[32378]: pgmap v352: 129 pgs: 129 active+clean; 638 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2024-06-10T00:25:31.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:31 smithi135 ceph-mon[36424]: pgmap v352: 129 pgs: 129 active+clean; 638 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2024-06-10T00:25:31.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:31 smithi193 ceph-mon[36716]: pgmap v352: 129 pgs: 129 active+clean; 638 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 4.3 KiB/s wr, 9 op/s 2024-06-10T00:25:31.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.690+0000 7faaa6afe700 1 -- 172.21.15.121:0/1237711135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faaa0101d50 msgr2=0x7faaa010e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:31.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.690+0000 7faaa6afe700 1 --2- 172.21.15.121:0/1237711135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faaa0101d50 0x7faaa010e240 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faa9800b0a0 tx=0x7faa9800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:25:31.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.690+0000 7faaa6afe700 1 -- 172.21.15.121:0/1237711135 shutdown_connections 2024-06-10T00:25:31.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.690+0000 7faaa6afe700 1 --2- 172.21.15.121:0/1237711135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faaa0101d50 0x7faaa010e240 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:31.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.690+0000 7faaa6afe700 1 --2- 172.21.15.121:0/1237711135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faaa0107400 0x7faaa01077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:31.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.690+0000 7faaa6afe700 1 --2- 172.21.15.121:0/1237711135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faaa0101390 0x7faaa0101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:31.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.690+0000 7faaa6afe700 1 -- 172.21.15.121:0/1237711135 >> 172.21.15.121:0/1237711135 conn(0x7faaa0078340 msgr2=0x7faaa0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:31.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.690+0000 7faaa6afe700 1 -- 172.21.15.121:0/1237711135 shutdown_connections 2024-06-10T00:25:31.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.690+0000 7faaa6afe700 1 -- 172.21.15.121:0/1237711135 wait complete. 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.690+0000 7faaa6afe700 1 Processor -- start 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.690+0000 7faaa6afe700 1 -- start start 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa6afe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faaa0101390 0x7faaa0199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa6afe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faaa0101d50 0x7faaa019a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa6afe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faaa0107400 0x7faaa01a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa6afe700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7faaa0111770 con 0x7faaa0107400 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa6afe700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7faaa01115f0 con 0x7faaa0101390 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa6afe700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faaa01118f0 con 0x7faaa0101d50 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa509b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faaa0107400 0x7faaa01a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa509b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faaa0107400 0x7faaa01a4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37942/0 (socket says 172.21.15.121:37942) 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa509b700 1 -- 172.21.15.121:0/3242547239 learned_addr learned my addr 172.21.15.121:0/3242547239 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:31.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa489a700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faaa0101390 0x7faaa0199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faa9ffff700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faaa0101d50 0x7faaa019a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa509b700 1 -- 172.21.15.121:0/3242547239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faaa0101d50 msgr2=0x7faaa019a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa509b700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faaa0101d50 0x7faaa019a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa509b700 1 -- 172.21.15.121:0/3242547239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faaa0101390 msgr2=0x7faaa0199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa509b700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faaa0101390 0x7faaa0199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa509b700 1 -- 172.21.15.121:0/3242547239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaa01a5090 con 0x7faaa0107400 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faa9ffff700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faaa0101d50 0x7faaa019a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa509b700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faaa0107400 0x7faaa01a4b50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faa980062a0 tx=0x7faa9800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faa9dffb700 1 -- 172.21.15.121:0/3242547239 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa98021070 con 0x7faaa0107400 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faa9dffb700 1 -- 172.21.15.121:0/3242547239 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa98007d00 con 0x7faaa0107400 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faaa6afe700 1 -- 172.21.15.121:0/3242547239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaa01a5260 con 0x7faaa0107400 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.691+0000 7faa9dffb700 1 -- 172.21.15.121:0/3242547239 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa98005460 con 0x7faaa0107400 2024-06-10T00:25:31.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.692+0000 7faaa6afe700 1 -- 172.21.15.121:0/3242547239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaa01a5710 con 0x7faaa0107400 2024-06-10T00:25:31.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.693+0000 7faaa6afe700 1 -- 172.21.15.121:0/3242547239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaa0109970 con 0x7faaa0107400 2024-06-10T00:25:31.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.694+0000 7faa9dffb700 1 -- 172.21.15.121:0/3242547239 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7faa98024070 con 0x7faaa0107400 2024-06-10T00:25:31.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.694+0000 7faa9dffb700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7faa900a2cb0 0x7faa900a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:31.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.695+0000 7faaa489a700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7faa900a2cb0 0x7faa900a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:31.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.695+0000 7faa9dffb700 1 -- 172.21.15.121:0/3242547239 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7faa9800f020 con 0x7faaa0107400 2024-06-10T00:25:31.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.695+0000 7faaa489a700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7faa900a2cb0 0x7faa900a5170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faa94009a10 tx=0x7faa94009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:31.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.698+0000 7faa9dffb700 1 -- 172.21.15.121:0/3242547239 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7faa9802b020 con 0x7faaa0107400 2024-06-10T00:25:31.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.890+0000 7faaa6afe700 1 -- 172.21.15.121:0/3242547239 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faaa006a780 con 0x7faa900a2cb0 2024-06-10T00:25:31.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.900+0000 7faa9dffb700 1 -- 172.21.15.121:0/3242547239 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340645 (secure 0 0 0) 0x7faaa006a780 con 0x7faa900a2cb0 2024-06-10T00:25:31.905 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 -- 172.21.15.121:0/3242547239 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7faa900a2cb0 msgr2=0x7faa900a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7faa900a2cb0 0x7faa900a5170 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faa94009a10 tx=0x7faa94009190 comp rx=0 tx=0).stop 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 -- 172.21.15.121:0/3242547239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faaa0107400 msgr2=0x7faaa01a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faaa0107400 0x7faaa01a4b50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faa980062a0 tx=0x7faa9800be60 comp rx=0 tx=0).stop 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 -- 172.21.15.121:0/3242547239 shutdown_connections 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7faa900a2cb0 0x7faa900a5170 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faaa0107400 0x7faaa01a4b50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faaa0101390 0x7faaa0199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 --2- 172.21.15.121:0/3242547239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faaa0101d50 0x7faaa019a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 -- 172.21.15.121:0/3242547239 >> 172.21.15.121:0/3242547239 conn(0x7faaa0078340 msgr2=0x7faaa00ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 -- 172.21.15.121:0/3242547239 shutdown_connections 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:31.903+0000 7faaa6afe700 1 -- 172.21.15.121:0/3242547239 wait complete. 2024-06-10T00:25:31.908 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-06-10T00:25:32.727 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-06-10T00:25:30.118081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652992,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":301,"num_read_kb":270,"num_write":351,"num_write_kb":3053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":282,"ondisk_log_size":282,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358096,"kb_used_data":13104,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583728,"statfs":{"total":1151940427776,"available":1151573737472,"internally_reserved":0,"allocated":13418496,"data_stored":8578007,"data_compressed":423552,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40076,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":94,"num_write":31,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259557+0000","last_change":"2024-06-10T00:23:57.564073+0000","last_active":"2024-06-10T00:24:43.259557+0000","last_peered":"2024-06-10T00:24:43.259557+0000","last_clean":"2024-06-10T00:24:43.259557+0000","last_became_active":"2024-06-10T00:23:57.563618+0000","last_became_peered":"2024-06-10T00:23:57.563618+0000","last_unstale":"2024-06-10T00:24:43.259557+0000","last_undegraded":"2024-06-10T00:24:43.259557+0000","last_fullsized":"2024-06-10T00:24:43.259557+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:26:25.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647887+0000","last_change":"2024-06-10T00:23:57.560148+0000","last_active":"2024-06-10T00:24:43.647887+0000","last_peered":"2024-06-10T00:24:43.647887+0000","last_clean":"2024-06-10T00:24:43.647887+0000","last_became_active":"2024-06-10T00:23:57.559794+0000","last_became_peered":"2024-06-10T00:23:57.559794+0000","last_unstale":"2024-06-10T00:24:43.647887+0000","last_undegraded":"2024-06-10T00:24:43.647887+0000","last_fullsized":"2024-06-10T00:24:43.647887+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:59:50.624542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647613+0000","last_change":"2024-06-10T00:23:57.563292+0000","last_active":"2024-06-10T00:24:43.647613+0000","last_peered":"2024-06-10T00:24:43.647613+0000","last_clean":"2024-06-10T00:24:43.647613+0000","last_became_active":"2024-06-10T00:23:57.563007+0000","last_became_peered":"2024-06-10T00:23:57.563007+0000","last_unstale":"2024-06-10T00:24:43.647613+0000","last_undegraded":"2024-06-10T00:24:43.647613+0000","last_fullsized":"2024-06-10T00:24:43.647613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:27:44.897770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259120+0000","last_change":"2024-06-10T00:23:57.558789+0000","last_active":"2024-06-10T00:24:43.259120+0000","last_peered":"2024-06-10T00:24:43.259120+0000","last_clean":"2024-06-10T00:24:43.259120+0000","last_became_active":"2024-06-10T00:23:57.557943+0000","last_became_peered":"2024-06-10T00:23:57.557943+0000","last_unstale":"2024-06-10T00:24:43.259120+0000","last_undegraded":"2024-06-10T00:24:43.259120+0000","last_fullsized":"2024-06-10T00:24:43.259120+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:10:40.081996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.751081+0000","last_change":"2024-06-10T00:23:57.550806+0000","last_active":"2024-06-10T00:24:43.751081+0000","last_peered":"2024-06-10T00:24:43.751081+0000","last_clean":"2024-06-10T00:24:43.751081+0000","last_became_active":"2024-06-10T00:23:57.550453+0000","last_became_peered":"2024-06-10T00:23:57.550453+0000","last_unstale":"2024-06-10T00:24:43.751081+0000","last_undegraded":"2024-06-10T00:24:43.751081+0000","last_fullsized":"2024-06-10T00:24:43.751081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:11:58.765099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257488+0000","last_change":"2024-06-10T00:23:57.567307+0000","last_active":"2024-06-10T00:24:43.257488+0000","last_peered":"2024-06-10T00:24:43.257488+0000","last_clean":"2024-06-10T00:24:43.257488+0000","last_became_active":"2024-06-10T00:23:57.566973+0000","last_became_peered":"2024-06-10T00:23:57.566973+0000","last_unstale":"2024-06-10T00:24:43.257488+0000","last_undegraded":"2024-06-10T00:24:43.257488+0000","last_fullsized":"2024-06-10T00:24:43.257488+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:27:25.135046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254872+0000","last_change":"2024-06-10T00:23:57.547998+0000","last_active":"2024-06-10T00:24:43.254872+0000","last_peered":"2024-06-10T00:24:43.254872+0000","last_clean":"2024-06-10T00:24:43.254872+0000","last_became_active":"2024-06-10T00:23:57.547593+0000","last_became_peered":"2024-06-10T00:23:57.547593+0000","last_unstale":"2024-06-10T00:24:43.254872+0000","last_undegraded":"2024-06-10T00:24:43.254872+0000","last_fullsized":"2024-06-10T00:24:43.254872+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:34:04.219399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254160+0000","last_change":"2024-06-10T00:23:57.560252+0000","last_active":"2024-06-10T00:24:43.254160+0000","last_peered":"2024-06-10T00:24:43.254160+0000","last_clean":"2024-06-10T00:24:43.254160+0000","last_became_active":"2024-06-10T00:23:57.559924+0000","last_became_peered":"2024-06-10T00:23:57.559924+0000","last_unstale":"2024-06-10T00:24:43.254160+0000","last_undegraded":"2024-06-10T00:24:43.254160+0000","last_fullsized":"2024-06-10T00:24:43.254160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:01:43.684262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767541+0000","last_change":"2024-06-10T00:23:57.564031+0000","last_active":"2024-06-10T00:24:43.767541+0000","last_peered":"2024-06-10T00:24:43.767541+0000","last_clean":"2024-06-10T00:24:43.767541+0000","last_became_active":"2024-06-10T00:23:57.563642+0000","last_became_peered":"2024-06-10T00:23:57.563642+0000","last_unstale":"2024-06-10T00:24:43.767541+0000","last_undegraded":"2024-06-10T00:24:43.767541+0000","last_fullsized":"2024-06-10T00:24:43.767541+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:16:23.003152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258104+0000","last_change":"2024-06-10T00:23:57.566877+0000","last_active":"2024-06-10T00:24:43.258104+0000","last_peered":"2024-06-10T00:24:43.258104+0000","last_clean":"2024-06-10T00:24:43.258104+0000","last_became_active":"2024-06-10T00:23:57.566529+0000","last_became_peered":"2024-06-10T00:23:57.566529+0000","last_unstale":"2024-06-10T00:24:43.258104+0000","last_undegraded":"2024-06-10T00:24:43.258104+0000","last_fullsized":"2024-06-10T00:24:43.258104+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:30:43.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255534+0000","last_change":"2024-06-10T00:23:57.564979+0000","last_active":"2024-06-10T00:24:43.255534+0000","last_peered":"2024-06-10T00:24:43.255534+0000","last_clean":"2024-06-10T00:24:43.255534+0000","last_became_active":"2024-06-10T00:23:57.564578+0000","last_became_peered":"2024-06-10T00:23:57.564578+0000","last_unstale":"2024-06-10T00:24:43.255534+0000","last_undegraded":"2024-06-10T00:24:43.255534+0000","last_fullsized":"2024-06-10T00:24:43.255534+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:40:53.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259073+0000","last_change":"2024-06-10T00:23:57.559126+0000","last_active":"2024-06-10T00:24:43.259073+0000","last_peered":"2024-06-10T00:24:43.259073+0000","last_clean":"2024-06-10T00:24:43.259073+0000","last_became_active":"2024-06-10T00:23:57.558401+0000","last_became_peered":"2024-06-10T00:23:57.558401+0000","last_unstale":"2024-06-10T00:24:43.259073+0000","last_undegraded":"2024-06-10T00:24:43.259073+0000","last_fullsized":"2024-06-10T00:24:43.259073+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:57.894137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257666+0000","last_change":"2024-06-10T00:23:57.559272+0000","last_active":"2024-06-10T00:24:43.257666+0000","last_peered":"2024-06-10T00:24:43.257666+0000","last_clean":"2024-06-10T00:24:43.257666+0000","last_became_active":"2024-06-10T00:23:57.558941+0000","last_became_peered":"2024-06-10T00:23:57.558941+0000","last_unstale":"2024-06-10T00:24:43.257666+0000","last_undegraded":"2024-06-10T00:24:43.257666+0000","last_fullsized":"2024-06-10T00:24:43.257666+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:36:48.088347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647320+0000","last_change":"2024-06-10T00:23:57.555875+0000","last_active":"2024-06-10T00:24:43.647320+0000","last_peered":"2024-06-10T00:24:43.647320+0000","last_clean":"2024-06-10T00:24:43.647320+0000","last_became_active":"2024-06-10T00:23:57.555492+0000","last_became_peered":"2024-06-10T00:23:57.555492+0000","last_unstale":"2024-06-10T00:24:43.647320+0000","last_undegraded":"2024-06-10T00:24:43.647320+0000","last_fullsized":"2024-06-10T00:24:43.647320+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:49:10.827458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647295+0000","last_change":"2024-06-10T00:23:57.555316+0000","last_active":"2024-06-10T00:24:43.647295+0000","last_peered":"2024-06-10T00:24:43.647295+0000","last_clean":"2024-06-10T00:24:43.647295+0000","last_became_active":"2024-06-10T00:23:57.554952+0000","last_became_peered":"2024-06-10T00:23:57.554952+0000","last_unstale":"2024-06-10T00:24:43.647295+0000","last_undegraded":"2024-06-10T00:24:43.647295+0000","last_fullsized":"2024-06-10T00:24:43.647295+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:03:57.557330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.766655+0000","last_change":"2024-06-10T00:23:57.562442+0000","last_active":"2024-06-10T00:24:43.766655+0000","last_peered":"2024-06-10T00:24:43.766655+0000","last_clean":"2024-06-10T00:24:43.766655+0000","last_became_active":"2024-06-10T00:23:57.561965+0000","last_became_peered":"2024-06-10T00:23:57.561965+0000","last_unstale":"2024-06-10T00:24:43.766655+0000","last_undegraded":"2024-06-10T00:24:43.766655+0000","last_fullsized":"2024-06-10T00:24:43.766655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:40:16.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259382+0000","last_change":"2024-06-10T00:23:57.564422+0000","last_active":"2024-06-10T00:24:43.259382+0000","last_peered":"2024-06-10T00:24:43.259382+0000","last_clean":"2024-06-10T00:24:43.259382+0000","last_became_active":"2024-06-10T00:23:57.564069+0000","last_became_peered":"2024-06-10T00:23:57.564069+0000","last_unstale":"2024-06-10T00:24:43.259382+0000","last_undegraded":"2024-06-10T00:24:43.259382+0000","last_fullsized":"2024-06-10T00:24:43.259382+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:07.662465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767925+0000","last_change":"2024-06-10T00:23:57.561284+0000","last_active":"2024-06-10T00:24:43.767925+0000","last_peered":"2024-06-10T00:24:43.767925+0000","last_clean":"2024-06-10T00:24:43.767925+0000","last_became_active":"2024-06-10T00:23:57.560933+0000","last_became_peered":"2024-06-10T00:23:57.560933+0000","last_unstale":"2024-06-10T00:24:43.767925+0000","last_undegraded":"2024-06-10T00:24:43.767925+0000","last_fullsized":"2024-06-10T00:24:43.767925+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:58:48.511774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.262098+0000","last_change":"2024-06-10T00:23:57.557752+0000","last_active":"2024-06-10T00:24:43.262098+0000","last_peered":"2024-06-10T00:24:43.262098+0000","last_clean":"2024-06-10T00:24:43.262098+0000","last_became_active":"2024-06-10T00:23:57.557419+0000","last_became_peered":"2024-06-10T00:23:57.557419+0000","last_unstale":"2024-06-10T00:24:43.262098+0000","last_undegraded":"2024-06-10T00:24:43.262098+0000","last_fullsized":"2024-06-10T00:24:43.262098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:48.234181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.261268+0000","last_change":"2024-06-10T00:24:09.177183+0000","last_active":"2024-06-10T00:24:43.261268+0000","last_peered":"2024-06-10T00:24:43.261268+0000","last_clean":"2024-06-10T00:24:43.261268+0000","last_became_active":"2024-06-10T00:24:09.176680+0000","last_became_peered":"2024-06-10T00:24:09.176680+0000","last_unstale":"2024-06-10T00:24:43.261268+0000","last_undegraded":"2024-06-10T00:24:43.261268+0000","last_fullsized":"2024-06-10T00:24:43.261268+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:17:06.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"98'9","reported_seq":50,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.265335+0000","last_change":"2024-06-10T00:23:54.426364+0000","last_active":"2024-06-10T00:25:03.265335+0000","last_peered":"2024-06-10T00:25:03.265335+0000","last_clean":"2024-06-10T00:25:03.265335+0000","last_became_active":"2024-06-10T00:23:54.426054+0000","last_became_peered":"2024-06-10T00:23:54.426054+0000","last_unstale":"2024-06-10T00:25:03.265335+0000","last_undegraded":"2024-06-10T00:25:03.265335+0000","last_fullsized":"2024-06-10T00:25:03.265335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:47.613402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255690+0000","last_change":"2024-06-10T00:23:57.557206+0000","last_active":"2024-06-10T00:24:43.255690+0000","last_peered":"2024-06-10T00:24:43.255690+0000","last_clean":"2024-06-10T00:24:43.255690+0000","last_became_active":"2024-06-10T00:23:57.556853+0000","last_became_peered":"2024-06-10T00:23:57.556853+0000","last_unstale":"2024-06-10T00:24:43.255690+0000","last_undegraded":"2024-06-10T00:24:43.255690+0000","last_fullsized":"2024-06-10T00:24:43.255690+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:09.686334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258363+0000","last_change":"2024-06-10T00:24:09.171047+0000","last_active":"2024-06-10T00:24:43.258363+0000","last_peered":"2024-06-10T00:24:43.258363+0000","last_clean":"2024-06-10T00:24:43.258363+0000","last_became_active":"2024-06-10T00:24:09.170501+0000","last_became_peered":"2024-06-10T00:24:09.170501+0000","last_unstale":"2024-06-10T00:24:43.258363+0000","last_undegraded":"2024-06-10T00:24:43.258363+0000","last_fullsized":"2024-06-10T00:24:43.258363+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:47:27.690439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"98'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255912+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:24:43.255912+0000","last_peered":"2024-06-10T00:24:43.255912+0000","last_clean":"2024-06-10T00:24:43.255912+0000","last_became_active":"2024-06-10T00:23:54.822447+0000","last_became_peered":"2024-06-10T00:23:54.822447+0000","last_unstale":"2024-06-10T00:24:43.255912+0000","last_undegraded":"2024-06-10T00:24:43.255912+0000","last_fullsized":"2024-06-10T00:24:43.255912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:45:14.956540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647840+0000","last_change":"2024-06-10T00:23:57.554958+0000","last_active":"2024-06-10T00:24:43.647840+0000","last_peered":"2024-06-10T00:24:43.647840+0000","last_clean":"2024-06-10T00:24:43.647840+0000","last_became_active":"2024-06-10T00:23:57.554477+0000","last_became_peered":"2024-06-10T00:23:57.554477+0000","last_unstale":"2024-06-10T00:24:43.647840+0000","last_undegraded":"2024-06-10T00:24:43.647840+0000","last_fullsized":"2024-06-10T00:24:43.647840+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:40:06.066967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259093+0000","last_change":"2024-06-10T00:24:09.169642+0000","last_active":"2024-06-10T00:24:43.259093+0000","last_peered":"2024-06-10T00:24:43.259093+0000","last_clean":"2024-06-10T00:24:43.259093+0000","last_became_active":"2024-06-10T00:24:09.168071+0000","last_became_peered":"2024-06-10T00:24:09.168071+0000","last_unstale":"2024-06-10T00:24:43.259093+0000","last_undegraded":"2024-06-10T00:24:43.259093+0000","last_fullsized":"2024-06-10T00:24:43.259093+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:44.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"98'4","reported_seq":44,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.263545+0000","last_change":"2024-06-10T00:23:54.430543+0000","last_active":"2024-06-10T00:25:03.263545+0000","last_peered":"2024-06-10T00:25:03.263545+0000","last_clean":"2024-06-10T00:25:03.263545+0000","last_became_active":"2024-06-10T00:23:54.430138+0000","last_became_peered":"2024-06-10T00:23:54.430138+0000","last_unstale":"2024-06-10T00:25:03.263545+0000","last_undegraded":"2024-06-10T00:25:03.263545+0000","last_fullsized":"2024-06-10T00:25:03.263545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:39:53.212607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767084+0000","last_change":"2024-06-10T00:23:57.562251+0000","last_active":"2024-06-10T00:24:43.767084+0000","last_peered":"2024-06-10T00:24:43.767084+0000","last_clean":"2024-06-10T00:24:43.767084+0000","last_became_active":"2024-06-10T00:23:57.561903+0000","last_became_peered":"2024-06-10T00:23:57.561903+0000","last_unstale":"2024-06-10T00:24:43.767084+0000","last_undegraded":"2024-06-10T00:24:43.767084+0000","last_fullsized":"2024-06-10T00:24:43.767084+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:59.489475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258542+0000","last_change":"2024-06-10T00:24:09.165894+0000","last_active":"2024-06-10T00:24:43.258542+0000","last_peered":"2024-06-10T00:24:43.258542+0000","last_clean":"2024-06-10T00:24:43.258542+0000","last_became_active":"2024-06-10T00:24:09.165435+0000","last_became_peered":"2024-06-10T00:24:09.165435+0000","last_unstale":"2024-06-10T00:24:43.258542+0000","last_undegraded":"2024-06-10T00:24:43.258542+0000","last_fullsized":"2024-06-10T00:24:43.258542+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:13:36.058375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.751350+0000","last_change":"2024-06-10T00:23:54.427868+0000","last_active":"2024-06-10T00:24:43.751350+0000","last_peered":"2024-06-10T00:24:43.751350+0000","last_clean":"2024-06-10T00:24:43.751350+0000","last_became_active":"2024-06-10T00:23:54.427366+0000","last_became_peered":"2024-06-10T00:23:54.427366+0000","last_unstale":"2024-06-10T00:24:43.751350+0000","last_undegraded":"2024-06-10T00:24:43.751350+0000","last_fullsized":"2024-06-10T00:24:43.751350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:15.499292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":40,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.768556+0000","last_change":"2024-06-10T00:24:43.768556+0000","last_active":"2024-06-10T00:24:43.768556+0000","last_peered":"2024-06-10T00:24:43.768556+0000","last_clean":"2024-06-10T00:24:43.768556+0000","last_became_active":"2024-06-10T00:24:43.768090+0000","last_became_peered":"2024-06-10T00:24:43.768090+0000","last_unstale":"2024-06-10T00:24:43.768556+0000","last_undegraded":"2024-06-10T00:24:43.768556+0000","last_fullsized":"2024-06-10T00:24:43.768556+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:05:41.604184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.262093+0000","last_change":"2024-06-10T00:24:09.170514+0000","last_active":"2024-06-10T00:24:43.262093+0000","last_peered":"2024-06-10T00:24:43.262093+0000","last_clean":"2024-06-10T00:24:43.262093+0000","last_became_active":"2024-06-10T00:24:09.168902+0000","last_became_peered":"2024-06-10T00:24:09.168902+0000","last_unstale":"2024-06-10T00:24:43.262093+0000","last_undegraded":"2024-06-10T00:24:43.262093+0000","last_fullsized":"2024-06-10T00:24:43.262093+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:33:10.534657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"98'6","reported_seq":70,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:24.313661+0000","last_change":"2024-06-10T00:23:54.426581+0000","last_active":"2024-06-10T00:25:24.313661+0000","last_peered":"2024-06-10T00:25:24.313661+0000","last_clean":"2024-06-10T00:25:24.313661+0000","last_became_active":"2024-06-10T00:23:54.426211+0000","last_became_peered":"2024-06-10T00:23:54.426211+0000","last_unstale":"2024-06-10T00:25:24.313661+0000","last_undegraded":"2024-06-10T00:25:24.313661+0000","last_fullsized":"2024-06-10T00:25:24.313661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:49:05.360742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.751249+0000","last_change":"2024-06-10T00:23:57.552308+0000","last_active":"2024-06-10T00:24:43.751249+0000","last_peered":"2024-06-10T00:24:43.751249+0000","last_clean":"2024-06-10T00:24:43.751249+0000","last_became_active":"2024-06-10T00:23:57.551762+0000","last_became_peered":"2024-06-10T00:23:57.551762+0000","last_unstale":"2024-06-10T00:24:43.751249+0000","last_undegraded":"2024-06-10T00:24:43.751249+0000","last_fullsized":"2024-06-10T00:24:43.751249+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:22:28.476019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.261639+0000","last_change":"2024-06-10T00:24:09.171456+0000","last_active":"2024-06-10T00:24:43.261639+0000","last_peered":"2024-06-10T00:24:43.261639+0000","last_clean":"2024-06-10T00:24:43.261639+0000","last_became_active":"2024-06-10T00:24:09.170969+0000","last_became_peered":"2024-06-10T00:24:09.170969+0000","last_unstale":"2024-06-10T00:24:43.261639+0000","last_undegraded":"2024-06-10T00:24:43.261639+0000","last_fullsized":"2024-06-10T00:24:43.261639+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:22:46.023897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"98'3","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.264693+0000","last_change":"2024-06-10T00:23:54.428384+0000","last_active":"2024-06-10T00:25:03.264693+0000","last_peered":"2024-06-10T00:25:03.264693+0000","last_clean":"2024-06-10T00:25:03.264693+0000","last_became_active":"2024-06-10T00:23:54.427967+0000","last_became_peered":"2024-06-10T00:23:54.427967+0000","last_unstale":"2024-06-10T00:25:03.264693+0000","last_undegraded":"2024-06-10T00:25:03.264693+0000","last_fullsized":"2024-06-10T00:25:03.264693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:54:30.408684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":40,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.262593+0000","last_change":"2024-06-10T00:24:43.262593+0000","last_active":"2024-06-10T00:24:43.262593+0000","last_peered":"2024-06-10T00:24:43.262593+0000","last_clean":"2024-06-10T00:24:43.262593+0000","last_became_active":"2024-06-10T00:24:43.262212+0000","last_became_peered":"2024-06-10T00:24:43.262212+0000","last_unstale":"2024-06-10T00:24:43.262593+0000","last_undegraded":"2024-06-10T00:24:43.262593+0000","last_fullsized":"2024-06-10T00:24:43.262593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:40:24.442712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":29,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.752805+0000","last_change":"2024-06-10T00:24:43.752805+0000","last_active":"2024-06-10T00:24:43.752805+0000","last_peered":"2024-06-10T00:24:43.752805+0000","last_clean":"2024-06-10T00:24:43.752805+0000","last_became_active":"2024-06-10T00:24:43.752251+0000","last_became_peered":"2024-06-10T00:24:43.752251+0000","last_unstale":"2024-06-10T00:24:43.752805+0000","last_undegraded":"2024-06-10T00:24:43.752805+0000","last_fullsized":"2024-06-10T00:24:43.752805+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:27:57.617443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"98'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.262940+0000","last_change":"2024-06-10T00:23:54.421493+0000","last_active":"2024-06-10T00:25:03.262940+0000","last_peered":"2024-06-10T00:25:03.262940+0000","last_clean":"2024-06-10T00:25:03.262940+0000","last_became_active":"2024-06-10T00:23:54.421151+0000","last_became_peered":"2024-06-10T00:23:54.421151+0000","last_unstale":"2024-06-10T00:25:03.262940+0000","last_undegraded":"2024-06-10T00:25:03.262940+0000","last_fullsized":"2024-06-10T00:25:03.262940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:03.995244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254283+0000","last_change":"2024-06-10T00:23:57.554020+0000","last_active":"2024-06-10T00:24:43.254283+0000","last_peered":"2024-06-10T00:24:43.254283+0000","last_clean":"2024-06-10T00:24:43.254283+0000","last_became_active":"2024-06-10T00:23:57.553690+0000","last_became_peered":"2024-06-10T00:23:57.553690+0000","last_unstale":"2024-06-10T00:24:43.254283+0000","last_undegraded":"2024-06-10T00:24:43.254283+0000","last_fullsized":"2024-06-10T00:24:43.254283+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:48.097074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647854+0000","last_change":"2024-06-10T00:24:09.166426+0000","last_active":"2024-06-10T00:24:43.647854+0000","last_peered":"2024-06-10T00:24:43.647854+0000","last_clean":"2024-06-10T00:24:43.647854+0000","last_became_active":"2024-06-10T00:24:09.166047+0000","last_became_peered":"2024-06-10T00:24:09.166047+0000","last_unstale":"2024-06-10T00:24:43.647854+0000","last_undegraded":"2024-06-10T00:24:43.647854+0000","last_fullsized":"2024-06-10T00:24:43.647854+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:56:11.608061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"98'1","reported_seq":41,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258220+0000","last_change":"2024-06-10T00:23:54.426409+0000","last_active":"2024-06-10T00:24:43.258220+0000","last_peered":"2024-06-10T00:24:43.258220+0000","last_clean":"2024-06-10T00:24:43.258220+0000","last_became_active":"2024-06-10T00:23:54.425385+0000","last_became_peered":"2024-06-10T00:23:54.425385+0000","last_unstale":"2024-06-10T00:24:43.258220+0000","last_undegraded":"2024-06-10T00:24:43.258220+0000","last_fullsized":"2024-06-10T00:24:43.258220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:27:00.379189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767320+0000","last_change":"2024-06-10T00:23:57.560132+0000","last_active":"2024-06-10T00:24:43.767320+0000","last_peered":"2024-06-10T00:24:43.767320+0000","last_clean":"2024-06-10T00:24:43.767320+0000","last_became_active":"2024-06-10T00:23:57.559803+0000","last_became_peered":"2024-06-10T00:23:57.559803+0000","last_unstale":"2024-06-10T00:24:43.767320+0000","last_undegraded":"2024-06-10T00:24:43.767320+0000","last_fullsized":"2024-06-10T00:24:43.767320+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:13:16.004233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254496+0000","last_change":"2024-06-10T00:24:09.170962+0000","last_active":"2024-06-10T00:24:43.254496+0000","last_peered":"2024-06-10T00:24:43.254496+0000","last_clean":"2024-06-10T00:24:43.254496+0000","last_became_active":"2024-06-10T00:24:09.170090+0000","last_became_peered":"2024-06-10T00:24:09.170090+0000","last_unstale":"2024-06-10T00:24:43.254496+0000","last_undegraded":"2024-06-10T00:24:43.254496+0000","last_fullsized":"2024-06-10T00:24:43.254496+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:07:33.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"98'3","reported_seq":67,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:27.317638+0000","last_change":"2024-06-10T00:23:54.421472+0000","last_active":"2024-06-10T00:25:27.317638+0000","last_peered":"2024-06-10T00:25:27.317638+0000","last_clean":"2024-06-10T00:25:27.317638+0000","last_became_active":"2024-06-10T00:23:54.421019+0000","last_became_peered":"2024-06-10T00:23:54.421019+0000","last_unstale":"2024-06-10T00:25:27.317638+0000","last_undegraded":"2024-06-10T00:25:27.317638+0000","last_fullsized":"2024-06-10T00:25:27.317638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:28:18.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.766989+0000","last_change":"2024-06-10T00:23:57.563010+0000","last_active":"2024-06-10T00:24:43.766989+0000","last_peered":"2024-06-10T00:24:43.766989+0000","last_clean":"2024-06-10T00:24:43.766989+0000","last_became_active":"2024-06-10T00:23:57.561986+0000","last_became_peered":"2024-06-10T00:23:57.561986+0000","last_unstale":"2024-06-10T00:24:43.766989+0000","last_undegraded":"2024-06-10T00:24:43.766989+0000","last_fullsized":"2024-06-10T00:24:43.766989+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:31:17.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767249+0000","last_change":"2024-06-10T00:24:09.171746+0000","last_active":"2024-06-10T00:24:43.767249+0000","last_peered":"2024-06-10T00:24:43.767249+0000","last_clean":"2024-06-10T00:24:43.767249+0000","last_became_active":"2024-06-10T00:24:09.171030+0000","last_became_peered":"2024-06-10T00:24:09.171030+0000","last_unstale":"2024-06-10T00:24:43.767249+0000","last_undegraded":"2024-06-10T00:24:43.767249+0000","last_fullsized":"2024-06-10T00:24:43.767249+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:41.669471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"98'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.257212+0000","last_change":"2024-06-10T00:23:54.428917+0000","last_active":"2024-06-10T00:25:02.257212+0000","last_peered":"2024-06-10T00:25:02.257212+0000","last_clean":"2024-06-10T00:25:02.257212+0000","last_became_active":"2024-06-10T00:23:54.428585+0000","last_became_peered":"2024-06-10T00:23:54.428585+0000","last_unstale":"2024-06-10T00:25:02.257212+0000","last_undegraded":"2024-06-10T00:25:02.257212+0000","last_fullsized":"2024-06-10T00:25:02.257212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:21:58.235848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647961+0000","last_change":"2024-06-10T00:23:57.551778+0000","last_active":"2024-06-10T00:24:43.647961+0000","last_peered":"2024-06-10T00:24:43.647961+0000","last_clean":"2024-06-10T00:24:43.647961+0000","last_became_active":"2024-06-10T00:23:57.551436+0000","last_became_peered":"2024-06-10T00:23:57.551436+0000","last_unstale":"2024-06-10T00:24:43.647961+0000","last_undegraded":"2024-06-10T00:24:43.647961+0000","last_fullsized":"2024-06-10T00:24:43.647961+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:30.059783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259187+0000","last_change":"2024-06-10T00:24:09.167547+0000","last_active":"2024-06-10T00:24:43.259187+0000","last_peered":"2024-06-10T00:24:43.259187+0000","last_clean":"2024-06-10T00:24:43.259187+0000","last_became_active":"2024-06-10T00:24:09.167132+0000","last_became_peered":"2024-06-10T00:24:09.167132+0000","last_unstale":"2024-06-10T00:24:43.259187+0000","last_undegraded":"2024-06-10T00:24:43.259187+0000","last_fullsized":"2024-06-10T00:24:43.259187+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:11.520330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"98'4","reported_seq":45,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.257047+0000","last_change":"2024-06-10T00:23:54.823754+0000","last_active":"2024-06-10T00:25:02.257047+0000","last_peered":"2024-06-10T00:25:02.257047+0000","last_clean":"2024-06-10T00:25:02.257047+0000","last_became_active":"2024-06-10T00:23:54.823362+0000","last_became_peered":"2024-06-10T00:23:54.823362+0000","last_unstale":"2024-06-10T00:25:02.257047+0000","last_undegraded":"2024-06-10T00:25:02.257047+0000","last_fullsized":"2024-06-10T00:25:02.257047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:48:50.915439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767273+0000","last_change":"2024-06-10T00:23:57.563313+0000","last_active":"2024-06-10T00:24:43.767273+0000","last_peered":"2024-06-10T00:24:43.767273+0000","last_clean":"2024-06-10T00:24:43.767273+0000","last_became_active":"2024-06-10T00:23:57.562307+0000","last_became_peered":"2024-06-10T00:23:57.562307+0000","last_unstale":"2024-06-10T00:24:43.767273+0000","last_undegraded":"2024-06-10T00:24:43.767273+0000","last_fullsized":"2024-06-10T00:24:43.767273+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:23.815267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767417+0000","last_change":"2024-06-10T00:24:09.174660+0000","last_active":"2024-06-10T00:24:43.767417+0000","last_peered":"2024-06-10T00:24:43.767417+0000","last_clean":"2024-06-10T00:24:43.767417+0000","last_became_active":"2024-06-10T00:24:09.174283+0000","last_became_peered":"2024-06-10T00:24:09.174283+0000","last_unstale":"2024-06-10T00:24:43.767417+0000","last_undegraded":"2024-06-10T00:24:43.767417+0000","last_fullsized":"2024-06-10T00:24:43.767417+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:15.236146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"98'32","reported_seq":128,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:28.319518+0000","last_change":"2024-06-10T00:23:54.426621+0000","last_active":"2024-06-10T00:25:28.319518+0000","last_peered":"2024-06-10T00:25:28.319518+0000","last_clean":"2024-06-10T00:25:28.319518+0000","last_became_active":"2024-06-10T00:23:54.426131+0000","last_became_peered":"2024-06-10T00:23:54.426131+0000","last_unstale":"2024-06-10T00:25:28.319518+0000","last_undegraded":"2024-06-10T00:25:28.319518+0000","last_fullsized":"2024-06-10T00:25:28.319518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:56:14.819454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58390,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":56,"num_read_kb":85,"num_write":33,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.751923+0000","last_change":"2024-06-10T00:23:57.552606+0000","last_active":"2024-06-10T00:24:43.751923+0000","last_peered":"2024-06-10T00:24:43.751923+0000","last_clean":"2024-06-10T00:24:43.751923+0000","last_became_active":"2024-06-10T00:23:57.552246+0000","last_became_peered":"2024-06-10T00:23:57.552246+0000","last_unstale":"2024-06-10T00:24:43.751923+0000","last_undegraded":"2024-06-10T00:24:43.751923+0000","last_fullsized":"2024-06-10T00:24:43.751923+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:19:21.508721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.260848+0000","last_change":"2024-06-10T00:24:09.174862+0000","last_active":"2024-06-10T00:24:43.260848+0000","last_peered":"2024-06-10T00:24:43.260848+0000","last_clean":"2024-06-10T00:24:43.260848+0000","last_became_active":"2024-06-10T00:24:09.173550+0000","last_became_peered":"2024-06-10T00:24:09.173550+0000","last_unstale":"2024-06-10T00:24:43.260848+0000","last_undegraded":"2024-06-10T00:24:43.260848+0000","last_fullsized":"2024-06-10T00:24:43.260848+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:49:25.549084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"98'6","reported_seq":47,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.264764+0000","last_change":"2024-06-10T00:23:54.426074+0000","last_active":"2024-06-10T00:25:03.264764+0000","last_peered":"2024-06-10T00:25:03.264764+0000","last_clean":"2024-06-10T00:25:03.264764+0000","last_became_active":"2024-06-10T00:23:54.425707+0000","last_became_peered":"2024-06-10T00:23:54.425707+0000","last_unstale":"2024-06-10T00:25:03.264764+0000","last_undegraded":"2024-06-10T00:25:03.264764+0000","last_fullsized":"2024-06-10T00:25:03.264764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:11:10.563595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257583+0000","last_change":"2024-06-10T00:23:57.549784+0000","last_active":"2024-06-10T00:24:43.257583+0000","last_peered":"2024-06-10T00:24:43.257583+0000","last_clean":"2024-06-10T00:24:43.257583+0000","last_became_active":"2024-06-10T00:23:57.549457+0000","last_became_peered":"2024-06-10T00:23:57.549457+0000","last_unstale":"2024-06-10T00:24:43.257583+0000","last_undegraded":"2024-06-10T00:24:43.257583+0000","last_fullsized":"2024-06-10T00:24:43.257583+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:59:27.352927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258047+0000","last_change":"2024-06-10T00:24:09.171028+0000","last_active":"2024-06-10T00:24:43.258047+0000","last_peered":"2024-06-10T00:24:43.258047+0000","last_clean":"2024-06-10T00:24:43.258047+0000","last_became_active":"2024-06-10T00:24:09.170543+0000","last_became_peered":"2024-06-10T00:24:09.170543+0000","last_unstale":"2024-06-10T00:24:43.258047+0000","last_undegraded":"2024-06-10T00:24:43.258047+0000","last_fullsized":"2024-06-10T00:24:43.258047+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:57:40.816585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"98'3","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.257814+0000","last_change":"2024-06-10T00:23:54.423332+0000","last_active":"2024-06-10T00:25:02.257814+0000","last_peered":"2024-06-10T00:25:02.257814+0000","last_clean":"2024-06-10T00:25:02.257814+0000","last_became_active":"2024-06-10T00:23:54.422792+0000","last_became_peered":"2024-06-10T00:23:54.422792+0000","last_unstale":"2024-06-10T00:25:02.257814+0000","last_undegraded":"2024-06-10T00:25:02.257814+0000","last_fullsized":"2024-06-10T00:25:02.257814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:50:52.655641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255110+0000","last_change":"2024-06-10T00:23:57.551411+0000","last_active":"2024-06-10T00:24:43.255110+0000","last_peered":"2024-06-10T00:24:43.255110+0000","last_clean":"2024-06-10T00:24:43.255110+0000","last_became_active":"2024-06-10T00:23:57.551053+0000","last_became_peered":"2024-06-10T00:23:57.551053+0000","last_unstale":"2024-06-10T00:24:43.255110+0000","last_undegraded":"2024-06-10T00:24:43.255110+0000","last_fullsized":"2024-06-10T00:24:43.255110+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:16:18.082777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.649288+0000","last_change":"2024-06-10T00:24:43.648767+0000","last_active":"2024-06-10T00:24:43.649288+0000","last_peered":"2024-06-10T00:24:43.649288+0000","last_clean":"2024-06-10T00:24:43.649288+0000","last_became_active":"2024-06-10T00:24:43.648267+0000","last_became_peered":"2024-06-10T00:24:43.648267+0000","last_unstale":"2024-06-10T00:24:43.649288+0000","last_undegraded":"2024-06-10T00:24:43.649288+0000","last_fullsized":"2024-06-10T00:24:43.649288+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:41:09.005202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"98'3","reported_seq":66,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:25.314518+0000","last_change":"2024-06-10T00:23:54.425963+0000","last_active":"2024-06-10T00:25:25.314518+0000","last_peered":"2024-06-10T00:25:25.314518+0000","last_clean":"2024-06-10T00:25:25.314518+0000","last_became_active":"2024-06-10T00:23:54.425581+0000","last_became_peered":"2024-06-10T00:23:54.425581+0000","last_unstale":"2024-06-10T00:25:25.314518+0000","last_undegraded":"2024-06-10T00:25:25.314518+0000","last_fullsized":"2024-06-10T00:25:25.314518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:54.760610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255715+0000","last_change":"2024-06-10T00:23:57.557464+0000","last_active":"2024-06-10T00:24:43.255715+0000","last_peered":"2024-06-10T00:24:43.255715+0000","last_clean":"2024-06-10T00:24:43.255715+0000","last_became_active":"2024-06-10T00:23:57.557130+0000","last_became_peered":"2024-06-10T00:23:57.557130+0000","last_unstale":"2024-06-10T00:24:43.255715+0000","last_undegraded":"2024-06-10T00:24:43.255715+0000","last_fullsized":"2024-06-10T00:24:43.255715+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:16:36.341210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.753291+0000","last_change":"2024-06-10T00:24:43.753291+0000","last_active":"2024-06-10T00:24:43.753291+0000","last_peered":"2024-06-10T00:24:43.753291+0000","last_clean":"2024-06-10T00:24:43.753291+0000","last_became_active":"2024-06-10T00:24:43.752765+0000","last_became_peered":"2024-06-10T00:24:43.752765+0000","last_unstale":"2024-06-10T00:24:43.753291+0000","last_undegraded":"2024-06-10T00:24:43.753291+0000","last_fullsized":"2024-06-10T00:24:43.753291+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:55:06.349564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"98'15","reported_seq":56,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.265792+0000","last_change":"2024-06-10T00:23:54.823489+0000","last_active":"2024-06-10T00:25:03.265792+0000","last_peered":"2024-06-10T00:25:03.265792+0000","last_clean":"2024-06-10T00:25:03.265792+0000","last_became_active":"2024-06-10T00:23:54.823073+0000","last_became_peered":"2024-06-10T00:23:54.823073+0000","last_unstale":"2024-06-10T00:25:03.265792+0000","last_undegraded":"2024-06-10T00:25:03.265792+0000","last_fullsized":"2024-06-10T00:25:03.265792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:07:52.563009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258634+0000","last_change":"2024-06-10T00:23:57.563388+0000","last_active":"2024-06-10T00:24:43.258634+0000","last_peered":"2024-06-10T00:24:43.258634+0000","last_clean":"2024-06-10T00:24:43.258634+0000","last_became_active":"2024-06-10T00:23:57.563004+0000","last_became_peered":"2024-06-10T00:23:57.563004+0000","last_unstale":"2024-06-10T00:24:43.258634+0000","last_undegraded":"2024-06-10T00:24:43.258634+0000","last_fullsized":"2024-06-10T00:24:43.258634+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:51:18.805662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258533+0000","last_change":"2024-06-10T00:24:09.168447+0000","last_active":"2024-06-10T00:24:43.258533+0000","last_peered":"2024-06-10T00:24:43.258533+0000","last_clean":"2024-06-10T00:24:43.258533+0000","last_became_active":"2024-06-10T00:24:09.167439+0000","last_became_peered":"2024-06-10T00:24:09.167439+0000","last_unstale":"2024-06-10T00:24:43.258533+0000","last_undegraded":"2024-06-10T00:24:43.258533+0000","last_fullsized":"2024-06-10T00:24:43.258533+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:52.319395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"98'2","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:05.259903+0000","last_change":"2024-06-10T00:23:54.427666+0000","last_active":"2024-06-10T00:25:05.259903+0000","last_peered":"2024-06-10T00:25:05.259903+0000","last_clean":"2024-06-10T00:25:05.259903+0000","last_became_active":"2024-06-10T00:23:54.427285+0000","last_became_peered":"2024-06-10T00:23:54.427285+0000","last_unstale":"2024-06-10T00:25:05.259903+0000","last_undegraded":"2024-06-10T00:25:05.259903+0000","last_fullsized":"2024-06-10T00:25:05.259903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:28:20.196168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258575+0000","last_change":"2024-06-10T00:23:57.558787+0000","last_active":"2024-06-10T00:24:43.258575+0000","last_peered":"2024-06-10T00:24:43.258575+0000","last_clean":"2024-06-10T00:24:43.258575+0000","last_became_active":"2024-06-10T00:23:57.558470+0000","last_became_peered":"2024-06-10T00:23:57.558470+0000","last_unstale":"2024-06-10T00:24:43.258575+0000","last_undegraded":"2024-06-10T00:24:43.258575+0000","last_fullsized":"2024-06-10T00:24:43.258575+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:02:14.758802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.751230+0000","last_change":"2024-06-10T00:24:09.264952+0000","last_active":"2024-06-10T00:24:43.751230+0000","last_peered":"2024-06-10T00:24:43.751230+0000","last_clean":"2024-06-10T00:24:43.751230+0000","last_became_active":"2024-06-10T00:24:09.264379+0000","last_became_peered":"2024-06-10T00:24:09.264379+0000","last_unstale":"2024-06-10T00:24:43.751230+0000","last_undegraded":"2024-06-10T00:24:43.751230+0000","last_fullsized":"2024-06-10T00:24:43.751230+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:25:40.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767755+0000","last_change":"2024-06-10T00:23:54.823298+0000","last_active":"2024-06-10T00:24:43.767755+0000","last_peered":"2024-06-10T00:24:43.767755+0000","last_clean":"2024-06-10T00:24:43.767755+0000","last_became_active":"2024-06-10T00:23:54.822917+0000","last_became_peered":"2024-06-10T00:23:54.822917+0000","last_unstale":"2024-06-10T00:24:43.767755+0000","last_undegraded":"2024-06-10T00:24:43.767755+0000","last_fullsized":"2024-06-10T00:24:43.767755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:07:52.500641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257683+0000","last_change":"2024-06-10T00:23:57.559202+0000","last_active":"2024-06-10T00:24:43.257683+0000","last_peered":"2024-06-10T00:24:43.257683+0000","last_clean":"2024-06-10T00:24:43.257683+0000","last_became_active":"2024-06-10T00:23:57.558800+0000","last_became_peered":"2024-06-10T00:23:57.558800+0000","last_unstale":"2024-06-10T00:24:43.257683+0000","last_undegraded":"2024-06-10T00:24:43.257683+0000","last_fullsized":"2024-06-10T00:24:43.257683+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:49:51.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257907+0000","last_change":"2024-06-10T00:24:09.168603+0000","last_active":"2024-06-10T00:24:43.257907+0000","last_peered":"2024-06-10T00:24:43.257907+0000","last_clean":"2024-06-10T00:24:43.257907+0000","last_became_active":"2024-06-10T00:24:09.168233+0000","last_became_peered":"2024-06-10T00:24:09.168233+0000","last_unstale":"2024-06-10T00:24:43.257907+0000","last_undegraded":"2024-06-10T00:24:43.257907+0000","last_fullsized":"2024-06-10T00:24:43.257907+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:59.585023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'1","reported_seq":41,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255080+0000","last_change":"2024-06-10T00:23:54.425505+0000","last_active":"2024-06-10T00:24:43.255080+0000","last_peered":"2024-06-10T00:24:43.255080+0000","last_clean":"2024-06-10T00:24:43.255080+0000","last_became_active":"2024-06-10T00:23:54.425058+0000","last_became_peered":"2024-06-10T00:23:54.425058+0000","last_unstale":"2024-06-10T00:24:43.255080+0000","last_undegraded":"2024-06-10T00:24:43.255080+0000","last_fullsized":"2024-06-10T00:24:43.255080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:48:05.605947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257635+0000","last_change":"2024-06-10T00:23:57.566060+0000","last_active":"2024-06-10T00:24:43.257635+0000","last_peered":"2024-06-10T00:24:43.257635+0000","last_clean":"2024-06-10T00:24:43.257635+0000","last_became_active":"2024-06-10T00:23:57.565099+0000","last_became_peered":"2024-06-10T00:23:57.565099+0000","last_unstale":"2024-06-10T00:24:43.257635+0000","last_undegraded":"2024-06-10T00:24:43.257635+0000","last_fullsized":"2024-06-10T00:24:43.257635+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:24:54.204399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258419+0000","last_change":"2024-06-10T00:24:09.264741+0000","last_active":"2024-06-10T00:24:43.258419+0000","last_peered":"2024-06-10T00:24:43.258419+0000","last_clean":"2024-06-10T00:24:43.258419+0000","last_became_active":"2024-06-10T00:24:09.264267+0000","last_became_peered":"2024-06-10T00:24:09.264267+0000","last_unstale":"2024-06-10T00:24:43.258419+0000","last_undegraded":"2024-06-10T00:24:43.258419+0000","last_fullsized":"2024-06-10T00:24:43.258419+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:27.627391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258589+0000","last_change":"2024-06-10T00:23:57.555569+0000","last_active":"2024-06-10T00:24:43.258589+0000","last_peered":"2024-06-10T00:24:43.258589+0000","last_clean":"2024-06-10T00:24:43.258589+0000","last_became_active":"2024-06-10T00:23:57.555265+0000","last_became_peered":"2024-06-10T00:23:57.555265+0000","last_unstale":"2024-06-10T00:24:43.258589+0000","last_undegraded":"2024-06-10T00:24:43.258589+0000","last_fullsized":"2024-06-10T00:24:43.258589+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:28:08.192083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"98'4","reported_seq":67,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:26.315554+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:25:26.315554+0000","last_peered":"2024-06-10T00:25:26.315554+0000","last_clean":"2024-06-10T00:25:26.315554+0000","last_became_active":"2024-06-10T00:23:54.822442+0000","last_became_peered":"2024-06-10T00:23:54.822442+0000","last_unstale":"2024-06-10T00:25:26.315554+0000","last_undegraded":"2024-06-10T00:25:26.315554+0000","last_fullsized":"2024-06-10T00:25:26.315554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:51:10.278257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"98'1","reported_seq":25,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255405+0000","last_change":"2024-06-10T00:24:09.168073+0000","last_active":"2024-06-10T00:24:43.255405+0000","last_peered":"2024-06-10T00:24:43.255405+0000","last_clean":"2024-06-10T00:24:43.255405+0000","last_became_active":"2024-06-10T00:24:09.167657+0000","last_became_peered":"2024-06-10T00:24:09.167657+0000","last_unstale":"2024-06-10T00:24:43.255405+0000","last_undegraded":"2024-06-10T00:24:43.255405+0000","last_fullsized":"2024-06-10T00:24:43.255405+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:20:23.481685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254627+0000","last_change":"2024-06-10T00:23:57.557817+0000","last_active":"2024-06-10T00:24:43.254627+0000","last_peered":"2024-06-10T00:24:43.254627+0000","last_clean":"2024-06-10T00:24:43.254627+0000","last_became_active":"2024-06-10T00:23:57.557155+0000","last_became_peered":"2024-06-10T00:23:57.557155+0000","last_unstale":"2024-06-10T00:24:43.254627+0000","last_undegraded":"2024-06-10T00:24:43.254627+0000","last_fullsized":"2024-06-10T00:24:43.254627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:41:33.913307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"98'3","reported_seq":44,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:05.281291+0000","last_change":"2024-06-10T00:23:54.423507+0000","last_active":"2024-06-10T00:25:05.281291+0000","last_peered":"2024-06-10T00:25:05.281291+0000","last_clean":"2024-06-10T00:25:05.281291+0000","last_became_active":"2024-06-10T00:23:54.423092+0000","last_became_peered":"2024-06-10T00:23:54.423092+0000","last_unstale":"2024-06-10T00:25:05.281291+0000","last_undegraded":"2024-06-10T00:25:05.281291+0000","last_fullsized":"2024-06-10T00:25:05.281291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:06:15.924827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":373,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255089+0000","last_change":"2024-06-10T00:22:28.025243+0000","last_active":"2024-06-10T00:24:43.255089+0000","last_peered":"2024-06-10T00:24:43.255089+0000","last_clean":"2024-06-10T00:24:43.255089+0000","last_became_active":"2024-06-10T00:22:28.024751+0000","last_became_peered":"2024-06-10T00:22:28.024751+0000","last_unstale":"2024-06-10T00:24:43.255089+0000","last_undegraded":"2024-06-10T00:24:43.255089+0000","last_fullsized":"2024-06-10T00:24:43.255089+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_clean_scrub_stamp":"2024-06-10T00:19:05.039631+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:02:36.557931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259322+0000","last_change":"2024-06-10T00:24:09.265084+0000","last_active":"2024-06-10T00:24:43.259322+0000","last_peered":"2024-06-10T00:24:43.259322+0000","last_clean":"2024-06-10T00:24:43.259322+0000","last_became_active":"2024-06-10T00:24:09.264425+0000","last_became_peered":"2024-06-10T00:24:09.264425+0000","last_unstale":"2024-06-10T00:24:43.259322+0000","last_undegraded":"2024-06-10T00:24:43.259322+0000","last_fullsized":"2024-06-10T00:24:43.259322+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:37:31.577688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257290+0000","last_change":"2024-06-10T00:23:57.558540+0000","last_active":"2024-06-10T00:24:43.257290+0000","last_peered":"2024-06-10T00:24:43.257290+0000","last_clean":"2024-06-10T00:24:43.257290+0000","last_became_active":"2024-06-10T00:23:57.558197+0000","last_became_peered":"2024-06-10T00:23:57.558197+0000","last_unstale":"2024-06-10T00:24:43.257290+0000","last_undegraded":"2024-06-10T00:24:43.257290+0000","last_fullsized":"2024-06-10T00:24:43.257290+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:04:25.566135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"98'3","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.264553+0000","last_change":"2024-06-10T00:23:54.428126+0000","last_active":"2024-06-10T00:25:03.264553+0000","last_peered":"2024-06-10T00:25:03.264553+0000","last_clean":"2024-06-10T00:25:03.264553+0000","last_became_active":"2024-06-10T00:23:54.427630+0000","last_became_peered":"2024-06-10T00:23:54.427630+0000","last_unstale":"2024-06-10T00:25:03.264553+0000","last_undegraded":"2024-06-10T00:25:03.264553+0000","last_fullsized":"2024-06-10T00:25:03.264553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:05:23.679704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258656+0000","last_change":"2024-06-10T00:24:09.169984+0000","last_active":"2024-06-10T00:24:43.258656+0000","last_peered":"2024-06-10T00:24:43.258656+0000","last_clean":"2024-06-10T00:24:43.258656+0000","last_became_active":"2024-06-10T00:24:09.169556+0000","last_became_peered":"2024-06-10T00:24:09.169556+0000","last_unstale":"2024-06-10T00:24:43.258656+0000","last_undegraded":"2024-06-10T00:24:43.258656+0000","last_fullsized":"2024-06-10T00:24:43.258656+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:54:13.575207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.255437+0000","last_change":"2024-06-10T00:24:43.767901+0000","last_active":"2024-06-10T00:25:02.255437+0000","last_peered":"2024-06-10T00:25:02.255437+0000","last_clean":"2024-06-10T00:25:02.255437+0000","last_became_active":"2024-06-10T00:24:43.767562+0000","last_became_peered":"2024-06-10T00:24:43.767562+0000","last_unstale":"2024-06-10T00:25:02.255437+0000","last_undegraded":"2024-06-10T00:25:02.255437+0000","last_fullsized":"2024-06-10T00:25:02.255437+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:46:36.099750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258091+0000","last_change":"2024-06-10T00:23:57.554906+0000","last_active":"2024-06-10T00:24:43.258091+0000","last_peered":"2024-06-10T00:24:43.258091+0000","last_clean":"2024-06-10T00:24:43.258091+0000","last_became_active":"2024-06-10T00:23:57.554608+0000","last_became_peered":"2024-06-10T00:23:57.554608+0000","last_unstale":"2024-06-10T00:24:43.258091+0000","last_undegraded":"2024-06-10T00:24:43.258091+0000","last_fullsized":"2024-06-10T00:24:43.258091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:24.690594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254956+0000","last_change":"2024-06-10T00:24:09.264676+0000","last_active":"2024-06-10T00:24:43.254956+0000","last_peered":"2024-06-10T00:24:43.254956+0000","last_clean":"2024-06-10T00:24:43.254956+0000","last_became_active":"2024-06-10T00:24:09.264259+0000","last_became_peered":"2024-06-10T00:24:09.264259+0000","last_unstale":"2024-06-10T00:24:43.254956+0000","last_undegraded":"2024-06-10T00:24:43.254956+0000","last_fullsized":"2024-06-10T00:24:43.254956+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:37:53.787039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"98'12","reported_seq":52,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:25.564411+0000","last_change":"2024-06-10T00:23:54.424201+0000","last_active":"2024-06-10T00:25:25.564411+0000","last_peered":"2024-06-10T00:25:25.564411+0000","last_clean":"2024-06-10T00:25:25.564411+0000","last_became_active":"2024-06-10T00:23:54.423853+0000","last_became_peered":"2024-06-10T00:23:54.423853+0000","last_unstale":"2024-06-10T00:25:25.564411+0000","last_undegraded":"2024-06-10T00:25:25.564411+0000","last_fullsized":"2024-06-10T00:25:25.564411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:00.884041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255024+0000","last_change":"2024-06-10T00:23:57.554276+0000","last_active":"2024-06-10T00:24:43.255024+0000","last_peered":"2024-06-10T00:24:43.255024+0000","last_clean":"2024-06-10T00:24:43.255024+0000","last_became_active":"2024-06-10T00:23:57.553964+0000","last_became_peered":"2024-06-10T00:23:57.553964+0000","last_unstale":"2024-06-10T00:24:43.255024+0000","last_undegraded":"2024-06-10T00:24:43.255024+0000","last_fullsized":"2024-06-10T00:24:43.255024+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:39:51.974770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.261159+0000","last_change":"2024-06-10T00:24:09.170249+0000","last_active":"2024-06-10T00:24:43.261159+0000","last_peered":"2024-06-10T00:24:43.261159+0000","last_clean":"2024-06-10T00:24:43.261159+0000","last_became_active":"2024-06-10T00:24:09.169830+0000","last_became_peered":"2024-06-10T00:24:09.169830+0000","last_unstale":"2024-06-10T00:24:43.261159+0000","last_undegraded":"2024-06-10T00:24:43.261159+0000","last_fullsized":"2024-06-10T00:24:43.261159+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.320609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"98'4","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.255865+0000","last_change":"2024-06-10T00:23:54.473943+0000","last_active":"2024-06-10T00:25:02.255865+0000","last_peered":"2024-06-10T00:25:02.255865+0000","last_clean":"2024-06-10T00:25:02.255865+0000","last_became_active":"2024-06-10T00:23:54.473652+0000","last_became_peered":"2024-06-10T00:23:54.473652+0000","last_unstale":"2024-06-10T00:25:02.255865+0000","last_undegraded":"2024-06-10T00:25:02.255865+0000","last_fullsized":"2024-06-10T00:25:02.255865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:52:06.773684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647984+0000","last_change":"2024-06-10T00:23:57.550799+0000","last_active":"2024-06-10T00:24:43.647984+0000","last_peered":"2024-06-10T00:24:43.647984+0000","last_clean":"2024-06-10T00:24:43.647984+0000","last_became_active":"2024-06-10T00:23:57.549252+0000","last_became_peered":"2024-06-10T00:23:57.549252+0000","last_unstale":"2024-06-10T00:24:43.647984+0000","last_undegraded":"2024-06-10T00:24:43.647984+0000","last_fullsized":"2024-06-10T00:24:43.647984+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:54:54.045006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.768837+0000","last_change":"2024-06-10T00:24:43.768837+0000","last_active":"2024-06-10T00:24:43.768837+0000","last_peered":"2024-06-10T00:24:43.768837+0000","last_clean":"2024-06-10T00:24:43.768837+0000","last_became_active":"2024-06-10T00:24:43.768295+0000","last_became_peered":"2024-06-10T00:24:43.768295+0000","last_unstale":"2024-06-10T00:24:43.768837+0000","last_undegraded":"2024-06-10T00:24:43.768837+0000","last_fullsized":"2024-06-10T00:24:43.768837+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:59.600864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"98'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.255542+0000","last_change":"2024-06-10T00:23:54.823448+0000","last_active":"2024-06-10T00:25:02.255542+0000","last_peered":"2024-06-10T00:25:02.255542+0000","last_clean":"2024-06-10T00:25:02.255542+0000","last_became_active":"2024-06-10T00:23:54.823107+0000","last_became_peered":"2024-06-10T00:23:54.823107+0000","last_unstale":"2024-06-10T00:25:02.255542+0000","last_undegraded":"2024-06-10T00:25:02.255542+0000","last_fullsized":"2024-06-10T00:25:02.255542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:50.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.750987+0000","last_change":"2024-06-10T00:23:57.556733+0000","last_active":"2024-06-10T00:24:43.750987+0000","last_peered":"2024-06-10T00:24:43.750987+0000","last_clean":"2024-06-10T00:24:43.750987+0000","last_became_active":"2024-06-10T00:23:57.556386+0000","last_became_peered":"2024-06-10T00:23:57.556386+0000","last_unstale":"2024-06-10T00:24:43.750987+0000","last_undegraded":"2024-06-10T00:24:43.750987+0000","last_fullsized":"2024-06-10T00:24:43.750987+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:39:00.572311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.768420+0000","last_change":"2024-06-10T00:24:09.174600+0000","last_active":"2024-06-10T00:24:43.768420+0000","last_peered":"2024-06-10T00:24:43.768420+0000","last_clean":"2024-06-10T00:24:43.768420+0000","last_became_active":"2024-06-10T00:24:09.174236+0000","last_became_peered":"2024-06-10T00:24:09.174236+0000","last_unstale":"2024-06-10T00:24:43.768420+0000","last_undegraded":"2024-06-10T00:24:43.768420+0000","last_fullsized":"2024-06-10T00:24:43.768420+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:58:18.447768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"98'3","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.257023+0000","last_change":"2024-06-10T00:23:54.425111+0000","last_active":"2024-06-10T00:25:02.257023+0000","last_peered":"2024-06-10T00:25:02.257023+0000","last_clean":"2024-06-10T00:25:02.257023+0000","last_became_active":"2024-06-10T00:23:54.424724+0000","last_became_peered":"2024-06-10T00:23:54.424724+0000","last_unstale":"2024-06-10T00:25:02.257023+0000","last_undegraded":"2024-06-10T00:25:02.257023+0000","last_fullsized":"2024-06-10T00:25:02.257023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:54.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258868+0000","last_change":"2024-06-10T00:23:57.556355+0000","last_active":"2024-06-10T00:24:43.258868+0000","last_peered":"2024-06-10T00:24:43.258868+0000","last_clean":"2024-06-10T00:24:43.258868+0000","last_became_active":"2024-06-10T00:23:57.555479+0000","last_became_peered":"2024-06-10T00:23:57.555479+0000","last_unstale":"2024-06-10T00:24:43.258868+0000","last_undegraded":"2024-06-10T00:24:43.258868+0000","last_fullsized":"2024-06-10T00:24:43.258868+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:08.817381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"98'1","reported_seq":23,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259000+0000","last_change":"2024-06-10T00:24:09.169589+0000","last_active":"2024-06-10T00:24:43.259000+0000","last_peered":"2024-06-10T00:24:43.259000+0000","last_clean":"2024-06-10T00:24:43.259000+0000","last_became_active":"2024-06-10T00:24:09.168436+0000","last_became_peered":"2024-06-10T00:24:09.168436+0000","last_unstale":"2024-06-10T00:24:43.259000+0000","last_undegraded":"2024-06-10T00:24:43.259000+0000","last_fullsized":"2024-06-10T00:24:43.259000+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:17:32.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"98'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:46.162418+0000","last_change":"2024-06-10T00:24:43.751399+0000","last_active":"2024-06-10T00:24:46.162418+0000","last_peered":"2024-06-10T00:24:46.162418+0000","last_clean":"2024-06-10T00:24:46.162418+0000","last_became_active":"2024-06-10T00:24:43.750987+0000","last_became_peered":"2024-06-10T00:24:43.750987+0000","last_unstale":"2024-06-10T00:24:46.162418+0000","last_undegraded":"2024-06-10T00:24:46.162418+0000","last_fullsized":"2024-06-10T00:24:46.162418+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:14:47.902017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255189+0000","last_change":"2024-06-10T00:23:57.560213+0000","last_active":"2024-06-10T00:24:43.255189+0000","last_peered":"2024-06-10T00:24:43.255189+0000","last_clean":"2024-06-10T00:24:43.255189+0000","last_became_active":"2024-06-10T00:23:57.559925+0000","last_became_peered":"2024-06-10T00:23:57.559925+0000","last_unstale":"2024-06-10T00:24:43.255189+0000","last_undegraded":"2024-06-10T00:24:43.255189+0000","last_fullsized":"2024-06-10T00:24:43.255189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:00:37.403087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258726+0000","last_change":"2024-06-10T00:24:09.264758+0000","last_active":"2024-06-10T00:24:43.258726+0000","last_peered":"2024-06-10T00:24:43.258726+0000","last_clean":"2024-06-10T00:24:43.258726+0000","last_became_active":"2024-06-10T00:24:09.263928+0000","last_became_peered":"2024-06-10T00:24:09.263928+0000","last_unstale":"2024-06-10T00:24:43.258726+0000","last_undegraded":"2024-06-10T00:24:43.258726+0000","last_fullsized":"2024-06-10T00:24:43.258726+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:23:41.083091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"98'4","reported_seq":45,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.258080+0000","last_change":"2024-06-10T00:23:54.428553+0000","last_active":"2024-06-10T00:25:02.258080+0000","last_peered":"2024-06-10T00:25:02.258080+0000","last_clean":"2024-06-10T00:25:02.258080+0000","last_became_active":"2024-06-10T00:23:54.428108+0000","last_became_peered":"2024-06-10T00:23:54.428108+0000","last_unstale":"2024-06-10T00:25:02.258080+0000","last_undegraded":"2024-06-10T00:25:02.258080+0000","last_fullsized":"2024-06-10T00:25:02.258080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:57:26.207915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255060+0000","last_change":"2024-06-10T00:23:57.560686+0000","last_active":"2024-06-10T00:24:43.255060+0000","last_peered":"2024-06-10T00:24:43.255060+0000","last_clean":"2024-06-10T00:24:43.255060+0000","last_became_active":"2024-06-10T00:23:57.560326+0000","last_became_peered":"2024-06-10T00:23:57.560326+0000","last_unstale":"2024-06-10T00:24:43.255060+0000","last_undegraded":"2024-06-10T00:24:43.255060+0000","last_fullsized":"2024-06-10T00:24:43.255060+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:08:39.506567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.766708+0000","last_change":"2024-06-10T00:24:09.166947+0000","last_active":"2024-06-10T00:24:43.766708+0000","last_peered":"2024-06-10T00:24:43.766708+0000","last_clean":"2024-06-10T00:24:43.766708+0000","last_became_active":"2024-06-10T00:24:09.166503+0000","last_became_peered":"2024-06-10T00:24:09.166503+0000","last_unstale":"2024-06-10T00:24:43.766708+0000","last_undegraded":"2024-06-10T00:24:43.766708+0000","last_fullsized":"2024-06-10T00:24:43.766708+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:16:45.094642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"98'3","reported_seq":44,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.265919+0000","last_change":"2024-06-10T00:23:54.823522+0000","last_active":"2024-06-10T00:25:03.265919+0000","last_peered":"2024-06-10T00:25:03.265919+0000","last_clean":"2024-06-10T00:25:03.265919+0000","last_became_active":"2024-06-10T00:23:54.823142+0000","last_became_peered":"2024-06-10T00:23:54.823142+0000","last_unstale":"2024-06-10T00:25:03.265919+0000","last_undegraded":"2024-06-10T00:25:03.265919+0000","last_fullsized":"2024-06-10T00:25:03.265919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:03.836850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255596+0000","last_change":"2024-06-10T00:23:57.563710+0000","last_active":"2024-06-10T00:24:43.255596+0000","last_peered":"2024-06-10T00:24:43.255596+0000","last_clean":"2024-06-10T00:24:43.255596+0000","last_became_active":"2024-06-10T00:23:57.563331+0000","last_became_peered":"2024-06-10T00:23:57.563331+0000","last_unstale":"2024-06-10T00:24:43.255596+0000","last_undegraded":"2024-06-10T00:24:43.255596+0000","last_fullsized":"2024-06-10T00:24:43.255596+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:08:48.061885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647343+0000","last_change":"2024-06-10T00:24:09.171203+0000","last_active":"2024-06-10T00:24:43.647343+0000","last_peered":"2024-06-10T00:24:43.647343+0000","last_clean":"2024-06-10T00:24:43.647343+0000","last_became_active":"2024-06-10T00:24:09.170824+0000","last_became_peered":"2024-06-10T00:24:09.170824+0000","last_unstale":"2024-06-10T00:24:43.647343+0000","last_undegraded":"2024-06-10T00:24:43.647343+0000","last_fullsized":"2024-06-10T00:24:43.647343+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:47:28.045602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.260674+0000","last_change":"2024-06-10T00:23:54.421307+0000","last_active":"2024-06-10T00:24:43.260674+0000","last_peered":"2024-06-10T00:24:43.260674+0000","last_clean":"2024-06-10T00:24:43.260674+0000","last_became_active":"2024-06-10T00:23:54.420928+0000","last_became_peered":"2024-06-10T00:23:54.420928+0000","last_unstale":"2024-06-10T00:24:43.260674+0000","last_undegraded":"2024-06-10T00:24:43.260674+0000","last_fullsized":"2024-06-10T00:24:43.260674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:59:41.433605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255327+0000","last_change":"2024-06-10T00:23:57.556494+0000","last_active":"2024-06-10T00:24:43.255327+0000","last_peered":"2024-06-10T00:24:43.255327+0000","last_clean":"2024-06-10T00:24:43.255327+0000","last_became_active":"2024-06-10T00:23:57.556088+0000","last_became_peered":"2024-06-10T00:23:57.556088+0000","last_unstale":"2024-06-10T00:24:43.255327+0000","last_undegraded":"2024-06-10T00:24:43.255327+0000","last_fullsized":"2024-06-10T00:24:43.255327+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:06:23.873281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257821+0000","last_change":"2024-06-10T00:24:09.166734+0000","last_active":"2024-06-10T00:24:43.257821+0000","last_peered":"2024-06-10T00:24:43.257821+0000","last_clean":"2024-06-10T00:24:43.257821+0000","last_became_active":"2024-06-10T00:24:09.166270+0000","last_became_peered":"2024-06-10T00:24:09.166270+0000","last_unstale":"2024-06-10T00:24:43.257821+0000","last_undegraded":"2024-06-10T00:24:43.257821+0000","last_fullsized":"2024-06-10T00:24:43.257821+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:38.039633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258279+0000","last_change":"2024-06-10T00:23:54.426226+0000","last_active":"2024-06-10T00:24:43.258279+0000","last_peered":"2024-06-10T00:24:43.258279+0000","last_clean":"2024-06-10T00:24:43.258279+0000","last_became_active":"2024-06-10T00:23:54.425879+0000","last_became_peered":"2024-06-10T00:23:54.425879+0000","last_unstale":"2024-06-10T00:24:43.258279+0000","last_undegraded":"2024-06-10T00:24:43.258279+0000","last_fullsized":"2024-06-10T00:24:43.258279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:15:32.936556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255588+0000","last_change":"2024-06-10T00:23:57.553814+0000","last_active":"2024-06-10T00:24:43.255588+0000","last_peered":"2024-06-10T00:24:43.255588+0000","last_clean":"2024-06-10T00:24:43.255588+0000","last_became_active":"2024-06-10T00:23:57.553478+0000","last_became_peered":"2024-06-10T00:23:57.553478+0000","last_unstale":"2024-06-10T00:24:43.255588+0000","last_undegraded":"2024-06-10T00:24:43.255588+0000","last_fullsized":"2024-06-10T00:24:43.255588+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:54:39.264526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258979+0000","last_change":"2024-06-10T00:23:57.556676+0000","last_active":"2024-06-10T00:24:43.258979+0000","last_peered":"2024-06-10T00:24:43.258979+0000","last_clean":"2024-06-10T00:24:43.258979+0000","last_became_active":"2024-06-10T00:23:57.555946+0000","last_became_peered":"2024-06-10T00:23:57.555946+0000","last_unstale":"2024-06-10T00:24:43.258979+0000","last_undegraded":"2024-06-10T00:24:43.258979+0000","last_fullsized":"2024-06-10T00:24:43.258979+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:10:10.868799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.261526+0000","last_change":"2024-06-10T00:23:57.565423+0000","last_active":"2024-06-10T00:24:43.261526+0000","last_peered":"2024-06-10T00:24:43.261526+0000","last_clean":"2024-06-10T00:24:43.261526+0000","last_became_active":"2024-06-10T00:23:57.564830+0000","last_became_peered":"2024-06-10T00:23:57.564830+0000","last_unstale":"2024-06-10T00:24:43.261526+0000","last_undegraded":"2024-06-10T00:24:43.261526+0000","last_fullsized":"2024-06-10T00:24:43.261526+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:11:06.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.261357+0000","last_change":"2024-06-10T00:23:57.558971+0000","last_active":"2024-06-10T00:24:43.261357+0000","last_peered":"2024-06-10T00:24:43.261357+0000","last_clean":"2024-06-10T00:24:43.261357+0000","last_became_active":"2024-06-10T00:23:57.558620+0000","last_became_peered":"2024-06-10T00:23:57.558620+0000","last_unstale":"2024-06-10T00:24:43.261357+0000","last_undegraded":"2024-06-10T00:24:43.261357+0000","last_fullsized":"2024-06-10T00:24:43.261357+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:45:43.308008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647564+0000","last_change":"2024-06-10T00:23:57.552341+0000","last_active":"2024-06-10T00:24:43.647564+0000","last_peered":"2024-06-10T00:24:43.647564+0000","last_clean":"2024-06-10T00:24:43.647564+0000","last_became_active":"2024-06-10T00:23:57.551464+0000","last_became_peered":"2024-06-10T00:23:57.551464+0000","last_unstale":"2024-06-10T00:24:43.647564+0000","last_undegraded":"2024-06-10T00:24:43.647564+0000","last_fullsized":"2024-06-10T00:24:43.647564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:44:41.063516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":40,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.768961+0000","last_change":"2024-06-10T00:24:43.768961+0000","last_active":"2024-06-10T00:24:43.768961+0000","last_peered":"2024-06-10T00:24:43.768961+0000","last_clean":"2024-06-10T00:24:43.768961+0000","last_became_active":"2024-06-10T00:24:43.768542+0000","last_became_peered":"2024-06-10T00:24:43.768542+0000","last_unstale":"2024-06-10T00:24:43.768961+0000","last_undegraded":"2024-06-10T00:24:43.768961+0000","last_fullsized":"2024-06-10T00:24:43.768961+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:23:58.412054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258512+0000","last_change":"2024-06-10T00:23:57.554924+0000","last_active":"2024-06-10T00:24:43.258512+0000","last_peered":"2024-06-10T00:24:43.258512+0000","last_clean":"2024-06-10T00:24:43.258512+0000","last_became_active":"2024-06-10T00:23:57.554515+0000","last_became_peered":"2024-06-10T00:23:57.554515+0000","last_unstale":"2024-06-10T00:24:43.258512+0000","last_undegraded":"2024-06-10T00:24:43.258512+0000","last_fullsized":"2024-06-10T00:24:43.258512+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:58:15.624775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.262115+0000","last_change":"2024-06-10T00:23:57.561940+0000","last_active":"2024-06-10T00:24:43.262115+0000","last_peered":"2024-06-10T00:24:43.262115+0000","last_clean":"2024-06-10T00:24:43.262115+0000","last_became_active":"2024-06-10T00:23:57.561512+0000","last_became_peered":"2024-06-10T00:23:57.561512+0000","last_unstale":"2024-06-10T00:24:43.262115+0000","last_undegraded":"2024-06-10T00:24:43.262115+0000","last_fullsized":"2024-06-10T00:24:43.262115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:57:57.147311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255250+0000","last_change":"2024-06-10T00:23:57.557229+0000","last_active":"2024-06-10T00:24:43.255250+0000","last_peered":"2024-06-10T00:24:43.255250+0000","last_clean":"2024-06-10T00:24:43.255250+0000","last_became_active":"2024-06-10T00:23:57.556905+0000","last_became_peered":"2024-06-10T00:23:57.556905+0000","last_unstale":"2024-06-10T00:24:43.255250+0000","last_undegraded":"2024-06-10T00:24:43.255250+0000","last_fullsized":"2024-06-10T00:24:43.255250+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:30.513517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.768509+0000","last_change":"2024-06-10T00:24:43.768509+0000","last_active":"2024-06-10T00:24:43.768509+0000","last_peered":"2024-06-10T00:24:43.768509+0000","last_clean":"2024-06-10T00:24:43.768509+0000","last_became_active":"2024-06-10T00:24:43.768038+0000","last_became_peered":"2024-06-10T00:24:43.768038+0000","last_unstale":"2024-06-10T00:24:43.768509+0000","last_undegraded":"2024-06-10T00:24:43.768509+0000","last_fullsized":"2024-06-10T00:24:43.768509+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:38.344375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254595+0000","last_change":"2024-06-10T00:23:57.557830+0000","last_active":"2024-06-10T00:24:43.254595+0000","last_peered":"2024-06-10T00:24:43.254595+0000","last_clean":"2024-06-10T00:24:43.254595+0000","last_became_active":"2024-06-10T00:23:57.556986+0000","last_became_peered":"2024-06-10T00:23:57.556986+0000","last_unstale":"2024-06-10T00:24:43.254595+0000","last_undegraded":"2024-06-10T00:24:43.254595+0000","last_fullsized":"2024-06-10T00:24:43.254595+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:53.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.766973+0000","last_change":"2024-06-10T00:23:57.565602+0000","last_active":"2024-06-10T00:24:43.766973+0000","last_peered":"2024-06-10T00:24:43.766973+0000","last_clean":"2024-06-10T00:24:43.766973+0000","last_became_active":"2024-06-10T00:23:57.565264+0000","last_became_peered":"2024-06-10T00:23:57.565264+0000","last_unstale":"2024-06-10T00:24:43.766973+0000","last_undegraded":"2024-06-10T00:24:43.766973+0000","last_fullsized":"2024-06-10T00:24:43.766973+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:09:12.773087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257833+0000","last_change":"2024-06-10T00:23:57.548516+0000","last_active":"2024-06-10T00:24:43.257833+0000","last_peered":"2024-06-10T00:24:43.257833+0000","last_clean":"2024-06-10T00:24:43.257833+0000","last_became_active":"2024-06-10T00:23:57.548202+0000","last_became_peered":"2024-06-10T00:23:57.548202+0000","last_unstale":"2024-06-10T00:24:43.257833+0000","last_undegraded":"2024-06-10T00:24:43.257833+0000","last_fullsized":"2024-06-10T00:24:43.257833+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:13.901478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258257+0000","last_change":"2024-06-10T00:23:57.565900+0000","last_active":"2024-06-10T00:24:43.258257+0000","last_peered":"2024-06-10T00:24:43.258257+0000","last_clean":"2024-06-10T00:24:43.258257+0000","last_became_active":"2024-06-10T00:23:57.565573+0000","last_became_peered":"2024-06-10T00:23:57.565573+0000","last_unstale":"2024-06-10T00:24:43.258257+0000","last_undegraded":"2024-06-10T00:24:43.258257+0000","last_fullsized":"2024-06-10T00:24:43.258257+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:08.255905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280680,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62390,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":165,"num_read_kb":151,"num_write":168,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":181767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":554064,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612647,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32368,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712784,"statfs":{"total":95995035648,"available":95961890816,"internally_reserved":0,"allocated":1220608,"data_stored":1199700,"data_compressed":62071,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776173,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":558374,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037809,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":554176,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299446,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1150106,"data_compressed":62071,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462971,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":549454,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":602351,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":550842,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":609494,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":549648,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888133,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":554104,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1150976,"data_stored":1145694,"data_compressed":62071,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":52987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":60130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":59968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-10T00:25:32.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-10T00:25:32.977 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:33.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:33 smithi121 ceph-mon[32378]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:25:33.355 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:33 smithi121 ceph-mon[32378]: pgmap v353: 129 pgs: 129 active+clean; 638 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-06-10T00:25:33.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:33 smithi135 ceph-mon[36424]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:25:33.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:33 smithi135 ceph-mon[36424]: pgmap v353: 129 pgs: 129 active+clean; 638 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-06-10T00:25:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:33 smithi193 ceph-mon[36716]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:25:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:33 smithi193 ceph-mon[36716]: pgmap v353: 129 pgs: 129 active+clean; 638 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-06-10T00:25:34.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.794+0000 7f178c8cf700 1 -- 172.21.15.121:0/1764252569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1788107840 msgr2=0x7f178810f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:34.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.794+0000 7f178c8cf700 1 --2- 172.21.15.121:0/1764252569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1788107840 0x7f178810f3c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f178000b0d0 tx=0x7f178000b3e0 comp rx=0 tx=0).stop 2024-06-10T00:25:34.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.794+0000 7f178c8cf700 1 -- 172.21.15.121:0/1764252569 shutdown_connections 2024-06-10T00:25:34.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.794+0000 7f178c8cf700 1 --2- 172.21.15.121:0/1764252569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1788107840 0x7f178810f3c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:34.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.794+0000 7f178c8cf700 1 --2- 172.21.15.121:0/1764252569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1788102740 0x7f1788102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:34.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.794+0000 7f178c8cf700 1 --2- 172.21.15.121:0/1764252569 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f17881030f0 0x7f1788107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:34.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.794+0000 7f178c8cf700 1 -- 172.21.15.121:0/1764252569 >> 172.21.15.121:0/1764252569 conn(0x7f17880795f0 msgr2=0x7f1788079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:34.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.794+0000 7f178c8cf700 1 -- 172.21.15.121:0/1764252569 shutdown_connections 2024-06-10T00:25:34.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.794+0000 7f178c8cf700 1 -- 172.21.15.121:0/1764252569 wait complete. 2024-06-10T00:25:34.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.795+0000 7f178c8cf700 1 Processor -- start 2024-06-10T00:25:34.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.795+0000 7f178c8cf700 1 -- start start 2024-06-10T00:25:34.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.795+0000 7f178c8cf700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1788102740 0x7f178819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:34.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.795+0000 7f178c8cf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17881030f0 0x7f178819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:34.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.795+0000 7f178c8cf700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1788107840 0x7f17881988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:34.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.795+0000 7f178c8cf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1788112780 con 0x7f17881030f0 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.795+0000 7f178c8cf700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f1788112600 con 0x7f1788102740 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.795+0000 7f178c8cf700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1788112900 con 0x7f1788107840 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.795+0000 7f1785d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17881030f0 0x7f178819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f1785d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17881030f0 0x7f178819ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37952/0 (socket says 172.21.15.121:37952) 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f1785d9b700 1 -- 172.21.15.121:0/3218176610 learned_addr learned my addr 172.21.15.121:0/3218176610 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f1786d9d700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1788107840 0x7f17881988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f178659c700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1788102740 0x7f178819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f1785d9b700 1 -- 172.21.15.121:0/3218176610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1788107840 msgr2=0x7f17881988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f1785d9b700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1788107840 0x7f17881988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f1785d9b700 1 -- 172.21.15.121:0/3218176610 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1788102740 msgr2=0x7f178819e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f1785d9b700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1788102740 0x7f178819e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:34.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f1785d9b700 1 -- 172.21.15.121:0/3218176610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1788199020 con 0x7f17881030f0 2024-06-10T00:25:34.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f178659c700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1788102740 0x7f178819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:25:34.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f1785d9b700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17881030f0 0x7f178819ec40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f177800c9b0 tx=0x7f177800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:34.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f17777fe700 1 -- 172.21.15.121:0/3218176610 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778007b00 con 0x7f17881030f0 2024-06-10T00:25:34.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f178c8cf700 1 -- 172.21.15.121:0/3218176610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1788199310 con 0x7f17881030f0 2024-06-10T00:25:34.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f17777fe700 1 -- 172.21.15.121:0/3218176610 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1778007ca0 con 0x7f17881030f0 2024-06-10T00:25:34.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f17777fe700 1 -- 172.21.15.121:0/3218176610 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778018730 con 0x7f17881030f0 2024-06-10T00:25:34.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.796+0000 7f178c8cf700 1 -- 172.21.15.121:0/3218176610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17880746c0 con 0x7f17881030f0 2024-06-10T00:25:34.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.797+0000 7f178c8cf700 1 -- 172.21.15.121:0/3218176610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1788069180 con 0x7f17881030f0 2024-06-10T00:25:34.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.798+0000 7f17777fe700 1 -- 172.21.15.121:0/3218176610 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f17780188d0 con 0x7f17881030f0 2024-06-10T00:25:34.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.799+0000 7f17777fe700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f17700a2cb0 0x7f17700a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:34.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.799+0000 7f178659c700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f17700a2cb0 0x7f17700a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:34.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.799+0000 7f17777fe700 1 -- 172.21.15.121:0/3218176610 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f17780cf790 con 0x7f17881030f0 2024-06-10T00:25:34.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.800+0000 7f178659c700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f17700a2cb0 0x7f17700a5170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1788199a00 tx=0x7f177c00b2c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:34.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.803+0000 7f17777fe700 1 -- 172.21.15.121:0/3218176610 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1778014030 con 0x7f17881030f0 2024-06-10T00:25:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:34.993+0000 7f178c8cf700 1 -- 172.21.15.121:0/3218176610 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1788100b60 con 0x7f17700a2cb0 2024-06-10T00:25:35.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.003+0000 7f17777fe700 1 -- 172.21.15.121:0/3218176610 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340645 (secure 0 0 0) 0x7f1788100b60 con 0x7f17700a2cb0 2024-06-10T00:25:35.007 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:25:35.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 -- 172.21.15.121:0/3218176610 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f17700a2cb0 msgr2=0x7f17700a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:35.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f17700a2cb0 0x7f17700a5170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1788199a00 tx=0x7f177c00b2c0 comp rx=0 tx=0).stop 2024-06-10T00:25:35.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 -- 172.21.15.121:0/3218176610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17881030f0 msgr2=0x7f178819ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:35.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17881030f0 0x7f178819ec40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f177800c9b0 tx=0x7f177800ccc0 comp rx=0 tx=0).stop 2024-06-10T00:25:35.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 -- 172.21.15.121:0/3218176610 shutdown_connections 2024-06-10T00:25:35.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f17700a2cb0 0x7f17700a5170 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:35.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1788107840 0x7f17881988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:35.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1788102740 0x7f178819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:35.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 --2- 172.21.15.121:0/3218176610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17881030f0 0x7f178819ec40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:35.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 -- 172.21.15.121:0/3218176610 >> 172.21.15.121:0/3218176610 conn(0x7f17880795f0 msgr2=0x7f1788100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:35.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 -- 172.21.15.121:0/3218176610 shutdown_connections 2024-06-10T00:25:35.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:35.006+0000 7f178c8cf700 1 -- 172.21.15.121:0/3218176610 wait complete. 2024-06-10T00:25:35.010 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-06-10T00:25:35.318 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:35 smithi121 ceph-mon[32378]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-06-10T00:25:35.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:35 smithi135 ceph-mon[36424]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-06-10T00:25:35.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:35 smithi193 ceph-mon[36716]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.3 KiB/s wr, 8 op/s 2024-06-10T00:25:35.680 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-06-10T00:25:34.119091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657257,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":332,"num_read_kb":290,"num_write":352,"num_write_kb":3058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":283,"ondisk_log_size":283,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358112,"kb_used_data":13120,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583712,"statfs":{"total":1151940427776,"available":1151573721088,"internally_reserved":0,"allocated":13434880,"data_stored":8591940,"data_compressed":423552,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32312,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":84,"num_read_kb":79,"num_write":23,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003109"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259557+0000","last_change":"2024-06-10T00:23:57.564073+0000","last_active":"2024-06-10T00:24:43.259557+0000","last_peered":"2024-06-10T00:24:43.259557+0000","last_clean":"2024-06-10T00:24:43.259557+0000","last_became_active":"2024-06-10T00:23:57.563618+0000","last_became_peered":"2024-06-10T00:23:57.563618+0000","last_unstale":"2024-06-10T00:24:43.259557+0000","last_undegraded":"2024-06-10T00:24:43.259557+0000","last_fullsized":"2024-06-10T00:24:43.259557+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:26:25.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647887+0000","last_change":"2024-06-10T00:23:57.560148+0000","last_active":"2024-06-10T00:24:43.647887+0000","last_peered":"2024-06-10T00:24:43.647887+0000","last_clean":"2024-06-10T00:24:43.647887+0000","last_became_active":"2024-06-10T00:23:57.559794+0000","last_became_peered":"2024-06-10T00:23:57.559794+0000","last_unstale":"2024-06-10T00:24:43.647887+0000","last_undegraded":"2024-06-10T00:24:43.647887+0000","last_fullsized":"2024-06-10T00:24:43.647887+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:59:50.624542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647613+0000","last_change":"2024-06-10T00:23:57.563292+0000","last_active":"2024-06-10T00:24:43.647613+0000","last_peered":"2024-06-10T00:24:43.647613+0000","last_clean":"2024-06-10T00:24:43.647613+0000","last_became_active":"2024-06-10T00:23:57.563007+0000","last_became_peered":"2024-06-10T00:23:57.563007+0000","last_unstale":"2024-06-10T00:24:43.647613+0000","last_undegraded":"2024-06-10T00:24:43.647613+0000","last_fullsized":"2024-06-10T00:24:43.647613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:27:44.897770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259120+0000","last_change":"2024-06-10T00:23:57.558789+0000","last_active":"2024-06-10T00:24:43.259120+0000","last_peered":"2024-06-10T00:24:43.259120+0000","last_clean":"2024-06-10T00:24:43.259120+0000","last_became_active":"2024-06-10T00:23:57.557943+0000","last_became_peered":"2024-06-10T00:23:57.557943+0000","last_unstale":"2024-06-10T00:24:43.259120+0000","last_undegraded":"2024-06-10T00:24:43.259120+0000","last_fullsized":"2024-06-10T00:24:43.259120+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:10:40.081996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.751081+0000","last_change":"2024-06-10T00:23:57.550806+0000","last_active":"2024-06-10T00:24:43.751081+0000","last_peered":"2024-06-10T00:24:43.751081+0000","last_clean":"2024-06-10T00:24:43.751081+0000","last_became_active":"2024-06-10T00:23:57.550453+0000","last_became_peered":"2024-06-10T00:23:57.550453+0000","last_unstale":"2024-06-10T00:24:43.751081+0000","last_undegraded":"2024-06-10T00:24:43.751081+0000","last_fullsized":"2024-06-10T00:24:43.751081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:11:58.765099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257488+0000","last_change":"2024-06-10T00:23:57.567307+0000","last_active":"2024-06-10T00:24:43.257488+0000","last_peered":"2024-06-10T00:24:43.257488+0000","last_clean":"2024-06-10T00:24:43.257488+0000","last_became_active":"2024-06-10T00:23:57.566973+0000","last_became_peered":"2024-06-10T00:23:57.566973+0000","last_unstale":"2024-06-10T00:24:43.257488+0000","last_undegraded":"2024-06-10T00:24:43.257488+0000","last_fullsized":"2024-06-10T00:24:43.257488+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:27:25.135046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254872+0000","last_change":"2024-06-10T00:23:57.547998+0000","last_active":"2024-06-10T00:24:43.254872+0000","last_peered":"2024-06-10T00:24:43.254872+0000","last_clean":"2024-06-10T00:24:43.254872+0000","last_became_active":"2024-06-10T00:23:57.547593+0000","last_became_peered":"2024-06-10T00:23:57.547593+0000","last_unstale":"2024-06-10T00:24:43.254872+0000","last_undegraded":"2024-06-10T00:24:43.254872+0000","last_fullsized":"2024-06-10T00:24:43.254872+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:34:04.219399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254160+0000","last_change":"2024-06-10T00:23:57.560252+0000","last_active":"2024-06-10T00:24:43.254160+0000","last_peered":"2024-06-10T00:24:43.254160+0000","last_clean":"2024-06-10T00:24:43.254160+0000","last_became_active":"2024-06-10T00:23:57.559924+0000","last_became_peered":"2024-06-10T00:23:57.559924+0000","last_unstale":"2024-06-10T00:24:43.254160+0000","last_undegraded":"2024-06-10T00:24:43.254160+0000","last_fullsized":"2024-06-10T00:24:43.254160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:01:43.684262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767541+0000","last_change":"2024-06-10T00:23:57.564031+0000","last_active":"2024-06-10T00:24:43.767541+0000","last_peered":"2024-06-10T00:24:43.767541+0000","last_clean":"2024-06-10T00:24:43.767541+0000","last_became_active":"2024-06-10T00:23:57.563642+0000","last_became_peered":"2024-06-10T00:23:57.563642+0000","last_unstale":"2024-06-10T00:24:43.767541+0000","last_undegraded":"2024-06-10T00:24:43.767541+0000","last_fullsized":"2024-06-10T00:24:43.767541+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:16:23.003152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258104+0000","last_change":"2024-06-10T00:23:57.566877+0000","last_active":"2024-06-10T00:24:43.258104+0000","last_peered":"2024-06-10T00:24:43.258104+0000","last_clean":"2024-06-10T00:24:43.258104+0000","last_became_active":"2024-06-10T00:23:57.566529+0000","last_became_peered":"2024-06-10T00:23:57.566529+0000","last_unstale":"2024-06-10T00:24:43.258104+0000","last_undegraded":"2024-06-10T00:24:43.258104+0000","last_fullsized":"2024-06-10T00:24:43.258104+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:30:43.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255534+0000","last_change":"2024-06-10T00:23:57.564979+0000","last_active":"2024-06-10T00:24:43.255534+0000","last_peered":"2024-06-10T00:24:43.255534+0000","last_clean":"2024-06-10T00:24:43.255534+0000","last_became_active":"2024-06-10T00:23:57.564578+0000","last_became_peered":"2024-06-10T00:23:57.564578+0000","last_unstale":"2024-06-10T00:24:43.255534+0000","last_undegraded":"2024-06-10T00:24:43.255534+0000","last_fullsized":"2024-06-10T00:24:43.255534+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:40:53.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259073+0000","last_change":"2024-06-10T00:23:57.559126+0000","last_active":"2024-06-10T00:24:43.259073+0000","last_peered":"2024-06-10T00:24:43.259073+0000","last_clean":"2024-06-10T00:24:43.259073+0000","last_became_active":"2024-06-10T00:23:57.558401+0000","last_became_peered":"2024-06-10T00:23:57.558401+0000","last_unstale":"2024-06-10T00:24:43.259073+0000","last_undegraded":"2024-06-10T00:24:43.259073+0000","last_fullsized":"2024-06-10T00:24:43.259073+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:57.894137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257666+0000","last_change":"2024-06-10T00:23:57.559272+0000","last_active":"2024-06-10T00:24:43.257666+0000","last_peered":"2024-06-10T00:24:43.257666+0000","last_clean":"2024-06-10T00:24:43.257666+0000","last_became_active":"2024-06-10T00:23:57.558941+0000","last_became_peered":"2024-06-10T00:23:57.558941+0000","last_unstale":"2024-06-10T00:24:43.257666+0000","last_undegraded":"2024-06-10T00:24:43.257666+0000","last_fullsized":"2024-06-10T00:24:43.257666+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:36:48.088347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647320+0000","last_change":"2024-06-10T00:23:57.555875+0000","last_active":"2024-06-10T00:24:43.647320+0000","last_peered":"2024-06-10T00:24:43.647320+0000","last_clean":"2024-06-10T00:24:43.647320+0000","last_became_active":"2024-06-10T00:23:57.555492+0000","last_became_peered":"2024-06-10T00:23:57.555492+0000","last_unstale":"2024-06-10T00:24:43.647320+0000","last_undegraded":"2024-06-10T00:24:43.647320+0000","last_fullsized":"2024-06-10T00:24:43.647320+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:49:10.827458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647295+0000","last_change":"2024-06-10T00:23:57.555316+0000","last_active":"2024-06-10T00:24:43.647295+0000","last_peered":"2024-06-10T00:24:43.647295+0000","last_clean":"2024-06-10T00:24:43.647295+0000","last_became_active":"2024-06-10T00:23:57.554952+0000","last_became_peered":"2024-06-10T00:23:57.554952+0000","last_unstale":"2024-06-10T00:24:43.647295+0000","last_undegraded":"2024-06-10T00:24:43.647295+0000","last_fullsized":"2024-06-10T00:24:43.647295+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:03:57.557330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.766655+0000","last_change":"2024-06-10T00:23:57.562442+0000","last_active":"2024-06-10T00:24:43.766655+0000","last_peered":"2024-06-10T00:24:43.766655+0000","last_clean":"2024-06-10T00:24:43.766655+0000","last_became_active":"2024-06-10T00:23:57.561965+0000","last_became_peered":"2024-06-10T00:23:57.561965+0000","last_unstale":"2024-06-10T00:24:43.766655+0000","last_undegraded":"2024-06-10T00:24:43.766655+0000","last_fullsized":"2024-06-10T00:24:43.766655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:40:16.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259382+0000","last_change":"2024-06-10T00:23:57.564422+0000","last_active":"2024-06-10T00:24:43.259382+0000","last_peered":"2024-06-10T00:24:43.259382+0000","last_clean":"2024-06-10T00:24:43.259382+0000","last_became_active":"2024-06-10T00:23:57.564069+0000","last_became_peered":"2024-06-10T00:23:57.564069+0000","last_unstale":"2024-06-10T00:24:43.259382+0000","last_undegraded":"2024-06-10T00:24:43.259382+0000","last_fullsized":"2024-06-10T00:24:43.259382+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:07.662465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767925+0000","last_change":"2024-06-10T00:23:57.561284+0000","last_active":"2024-06-10T00:24:43.767925+0000","last_peered":"2024-06-10T00:24:43.767925+0000","last_clean":"2024-06-10T00:24:43.767925+0000","last_became_active":"2024-06-10T00:23:57.560933+0000","last_became_peered":"2024-06-10T00:23:57.560933+0000","last_unstale":"2024-06-10T00:24:43.767925+0000","last_undegraded":"2024-06-10T00:24:43.767925+0000","last_fullsized":"2024-06-10T00:24:43.767925+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:58:48.511774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.262098+0000","last_change":"2024-06-10T00:23:57.557752+0000","last_active":"2024-06-10T00:24:43.262098+0000","last_peered":"2024-06-10T00:24:43.262098+0000","last_clean":"2024-06-10T00:24:43.262098+0000","last_became_active":"2024-06-10T00:23:57.557419+0000","last_became_peered":"2024-06-10T00:23:57.557419+0000","last_unstale":"2024-06-10T00:24:43.262098+0000","last_undegraded":"2024-06-10T00:24:43.262098+0000","last_fullsized":"2024-06-10T00:24:43.262098+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:48.234181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.261268+0000","last_change":"2024-06-10T00:24:09.177183+0000","last_active":"2024-06-10T00:24:43.261268+0000","last_peered":"2024-06-10T00:24:43.261268+0000","last_clean":"2024-06-10T00:24:43.261268+0000","last_became_active":"2024-06-10T00:24:09.176680+0000","last_became_peered":"2024-06-10T00:24:09.176680+0000","last_unstale":"2024-06-10T00:24:43.261268+0000","last_undegraded":"2024-06-10T00:24:43.261268+0000","last_fullsized":"2024-06-10T00:24:43.261268+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:17:06.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"98'9","reported_seq":50,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.265335+0000","last_change":"2024-06-10T00:23:54.426364+0000","last_active":"2024-06-10T00:25:03.265335+0000","last_peered":"2024-06-10T00:25:03.265335+0000","last_clean":"2024-06-10T00:25:03.265335+0000","last_became_active":"2024-06-10T00:23:54.426054+0000","last_became_peered":"2024-06-10T00:23:54.426054+0000","last_unstale":"2024-06-10T00:25:03.265335+0000","last_undegraded":"2024-06-10T00:25:03.265335+0000","last_fullsized":"2024-06-10T00:25:03.265335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:47.613402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255690+0000","last_change":"2024-06-10T00:23:57.557206+0000","last_active":"2024-06-10T00:24:43.255690+0000","last_peered":"2024-06-10T00:24:43.255690+0000","last_clean":"2024-06-10T00:24:43.255690+0000","last_became_active":"2024-06-10T00:23:57.556853+0000","last_became_peered":"2024-06-10T00:23:57.556853+0000","last_unstale":"2024-06-10T00:24:43.255690+0000","last_undegraded":"2024-06-10T00:24:43.255690+0000","last_fullsized":"2024-06-10T00:24:43.255690+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:09.686334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258363+0000","last_change":"2024-06-10T00:24:09.171047+0000","last_active":"2024-06-10T00:24:43.258363+0000","last_peered":"2024-06-10T00:24:43.258363+0000","last_clean":"2024-06-10T00:24:43.258363+0000","last_became_active":"2024-06-10T00:24:09.170501+0000","last_became_peered":"2024-06-10T00:24:09.170501+0000","last_unstale":"2024-06-10T00:24:43.258363+0000","last_undegraded":"2024-06-10T00:24:43.258363+0000","last_fullsized":"2024-06-10T00:24:43.258363+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:47:27.690439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"98'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255912+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:24:43.255912+0000","last_peered":"2024-06-10T00:24:43.255912+0000","last_clean":"2024-06-10T00:24:43.255912+0000","last_became_active":"2024-06-10T00:23:54.822447+0000","last_became_peered":"2024-06-10T00:23:54.822447+0000","last_unstale":"2024-06-10T00:24:43.255912+0000","last_undegraded":"2024-06-10T00:24:43.255912+0000","last_fullsized":"2024-06-10T00:24:43.255912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:45:14.956540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647840+0000","last_change":"2024-06-10T00:23:57.554958+0000","last_active":"2024-06-10T00:24:43.647840+0000","last_peered":"2024-06-10T00:24:43.647840+0000","last_clean":"2024-06-10T00:24:43.647840+0000","last_became_active":"2024-06-10T00:23:57.554477+0000","last_became_peered":"2024-06-10T00:23:57.554477+0000","last_unstale":"2024-06-10T00:24:43.647840+0000","last_undegraded":"2024-06-10T00:24:43.647840+0000","last_fullsized":"2024-06-10T00:24:43.647840+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:40:06.066967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259093+0000","last_change":"2024-06-10T00:24:09.169642+0000","last_active":"2024-06-10T00:24:43.259093+0000","last_peered":"2024-06-10T00:24:43.259093+0000","last_clean":"2024-06-10T00:24:43.259093+0000","last_became_active":"2024-06-10T00:24:09.168071+0000","last_became_peered":"2024-06-10T00:24:09.168071+0000","last_unstale":"2024-06-10T00:24:43.259093+0000","last_undegraded":"2024-06-10T00:24:43.259093+0000","last_fullsized":"2024-06-10T00:24:43.259093+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:44.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"98'4","reported_seq":44,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.263545+0000","last_change":"2024-06-10T00:23:54.430543+0000","last_active":"2024-06-10T00:25:03.263545+0000","last_peered":"2024-06-10T00:25:03.263545+0000","last_clean":"2024-06-10T00:25:03.263545+0000","last_became_active":"2024-06-10T00:23:54.430138+0000","last_became_peered":"2024-06-10T00:23:54.430138+0000","last_unstale":"2024-06-10T00:25:03.263545+0000","last_undegraded":"2024-06-10T00:25:03.263545+0000","last_fullsized":"2024-06-10T00:25:03.263545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:39:53.212607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767084+0000","last_change":"2024-06-10T00:23:57.562251+0000","last_active":"2024-06-10T00:24:43.767084+0000","last_peered":"2024-06-10T00:24:43.767084+0000","last_clean":"2024-06-10T00:24:43.767084+0000","last_became_active":"2024-06-10T00:23:57.561903+0000","last_became_peered":"2024-06-10T00:23:57.561903+0000","last_unstale":"2024-06-10T00:24:43.767084+0000","last_undegraded":"2024-06-10T00:24:43.767084+0000","last_fullsized":"2024-06-10T00:24:43.767084+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:59.489475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258542+0000","last_change":"2024-06-10T00:24:09.165894+0000","last_active":"2024-06-10T00:24:43.258542+0000","last_peered":"2024-06-10T00:24:43.258542+0000","last_clean":"2024-06-10T00:24:43.258542+0000","last_became_active":"2024-06-10T00:24:09.165435+0000","last_became_peered":"2024-06-10T00:24:09.165435+0000","last_unstale":"2024-06-10T00:24:43.258542+0000","last_undegraded":"2024-06-10T00:24:43.258542+0000","last_fullsized":"2024-06-10T00:24:43.258542+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:13:36.058375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.751350+0000","last_change":"2024-06-10T00:23:54.427868+0000","last_active":"2024-06-10T00:24:43.751350+0000","last_peered":"2024-06-10T00:24:43.751350+0000","last_clean":"2024-06-10T00:24:43.751350+0000","last_became_active":"2024-06-10T00:23:54.427366+0000","last_became_peered":"2024-06-10T00:23:54.427366+0000","last_unstale":"2024-06-10T00:24:43.751350+0000","last_undegraded":"2024-06-10T00:24:43.751350+0000","last_fullsized":"2024-06-10T00:24:43.751350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:15.499292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":40,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.768556+0000","last_change":"2024-06-10T00:24:43.768556+0000","last_active":"2024-06-10T00:24:43.768556+0000","last_peered":"2024-06-10T00:24:43.768556+0000","last_clean":"2024-06-10T00:24:43.768556+0000","last_became_active":"2024-06-10T00:24:43.768090+0000","last_became_peered":"2024-06-10T00:24:43.768090+0000","last_unstale":"2024-06-10T00:24:43.768556+0000","last_undegraded":"2024-06-10T00:24:43.768556+0000","last_fullsized":"2024-06-10T00:24:43.768556+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:05:41.604184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.262093+0000","last_change":"2024-06-10T00:24:09.170514+0000","last_active":"2024-06-10T00:24:43.262093+0000","last_peered":"2024-06-10T00:24:43.262093+0000","last_clean":"2024-06-10T00:24:43.262093+0000","last_became_active":"2024-06-10T00:24:09.168902+0000","last_became_peered":"2024-06-10T00:24:09.168902+0000","last_unstale":"2024-06-10T00:24:43.262093+0000","last_undegraded":"2024-06-10T00:24:43.262093+0000","last_fullsized":"2024-06-10T00:24:43.262093+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:33:10.534657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"98'6","reported_seq":75,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:29.321294+0000","last_change":"2024-06-10T00:23:54.426581+0000","last_active":"2024-06-10T00:25:29.321294+0000","last_peered":"2024-06-10T00:25:29.321294+0000","last_clean":"2024-06-10T00:25:29.321294+0000","last_became_active":"2024-06-10T00:23:54.426211+0000","last_became_peered":"2024-06-10T00:23:54.426211+0000","last_unstale":"2024-06-10T00:25:29.321294+0000","last_undegraded":"2024-06-10T00:25:29.321294+0000","last_fullsized":"2024-06-10T00:25:29.321294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:49:05.360742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.751249+0000","last_change":"2024-06-10T00:23:57.552308+0000","last_active":"2024-06-10T00:24:43.751249+0000","last_peered":"2024-06-10T00:24:43.751249+0000","last_clean":"2024-06-10T00:24:43.751249+0000","last_became_active":"2024-06-10T00:23:57.551762+0000","last_became_peered":"2024-06-10T00:23:57.551762+0000","last_unstale":"2024-06-10T00:24:43.751249+0000","last_undegraded":"2024-06-10T00:24:43.751249+0000","last_fullsized":"2024-06-10T00:24:43.751249+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:22:28.476019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.261639+0000","last_change":"2024-06-10T00:24:09.171456+0000","last_active":"2024-06-10T00:24:43.261639+0000","last_peered":"2024-06-10T00:24:43.261639+0000","last_clean":"2024-06-10T00:24:43.261639+0000","last_became_active":"2024-06-10T00:24:09.170969+0000","last_became_peered":"2024-06-10T00:24:09.170969+0000","last_unstale":"2024-06-10T00:24:43.261639+0000","last_undegraded":"2024-06-10T00:24:43.261639+0000","last_fullsized":"2024-06-10T00:24:43.261639+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:22:46.023897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"98'3","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.264693+0000","last_change":"2024-06-10T00:23:54.428384+0000","last_active":"2024-06-10T00:25:03.264693+0000","last_peered":"2024-06-10T00:25:03.264693+0000","last_clean":"2024-06-10T00:25:03.264693+0000","last_became_active":"2024-06-10T00:23:54.427967+0000","last_became_peered":"2024-06-10T00:23:54.427967+0000","last_unstale":"2024-06-10T00:25:03.264693+0000","last_undegraded":"2024-06-10T00:25:03.264693+0000","last_fullsized":"2024-06-10T00:25:03.264693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:54:30.408684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":40,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.262593+0000","last_change":"2024-06-10T00:24:43.262593+0000","last_active":"2024-06-10T00:24:43.262593+0000","last_peered":"2024-06-10T00:24:43.262593+0000","last_clean":"2024-06-10T00:24:43.262593+0000","last_became_active":"2024-06-10T00:24:43.262212+0000","last_became_peered":"2024-06-10T00:24:43.262212+0000","last_unstale":"2024-06-10T00:24:43.262593+0000","last_undegraded":"2024-06-10T00:24:43.262593+0000","last_fullsized":"2024-06-10T00:24:43.262593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:40:24.442712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":29,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.752805+0000","last_change":"2024-06-10T00:24:43.752805+0000","last_active":"2024-06-10T00:24:43.752805+0000","last_peered":"2024-06-10T00:24:43.752805+0000","last_clean":"2024-06-10T00:24:43.752805+0000","last_became_active":"2024-06-10T00:24:43.752251+0000","last_became_peered":"2024-06-10T00:24:43.752251+0000","last_unstale":"2024-06-10T00:24:43.752805+0000","last_undegraded":"2024-06-10T00:24:43.752805+0000","last_fullsized":"2024-06-10T00:24:43.752805+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:27:57.617443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"98'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.262940+0000","last_change":"2024-06-10T00:23:54.421493+0000","last_active":"2024-06-10T00:25:03.262940+0000","last_peered":"2024-06-10T00:25:03.262940+0000","last_clean":"2024-06-10T00:25:03.262940+0000","last_became_active":"2024-06-10T00:23:54.421151+0000","last_became_peered":"2024-06-10T00:23:54.421151+0000","last_unstale":"2024-06-10T00:25:03.262940+0000","last_undegraded":"2024-06-10T00:25:03.262940+0000","last_fullsized":"2024-06-10T00:25:03.262940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:03.995244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254283+0000","last_change":"2024-06-10T00:23:57.554020+0000","last_active":"2024-06-10T00:24:43.254283+0000","last_peered":"2024-06-10T00:24:43.254283+0000","last_clean":"2024-06-10T00:24:43.254283+0000","last_became_active":"2024-06-10T00:23:57.553690+0000","last_became_peered":"2024-06-10T00:23:57.553690+0000","last_unstale":"2024-06-10T00:24:43.254283+0000","last_undegraded":"2024-06-10T00:24:43.254283+0000","last_fullsized":"2024-06-10T00:24:43.254283+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:48.097074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647854+0000","last_change":"2024-06-10T00:24:09.166426+0000","last_active":"2024-06-10T00:24:43.647854+0000","last_peered":"2024-06-10T00:24:43.647854+0000","last_clean":"2024-06-10T00:24:43.647854+0000","last_became_active":"2024-06-10T00:24:09.166047+0000","last_became_peered":"2024-06-10T00:24:09.166047+0000","last_unstale":"2024-06-10T00:24:43.647854+0000","last_undegraded":"2024-06-10T00:24:43.647854+0000","last_fullsized":"2024-06-10T00:24:43.647854+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:56:11.608061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"98'1","reported_seq":41,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258220+0000","last_change":"2024-06-10T00:23:54.426409+0000","last_active":"2024-06-10T00:24:43.258220+0000","last_peered":"2024-06-10T00:24:43.258220+0000","last_clean":"2024-06-10T00:24:43.258220+0000","last_became_active":"2024-06-10T00:23:54.425385+0000","last_became_peered":"2024-06-10T00:23:54.425385+0000","last_unstale":"2024-06-10T00:24:43.258220+0000","last_undegraded":"2024-06-10T00:24:43.258220+0000","last_fullsized":"2024-06-10T00:24:43.258220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:27:00.379189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767320+0000","last_change":"2024-06-10T00:23:57.560132+0000","last_active":"2024-06-10T00:24:43.767320+0000","last_peered":"2024-06-10T00:24:43.767320+0000","last_clean":"2024-06-10T00:24:43.767320+0000","last_became_active":"2024-06-10T00:23:57.559803+0000","last_became_peered":"2024-06-10T00:23:57.559803+0000","last_unstale":"2024-06-10T00:24:43.767320+0000","last_undegraded":"2024-06-10T00:24:43.767320+0000","last_fullsized":"2024-06-10T00:24:43.767320+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:13:16.004233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254496+0000","last_change":"2024-06-10T00:24:09.170962+0000","last_active":"2024-06-10T00:24:43.254496+0000","last_peered":"2024-06-10T00:24:43.254496+0000","last_clean":"2024-06-10T00:24:43.254496+0000","last_became_active":"2024-06-10T00:24:09.170090+0000","last_became_peered":"2024-06-10T00:24:09.170090+0000","last_unstale":"2024-06-10T00:24:43.254496+0000","last_undegraded":"2024-06-10T00:24:43.254496+0000","last_fullsized":"2024-06-10T00:24:43.254496+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:07:33.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"98'3","reported_seq":72,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:32.325125+0000","last_change":"2024-06-10T00:23:54.421472+0000","last_active":"2024-06-10T00:25:32.325125+0000","last_peered":"2024-06-10T00:25:32.325125+0000","last_clean":"2024-06-10T00:25:32.325125+0000","last_became_active":"2024-06-10T00:23:54.421019+0000","last_became_peered":"2024-06-10T00:23:54.421019+0000","last_unstale":"2024-06-10T00:25:32.325125+0000","last_undegraded":"2024-06-10T00:25:32.325125+0000","last_fullsized":"2024-06-10T00:25:32.325125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:28:18.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.766989+0000","last_change":"2024-06-10T00:23:57.563010+0000","last_active":"2024-06-10T00:24:43.766989+0000","last_peered":"2024-06-10T00:24:43.766989+0000","last_clean":"2024-06-10T00:24:43.766989+0000","last_became_active":"2024-06-10T00:23:57.561986+0000","last_became_peered":"2024-06-10T00:23:57.561986+0000","last_unstale":"2024-06-10T00:24:43.766989+0000","last_undegraded":"2024-06-10T00:24:43.766989+0000","last_fullsized":"2024-06-10T00:24:43.766989+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:31:17.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767249+0000","last_change":"2024-06-10T00:24:09.171746+0000","last_active":"2024-06-10T00:24:43.767249+0000","last_peered":"2024-06-10T00:24:43.767249+0000","last_clean":"2024-06-10T00:24:43.767249+0000","last_became_active":"2024-06-10T00:24:09.171030+0000","last_became_peered":"2024-06-10T00:24:09.171030+0000","last_unstale":"2024-06-10T00:24:43.767249+0000","last_undegraded":"2024-06-10T00:24:43.767249+0000","last_fullsized":"2024-06-10T00:24:43.767249+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:41.669471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"98'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.257212+0000","last_change":"2024-06-10T00:23:54.428917+0000","last_active":"2024-06-10T00:25:02.257212+0000","last_peered":"2024-06-10T00:25:02.257212+0000","last_clean":"2024-06-10T00:25:02.257212+0000","last_became_active":"2024-06-10T00:23:54.428585+0000","last_became_peered":"2024-06-10T00:23:54.428585+0000","last_unstale":"2024-06-10T00:25:02.257212+0000","last_undegraded":"2024-06-10T00:25:02.257212+0000","last_fullsized":"2024-06-10T00:25:02.257212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:21:58.235848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647961+0000","last_change":"2024-06-10T00:23:57.551778+0000","last_active":"2024-06-10T00:24:43.647961+0000","last_peered":"2024-06-10T00:24:43.647961+0000","last_clean":"2024-06-10T00:24:43.647961+0000","last_became_active":"2024-06-10T00:23:57.551436+0000","last_became_peered":"2024-06-10T00:23:57.551436+0000","last_unstale":"2024-06-10T00:24:43.647961+0000","last_undegraded":"2024-06-10T00:24:43.647961+0000","last_fullsized":"2024-06-10T00:24:43.647961+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:30.059783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259187+0000","last_change":"2024-06-10T00:24:09.167547+0000","last_active":"2024-06-10T00:24:43.259187+0000","last_peered":"2024-06-10T00:24:43.259187+0000","last_clean":"2024-06-10T00:24:43.259187+0000","last_became_active":"2024-06-10T00:24:09.167132+0000","last_became_peered":"2024-06-10T00:24:09.167132+0000","last_unstale":"2024-06-10T00:24:43.259187+0000","last_undegraded":"2024-06-10T00:24:43.259187+0000","last_fullsized":"2024-06-10T00:24:43.259187+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:11.520330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"98'4","reported_seq":45,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.257047+0000","last_change":"2024-06-10T00:23:54.823754+0000","last_active":"2024-06-10T00:25:02.257047+0000","last_peered":"2024-06-10T00:25:02.257047+0000","last_clean":"2024-06-10T00:25:02.257047+0000","last_became_active":"2024-06-10T00:23:54.823362+0000","last_became_peered":"2024-06-10T00:23:54.823362+0000","last_unstale":"2024-06-10T00:25:02.257047+0000","last_undegraded":"2024-06-10T00:25:02.257047+0000","last_fullsized":"2024-06-10T00:25:02.257047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:48:50.915439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767273+0000","last_change":"2024-06-10T00:23:57.563313+0000","last_active":"2024-06-10T00:24:43.767273+0000","last_peered":"2024-06-10T00:24:43.767273+0000","last_clean":"2024-06-10T00:24:43.767273+0000","last_became_active":"2024-06-10T00:23:57.562307+0000","last_became_peered":"2024-06-10T00:23:57.562307+0000","last_unstale":"2024-06-10T00:24:43.767273+0000","last_undegraded":"2024-06-10T00:24:43.767273+0000","last_fullsized":"2024-06-10T00:24:43.767273+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:23.815267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767417+0000","last_change":"2024-06-10T00:24:09.174660+0000","last_active":"2024-06-10T00:24:43.767417+0000","last_peered":"2024-06-10T00:24:43.767417+0000","last_clean":"2024-06-10T00:24:43.767417+0000","last_became_active":"2024-06-10T00:24:09.174283+0000","last_became_peered":"2024-06-10T00:24:09.174283+0000","last_unstale":"2024-06-10T00:24:43.767417+0000","last_undegraded":"2024-06-10T00:24:43.767417+0000","last_fullsized":"2024-06-10T00:24:43.767417+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:15.236146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"98'33","reported_seq":140,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:33.327049+0000","last_change":"2024-06-10T00:23:54.426621+0000","last_active":"2024-06-10T00:25:33.327049+0000","last_peered":"2024-06-10T00:25:33.327049+0000","last_clean":"2024-06-10T00:25:33.327049+0000","last_became_active":"2024-06-10T00:23:54.426131+0000","last_became_peered":"2024-06-10T00:23:54.426131+0000","last_unstale":"2024-06-10T00:25:33.327049+0000","last_undegraded":"2024-06-10T00:25:33.327049+0000","last_fullsized":"2024-06-10T00:25:33.327049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:56:14.819454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62655,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":67,"num_read_kb":95,"num_write":34,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.751923+0000","last_change":"2024-06-10T00:23:57.552606+0000","last_active":"2024-06-10T00:24:43.751923+0000","last_peered":"2024-06-10T00:24:43.751923+0000","last_clean":"2024-06-10T00:24:43.751923+0000","last_became_active":"2024-06-10T00:23:57.552246+0000","last_became_peered":"2024-06-10T00:23:57.552246+0000","last_unstale":"2024-06-10T00:24:43.751923+0000","last_undegraded":"2024-06-10T00:24:43.751923+0000","last_fullsized":"2024-06-10T00:24:43.751923+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:19:21.508721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.260848+0000","last_change":"2024-06-10T00:24:09.174862+0000","last_active":"2024-06-10T00:24:43.260848+0000","last_peered":"2024-06-10T00:24:43.260848+0000","last_clean":"2024-06-10T00:24:43.260848+0000","last_became_active":"2024-06-10T00:24:09.173550+0000","last_became_peered":"2024-06-10T00:24:09.173550+0000","last_unstale":"2024-06-10T00:24:43.260848+0000","last_undegraded":"2024-06-10T00:24:43.260848+0000","last_fullsized":"2024-06-10T00:24:43.260848+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:49:25.549084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"98'6","reported_seq":47,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.264764+0000","last_change":"2024-06-10T00:23:54.426074+0000","last_active":"2024-06-10T00:25:03.264764+0000","last_peered":"2024-06-10T00:25:03.264764+0000","last_clean":"2024-06-10T00:25:03.264764+0000","last_became_active":"2024-06-10T00:23:54.425707+0000","last_became_peered":"2024-06-10T00:23:54.425707+0000","last_unstale":"2024-06-10T00:25:03.264764+0000","last_undegraded":"2024-06-10T00:25:03.264764+0000","last_fullsized":"2024-06-10T00:25:03.264764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:11:10.563595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257583+0000","last_change":"2024-06-10T00:23:57.549784+0000","last_active":"2024-06-10T00:24:43.257583+0000","last_peered":"2024-06-10T00:24:43.257583+0000","last_clean":"2024-06-10T00:24:43.257583+0000","last_became_active":"2024-06-10T00:23:57.549457+0000","last_became_peered":"2024-06-10T00:23:57.549457+0000","last_unstale":"2024-06-10T00:24:43.257583+0000","last_undegraded":"2024-06-10T00:24:43.257583+0000","last_fullsized":"2024-06-10T00:24:43.257583+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:59:27.352927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258047+0000","last_change":"2024-06-10T00:24:09.171028+0000","last_active":"2024-06-10T00:24:43.258047+0000","last_peered":"2024-06-10T00:24:43.258047+0000","last_clean":"2024-06-10T00:24:43.258047+0000","last_became_active":"2024-06-10T00:24:09.170543+0000","last_became_peered":"2024-06-10T00:24:09.170543+0000","last_unstale":"2024-06-10T00:24:43.258047+0000","last_undegraded":"2024-06-10T00:24:43.258047+0000","last_fullsized":"2024-06-10T00:24:43.258047+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:57:40.816585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"98'3","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.257814+0000","last_change":"2024-06-10T00:23:54.423332+0000","last_active":"2024-06-10T00:25:02.257814+0000","last_peered":"2024-06-10T00:25:02.257814+0000","last_clean":"2024-06-10T00:25:02.257814+0000","last_became_active":"2024-06-10T00:23:54.422792+0000","last_became_peered":"2024-06-10T00:23:54.422792+0000","last_unstale":"2024-06-10T00:25:02.257814+0000","last_undegraded":"2024-06-10T00:25:02.257814+0000","last_fullsized":"2024-06-10T00:25:02.257814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:50:52.655641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255110+0000","last_change":"2024-06-10T00:23:57.551411+0000","last_active":"2024-06-10T00:24:43.255110+0000","last_peered":"2024-06-10T00:24:43.255110+0000","last_clean":"2024-06-10T00:24:43.255110+0000","last_became_active":"2024-06-10T00:23:57.551053+0000","last_became_peered":"2024-06-10T00:23:57.551053+0000","last_unstale":"2024-06-10T00:24:43.255110+0000","last_undegraded":"2024-06-10T00:24:43.255110+0000","last_fullsized":"2024-06-10T00:24:43.255110+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:16:18.082777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.649288+0000","last_change":"2024-06-10T00:24:43.648767+0000","last_active":"2024-06-10T00:24:43.649288+0000","last_peered":"2024-06-10T00:24:43.649288+0000","last_clean":"2024-06-10T00:24:43.649288+0000","last_became_active":"2024-06-10T00:24:43.648267+0000","last_became_peered":"2024-06-10T00:24:43.648267+0000","last_unstale":"2024-06-10T00:24:43.649288+0000","last_undegraded":"2024-06-10T00:24:43.649288+0000","last_fullsized":"2024-06-10T00:24:43.649288+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:41:09.005202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"98'3","reported_seq":71,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:30.322194+0000","last_change":"2024-06-10T00:23:54.425963+0000","last_active":"2024-06-10T00:25:30.322194+0000","last_peered":"2024-06-10T00:25:30.322194+0000","last_clean":"2024-06-10T00:25:30.322194+0000","last_became_active":"2024-06-10T00:23:54.425581+0000","last_became_peered":"2024-06-10T00:23:54.425581+0000","last_unstale":"2024-06-10T00:25:30.322194+0000","last_undegraded":"2024-06-10T00:25:30.322194+0000","last_fullsized":"2024-06-10T00:25:30.322194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:54.760610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255715+0000","last_change":"2024-06-10T00:23:57.557464+0000","last_active":"2024-06-10T00:24:43.255715+0000","last_peered":"2024-06-10T00:24:43.255715+0000","last_clean":"2024-06-10T00:24:43.255715+0000","last_became_active":"2024-06-10T00:23:57.557130+0000","last_became_peered":"2024-06-10T00:23:57.557130+0000","last_unstale":"2024-06-10T00:24:43.255715+0000","last_undegraded":"2024-06-10T00:24:43.255715+0000","last_fullsized":"2024-06-10T00:24:43.255715+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:16:36.341210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.753291+0000","last_change":"2024-06-10T00:24:43.753291+0000","last_active":"2024-06-10T00:24:43.753291+0000","last_peered":"2024-06-10T00:24:43.753291+0000","last_clean":"2024-06-10T00:24:43.753291+0000","last_became_active":"2024-06-10T00:24:43.752765+0000","last_became_peered":"2024-06-10T00:24:43.752765+0000","last_unstale":"2024-06-10T00:24:43.753291+0000","last_undegraded":"2024-06-10T00:24:43.753291+0000","last_fullsized":"2024-06-10T00:24:43.753291+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:55:06.349564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"98'15","reported_seq":56,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.265792+0000","last_change":"2024-06-10T00:23:54.823489+0000","last_active":"2024-06-10T00:25:03.265792+0000","last_peered":"2024-06-10T00:25:03.265792+0000","last_clean":"2024-06-10T00:25:03.265792+0000","last_became_active":"2024-06-10T00:23:54.823073+0000","last_became_peered":"2024-06-10T00:23:54.823073+0000","last_unstale":"2024-06-10T00:25:03.265792+0000","last_undegraded":"2024-06-10T00:25:03.265792+0000","last_fullsized":"2024-06-10T00:25:03.265792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:07:52.563009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258634+0000","last_change":"2024-06-10T00:23:57.563388+0000","last_active":"2024-06-10T00:24:43.258634+0000","last_peered":"2024-06-10T00:24:43.258634+0000","last_clean":"2024-06-10T00:24:43.258634+0000","last_became_active":"2024-06-10T00:23:57.563004+0000","last_became_peered":"2024-06-10T00:23:57.563004+0000","last_unstale":"2024-06-10T00:24:43.258634+0000","last_undegraded":"2024-06-10T00:24:43.258634+0000","last_fullsized":"2024-06-10T00:24:43.258634+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:51:18.805662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258533+0000","last_change":"2024-06-10T00:24:09.168447+0000","last_active":"2024-06-10T00:24:43.258533+0000","last_peered":"2024-06-10T00:24:43.258533+0000","last_clean":"2024-06-10T00:24:43.258533+0000","last_became_active":"2024-06-10T00:24:09.167439+0000","last_became_peered":"2024-06-10T00:24:09.167439+0000","last_unstale":"2024-06-10T00:24:43.258533+0000","last_undegraded":"2024-06-10T00:24:43.258533+0000","last_fullsized":"2024-06-10T00:24:43.258533+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:52.319395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"98'2","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:05.259903+0000","last_change":"2024-06-10T00:23:54.427666+0000","last_active":"2024-06-10T00:25:05.259903+0000","last_peered":"2024-06-10T00:25:05.259903+0000","last_clean":"2024-06-10T00:25:05.259903+0000","last_became_active":"2024-06-10T00:23:54.427285+0000","last_became_peered":"2024-06-10T00:23:54.427285+0000","last_unstale":"2024-06-10T00:25:05.259903+0000","last_undegraded":"2024-06-10T00:25:05.259903+0000","last_fullsized":"2024-06-10T00:25:05.259903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:28:20.196168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258575+0000","last_change":"2024-06-10T00:23:57.558787+0000","last_active":"2024-06-10T00:24:43.258575+0000","last_peered":"2024-06-10T00:24:43.258575+0000","last_clean":"2024-06-10T00:24:43.258575+0000","last_became_active":"2024-06-10T00:23:57.558470+0000","last_became_peered":"2024-06-10T00:23:57.558470+0000","last_unstale":"2024-06-10T00:24:43.258575+0000","last_undegraded":"2024-06-10T00:24:43.258575+0000","last_fullsized":"2024-06-10T00:24:43.258575+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:02:14.758802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.751230+0000","last_change":"2024-06-10T00:24:09.264952+0000","last_active":"2024-06-10T00:24:43.751230+0000","last_peered":"2024-06-10T00:24:43.751230+0000","last_clean":"2024-06-10T00:24:43.751230+0000","last_became_active":"2024-06-10T00:24:09.264379+0000","last_became_peered":"2024-06-10T00:24:09.264379+0000","last_unstale":"2024-06-10T00:24:43.751230+0000","last_undegraded":"2024-06-10T00:24:43.751230+0000","last_fullsized":"2024-06-10T00:24:43.751230+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:25:40.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":40,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.767755+0000","last_change":"2024-06-10T00:23:54.823298+0000","last_active":"2024-06-10T00:24:43.767755+0000","last_peered":"2024-06-10T00:24:43.767755+0000","last_clean":"2024-06-10T00:24:43.767755+0000","last_became_active":"2024-06-10T00:23:54.822917+0000","last_became_peered":"2024-06-10T00:23:54.822917+0000","last_unstale":"2024-06-10T00:24:43.767755+0000","last_undegraded":"2024-06-10T00:24:43.767755+0000","last_fullsized":"2024-06-10T00:24:43.767755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:07:52.500641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257683+0000","last_change":"2024-06-10T00:23:57.559202+0000","last_active":"2024-06-10T00:24:43.257683+0000","last_peered":"2024-06-10T00:24:43.257683+0000","last_clean":"2024-06-10T00:24:43.257683+0000","last_became_active":"2024-06-10T00:23:57.558800+0000","last_became_peered":"2024-06-10T00:23:57.558800+0000","last_unstale":"2024-06-10T00:24:43.257683+0000","last_undegraded":"2024-06-10T00:24:43.257683+0000","last_fullsized":"2024-06-10T00:24:43.257683+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:49:51.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257907+0000","last_change":"2024-06-10T00:24:09.168603+0000","last_active":"2024-06-10T00:24:43.257907+0000","last_peered":"2024-06-10T00:24:43.257907+0000","last_clean":"2024-06-10T00:24:43.257907+0000","last_became_active":"2024-06-10T00:24:09.168233+0000","last_became_peered":"2024-06-10T00:24:09.168233+0000","last_unstale":"2024-06-10T00:24:43.257907+0000","last_undegraded":"2024-06-10T00:24:43.257907+0000","last_fullsized":"2024-06-10T00:24:43.257907+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:59.585023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'1","reported_seq":41,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255080+0000","last_change":"2024-06-10T00:23:54.425505+0000","last_active":"2024-06-10T00:24:43.255080+0000","last_peered":"2024-06-10T00:24:43.255080+0000","last_clean":"2024-06-10T00:24:43.255080+0000","last_became_active":"2024-06-10T00:23:54.425058+0000","last_became_peered":"2024-06-10T00:23:54.425058+0000","last_unstale":"2024-06-10T00:24:43.255080+0000","last_undegraded":"2024-06-10T00:24:43.255080+0000","last_fullsized":"2024-06-10T00:24:43.255080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:48:05.605947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257635+0000","last_change":"2024-06-10T00:23:57.566060+0000","last_active":"2024-06-10T00:24:43.257635+0000","last_peered":"2024-06-10T00:24:43.257635+0000","last_clean":"2024-06-10T00:24:43.257635+0000","last_became_active":"2024-06-10T00:23:57.565099+0000","last_became_peered":"2024-06-10T00:23:57.565099+0000","last_unstale":"2024-06-10T00:24:43.257635+0000","last_undegraded":"2024-06-10T00:24:43.257635+0000","last_fullsized":"2024-06-10T00:24:43.257635+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:24:54.204399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258419+0000","last_change":"2024-06-10T00:24:09.264741+0000","last_active":"2024-06-10T00:24:43.258419+0000","last_peered":"2024-06-10T00:24:43.258419+0000","last_clean":"2024-06-10T00:24:43.258419+0000","last_became_active":"2024-06-10T00:24:09.264267+0000","last_became_peered":"2024-06-10T00:24:09.264267+0000","last_unstale":"2024-06-10T00:24:43.258419+0000","last_undegraded":"2024-06-10T00:24:43.258419+0000","last_fullsized":"2024-06-10T00:24:43.258419+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:27.627391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258589+0000","last_change":"2024-06-10T00:23:57.555569+0000","last_active":"2024-06-10T00:24:43.258589+0000","last_peered":"2024-06-10T00:24:43.258589+0000","last_clean":"2024-06-10T00:24:43.258589+0000","last_became_active":"2024-06-10T00:23:57.555265+0000","last_became_peered":"2024-06-10T00:23:57.555265+0000","last_unstale":"2024-06-10T00:24:43.258589+0000","last_undegraded":"2024-06-10T00:24:43.258589+0000","last_fullsized":"2024-06-10T00:24:43.258589+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:28:08.192083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"98'4","reported_seq":72,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:31.322954+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:25:31.322954+0000","last_peered":"2024-06-10T00:25:31.322954+0000","last_clean":"2024-06-10T00:25:31.322954+0000","last_became_active":"2024-06-10T00:23:54.822442+0000","last_became_peered":"2024-06-10T00:23:54.822442+0000","last_unstale":"2024-06-10T00:25:31.322954+0000","last_undegraded":"2024-06-10T00:25:31.322954+0000","last_fullsized":"2024-06-10T00:25:31.322954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:51:10.278257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"98'1","reported_seq":25,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255405+0000","last_change":"2024-06-10T00:24:09.168073+0000","last_active":"2024-06-10T00:24:43.255405+0000","last_peered":"2024-06-10T00:24:43.255405+0000","last_clean":"2024-06-10T00:24:43.255405+0000","last_became_active":"2024-06-10T00:24:09.167657+0000","last_became_peered":"2024-06-10T00:24:09.167657+0000","last_unstale":"2024-06-10T00:24:43.255405+0000","last_undegraded":"2024-06-10T00:24:43.255405+0000","last_fullsized":"2024-06-10T00:24:43.255405+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:20:23.481685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254627+0000","last_change":"2024-06-10T00:23:57.557817+0000","last_active":"2024-06-10T00:24:43.254627+0000","last_peered":"2024-06-10T00:24:43.254627+0000","last_clean":"2024-06-10T00:24:43.254627+0000","last_became_active":"2024-06-10T00:23:57.557155+0000","last_became_peered":"2024-06-10T00:23:57.557155+0000","last_unstale":"2024-06-10T00:24:43.254627+0000","last_undegraded":"2024-06-10T00:24:43.254627+0000","last_fullsized":"2024-06-10T00:24:43.254627+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:41:33.913307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"98'3","reported_seq":44,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:05.281291+0000","last_change":"2024-06-10T00:23:54.423507+0000","last_active":"2024-06-10T00:25:05.281291+0000","last_peered":"2024-06-10T00:25:05.281291+0000","last_clean":"2024-06-10T00:25:05.281291+0000","last_became_active":"2024-06-10T00:23:54.423092+0000","last_became_peered":"2024-06-10T00:23:54.423092+0000","last_unstale":"2024-06-10T00:25:05.281291+0000","last_undegraded":"2024-06-10T00:25:05.281291+0000","last_fullsized":"2024-06-10T00:25:05.281291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:06:15.924827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":373,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255089+0000","last_change":"2024-06-10T00:22:28.025243+0000","last_active":"2024-06-10T00:24:43.255089+0000","last_peered":"2024-06-10T00:24:43.255089+0000","last_clean":"2024-06-10T00:24:43.255089+0000","last_became_active":"2024-06-10T00:22:28.024751+0000","last_became_peered":"2024-06-10T00:22:28.024751+0000","last_unstale":"2024-06-10T00:24:43.255089+0000","last_undegraded":"2024-06-10T00:24:43.255089+0000","last_fullsized":"2024-06-10T00:24:43.255089+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_clean_scrub_stamp":"2024-06-10T00:19:05.039631+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:02:36.557931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259322+0000","last_change":"2024-06-10T00:24:09.265084+0000","last_active":"2024-06-10T00:24:43.259322+0000","last_peered":"2024-06-10T00:24:43.259322+0000","last_clean":"2024-06-10T00:24:43.259322+0000","last_became_active":"2024-06-10T00:24:09.264425+0000","last_became_peered":"2024-06-10T00:24:09.264425+0000","last_unstale":"2024-06-10T00:24:43.259322+0000","last_undegraded":"2024-06-10T00:24:43.259322+0000","last_fullsized":"2024-06-10T00:24:43.259322+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:37:31.577688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257290+0000","last_change":"2024-06-10T00:23:57.558540+0000","last_active":"2024-06-10T00:24:43.257290+0000","last_peered":"2024-06-10T00:24:43.257290+0000","last_clean":"2024-06-10T00:24:43.257290+0000","last_became_active":"2024-06-10T00:23:57.558197+0000","last_became_peered":"2024-06-10T00:23:57.558197+0000","last_unstale":"2024-06-10T00:24:43.257290+0000","last_undegraded":"2024-06-10T00:24:43.257290+0000","last_fullsized":"2024-06-10T00:24:43.257290+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:04:25.566135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"98'3","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.264553+0000","last_change":"2024-06-10T00:23:54.428126+0000","last_active":"2024-06-10T00:25:03.264553+0000","last_peered":"2024-06-10T00:25:03.264553+0000","last_clean":"2024-06-10T00:25:03.264553+0000","last_became_active":"2024-06-10T00:23:54.427630+0000","last_became_peered":"2024-06-10T00:23:54.427630+0000","last_unstale":"2024-06-10T00:25:03.264553+0000","last_undegraded":"2024-06-10T00:25:03.264553+0000","last_fullsized":"2024-06-10T00:25:03.264553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:05:23.679704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258656+0000","last_change":"2024-06-10T00:24:09.169984+0000","last_active":"2024-06-10T00:24:43.258656+0000","last_peered":"2024-06-10T00:24:43.258656+0000","last_clean":"2024-06-10T00:24:43.258656+0000","last_became_active":"2024-06-10T00:24:09.169556+0000","last_became_peered":"2024-06-10T00:24:09.169556+0000","last_unstale":"2024-06-10T00:24:43.258656+0000","last_undegraded":"2024-06-10T00:24:43.258656+0000","last_fullsized":"2024-06-10T00:24:43.258656+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:54:13.575207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.255437+0000","last_change":"2024-06-10T00:24:43.767901+0000","last_active":"2024-06-10T00:25:02.255437+0000","last_peered":"2024-06-10T00:25:02.255437+0000","last_clean":"2024-06-10T00:25:02.255437+0000","last_became_active":"2024-06-10T00:24:43.767562+0000","last_became_peered":"2024-06-10T00:24:43.767562+0000","last_unstale":"2024-06-10T00:25:02.255437+0000","last_undegraded":"2024-06-10T00:25:02.255437+0000","last_fullsized":"2024-06-10T00:25:02.255437+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:46:36.099750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258091+0000","last_change":"2024-06-10T00:23:57.554906+0000","last_active":"2024-06-10T00:24:43.258091+0000","last_peered":"2024-06-10T00:24:43.258091+0000","last_clean":"2024-06-10T00:24:43.258091+0000","last_became_active":"2024-06-10T00:23:57.554608+0000","last_became_peered":"2024-06-10T00:23:57.554608+0000","last_unstale":"2024-06-10T00:24:43.258091+0000","last_undegraded":"2024-06-10T00:24:43.258091+0000","last_fullsized":"2024-06-10T00:24:43.258091+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:24.690594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254956+0000","last_change":"2024-06-10T00:24:09.264676+0000","last_active":"2024-06-10T00:24:43.254956+0000","last_peered":"2024-06-10T00:24:43.254956+0000","last_clean":"2024-06-10T00:24:43.254956+0000","last_became_active":"2024-06-10T00:24:09.264259+0000","last_became_peered":"2024-06-10T00:24:09.264259+0000","last_unstale":"2024-06-10T00:24:43.254956+0000","last_undegraded":"2024-06-10T00:24:43.254956+0000","last_fullsized":"2024-06-10T00:24:43.254956+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:37:53.787039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"98'12","reported_seq":52,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:25.564411+0000","last_change":"2024-06-10T00:23:54.424201+0000","last_active":"2024-06-10T00:25:25.564411+0000","last_peered":"2024-06-10T00:25:25.564411+0000","last_clean":"2024-06-10T00:25:25.564411+0000","last_became_active":"2024-06-10T00:23:54.423853+0000","last_became_peered":"2024-06-10T00:23:54.423853+0000","last_unstale":"2024-06-10T00:25:25.564411+0000","last_undegraded":"2024-06-10T00:25:25.564411+0000","last_fullsized":"2024-06-10T00:25:25.564411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:00.884041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255024+0000","last_change":"2024-06-10T00:23:57.554276+0000","last_active":"2024-06-10T00:24:43.255024+0000","last_peered":"2024-06-10T00:24:43.255024+0000","last_clean":"2024-06-10T00:24:43.255024+0000","last_became_active":"2024-06-10T00:23:57.553964+0000","last_became_peered":"2024-06-10T00:23:57.553964+0000","last_unstale":"2024-06-10T00:24:43.255024+0000","last_undegraded":"2024-06-10T00:24:43.255024+0000","last_fullsized":"2024-06-10T00:24:43.255024+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:39:51.974770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.261159+0000","last_change":"2024-06-10T00:24:09.170249+0000","last_active":"2024-06-10T00:24:43.261159+0000","last_peered":"2024-06-10T00:24:43.261159+0000","last_clean":"2024-06-10T00:24:43.261159+0000","last_became_active":"2024-06-10T00:24:09.169830+0000","last_became_peered":"2024-06-10T00:24:09.169830+0000","last_unstale":"2024-06-10T00:24:43.261159+0000","last_undegraded":"2024-06-10T00:24:43.261159+0000","last_fullsized":"2024-06-10T00:24:43.261159+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.320609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"98'4","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.255865+0000","last_change":"2024-06-10T00:23:54.473943+0000","last_active":"2024-06-10T00:25:02.255865+0000","last_peered":"2024-06-10T00:25:02.255865+0000","last_clean":"2024-06-10T00:25:02.255865+0000","last_became_active":"2024-06-10T00:23:54.473652+0000","last_became_peered":"2024-06-10T00:23:54.473652+0000","last_unstale":"2024-06-10T00:25:02.255865+0000","last_undegraded":"2024-06-10T00:25:02.255865+0000","last_fullsized":"2024-06-10T00:25:02.255865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:52:06.773684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647984+0000","last_change":"2024-06-10T00:23:57.550799+0000","last_active":"2024-06-10T00:24:43.647984+0000","last_peered":"2024-06-10T00:24:43.647984+0000","last_clean":"2024-06-10T00:24:43.647984+0000","last_became_active":"2024-06-10T00:23:57.549252+0000","last_became_peered":"2024-06-10T00:23:57.549252+0000","last_unstale":"2024-06-10T00:24:43.647984+0000","last_undegraded":"2024-06-10T00:24:43.647984+0000","last_fullsized":"2024-06-10T00:24:43.647984+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:54:54.045006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.768837+0000","last_change":"2024-06-10T00:24:43.768837+0000","last_active":"2024-06-10T00:24:43.768837+0000","last_peered":"2024-06-10T00:24:43.768837+0000","last_clean":"2024-06-10T00:24:43.768837+0000","last_became_active":"2024-06-10T00:24:43.768295+0000","last_became_peered":"2024-06-10T00:24:43.768295+0000","last_unstale":"2024-06-10T00:24:43.768837+0000","last_undegraded":"2024-06-10T00:24:43.768837+0000","last_fullsized":"2024-06-10T00:24:43.768837+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:59.600864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"98'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.255542+0000","last_change":"2024-06-10T00:23:54.823448+0000","last_active":"2024-06-10T00:25:02.255542+0000","last_peered":"2024-06-10T00:25:02.255542+0000","last_clean":"2024-06-10T00:25:02.255542+0000","last_became_active":"2024-06-10T00:23:54.823107+0000","last_became_peered":"2024-06-10T00:23:54.823107+0000","last_unstale":"2024-06-10T00:25:02.255542+0000","last_undegraded":"2024-06-10T00:25:02.255542+0000","last_fullsized":"2024-06-10T00:25:02.255542+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:50.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.750987+0000","last_change":"2024-06-10T00:23:57.556733+0000","last_active":"2024-06-10T00:24:43.750987+0000","last_peered":"2024-06-10T00:24:43.750987+0000","last_clean":"2024-06-10T00:24:43.750987+0000","last_became_active":"2024-06-10T00:23:57.556386+0000","last_became_peered":"2024-06-10T00:23:57.556386+0000","last_unstale":"2024-06-10T00:24:43.750987+0000","last_undegraded":"2024-06-10T00:24:43.750987+0000","last_fullsized":"2024-06-10T00:24:43.750987+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:39:00.572311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.768420+0000","last_change":"2024-06-10T00:24:09.174600+0000","last_active":"2024-06-10T00:24:43.768420+0000","last_peered":"2024-06-10T00:24:43.768420+0000","last_clean":"2024-06-10T00:24:43.768420+0000","last_became_active":"2024-06-10T00:24:09.174236+0000","last_became_peered":"2024-06-10T00:24:09.174236+0000","last_unstale":"2024-06-10T00:24:43.768420+0000","last_undegraded":"2024-06-10T00:24:43.768420+0000","last_fullsized":"2024-06-10T00:24:43.768420+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:58:18.447768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"98'3","reported_seq":43,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.257023+0000","last_change":"2024-06-10T00:23:54.425111+0000","last_active":"2024-06-10T00:25:02.257023+0000","last_peered":"2024-06-10T00:25:02.257023+0000","last_clean":"2024-06-10T00:25:02.257023+0000","last_became_active":"2024-06-10T00:23:54.424724+0000","last_became_peered":"2024-06-10T00:23:54.424724+0000","last_unstale":"2024-06-10T00:25:02.257023+0000","last_undegraded":"2024-06-10T00:25:02.257023+0000","last_fullsized":"2024-06-10T00:25:02.257023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:54.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258868+0000","last_change":"2024-06-10T00:23:57.556355+0000","last_active":"2024-06-10T00:24:43.258868+0000","last_peered":"2024-06-10T00:24:43.258868+0000","last_clean":"2024-06-10T00:24:43.258868+0000","last_became_active":"2024-06-10T00:23:57.555479+0000","last_became_peered":"2024-06-10T00:23:57.555479+0000","last_unstale":"2024-06-10T00:24:43.258868+0000","last_undegraded":"2024-06-10T00:24:43.258868+0000","last_fullsized":"2024-06-10T00:24:43.258868+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:08.817381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"98'1","reported_seq":23,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.259000+0000","last_change":"2024-06-10T00:24:09.169589+0000","last_active":"2024-06-10T00:24:43.259000+0000","last_peered":"2024-06-10T00:24:43.259000+0000","last_clean":"2024-06-10T00:24:43.259000+0000","last_became_active":"2024-06-10T00:24:09.168436+0000","last_became_peered":"2024-06-10T00:24:09.168436+0000","last_unstale":"2024-06-10T00:24:43.259000+0000","last_undegraded":"2024-06-10T00:24:43.259000+0000","last_fullsized":"2024-06-10T00:24:43.259000+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:17:32.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"98'1","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:46.162418+0000","last_change":"2024-06-10T00:24:43.751399+0000","last_active":"2024-06-10T00:24:46.162418+0000","last_peered":"2024-06-10T00:24:46.162418+0000","last_clean":"2024-06-10T00:24:46.162418+0000","last_became_active":"2024-06-10T00:24:43.750987+0000","last_became_peered":"2024-06-10T00:24:43.750987+0000","last_unstale":"2024-06-10T00:24:46.162418+0000","last_undegraded":"2024-06-10T00:24:46.162418+0000","last_fullsized":"2024-06-10T00:24:46.162418+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:14:47.902017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255189+0000","last_change":"2024-06-10T00:23:57.560213+0000","last_active":"2024-06-10T00:24:43.255189+0000","last_peered":"2024-06-10T00:24:43.255189+0000","last_clean":"2024-06-10T00:24:43.255189+0000","last_became_active":"2024-06-10T00:23:57.559925+0000","last_became_peered":"2024-06-10T00:23:57.559925+0000","last_unstale":"2024-06-10T00:24:43.255189+0000","last_undegraded":"2024-06-10T00:24:43.255189+0000","last_fullsized":"2024-06-10T00:24:43.255189+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:00:37.403087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258726+0000","last_change":"2024-06-10T00:24:09.264758+0000","last_active":"2024-06-10T00:24:43.258726+0000","last_peered":"2024-06-10T00:24:43.258726+0000","last_clean":"2024-06-10T00:24:43.258726+0000","last_became_active":"2024-06-10T00:24:09.263928+0000","last_became_peered":"2024-06-10T00:24:09.263928+0000","last_unstale":"2024-06-10T00:24:43.258726+0000","last_undegraded":"2024-06-10T00:24:43.258726+0000","last_fullsized":"2024-06-10T00:24:43.258726+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:23:41.083091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"98'4","reported_seq":45,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:02.258080+0000","last_change":"2024-06-10T00:23:54.428553+0000","last_active":"2024-06-10T00:25:02.258080+0000","last_peered":"2024-06-10T00:25:02.258080+0000","last_clean":"2024-06-10T00:25:02.258080+0000","last_became_active":"2024-06-10T00:23:54.428108+0000","last_became_peered":"2024-06-10T00:23:54.428108+0000","last_unstale":"2024-06-10T00:25:02.258080+0000","last_undegraded":"2024-06-10T00:25:02.258080+0000","last_fullsized":"2024-06-10T00:25:02.258080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:57:26.207915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255060+0000","last_change":"2024-06-10T00:23:57.560686+0000","last_active":"2024-06-10T00:24:43.255060+0000","last_peered":"2024-06-10T00:24:43.255060+0000","last_clean":"2024-06-10T00:24:43.255060+0000","last_became_active":"2024-06-10T00:23:57.560326+0000","last_became_peered":"2024-06-10T00:23:57.560326+0000","last_unstale":"2024-06-10T00:24:43.255060+0000","last_undegraded":"2024-06-10T00:24:43.255060+0000","last_fullsized":"2024-06-10T00:24:43.255060+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:08:39.506567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.766708+0000","last_change":"2024-06-10T00:24:09.166947+0000","last_active":"2024-06-10T00:24:43.766708+0000","last_peered":"2024-06-10T00:24:43.766708+0000","last_clean":"2024-06-10T00:24:43.766708+0000","last_became_active":"2024-06-10T00:24:09.166503+0000","last_became_peered":"2024-06-10T00:24:09.166503+0000","last_unstale":"2024-06-10T00:24:43.766708+0000","last_undegraded":"2024-06-10T00:24:43.766708+0000","last_fullsized":"2024-06-10T00:24:43.766708+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:16:45.094642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"98'3","reported_seq":44,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:25:03.265919+0000","last_change":"2024-06-10T00:23:54.823522+0000","last_active":"2024-06-10T00:25:03.265919+0000","last_peered":"2024-06-10T00:25:03.265919+0000","last_clean":"2024-06-10T00:25:03.265919+0000","last_became_active":"2024-06-10T00:23:54.823142+0000","last_became_peered":"2024-06-10T00:23:54.823142+0000","last_unstale":"2024-06-10T00:25:03.265919+0000","last_undegraded":"2024-06-10T00:25:03.265919+0000","last_fullsized":"2024-06-10T00:25:03.265919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:03.836850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255596+0000","last_change":"2024-06-10T00:23:57.563710+0000","last_active":"2024-06-10T00:24:43.255596+0000","last_peered":"2024-06-10T00:24:43.255596+0000","last_clean":"2024-06-10T00:24:43.255596+0000","last_became_active":"2024-06-10T00:23:57.563331+0000","last_became_peered":"2024-06-10T00:23:57.563331+0000","last_unstale":"2024-06-10T00:24:43.255596+0000","last_undegraded":"2024-06-10T00:24:43.255596+0000","last_fullsized":"2024-06-10T00:24:43.255596+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:08:48.061885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647343+0000","last_change":"2024-06-10T00:24:09.171203+0000","last_active":"2024-06-10T00:24:43.647343+0000","last_peered":"2024-06-10T00:24:43.647343+0000","last_clean":"2024-06-10T00:24:43.647343+0000","last_became_active":"2024-06-10T00:24:09.170824+0000","last_became_peered":"2024-06-10T00:24:09.170824+0000","last_unstale":"2024-06-10T00:24:43.647343+0000","last_undegraded":"2024-06-10T00:24:43.647343+0000","last_fullsized":"2024-06-10T00:24:43.647343+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:47:28.045602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.260674+0000","last_change":"2024-06-10T00:23:54.421307+0000","last_active":"2024-06-10T00:24:43.260674+0000","last_peered":"2024-06-10T00:24:43.260674+0000","last_clean":"2024-06-10T00:24:43.260674+0000","last_became_active":"2024-06-10T00:23:54.420928+0000","last_became_peered":"2024-06-10T00:23:54.420928+0000","last_unstale":"2024-06-10T00:24:43.260674+0000","last_undegraded":"2024-06-10T00:24:43.260674+0000","last_fullsized":"2024-06-10T00:24:43.260674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:59:41.433605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255327+0000","last_change":"2024-06-10T00:23:57.556494+0000","last_active":"2024-06-10T00:24:43.255327+0000","last_peered":"2024-06-10T00:24:43.255327+0000","last_clean":"2024-06-10T00:24:43.255327+0000","last_became_active":"2024-06-10T00:23:57.556088+0000","last_became_peered":"2024-06-10T00:23:57.556088+0000","last_unstale":"2024-06-10T00:24:43.255327+0000","last_undegraded":"2024-06-10T00:24:43.255327+0000","last_fullsized":"2024-06-10T00:24:43.255327+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:06:23.873281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":24,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257821+0000","last_change":"2024-06-10T00:24:09.166734+0000","last_active":"2024-06-10T00:24:43.257821+0000","last_peered":"2024-06-10T00:24:43.257821+0000","last_clean":"2024-06-10T00:24:43.257821+0000","last_became_active":"2024-06-10T00:24:09.166270+0000","last_became_peered":"2024-06-10T00:24:09.166270+0000","last_unstale":"2024-06-10T00:24:43.257821+0000","last_undegraded":"2024-06-10T00:24:43.257821+0000","last_fullsized":"2024-06-10T00:24:43.257821+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:38.039633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":42,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258279+0000","last_change":"2024-06-10T00:23:54.426226+0000","last_active":"2024-06-10T00:24:43.258279+0000","last_peered":"2024-06-10T00:24:43.258279+0000","last_clean":"2024-06-10T00:24:43.258279+0000","last_became_active":"2024-06-10T00:23:54.425879+0000","last_became_peered":"2024-06-10T00:23:54.425879+0000","last_unstale":"2024-06-10T00:24:43.258279+0000","last_undegraded":"2024-06-10T00:24:43.258279+0000","last_fullsized":"2024-06-10T00:24:43.258279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:15:32.936556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255588+0000","last_change":"2024-06-10T00:23:57.553814+0000","last_active":"2024-06-10T00:24:43.255588+0000","last_peered":"2024-06-10T00:24:43.255588+0000","last_clean":"2024-06-10T00:24:43.255588+0000","last_became_active":"2024-06-10T00:23:57.553478+0000","last_became_peered":"2024-06-10T00:23:57.553478+0000","last_unstale":"2024-06-10T00:24:43.255588+0000","last_undegraded":"2024-06-10T00:24:43.255588+0000","last_fullsized":"2024-06-10T00:24:43.255588+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:54:39.264526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258979+0000","last_change":"2024-06-10T00:23:57.556676+0000","last_active":"2024-06-10T00:24:43.258979+0000","last_peered":"2024-06-10T00:24:43.258979+0000","last_clean":"2024-06-10T00:24:43.258979+0000","last_became_active":"2024-06-10T00:23:57.555946+0000","last_became_peered":"2024-06-10T00:23:57.555946+0000","last_unstale":"2024-06-10T00:24:43.258979+0000","last_undegraded":"2024-06-10T00:24:43.258979+0000","last_fullsized":"2024-06-10T00:24:43.258979+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:10:10.868799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.261526+0000","last_change":"2024-06-10T00:23:57.565423+0000","last_active":"2024-06-10T00:24:43.261526+0000","last_peered":"2024-06-10T00:24:43.261526+0000","last_clean":"2024-06-10T00:24:43.261526+0000","last_became_active":"2024-06-10T00:23:57.564830+0000","last_became_peered":"2024-06-10T00:23:57.564830+0000","last_unstale":"2024-06-10T00:24:43.261526+0000","last_undegraded":"2024-06-10T00:24:43.261526+0000","last_fullsized":"2024-06-10T00:24:43.261526+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:11:06.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.261357+0000","last_change":"2024-06-10T00:23:57.558971+0000","last_active":"2024-06-10T00:24:43.261357+0000","last_peered":"2024-06-10T00:24:43.261357+0000","last_clean":"2024-06-10T00:24:43.261357+0000","last_became_active":"2024-06-10T00:23:57.558620+0000","last_became_peered":"2024-06-10T00:23:57.558620+0000","last_unstale":"2024-06-10T00:24:43.261357+0000","last_undegraded":"2024-06-10T00:24:43.261357+0000","last_fullsized":"2024-06-10T00:24:43.261357+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:45:43.308008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.647564+0000","last_change":"2024-06-10T00:23:57.552341+0000","last_active":"2024-06-10T00:24:43.647564+0000","last_peered":"2024-06-10T00:24:43.647564+0000","last_clean":"2024-06-10T00:24:43.647564+0000","last_became_active":"2024-06-10T00:23:57.551464+0000","last_became_peered":"2024-06-10T00:23:57.551464+0000","last_unstale":"2024-06-10T00:24:43.647564+0000","last_undegraded":"2024-06-10T00:24:43.647564+0000","last_fullsized":"2024-06-10T00:24:43.647564+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:44:41.063516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":40,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.768961+0000","last_change":"2024-06-10T00:24:43.768961+0000","last_active":"2024-06-10T00:24:43.768961+0000","last_peered":"2024-06-10T00:24:43.768961+0000","last_clean":"2024-06-10T00:24:43.768961+0000","last_became_active":"2024-06-10T00:24:43.768542+0000","last_became_peered":"2024-06-10T00:24:43.768542+0000","last_unstale":"2024-06-10T00:24:43.768961+0000","last_undegraded":"2024-06-10T00:24:43.768961+0000","last_fullsized":"2024-06-10T00:24:43.768961+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:23:58.412054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258512+0000","last_change":"2024-06-10T00:23:57.554924+0000","last_active":"2024-06-10T00:24:43.258512+0000","last_peered":"2024-06-10T00:24:43.258512+0000","last_clean":"2024-06-10T00:24:43.258512+0000","last_became_active":"2024-06-10T00:23:57.554515+0000","last_became_peered":"2024-06-10T00:23:57.554515+0000","last_unstale":"2024-06-10T00:24:43.258512+0000","last_undegraded":"2024-06-10T00:24:43.258512+0000","last_fullsized":"2024-06-10T00:24:43.258512+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:58:15.624775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.262115+0000","last_change":"2024-06-10T00:23:57.561940+0000","last_active":"2024-06-10T00:24:43.262115+0000","last_peered":"2024-06-10T00:24:43.262115+0000","last_clean":"2024-06-10T00:24:43.262115+0000","last_became_active":"2024-06-10T00:23:57.561512+0000","last_became_peered":"2024-06-10T00:23:57.561512+0000","last_unstale":"2024-06-10T00:24:43.262115+0000","last_undegraded":"2024-06-10T00:24:43.262115+0000","last_fullsized":"2024-06-10T00:24:43.262115+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:57:57.147311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.255250+0000","last_change":"2024-06-10T00:23:57.557229+0000","last_active":"2024-06-10T00:24:43.255250+0000","last_peered":"2024-06-10T00:24:43.255250+0000","last_clean":"2024-06-10T00:24:43.255250+0000","last_became_active":"2024-06-10T00:23:57.556905+0000","last_became_peered":"2024-06-10T00:23:57.556905+0000","last_unstale":"2024-06-10T00:24:43.255250+0000","last_undegraded":"2024-06-10T00:24:43.255250+0000","last_fullsized":"2024-06-10T00:24:43.255250+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:30.513517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.768509+0000","last_change":"2024-06-10T00:24:43.768509+0000","last_active":"2024-06-10T00:24:43.768509+0000","last_peered":"2024-06-10T00:24:43.768509+0000","last_clean":"2024-06-10T00:24:43.768509+0000","last_became_active":"2024-06-10T00:24:43.768038+0000","last_became_peered":"2024-06-10T00:24:43.768038+0000","last_unstale":"2024-06-10T00:24:43.768509+0000","last_undegraded":"2024-06-10T00:24:43.768509+0000","last_fullsized":"2024-06-10T00:24:43.768509+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:38.344375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.254595+0000","last_change":"2024-06-10T00:23:57.557830+0000","last_active":"2024-06-10T00:24:43.254595+0000","last_peered":"2024-06-10T00:24:43.254595+0000","last_clean":"2024-06-10T00:24:43.254595+0000","last_became_active":"2024-06-10T00:23:57.556986+0000","last_became_peered":"2024-06-10T00:23:57.556986+0000","last_unstale":"2024-06-10T00:24:43.254595+0000","last_undegraded":"2024-06-10T00:24:43.254595+0000","last_fullsized":"2024-06-10T00:24:43.254595+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:53.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.766973+0000","last_change":"2024-06-10T00:23:57.565602+0000","last_active":"2024-06-10T00:24:43.766973+0000","last_peered":"2024-06-10T00:24:43.766973+0000","last_clean":"2024-06-10T00:24:43.766973+0000","last_became_active":"2024-06-10T00:23:57.565264+0000","last_became_peered":"2024-06-10T00:23:57.565264+0000","last_unstale":"2024-06-10T00:24:43.766973+0000","last_undegraded":"2024-06-10T00:24:43.766973+0000","last_fullsized":"2024-06-10T00:24:43.766973+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:09:12.773087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":34,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.257833+0000","last_change":"2024-06-10T00:23:57.548516+0000","last_active":"2024-06-10T00:24:43.257833+0000","last_peered":"2024-06-10T00:24:43.257833+0000","last_clean":"2024-06-10T00:24:43.257833+0000","last_became_active":"2024-06-10T00:23:57.548202+0000","last_became_peered":"2024-06-10T00:23:57.548202+0000","last_unstale":"2024-06-10T00:24:43.257833+0000","last_undegraded":"2024-06-10T00:24:43.257833+0000","last_fullsized":"2024-06-10T00:24:43.257833+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:13.901478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":33,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-06-10T00:24:43.258257+0000","last_change":"2024-06-10T00:23:57.565900+0000","last_active":"2024-06-10T00:24:43.258257+0000","last_peered":"2024-06-10T00:24:43.258257+0000","last_clean":"2024-06-10T00:24:43.258257+0000","last_became_active":"2024-06-10T00:23:57.565573+0000","last_became_peered":"2024-06-10T00:23:57.565573+0000","last_unstale":"2024-06-10T00:24:43.258257+0000","last_undegraded":"2024-06-10T00:24:43.258257+0000","last_fullsized":"2024-06-10T00:24:43.258257+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:08.255905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280680,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66655,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":196,"num_read_kb":171,"num_write":169,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":425984,"data_stored":195700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351011,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":554064,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612648,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32372,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712780,"statfs":{"total":95995035648,"available":95961886720,"internally_reserved":0,"allocated":1224704,"data_stored":1203965,"data_compressed":62071,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776173,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":558374,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037810,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":554176,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299447,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28132,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717020,"statfs":{"total":95995035648,"available":95966228480,"internally_reserved":0,"allocated":1142784,"data_stored":1150106,"data_compressed":62071,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1073152,"data_stored":549454,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":607754,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":550842,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32316,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712836,"statfs":{"total":95995035648,"available":95961944064,"internally_reserved":0,"allocated":1167360,"data_stored":613759,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":549648,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888134,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":554104,"data_compressed":26371,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724609,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1150976,"data_stored":1145694,"data_compressed":62071,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-10T00:25:35.687 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-10T00:25:35.687 INFO:tasks.cephadm:Setting up client nodes... 2024-06-10T00:25:35.687 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-10T00:25:35.927 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:36.301 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:36 smithi121 ceph-mon[32378]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:25:36.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:36 smithi135 ceph-mon[36424]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:25:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:36 smithi193 ceph-mon[36716]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:25:37.341 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:37 smithi121 ceph-mon[32378]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-06-10T00:25:37.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:37 smithi135 ceph-mon[36424]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-06-10T00:25:37.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:37 smithi193 ceph-mon[36716]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-06-10T00:25:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.769+0000 7f5872897700 1 -- 172.21.15.121:0/3314734008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f586c104230 msgr2=0x7f586c10aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.769+0000 7f5872897700 1 --2- 172.21.15.121:0/3314734008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f586c104230 0x7f586c10aae0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f586400b0d0 tx=0x7f586400b3e0 comp rx=0 tx=0).stop 2024-06-10T00:25:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.769+0000 7f5872897700 1 -- 172.21.15.121:0/3314734008 shutdown_connections 2024-06-10T00:25:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.769+0000 7f5872897700 1 --2- 172.21.15.121:0/3314734008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f586c104230 0x7f586c10aae0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.769+0000 7f5872897700 1 --2- 172.21.15.121:0/3314734008 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f586c100810 0x7f586c100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.769+0000 7f5872897700 1 --2- 172.21.15.121:0/3314734008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f586c103890 0x7f586c103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.769+0000 7f5872897700 1 -- 172.21.15.121:0/3314734008 >> 172.21.15.121:0/3314734008 conn(0x7f586c0fc170 msgr2=0x7f586c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.769+0000 7f5872897700 1 -- 172.21.15.121:0/3314734008 shutdown_connections 2024-06-10T00:25:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.769+0000 7f5872897700 1 -- 172.21.15.121:0/3314734008 wait complete. 2024-06-10T00:25:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.769+0000 7f5872897700 1 Processor -- start 2024-06-10T00:25:37.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.769+0000 7f5872897700 1 -- start start 2024-06-10T00:25:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5872897700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f586c100810 0x7f586c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5872897700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f586c103890 0x7f586c198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5872897700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f586c104230 0x7f586c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:37.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5872897700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f586c10e170 con 0x7f586c104230 2024-06-10T00:25:37.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5872897700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f586c10dff0 con 0x7f586c103890 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5872897700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f586c10e2f0 con 0x7f586c100810 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5870e34700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f586c104230 0x7f586c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5870e34700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f586c104230 0x7f586c19fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37956/0 (socket says 172.21.15.121:37956) 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5870e34700 1 -- 172.21.15.121:0/3804469181 learned_addr learned my addr 172.21.15.121:0/3804469181 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f586b7fe700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f586c103890 0x7f586c198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f586bfff700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f586c100810 0x7f586c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5870e34700 1 -- 172.21.15.121:0/3804469181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f586c100810 msgr2=0x7f586c1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5870e34700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f586c100810 0x7f586c1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5870e34700 1 -- 172.21.15.121:0/3804469181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f586c103890 msgr2=0x7f586c198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5870e34700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f586c103890 0x7f586c198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.770+0000 7f5870e34700 1 -- 172.21.15.121:0/3804469181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586c1a0250 con 0x7f586c104230 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.771+0000 7f5870e34700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f586c104230 0x7f586c19fbb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5864007d10 tx=0x7f5864007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:37.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.771+0000 7f58697fa700 1 -- 172.21.15.121:0/3804469181 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864021020 con 0x7f586c104230 2024-06-10T00:25:37.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.771+0000 7f5872897700 1 -- 172.21.15.121:0/3804469181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586c1a04e0 con 0x7f586c104230 2024-06-10T00:25:37.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.771+0000 7f58697fa700 1 -- 172.21.15.121:0/3804469181 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f586401d950 con 0x7f586c104230 2024-06-10T00:25:37.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.771+0000 7f58697fa700 1 -- 172.21.15.121:0/3804469181 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58640053b0 con 0x7f586c104230 2024-06-10T00:25:37.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.771+0000 7f5872897700 1 -- 172.21.15.121:0/3804469181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f586c1a0af0 con 0x7f586c104230 2024-06-10T00:25:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.772+0000 7f5872897700 1 -- 172.21.15.121:0/3804469181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f586c050a00 con 0x7f586c104230 2024-06-10T00:25:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.773+0000 7f58697fa700 1 -- 172.21.15.121:0/3804469181 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f5864024020 con 0x7f586c104230 2024-06-10T00:25:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.774+0000 7f58697fa700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f58540a2d60 0x7f58540a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.774+0000 7f586bfff700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f58540a2d60 0x7f58540a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.774+0000 7f58697fa700 1 -- 172.21.15.121:0/3804469181 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f586400f020 con 0x7f586c104230 2024-06-10T00:25:37.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.774+0000 7f586bfff700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f58540a2d60 0x7f58540a5220 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f586c06a940 tx=0x7f585c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:37.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:37.778+0000 7f58697fa700 1 -- 172.21.15.121:0/3804469181 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f586409d240 con 0x7f586c104230 2024-06-10T00:25:38.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.060+0000 7f5872897700 1 -- 172.21.15.121:0/3804469181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f586c02dd40 con 0x7f586c104230 2024-06-10T00:25:38.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.062+0000 7f58697fa700 1 -- 172.21.15.121:0/3804469181 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f586c02dd40 con 0x7f586c104230 2024-06-10T00:25:38.062 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-06-10T00:25:38.062 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQACSGZmA7SwAxAAJqv4eNQwy833e9aIAgimvw== 2024-06-10T00:25:38.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 -- 172.21.15.121:0/3804469181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f58540a2d60 msgr2=0x7f58540a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:38.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f58540a2d60 0x7f58540a5220 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f586c06a940 tx=0x7f585c00b410 comp rx=0 tx=0).stop 2024-06-10T00:25:38.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 -- 172.21.15.121:0/3804469181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f586c104230 msgr2=0x7f586c19fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:38.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f586c104230 0x7f586c19fbb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5864007d10 tx=0x7f5864007d40 comp rx=0 tx=0).stop 2024-06-10T00:25:38.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 -- 172.21.15.121:0/3804469181 shutdown_connections 2024-06-10T00:25:38.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f58540a2d60 0x7f58540a5220 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:38.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f586c104230 0x7f586c19fbb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:38.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f586c100810 0x7f586c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:38.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 --2- 172.21.15.121:0/3804469181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f586c103890 0x7f586c198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:38.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 -- 172.21.15.121:0/3804469181 >> 172.21.15.121:0/3804469181 conn(0x7f586c0fc170 msgr2=0x7f586c106ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:38.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 -- 172.21.15.121:0/3804469181 shutdown_connections 2024-06-10T00:25:38.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:38.065+0000 7f5872897700 1 -- 172.21.15.121:0/3804469181 wait complete. 2024-06-10T00:25:38.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:38 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3804469181' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-10T00:25:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:38 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3804469181' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-10T00:25:38.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:38 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3804469181' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-10T00:25:38.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:38 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3804469181' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-10T00:25:38.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:38 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3804469181' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-10T00:25:38.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:38 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3804469181' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-10T00:25:38.877 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:25:38.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-10T00:25:38.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-10T00:25:38.917 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-10T00:25:38.917 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-10T00:25:38.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-06-10T00:25:39.197 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:39.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:39 smithi135 ceph-mon[36424]: pgmap v356: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-06-10T00:25:39.561 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:39 smithi121 ceph-mon[32378]: pgmap v356: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-06-10T00:25:39.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:39 smithi193 ceph-mon[36716]: pgmap v356: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-06-10T00:25:40.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.948+0000 7f73bba55700 1 -- 172.21.15.121:0/2161599519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73b4102e80 msgr2=0x7f73b410f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.948+0000 7f73bba55700 1 --2- 172.21.15.121:0/2161599519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73b4102e80 0x7f73b410f330 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00b0a0 tx=0x7f73ac00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.948+0000 7f73bba55700 1 -- 172.21.15.121:0/2161599519 shutdown_connections 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.948+0000 7f73bba55700 1 --2- 172.21.15.121:0/2161599519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73b4102e80 0x7f73b410f330 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.948+0000 7f73bba55700 1 --2- 172.21.15.121:0/2161599519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73b41084f0 0x7f73b41088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.948+0000 7f73bba55700 1 --2- 172.21.15.121:0/2161599519 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73b41024e0 0x7f73b4102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.948+0000 7f73bba55700 1 -- 172.21.15.121:0/2161599519 >> 172.21.15.121:0/2161599519 conn(0x7f73b4079530 msgr2=0x7f73b4079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.948+0000 7f73bba55700 1 -- 172.21.15.121:0/2161599519 shutdown_connections 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.948+0000 7f73bba55700 1 -- 172.21.15.121:0/2161599519 wait complete. 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.948+0000 7f73bba55700 1 Processor -- start 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.948+0000 7f73bba55700 1 -- start start 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73bba55700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73b41024e0 0x7f73b419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:40.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73bba55700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73b4102e80 0x7f73b419ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73bba55700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73b41084f0 0x7f73b4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73bba55700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f73b41127c0 con 0x7f73b4102e80 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73bba55700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f73b4112640 con 0x7f73b41084f0 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73bba55700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f73b4112940 con 0x7f73b41024e0 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73b8ff0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73b4102e80 0x7f73b419ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73b8ff0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73b4102e80 0x7f73b419ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38304/0 (socket says 172.21.15.121:38304) 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73b8ff0700 1 -- 172.21.15.121:0/3075517286 learned_addr learned my addr 172.21.15.121:0/3075517286 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73b9ff2700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73b41084f0 0x7f73b4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73b97f1700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73b41024e0 0x7f73b419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73b8ff0700 1 -- 172.21.15.121:0/3075517286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73b41024e0 msgr2=0x7f73b419e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73b8ff0700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73b41024e0 0x7f73b419e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73b8ff0700 1 -- 172.21.15.121:0/3075517286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73b41084f0 msgr2=0x7f73b4198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:40.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73b8ff0700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73b41084f0 0x7f73b4198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:40.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73b8ff0700 1 -- 172.21.15.121:0/3075517286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73b4199280 con 0x7f73b4102e80 2024-06-10T00:25:40.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73b8ff0700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73b4102e80 0x7f73b419ee50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f73a400d950 tx=0x7f73a400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:40.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73aa7fc700 1 -- 172.21.15.121:0/3075517286 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a4016070 con 0x7f73b4102e80 2024-06-10T00:25:40.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73bba55700 1 -- 172.21.15.121:0/3075517286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73b4199570 con 0x7f73b4102e80 2024-06-10T00:25:40.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73aa7fc700 1 -- 172.21.15.121:0/3075517286 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73a4009e20 con 0x7f73b4102e80 2024-06-10T00:25:40.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73aa7fc700 1 -- 172.21.15.121:0/3075517286 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a400f620 con 0x7f73b4102e80 2024-06-10T00:25:40.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.949+0000 7f73bba55700 1 -- 172.21.15.121:0/3075517286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73b41a57e0 con 0x7f73b4102e80 2024-06-10T00:25:40.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.950+0000 7f73bba55700 1 -- 172.21.15.121:0/3075517286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73b4069180 con 0x7f73b4102e80 2024-06-10T00:25:40.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.952+0000 7f73aa7fc700 1 -- 172.21.15.121:0/3075517286 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f73a4019070 con 0x7f73b4102e80 2024-06-10T00:25:40.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.952+0000 7f73aa7fc700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f73a00a2cb0 0x7f73a00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:40.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.952+0000 7f73b97f1700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f73a00a2cb0 0x7f73a00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:40.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.952+0000 7f73aa7fc700 1 -- 172.21.15.121:0/3075517286 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9092+0+0 (secure 0 0 0) 0x7f73a40cf0f0 con 0x7f73b4102e80 2024-06-10T00:25:40.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.953+0000 7f73b97f1700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f73a00a2cb0 0x7f73a00a5170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f73b0006fd0 tx=0x7f73b0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:40.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:40.956+0000 7f73aa7fc700 1 -- 172.21.15.121:0/3075517286 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f73a409bb90 con 0x7f73b4102e80 2024-06-10T00:25:41.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.210+0000 7f73bba55700 1 -- 172.21.15.121:0/3075517286 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f73b402dca0 con 0x7f73b4102e80 2024-06-10T00:25:41.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.215+0000 7f73aa7fc700 1 -- 172.21.15.121:0/3075517286 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263207 (secure 0 0 0) 0x7f73a409b490 con 0x7f73b4102e80 2024-06-10T00:25:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:25:41.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.217+0000 7f73bba55700 1 -- 172.21.15.121:0/3075517286 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f73a00a2cb0 msgr2=0x7f73a00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:41.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.217+0000 7f73bba55700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f73a00a2cb0 0x7f73a00a5170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f73b0006fd0 tx=0x7f73b0008040 comp rx=0 tx=0).stop 2024-06-10T00:25:41.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.217+0000 7f73bba55700 1 -- 172.21.15.121:0/3075517286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73b4102e80 msgr2=0x7f73b419ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:41.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.217+0000 7f73bba55700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73b4102e80 0x7f73b419ee50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f73a400d950 tx=0x7f73a400dc60 comp rx=0 tx=0).stop 2024-06-10T00:25:41.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.217+0000 7f73bba55700 1 -- 172.21.15.121:0/3075517286 shutdown_connections 2024-06-10T00:25:41.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.217+0000 7f73bba55700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f73a00a2cb0 0x7f73a00a5170 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:41.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.217+0000 7f73bba55700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f73b41084f0 0x7f73b4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:41.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.217+0000 7f73bba55700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73b41024e0 0x7f73b419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:41.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.217+0000 7f73bba55700 1 --2- 172.21.15.121:0/3075517286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73b4102e80 0x7f73b419ee50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:41.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.217+0000 7f73bba55700 1 -- 172.21.15.121:0/3075517286 >> 172.21.15.121:0/3075517286 conn(0x7f73b4079530 msgr2=0x7f73b41005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:41.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.218+0000 7f73bba55700 1 -- 172.21.15.121:0/3075517286 shutdown_connections 2024-06-10T00:25:41.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:41.218+0000 7f73bba55700 1 -- 172.21.15.121:0/3075517286 wait complete. 2024-06-10T00:25:41.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:41 smithi193 ceph-mon[36716]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-06-10T00:25:41.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:41 smithi121 ceph-mon[32378]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-06-10T00:25:41.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:41 smithi135 ceph-mon[36424]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-06-10T00:25:41.938 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506007074},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506007074}]},"active_addr":"172.21.15.121:6801/3506007074","active_change":"2024-06-10T00:16:42.011187+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":14214,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.121:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":58990727}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":4078460254}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":4266501776}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":1002791322}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":128771491}]}]} 2024-06-10T00:25:41.943 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-10T00:25:41.943 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-10T00:25:41.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:25:42.187 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:42.198 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:42 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3075517286' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-10T00:25:42.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:42 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3075517286' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-10T00:25:42.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:42 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3075517286' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-10T00:25:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:43 smithi193 ceph-mon[36716]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-06-10T00:25:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-06-10T00:25:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-06-10T00:25:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:43 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2024-06-10T00:25:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:43 smithi121 ceph-mon[32378]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-06-10T00:25:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-06-10T00:25:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-06-10T00:25:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:43 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2024-06-10T00:25:43.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:43 smithi135 ceph-mon[36424]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-06-10T00:25:43.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-06-10T00:25:43.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-06-10T00:25:43.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:43 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2024-06-10T00:25:44.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.033+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2259188116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff58102ed0 msgr2=0x7eff5810f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:44.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.033+0000 7eff5ef1a700 1 --2- 172.21.15.121:0/2259188116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff58102ed0 0x7eff5810f360 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7eff5000b0a0 tx=0x7eff5000b3b0 comp rx=0 tx=0).stop 2024-06-10T00:25:44.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.033+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2259188116 shutdown_connections 2024-06-10T00:25:44.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.033+0000 7eff5ef1a700 1 --2- 172.21.15.121:0/2259188116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff58102ed0 0x7eff5810f360 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:44.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.033+0000 7eff5ef1a700 1 --2- 172.21.15.121:0/2259188116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff58108520 0x7eff58108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:44.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.033+0000 7eff5ef1a700 1 --2- 172.21.15.121:0/2259188116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff58102510 0x7eff58102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:44.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.033+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2259188116 >> 172.21.15.121:0/2259188116 conn(0x7eff58079560 msgr2=0x7eff58079970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:44.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.034+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2259188116 shutdown_connections 2024-06-10T00:25:44.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.034+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2259188116 wait complete. 2024-06-10T00:25:44.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.034+0000 7eff5ef1a700 1 Processor -- start 2024-06-10T00:25:44.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.034+0000 7eff5ef1a700 1 -- start start 2024-06-10T00:25:44.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.034+0000 7eff5ef1a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff58102510 0x7eff5819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:44.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.034+0000 7eff5ef1a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff58102ed0 0x7eff5819ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:44.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.034+0000 7eff5ef1a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff58108520 0x7eff58198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:44.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.034+0000 7eff5ef1a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7eff581127b0 con 0x7eff58108520 2024-06-10T00:25:44.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.034+0000 7eff5ef1a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7eff58112630 con 0x7eff58102510 2024-06-10T00:25:44.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.034+0000 7eff5ef1a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff58112930 con 0x7eff58102ed0 2024-06-10T00:25:44.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff5d4b7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff58108520 0x7eff58198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:44.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff5d4b7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff58108520 0x7eff58198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38324/0 (socket says 172.21.15.121:38324) 2024-06-10T00:25:44.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff5d4b7700 1 -- 172.21.15.121:0/2147768933 learned_addr learned my addr 172.21.15.121:0/2147768933 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:44.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff5ccb6700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff58102510 0x7eff5819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:44.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff57fff700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff58102ed0 0x7eff5819ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:44.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff5d4b7700 1 -- 172.21.15.121:0/2147768933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff58102ed0 msgr2=0x7eff5819ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:44.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff5d4b7700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff58102ed0 0x7eff5819ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:44.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff5d4b7700 1 -- 172.21.15.121:0/2147768933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff58102510 msgr2=0x7eff5819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:44.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff5d4b7700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff58102510 0x7eff5819e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:44.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff5d4b7700 1 -- 172.21.15.121:0/2147768933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff581990f0 con 0x7eff58108520 2024-06-10T00:25:44.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff5d4b7700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff58108520 0x7eff58198990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7eff500062a0 tx=0x7eff5000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:44.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff55ffb700 1 -- 172.21.15.121:0/2147768933 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff50021070 con 0x7eff58108520 2024-06-10T00:25:44.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2147768933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff58199380 con 0x7eff58108520 2024-06-10T00:25:44.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff55ffb700 1 -- 172.21.15.121:0/2147768933 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff50007c80 con 0x7eff58108520 2024-06-10T00:25:44.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.035+0000 7eff55ffb700 1 -- 172.21.15.121:0/2147768933 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff50005420 con 0x7eff58108520 2024-06-10T00:25:44.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.036+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2147768933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff581a5650 con 0x7eff58108520 2024-06-10T00:25:44.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.036+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2147768933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff58069180 con 0x7eff58108520 2024-06-10T00:25:44.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.038+0000 7eff55ffb700 1 -- 172.21.15.121:0/2147768933 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7eff50024070 con 0x7eff58108520 2024-06-10T00:25:44.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.038+0000 7eff55ffb700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7eff480a2cb0 0x7eff480a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:44.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.038+0000 7eff5ccb6700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7eff480a2cb0 0x7eff480a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:44.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.038+0000 7eff55ffb700 1 -- 172.21.15.121:0/2147768933 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9179+0+0 (secure 0 0 0) 0x7eff5000f020 con 0x7eff58108520 2024-06-10T00:25:44.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.039+0000 7eff5ccb6700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7eff480a2cb0 0x7eff480a5170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7eff58108dd0 tx=0x7eff4c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:44.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.042+0000 7eff55ffb700 1 -- 172.21.15.121:0/2147768933 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7eff5002b030 con 0x7eff58108520 2024-06-10T00:25:44.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.248+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2147768933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff58199e70 con 0x7eff58108520 2024-06-10T00:25:44.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.249+0000 7eff55ffb700 1 -- 172.21.15.121:0/2147768933 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+24919 (secure 0 0 0) 0x7eff5009d340 con 0x7eff58108520 2024-06-10T00:25:44.250 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:25:44.250 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":100,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:25:44.074803+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:25:44.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2147768933 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7eff480a2cb0 msgr2=0x7eff480a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7eff480a2cb0 0x7eff480a5170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7eff58108dd0 tx=0x7eff4c00a380 comp rx=0 tx=0).stop 2024-06-10T00:25:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2147768933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff58108520 msgr2=0x7eff58198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff58108520 0x7eff58198990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7eff500062a0 tx=0x7eff5000be60 comp rx=0 tx=0).stop 2024-06-10T00:25:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2147768933 shutdown_connections 2024-06-10T00:25:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7eff480a2cb0 0x7eff480a5170 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff58108520 0x7eff58198990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7eff58102510 0x7eff5819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 --2- 172.21.15.121:0/2147768933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff58102ed0 0x7eff5819ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2147768933 >> 172.21.15.121:0/2147768933 conn(0x7eff58079560 msgr2=0x7eff58100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2147768933 shutdown_connections 2024-06-10T00:25:44.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:44.252+0000 7eff5ef1a700 1 -- 172.21.15.121:0/2147768933 wait complete. 2024-06-10T00:25:44.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-06-10T00:25:44.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-06-10T00:25:44.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2024-06-10T00:25:44.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:44 smithi193 ceph-mon[36716]: osdmap e99: 12 total, 12 up, 12 in 2024-06-10T00:25:44.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-06-10T00:25:44.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-06-10T00:25:44.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2024-06-10T00:25:44.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:44 smithi121 ceph-mon[32378]: osdmap e99: 12 total, 12 up, 12 in 2024-06-10T00:25:44.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-06-10T00:25:44.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-06-10T00:25:44.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2024-06-10T00:25:44.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:44 smithi135 ceph-mon[36424]: osdmap e99: 12 total, 12 up, 12 in 2024-06-10T00:25:44.973 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-10T00:25:44.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:25:45.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:45 smithi121 ceph-mon[32378]: osdmap e100: 12 total, 12 up, 12 in 2024-06-10T00:25:45.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:45 smithi121 ceph-mon[32378]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:25:45.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:45 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2147768933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:25:45.219 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:45 smithi193 ceph-mon[36716]: osdmap e100: 12 total, 12 up, 12 in 2024-06-10T00:25:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:45 smithi193 ceph-mon[36716]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:25:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:45 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2147768933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:25:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:45 smithi135 ceph-mon[36424]: osdmap e100: 12 total, 12 up, 12 in 2024-06-10T00:25:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:45 smithi135 ceph-mon[36424]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:25:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:45 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2147768933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:25:46.370 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:46 smithi121 ceph-mon[32378]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-06-10T00:25:46.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:46 smithi135 ceph-mon[36424]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-06-10T00:25:46.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:46 smithi193 ceph-mon[36716]: Health check failed: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2024-06-10T00:25:46.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.943+0000 7f6fdf12a700 1 -- 172.21.15.121:0/1370692967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8100a80 msgr2=0x7f6fd8100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.943+0000 7f6fdf12a700 1 --2- 172.21.15.121:0/1370692967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8100a80 0x7f6fd8100f00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0009990 tx=0x7f6fc0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:25:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.943+0000 7f6fdf12a700 1 -- 172.21.15.121:0/1370692967 shutdown_connections 2024-06-10T00:25:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.943+0000 7f6fdf12a700 1 --2- 172.21.15.121:0/1370692967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fd8101440 0x7f6fd810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.943+0000 7f6fdf12a700 1 --2- 172.21.15.121:0/1370692967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fd8108510 0x7f6fd81088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.943+0000 7f6fdf12a700 1 --2- 172.21.15.121:0/1370692967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8100a80 0x7f6fd8100f00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.943+0000 7f6fdf12a700 1 -- 172.21.15.121:0/1370692967 >> 172.21.15.121:0/1370692967 conn(0x7f6fd8077a00 msgr2=0x7f6fd80fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.943+0000 7f6fdf12a700 1 -- 172.21.15.121:0/1370692967 shutdown_connections 2024-06-10T00:25:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.943+0000 7f6fdf12a700 1 -- 172.21.15.121:0/1370692967 wait complete. 2024-06-10T00:25:46.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdf12a700 1 Processor -- start 2024-06-10T00:25:46.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdf12a700 1 -- start start 2024-06-10T00:25:46.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdf12a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fd8100a80 0x7f6fd819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:46.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdf12a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fd8101440 0x7f6fd819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:46.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdf12a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8108510 0x7f6fd8198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:46.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdf12a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd81127f0 con 0x7f6fd8108510 2024-06-10T00:25:46.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdf12a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8112670 con 0x7f6fd8101440 2024-06-10T00:25:46.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdf12a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd8112970 con 0x7f6fd8100a80 2024-06-10T00:25:46.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdd6c7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8108510 0x7f6fd8198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:46.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdd6c7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8108510 0x7f6fd8198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38328/0 (socket says 172.21.15.121:38328) 2024-06-10T00:25:46.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdd6c7700 1 -- 172.21.15.121:0/3016028741 learned_addr learned my addr 172.21.15.121:0/3016028741 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:46.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fd7fff700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fd8101440 0x7f6fd819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdcec6700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fd8100a80 0x7f6fd819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdd6c7700 1 -- 172.21.15.121:0/3016028741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fd8100a80 msgr2=0x7f6fd819e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdd6c7700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fd8100a80 0x7f6fd819e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdd6c7700 1 -- 172.21.15.121:0/3016028741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fd8101440 msgr2=0x7f6fd819ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdd6c7700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fd8101440 0x7f6fd819ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.944+0000 7f6fdd6c7700 1 -- 172.21.15.121:0/3016028741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd8199170 con 0x7f6fd8108510 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.945+0000 7f6fdd6c7700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8108510 0x7f6fd8198a10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6fc800b970 tx=0x7f6fc800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.945+0000 7f6fd5ffb700 1 -- 172.21.15.121:0/3016028741 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8019070 con 0x7f6fd8108510 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.945+0000 7f6fdf12a700 1 -- 172.21.15.121:0/3016028741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fd8199460 con 0x7f6fd8108510 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.945+0000 7f6fd5ffb700 1 -- 172.21.15.121:0/3016028741 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fc800ce10 con 0x7f6fd8108510 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.945+0000 7f6fd5ffb700 1 -- 172.21.15.121:0/3016028741 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8007370 con 0x7f6fd8108510 2024-06-10T00:25:46.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.945+0000 7f6fdf12a700 1 -- 172.21.15.121:0/3016028741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fd8101b90 con 0x7f6fd8108510 2024-06-10T00:25:46.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.946+0000 7f6fdf12a700 1 -- 172.21.15.121:0/3016028741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fd8199f40 con 0x7f6fd8108510 2024-06-10T00:25:46.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.947+0000 7f6fd5ffb700 1 -- 172.21.15.121:0/3016028741 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f6fc801c070 con 0x7f6fd8108510 2024-06-10T00:25:46.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.948+0000 7f6fd5ffb700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6fc40a2cb0 0x7f6fc40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:46.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.948+0000 7f6fdcec6700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6fc40a2cb0 0x7f6fc40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:46.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.948+0000 7f6fd5ffb700 1 -- 172.21.15.121:0/3016028741 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f6fc80ce830 con 0x7f6fd8108510 2024-06-10T00:25:46.950 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.948+0000 7f6fdcec6700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6fc40a2cb0 0x7f6fc40a5170 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0005ea0 tx=0x7f6fd0005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:46.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:46.952+0000 7f6fd5ffb700 1 -- 172.21.15.121:0/3016028741 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6fc809b390 con 0x7f6fd8108510 2024-06-10T00:25:47.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.150+0000 7f6fdf12a700 1 -- 172.21.15.121:0/3016028741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6fd802dca0 con 0x7f6fd8108510 2024-06-10T00:25:47.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.152+0000 7f6fd5ffb700 1 -- 172.21.15.121:0/3016028741 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+24919 (secure 0 0 0) 0x7f6fc809ac90 con 0x7f6fd8108510 2024-06-10T00:25:47.152 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:25:47.152 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":100,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:25:44.074803+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:25:47.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 -- 172.21.15.121:0/3016028741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6fc40a2cb0 msgr2=0x7f6fc40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:47.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6fc40a2cb0 0x7f6fc40a5170 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0005ea0 tx=0x7f6fd0005e30 comp rx=0 tx=0).stop 2024-06-10T00:25:47.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 -- 172.21.15.121:0/3016028741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8108510 msgr2=0x7f6fd8198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:47.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8108510 0x7f6fd8198a10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6fc800b970 tx=0x7f6fc800bc80 comp rx=0 tx=0).stop 2024-06-10T00:25:47.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 -- 172.21.15.121:0/3016028741 shutdown_connections 2024-06-10T00:25:47.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f6fc40a2cb0 0x7f6fc40a5170 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:47.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fd8108510 0x7f6fd8198a10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:47.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fd8100a80 0x7f6fd819e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:47.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 --2- 172.21.15.121:0/3016028741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fd8101440 0x7f6fd819ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:47.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 -- 172.21.15.121:0/3016028741 >> 172.21.15.121:0/3016028741 conn(0x7f6fd8077a00 msgr2=0x7f6fd80ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:47.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 -- 172.21.15.121:0/3016028741 shutdown_connections 2024-06-10T00:25:47.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:47.155+0000 7f6fdf12a700 1 -- 172.21.15.121:0/3016028741 wait complete. 2024-06-10T00:25:47.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:47 smithi135 ceph-mon[36424]: pgmap v362: 129 pgs: 2 activating, 127 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:25:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:47 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3016028741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:25:47.590 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:47 smithi121 ceph-mon[32378]: pgmap v362: 129 pgs: 2 activating, 127 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:25:47.590 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:47 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3016028741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:25:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:47 smithi193 ceph-mon[36716]: pgmap v362: 129 pgs: 2 activating, 127 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:25:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:47 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3016028741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:25:47.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-10T00:25:47.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-10T00:25:47.851 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-10T00:25:47.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-10T00:25:47.852 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-10T00:25:47.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-10T00:25:47.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-10T00:25:47.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-10T00:25:47.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-06-10T00:25:47.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-06-10T00:25:47.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-06-10T00:25:47.855 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-06-10T00:25:48.496 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:48.497 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:49.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:49 smithi135 ceph-mon[36424]: pgmap v363: 129 pgs: 2 activating, 127 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-06-10T00:25:49.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:49 smithi193 ceph-mon[36716]: pgmap v363: 129 pgs: 2 activating, 127 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-06-10T00:25:49.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:49 smithi121 ceph-mon[32378]: pgmap v363: 129 pgs: 2 activating, 127 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-06-10T00:25:49.899 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:49.904 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:49.906 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:49.907 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:49.907 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:49.909 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:49.909 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:50.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:50 smithi121 ceph-mon[32378]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-06-10T00:25:50.580 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:50 smithi121 ceph-mon[32378]: Cluster is now healthy 2024-06-10T00:25:50.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:50 smithi193 ceph-mon[36716]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-06-10T00:25:50.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:50 smithi193 ceph-mon[36716]: Cluster is now healthy 2024-06-10T00:25:50.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:50 smithi135 ceph-mon[36424]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive) 2024-06-10T00:25:50.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:50 smithi135 ceph-mon[36424]: Cluster is now healthy 2024-06-10T00:25:51.532 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:51 smithi193 ceph-mon[36716]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:25:51.532 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:51 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:25:51.613 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:51 smithi121 ceph-mon[32378]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:25:51.614 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:51 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:25:51.658 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:51 smithi135 ceph-mon[36424]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:25:51.659 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:51 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:25:52.047 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:52.047 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:52.048 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:25:53.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:53 smithi121 ceph-mon[32378]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:25:53.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:53 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:53.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:53 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:53.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:53 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:53.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:53 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:53.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:53 smithi193 ceph-mon[36716]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:25:53.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:53 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:53.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:53 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:53.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:53 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:53.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:53 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:53.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:53 smithi135 ceph-mon[36424]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:25:53.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:53 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:53.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:53 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:53.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:53 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:53.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:53 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:25:55.443 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:55 smithi121 ceph-mon[32378]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-06-10T00:25:55.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:55 smithi193 ceph-mon[36716]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-06-10T00:25:55.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:55 smithi135 ceph-mon[36424]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-06-10T00:25:57.392 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:57 smithi121 ceph-mon[32378]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:25:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.689+0000 7f4c87116700 1 -- 172.21.15.121:0/407390126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c80101df0 msgr2=0x7f4c801021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.689+0000 7f4c87116700 1 --2- 172.21.15.121:0/407390126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c80101df0 0x7f4c801021d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4c740099c0 tx=0x7f4c74009cd0 comp rx=0 tx=0).stop 2024-06-10T00:25:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.689+0000 7f4c87116700 1 -- 172.21.15.121:0/407390126 shutdown_connections 2024-06-10T00:25:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.689+0000 7f4c87116700 1 --2- 172.21.15.121:0/407390126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c80106750 0x7f4c8010e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.689+0000 7f4c87116700 1 --2- 172.21.15.121:0/407390126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c80101df0 0x7f4c801021d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.689+0000 7f4c87116700 1 --2- 172.21.15.121:0/407390126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c801027a0 0x7f4c80106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.689+0000 7f4c87116700 1 -- 172.21.15.121:0/407390126 >> 172.21.15.121:0/407390126 conn(0x7f4c80077a00 msgr2=0x7f4c800fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.689+0000 7f4c87116700 1 -- 172.21.15.121:0/407390126 shutdown_connections 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.689+0000 7f4c87116700 1 -- 172.21.15.121:0/407390126 wait complete. 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.689+0000 7f4c87116700 1 Processor -- start 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c87116700 1 -- start start 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c87116700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c80101df0 0x7f4c80199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c87116700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c801027a0 0x7f4c80199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c87116700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c80106750 0x7f4c801a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c87116700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4c801116e0 con 0x7f4c80106750 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c87116700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4c80111560 con 0x7f4c801027a0 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c87116700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4c80111860 con 0x7f4c80101df0 2024-06-10T00:25:57.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c856b3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c80106750 0x7f4c801a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c856b3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c80106750 0x7f4c801a4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53246/0 (socket says 172.21.15.121:53246) 2024-06-10T00:25:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c856b3700 1 -- 172.21.15.121:0/222579475 learned_addr learned my addr 172.21.15.121:0/222579475 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:25:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c84eb2700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c80101df0 0x7f4c80199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c7ffff700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c801027a0 0x7f4c80199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c856b3700 1 -- 172.21.15.121:0/222579475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c80101df0 msgr2=0x7f4c80199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c856b3700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4c80101df0 0x7f4c80199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:57.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c856b3700 1 -- 172.21.15.121:0/222579475 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c801027a0 msgr2=0x7f4c80199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c856b3700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c801027a0 0x7f4c80199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c856b3700 1 -- 172.21.15.121:0/222579475 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c801a5020 con 0x7f4c80106750 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c7ffff700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4c801027a0 0x7f4c80199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.690+0000 7f4c856b3700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c80106750 0x7f4c801a4ae0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4c78007ee0 tx=0x7f4c7800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.691+0000 7f4c7dffb700 1 -- 172.21.15.121:0/222579475 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7800dbd0 con 0x7f4c80106750 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.691+0000 7f4c7dffb700 1 -- 172.21.15.121:0/222579475 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c7800dd70 con 0x7f4c80106750 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.691+0000 7f4c87116700 1 -- 172.21.15.121:0/222579475 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c801a5250 con 0x7f4c80106750 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.691+0000 7f4c7dffb700 1 -- 172.21.15.121:0/222579475 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c78005a60 con 0x7f4c80106750 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.691+0000 7f4c87116700 1 -- 172.21.15.121:0/222579475 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c801a5790 con 0x7f4c80106750 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.692+0000 7f4c87116700 1 -- 172.21.15.121:0/222579475 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4c80062c20 con 0x7f4c80106750 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.693+0000 7f4c7dffb700 1 -- 172.21.15.121:0/222579475 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f4c7801a020 con 0x7f4c80106750 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.693+0000 7f4c7dffb700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4c700a2d30 0x7f4c700a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:57.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.694+0000 7f4c84eb2700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4c700a2d30 0x7f4c700a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.694+0000 7f4c7dffb700 1 -- 172.21.15.121:0/222579475 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f4c780cf320 con 0x7f4c80106750 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.694+0000 7f4c7dffb700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186] conn(0x7f4c700ae010 0x7f4c700b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.694+0000 7f4c7dffb700 1 -- 172.21.15.121:0/222579475 --> [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c700b0b80 con 0x7f4c700ae010 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.694+0000 7f4c7dffb700 1 -- 172.21.15.121:0/222579475 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f4c780cf770 con 0x7f4c80106750 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.694+0000 7f4c84eb2700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4c700a2d30 0x7f4c700a51f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4c74000c00 tx=0x7f4c74005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.694+0000 7f4c7ffff700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186] conn(0x7f4c700ae010 0x7f4c700b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.695+0000 7f4c7ffff700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186] conn(0x7f4c700ae010 0x7f4c700b04b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.695+0000 7f4c7dffb700 1 -- 172.21.15.121:0/222579475 <== osd.8 v2:172.21.15.193:6816/3981402186 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4c700b0b80 con 0x7f4c700ae010 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.711+0000 7f4c87116700 1 -- 172.21.15.121:0/222579475 --> [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4c800630a0 con 0x7f4c700ae010 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.712+0000 7f4c7dffb700 1 -- 172.21.15.121:0/222579475 <== osd.8 v2:172.21.15.193:6816/3981402186 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4c800630a0 con 0x7f4c700ae010 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.712+0000 7f4c87116700 1 -- 172.21.15.121:0/222579475 >> [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186] conn(0x7f4c700ae010 msgr2=0x7f4c700b04b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.712+0000 7f4c87116700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186] conn(0x7f4c700ae010 0x7f4c700b04b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.712+0000 7f4c87116700 1 -- 172.21.15.121:0/222579475 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4c700a2d30 msgr2=0x7f4c700a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.712+0000 7f4c87116700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f4c700a2d30 0x7f4c700a51f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4c74000c00 tx=0x7f4c74005fb0 comp rx=0 tx=0).stop 2024-06-10T00:25:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.712+0000 7f4c87116700 1 -- 172.21.15.121:0/222579475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c80106750 msgr2=0x7f4c801a4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:25:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.712+0000 7f4c87116700 1 --2- 172.21.15.121:0/222579475 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c80106750 0x7f4c801a4ae0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4c78007ee0 tx=0x7f4c7800d3b0 comp rx=0 tx=0).stop 2024-06-10T00:25:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.712+0000 7f4c856b3700 1 -- 172.21.15.121:0/222579475 reap_dead start 2024-06-10T00:25:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.713+0000 7f4c87116700 1 -- 172.21.15.121:0/222579475 shutdown_connections 2024-06-10T00:25:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.713+0000 7f4c87116700 1 -- 172.21.15.121:0/222579475 >> 172.21.15.121:0/222579475 conn(0x7f4c80077a00 msgr2=0x7f4c800fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:25:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.713+0000 7f4c87116700 1 -- 172.21.15.121:0/222579475 shutdown_connections 2024-06-10T00:25:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:25:55.713+0000 7f4c87116700 1 -- 172.21.15.121:0/222579475 wait complete. 2024-06-10T00:25:57.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:57 smithi193 ceph-mon[36716]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:25:57.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:57 smithi135 ceph-mon[36424]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:25:58.600 INFO:teuthology.orchestra.run.smithi121.stdout:257698037815 2024-06-10T00:25:58.600 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-06-10T00:25:59.598 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:25:59 smithi121 ceph-mon[32378]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:25:59.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:25:59 smithi193 ceph-mon[36716]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:25:59.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:25:59 smithi135 ceph-mon[36424]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:01.611 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:01 smithi121 ceph-mon[32378]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:01.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:01 smithi193 ceph-mon[36716]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:01.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:01 smithi135 ceph-mon[36424]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:02.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.147+0000 7fa49cb5f700 1 -- 172.21.15.121:0/1306337358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa49806ab30 msgr2=0x7fa49810f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.147+0000 7fa49cb5f700 1 --2- 172.21.15.121:0/1306337358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa49806ab30 0x7fa49810f320 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa49000b0a0 tx=0x7fa49000b3b0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.147+0000 7fa49cb5f700 1 -- 172.21.15.121:0/1306337358 shutdown_connections 2024-06-10T00:26:02.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.147+0000 7fa49cb5f700 1 --2- 172.21.15.121:0/1306337358 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa49806ab30 0x7fa49810f320 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.147+0000 7fa49cb5f700 1 --2- 172.21.15.121:0/1306337358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa498108500 0x7fa4981088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.147+0000 7fa49cb5f700 1 --2- 172.21.15.121:0/1306337358 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa49806a170 0x7fa49806a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.147+0000 7fa49cb5f700 1 -- 172.21.15.121:0/1306337358 >> 172.21.15.121:0/1306337358 conn(0x7fa498078340 msgr2=0x7fa498078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:02.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.147+0000 7fa49cb5f700 1 -- 172.21.15.121:0/1306337358 shutdown_connections 2024-06-10T00:26:02.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.147+0000 7fa49cb5f700 1 -- 172.21.15.121:0/1306337358 wait complete. 2024-06-10T00:26:02.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa49cb5f700 1 Processor -- start 2024-06-10T00:26:02.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa49cb5f700 1 -- start start 2024-06-10T00:26:02.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa49cb5f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa49806a170 0x7fa49819e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa49cb5f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa49806ab30 0x7fa49819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa49cb5f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa498108500 0x7fa498198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa49cb5f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa498112780 con 0x7fa498108500 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa49cb5f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fa498112600 con 0x7fa49806ab30 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa49cb5f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa498112900 con 0x7fa49806a170 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa496d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa498108500 0x7fa498198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa49659c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa49806a170 0x7fa49819e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa496d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa498108500 0x7fa498198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46344/0 (socket says 172.21.15.121:46344) 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa496d9d700 1 -- 172.21.15.121:0/1569692967 learned_addr learned my addr 172.21.15.121:0/1569692967 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa48dd9b700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa49806ab30 0x7fa49819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa496d9d700 1 -- 172.21.15.121:0/1569692967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa49806a170 msgr2=0x7fa49819e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa496d9d700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa49806a170 0x7fa49819e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa496d9d700 1 -- 172.21.15.121:0/1569692967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa49806ab30 msgr2=0x7fa49819edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa496d9d700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa49806ab30 0x7fa49819edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.148+0000 7fa496d9d700 1 -- 172.21.15.121:0/1569692967 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa498199220 con 0x7fa498108500 2024-06-10T00:26:02.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.149+0000 7fa496d9d700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa498108500 0x7fa498198ac0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa4900062a0 tx=0x7fa490004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:02.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.149+0000 7fa48ffff700 1 -- 172.21.15.121:0/1569692967 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490021070 con 0x7fa498108500 2024-06-10T00:26:02.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.149+0000 7fa49cb5f700 1 -- 172.21.15.121:0/1569692967 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4981994b0 con 0x7fa498108500 2024-06-10T00:26:02.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.149+0000 7fa48ffff700 1 -- 172.21.15.121:0/1569692967 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa490003c00 con 0x7fa498108500 2024-06-10T00:26:02.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.149+0000 7fa48ffff700 1 -- 172.21.15.121:0/1569692967 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490004880 con 0x7fa498108500 2024-06-10T00:26:02.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.149+0000 7fa49cb5f700 1 -- 172.21.15.121:0/1569692967 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4981a5780 con 0x7fa498108500 2024-06-10T00:26:02.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.150+0000 7fa48d59a700 1 -- 172.21.15.121:0/1569692967 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa47c001030 con 0x7fa498108500 2024-06-10T00:26:02.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.151+0000 7fa48ffff700 1 -- 172.21.15.121:0/1569692967 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fa49001d450 con 0x7fa498108500 2024-06-10T00:26:02.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.152+0000 7fa48ffff700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa4840a2d30 0x7fa4840a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:02.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.152+0000 7fa49659c700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa4840a2d30 0x7fa4840a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:02.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.152+0000 7fa48ffff700 1 -- 172.21.15.121:0/1569692967 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fa49000f020 con 0x7fa498108500 2024-06-10T00:26:02.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.153+0000 7fa48ffff700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040] conn(0x7fa4840ae010 0x7fa4840b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:02.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.153+0000 7fa48ffff700 1 -- 172.21.15.121:0/1569692967 --> [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa49001fe50 con 0x7fa4840ae010 2024-06-10T00:26:02.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.153+0000 7fa48ffff700 1 -- 172.21.15.121:0/1569692967 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7fa4900d1140 con 0x7fa498108500 2024-06-10T00:26:02.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.153+0000 7fa49659c700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa4840a2d30 0x7fa4840a51f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa4980ffa10 tx=0x7fa48800b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:02.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.153+0000 7fa48dd9b700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040] conn(0x7fa4840ae010 0x7fa4840b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:02.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.153+0000 7fa48dd9b700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040] conn(0x7fa4840ae010 0x7fa4840b04b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:02.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.154+0000 7fa48ffff700 1 -- 172.21.15.121:0/1569692967 <== osd.0 v2:172.21.15.121:6802/1987481040 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa49001fe50 con 0x7fa4840ae010 2024-06-10T00:26:02.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.170+0000 7fa48d59a700 1 -- 172.21.15.121:0/1569692967 --> [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa47c002e60 con 0x7fa4840ae010 2024-06-10T00:26:02.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa48ffff700 1 -- 172.21.15.121:0/1569692967 <== osd.0 v2:172.21.15.121:6802/1987481040 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa47c002e60 con 0x7fa4840ae010 2024-06-10T00:26:02.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa48d59a700 1 -- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040] conn(0x7fa4840ae010 msgr2=0x7fa4840b04b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa48d59a700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040] conn(0x7fa4840ae010 0x7fa4840b04b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa48d59a700 1 -- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa4840a2d30 msgr2=0x7fa4840a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa48d59a700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa4840a2d30 0x7fa4840a51f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa4980ffa10 tx=0x7fa48800b3c0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa48d59a700 1 -- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa498108500 msgr2=0x7fa498198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa48d59a700 1 --2- 172.21.15.121:0/1569692967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa498108500 0x7fa498198ac0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa4900062a0 tx=0x7fa490004d70 comp rx=0 tx=0).stop 2024-06-10T00:26:02.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa496d9d700 1 -- 172.21.15.121:0/1569692967 reap_dead start 2024-06-10T00:26:02.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa48d59a700 1 -- 172.21.15.121:0/1569692967 shutdown_connections 2024-06-10T00:26:02.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa48d59a700 1 -- 172.21.15.121:0/1569692967 >> 172.21.15.121:0/1569692967 conn(0x7fa498078340 msgr2=0x7fa498100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:02.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa48d59a700 1 -- 172.21.15.121:0/1569692967 shutdown_connections 2024-06-10T00:26:02.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.171+0000 7fa48d59a700 1 -- 172.21.15.121:0/1569692967 wait complete. 2024-06-10T00:26:02.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.365+0000 7fa121baf700 1 -- 172.21.15.121:0/3051995728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa11c102740 msgr2=0x7fa11c102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.365+0000 7fa121baf700 1 --2- 172.21.15.121:0/3051995728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa11c102740 0x7fa11c102b20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0099c0 tx=0x7fa10c009cd0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.366+0000 7fa121baf700 1 -- 172.21.15.121:0/3051995728 shutdown_connections 2024-06-10T00:26:02.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.366+0000 7fa121baf700 1 --2- 172.21.15.121:0/3051995728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa11c107840 0x7fa11c10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.366+0000 7fa121baf700 1 --2- 172.21.15.121:0/3051995728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa11c102740 0x7fa11c102b20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.366+0000 7fa121baf700 1 --2- 172.21.15.121:0/3051995728 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa11c1030f0 0x7fa11c107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.366+0000 7fa121baf700 1 -- 172.21.15.121:0/3051995728 >> 172.21.15.121:0/3051995728 conn(0x7fa11c0795f0 msgr2=0x7fa11c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.366+0000 7fa121baf700 1 -- 172.21.15.121:0/3051995728 shutdown_connections 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.366+0000 7fa121baf700 1 -- 172.21.15.121:0/3051995728 wait complete. 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.366+0000 7fa121baf700 1 Processor -- start 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.366+0000 7fa121baf700 1 -- start start 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa121baf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa11c102740 0x7fa11c19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa121baf700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa11c1030f0 0x7fa11c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa121baf700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa11c107840 0x7fa11c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa121baf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c112760 con 0x7fa11c102740 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa121baf700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c1125e0 con 0x7fa11c107840 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa121baf700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa11c1128e0 con 0x7fa11c1030f0 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa11c102740 0x7fa11c19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa11c102740 0x7fa11c19e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46348/0 (socket says 172.21.15.121:46348) 2024-06-10T00:26:02.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11b7fe700 1 -- 172.21.15.121:0/3897318783 learned_addr learned my addr 172.21.15.121:0/3897318783 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11affd700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa11c1030f0 0x7fa11c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11bfff700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa11c107840 0x7fa11c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11b7fe700 1 -- 172.21.15.121:0/3897318783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa11c1030f0 msgr2=0x7fa11c19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11b7fe700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa11c1030f0 0x7fa11c19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11b7fe700 1 -- 172.21.15.121:0/3897318783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa11c107840 msgr2=0x7fa11c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11b7fe700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa11c107840 0x7fa11c198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11b7fe700 1 -- 172.21.15.121:0/3897318783 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11c199200 con 0x7fa11c102740 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11bfff700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa11c107840 0x7fa11c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11affd700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa11c1030f0 0x7fa11c19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa11b7fe700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa11c102740 0x7fa11c19e8e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa10c006010 tx=0x7fa10c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa118ff9700 1 -- 172.21.15.121:0/3897318783 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c027070 con 0x7fa11c102740 2024-06-10T00:26:02.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa121baf700 1 -- 172.21.15.121:0/3897318783 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa11c199490 con 0x7fa11c102740 2024-06-10T00:26:02.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa118ff9700 1 -- 172.21.15.121:0/3897318783 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa10c004050 con 0x7fa11c102740 2024-06-10T00:26:02.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa118ff9700 1 -- 172.21.15.121:0/3897318783 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa10c022e10 con 0x7fa11c102740 2024-06-10T00:26:02.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.367+0000 7fa121baf700 1 -- 172.21.15.121:0/3897318783 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa11c1a57b0 con 0x7fa11c102740 2024-06-10T00:26:02.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.368+0000 7fa121baf700 1 -- 172.21.15.121:0/3897318783 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa11c062c20 con 0x7fa11c102740 2024-06-10T00:26:02.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.369+0000 7fa118ff9700 1 -- 172.21.15.121:0/3897318783 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fa10c02a070 con 0x7fa11c102740 2024-06-10T00:26:02.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.370+0000 7fa118ff9700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa1040a2cb0 0x7fa1040a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:02.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.370+0000 7fa11affd700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa1040a2cb0 0x7fa1040a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:02.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.370+0000 7fa118ff9700 1 -- 172.21.15.121:0/3897318783 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fa10c0d0b50 con 0x7fa11c102740 2024-06-10T00:26:02.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.370+0000 7fa118ff9700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681] conn(0x7fa1040adf90 0x7fa1040b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:02.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.370+0000 7fa118ff9700 1 -- 172.21.15.121:0/3897318783 --> [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa10c00b0b0 con 0x7fa1040adf90 2024-06-10T00:26:02.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.370+0000 7fa118ff9700 1 -- 172.21.15.121:0/3897318783 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7fa10c0d0fa0 con 0x7fa11c102740 2024-06-10T00:26:02.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.371+0000 7fa11bfff700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681] conn(0x7fa1040adf90 0x7fa1040b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:02.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.371+0000 7fa11affd700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa1040a2cb0 0x7fa1040a5170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa110005950 tx=0x7fa1100058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:02.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.371+0000 7fa11bfff700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681] conn(0x7fa1040adf90 0x7fa1040b0430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:02.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.372+0000 7fa118ff9700 1 -- 172.21.15.121:0/3897318783 <== osd.6 v2:172.21.15.121:6818/2905970681 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa10c00b0b0 con 0x7fa1040adf90 2024-06-10T00:26:02.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.388+0000 7fa121baf700 1 -- 172.21.15.121:0/3897318783 --> [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa11c063020 con 0x7fa1040adf90 2024-06-10T00:26:02.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.388+0000 7fa118ff9700 1 -- 172.21.15.121:0/3897318783 <== osd.6 v2:172.21.15.121:6818/2905970681 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa11c063020 con 0x7fa1040adf90 2024-06-10T00:26:02.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.389+0000 7fa121baf700 1 -- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681] conn(0x7fa1040adf90 msgr2=0x7fa1040b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.389+0000 7fa121baf700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681] conn(0x7fa1040adf90 0x7fa1040b0430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.389+0000 7fa121baf700 1 -- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa1040a2cb0 msgr2=0x7fa1040a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.389+0000 7fa121baf700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa1040a2cb0 0x7fa1040a5170 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa110005950 tx=0x7fa1100058e0 comp rx=0 tx=0).stop 2024-06-10T00:26:02.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.389+0000 7fa121baf700 1 -- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa11c102740 msgr2=0x7fa11c19e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:02.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.389+0000 7fa121baf700 1 --2- 172.21.15.121:0/3897318783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa11c102740 0x7fa11c19e8e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa10c006010 tx=0x7fa10c009f80 comp rx=0 tx=0).stop 2024-06-10T00:26:02.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.389+0000 7fa11bfff700 1 -- 172.21.15.121:0/3897318783 reap_dead start 2024-06-10T00:26:02.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.389+0000 7fa121baf700 1 -- 172.21.15.121:0/3897318783 shutdown_connections 2024-06-10T00:26:02.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.389+0000 7fa121baf700 1 -- 172.21.15.121:0/3897318783 >> 172.21.15.121:0/3897318783 conn(0x7fa11c0795f0 msgr2=0x7fa11c101ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:02.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.389+0000 7fa121baf700 1 -- 172.21.15.121:0/3897318783 shutdown_connections 2024-06-10T00:26:02.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:02.389+0000 7fa121baf700 1 -- 172.21.15.121:0/3897318783 wait complete. 2024-06-10T00:26:02.578 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:02 smithi121 ceph-mon[32378]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-06-10T00:26:02.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:02 smithi193 ceph-mon[36716]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-06-10T00:26:02.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:02 smithi135 ceph-mon[36424]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-06-10T00:26:03.985 INFO:teuthology.orchestra.run.smithi121.stdout:201863462978 2024-06-10T00:26:03.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-10T00:26:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.131+0000 7fc55942f700 1 -- 172.21.15.121:0/938919758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc554101370 msgr2=0x7fc5541017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.131+0000 7fc55942f700 1 --2- 172.21.15.121:0/938919758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc554101370 0x7fc5541017f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc544009990 tx=0x7fc544009ca0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.131+0000 7fc55942f700 1 -- 172.21.15.121:0/938919758 shutdown_connections 2024-06-10T00:26:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.131+0000 7fc55942f700 1 --2- 172.21.15.121:0/938919758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc554101d30 0x7fc55410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.131+0000 7fc55942f700 1 --2- 172.21.15.121:0/938919758 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc554107420 0x7fc554107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.131+0000 7fc55942f700 1 --2- 172.21.15.121:0/938919758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc554101370 0x7fc5541017f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.131+0000 7fc55942f700 1 -- 172.21.15.121:0/938919758 >> 172.21.15.121:0/938919758 conn(0x7fc5540783a0 msgr2=0x7fc5540787b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.131+0000 7fc55942f700 1 -- 172.21.15.121:0/938919758 shutdown_connections 2024-06-10T00:26:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.131+0000 7fc55942f700 1 -- 172.21.15.121:0/938919758 wait complete. 2024-06-10T00:26:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.132+0000 7fc55942f700 1 Processor -- start 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.132+0000 7fc55942f700 1 -- start start 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc55942f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc554101370 0x7fc5541999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc55942f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc554101d30 0x7fc554199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc55942f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc554107420 0x7fc5541a4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc55942f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc554111660 con 0x7fc554101d30 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc55942f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc5541114e0 con 0x7fc554107420 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc55942f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc5541117e0 con 0x7fc554101370 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc5527fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc554101d30 0x7fc554199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc5527fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc554101d30 0x7fc554199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46334/0 (socket says 172.21.15.121:46334) 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc5527fc700 1 -- 172.21.15.121:0/3704926192 learned_addr learned my addr 172.21.15.121:0/3704926192 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc552ffd700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc554101370 0x7fc5541999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc5537fe700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc554107420 0x7fc5541a4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc5527fc700 1 -- 172.21.15.121:0/3704926192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc554101370 msgr2=0x7fc5541999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc5527fc700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc554101370 0x7fc5541999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc5527fc700 1 -- 172.21.15.121:0/3704926192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc554107420 msgr2=0x7fc5541a4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc5527fc700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc554107420 0x7fc5541a4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc5527fc700 1 -- 172.21.15.121:0/3704926192 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5541a4f80 con 0x7fc554101d30 2024-06-10T00:26:04.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc5537fe700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc554107420 0x7fc5541a4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:26:04.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc5527fc700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc554101d30 0x7fc554199f00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc544000c00 tx=0x7fc5440047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:04.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc54bfff700 1 -- 172.21.15.121:0/3704926192 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc544027070 con 0x7fc554101d30 2024-06-10T00:26:04.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc54bfff700 1 -- 172.21.15.121:0/3704926192 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc544003780 con 0x7fc554101d30 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc55942f700 1 -- 172.21.15.121:0/3704926192 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5541a51b0 con 0x7fc554101d30 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc54bfff700 1 -- 172.21.15.121:0/3704926192 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc544022dd0 con 0x7fc554101d30 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.133+0000 7fc55942f700 1 -- 172.21.15.121:0/3704926192 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5541a5660 con 0x7fc554101d30 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.134+0000 7fc55942f700 1 -- 172.21.15.121:0/3704926192 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc554062be0 con 0x7fc554101d30 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.135+0000 7fc54bfff700 1 -- 172.21.15.121:0/3704926192 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc54402a070 con 0x7fc554101d30 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.136+0000 7fc54bfff700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc5400a2cb0 0x7fc5400a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.136+0000 7fc552ffd700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc5400a2cb0 0x7fc5400a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.136+0000 7fc54bfff700 1 -- 172.21.15.121:0/3704926192 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fc5440d0860 con 0x7fc554101d30 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.136+0000 7fc54bfff700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287] conn(0x7fc5400adf90 0x7fc5400b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.136+0000 7fc54bfff700 1 -- 172.21.15.121:0/3704926192 --> [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc544016e70 con 0x7fc5400adf90 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.136+0000 7fc54bfff700 1 -- 172.21.15.121:0/3704926192 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7fc5440d0cb0 con 0x7fc554101d30 2024-06-10T00:26:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.137+0000 7fc552ffd700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc5400a2cb0 0x7fc5400a5170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc53c007b60 tx=0x7fc53c0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.137+0000 7fc5537fe700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287] conn(0x7fc5400adf90 0x7fc5400b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.137+0000 7fc5537fe700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287] conn(0x7fc5400adf90 0x7fc5400b0430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.138+0000 7fc54bfff700 1 -- 172.21.15.121:0/3704926192 <== osd.10 v2:172.21.15.135:6824/937452287 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc544016e70 con 0x7fc5400adf90 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.154+0000 7fc55942f700 1 -- 172.21.15.121:0/3704926192 --> [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc554063090 con 0x7fc5400adf90 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.154+0000 7fc54bfff700 1 -- 172.21.15.121:0/3704926192 <== osd.10 v2:172.21.15.135:6824/937452287 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc554063090 con 0x7fc5400adf90 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.155+0000 7fc55942f700 1 -- 172.21.15.121:0/3704926192 >> [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287] conn(0x7fc5400adf90 msgr2=0x7fc5400b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.155+0000 7fc55942f700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287] conn(0x7fc5400adf90 0x7fc5400b0430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.155+0000 7fc55942f700 1 -- 172.21.15.121:0/3704926192 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc5400a2cb0 msgr2=0x7fc5400a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.155+0000 7fc55942f700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc5400a2cb0 0x7fc5400a5170 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc53c007b60 tx=0x7fc53c0073d0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.155+0000 7fc55942f700 1 -- 172.21.15.121:0/3704926192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc554101d30 msgr2=0x7fc554199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.155+0000 7fc55942f700 1 --2- 172.21.15.121:0/3704926192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc554101d30 0x7fc554199f00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc544000c00 tx=0x7fc5440047b0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.155+0000 7fc5537fe700 1 -- 172.21.15.121:0/3704926192 reap_dead start 2024-06-10T00:26:04.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.155+0000 7fc55942f700 1 -- 172.21.15.121:0/3704926192 shutdown_connections 2024-06-10T00:26:04.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.155+0000 7fc55942f700 1 -- 172.21.15.121:0/3704926192 >> 172.21.15.121:0/3704926192 conn(0x7fc5540783a0 msgr2=0x7fc554100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:04.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.155+0000 7fc55942f700 1 -- 172.21.15.121:0/3704926192 shutdown_connections 2024-06-10T00:26:04.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:00.155+0000 7fc55942f700 1 -- 172.21.15.121:0/3704926192 wait complete. 2024-06-10T00:26:04.674 INFO:teuthology.orchestra.run.smithi121.stdout:313532612654 2024-06-10T00:26:04.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-06-10T00:26:04.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.880+0000 7f0d30437700 1 -- 172.21.15.121:0/1231890980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d28102ef0 msgr2=0x7f0d2810f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.880+0000 7f0d30437700 1 --2- 172.21.15.121:0/1231890980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d28102ef0 0x7f0d2810f3a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800b0d0 tx=0x7f0d1800b3e0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.880+0000 7f0d30437700 1 -- 172.21.15.121:0/1231890980 shutdown_connections 2024-06-10T00:26:04.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.880+0000 7f0d30437700 1 --2- 172.21.15.121:0/1231890980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d28102ef0 0x7f0d2810f3a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.880+0000 7f0d30437700 1 --2- 172.21.15.121:0/1231890980 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d281085d0 0x7f0d281089b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.880+0000 7f0d30437700 1 --2- 172.21.15.121:0/1231890980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d28102530 0x7f0d281029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.880+0000 7f0d30437700 1 -- 172.21.15.121:0/1231890980 >> 172.21.15.121:0/1231890980 conn(0x7f0d28079640 msgr2=0x7f0d28079a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:04.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.880+0000 7f0d30437700 1 -- 172.21.15.121:0/1231890980 shutdown_connections 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.880+0000 7f0d30437700 1 -- 172.21.15.121:0/1231890980 wait complete. 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d30437700 1 Processor -- start 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d30437700 1 -- start start 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d30437700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d28102530 0x7f0d2819e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d30437700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d28102ef0 0x7f0d2819ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d30437700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d281085d0 0x7f0d28198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d30437700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0d28112810 con 0x7f0d281085d0 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d30437700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f0d28112690 con 0x7f0d28102ef0 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d30437700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d28112990 con 0x7f0d28102530 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d2e9d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d281085d0 0x7f0d28198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d2e9d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d281085d0 0x7f0d28198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46358/0 (socket says 172.21.15.121:46358) 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d2e9d4700 1 -- 172.21.15.121:0/4037049006 learned_addr learned my addr 172.21.15.121:0/4037049006 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:04.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d2e1d3700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d28102530 0x7f0d2819e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:04.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d2d9d2700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d28102ef0 0x7f0d2819ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:04.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d2e9d4700 1 -- 172.21.15.121:0/4037049006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d28102530 msgr2=0x7f0d2819e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d2e9d4700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d28102530 0x7f0d2819e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d2e9d4700 1 -- 172.21.15.121:0/4037049006 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d28102ef0 msgr2=0x7f0d2819ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d2e9d4700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d28102ef0 0x7f0d2819ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d2e9d4700 1 -- 172.21.15.121:0/4037049006 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d281990b0 con 0x7f0d281085d0 2024-06-10T00:26:04.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.881+0000 7f0d2e1d3700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d28102530 0x7f0d2819e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:26:04.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.882+0000 7f0d2e9d4700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d281085d0 0x7f0d28198950 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0d18006070 tx=0x7f0d18004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:04.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.882+0000 7f0d277fe700 1 -- 172.21.15.121:0/4037049006 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d18021070 con 0x7f0d281085d0 2024-06-10T00:26:04.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.882+0000 7f0d30437700 1 -- 172.21.15.121:0/4037049006 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d28199340 con 0x7f0d281085d0 2024-06-10T00:26:04.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.882+0000 7f0d277fe700 1 -- 172.21.15.121:0/4037049006 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d18007df0 con 0x7f0d281085d0 2024-06-10T00:26:04.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.882+0000 7f0d277fe700 1 -- 172.21.15.121:0/4037049006 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d18005470 con 0x7f0d281085d0 2024-06-10T00:26:04.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.882+0000 7f0d30437700 1 -- 172.21.15.121:0/4037049006 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d281a5660 con 0x7f0d281085d0 2024-06-10T00:26:04.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.883+0000 7f0d30437700 1 -- 172.21.15.121:0/4037049006 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0d28101770 con 0x7f0d281085d0 2024-06-10T00:26:04.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.884+0000 7f0d277fe700 1 -- 172.21.15.121:0/4037049006 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f0d18024070 con 0x7f0d281085d0 2024-06-10T00:26:04.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.885+0000 7f0d277fe700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0d100a2cb0 0x7f0d100a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:04.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.885+0000 7f0d2e1d3700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0d100a2cb0 0x7f0d100a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:04.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.885+0000 7f0d277fe700 1 -- 172.21.15.121:0/4037049006 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f0d1800f020 con 0x7f0d281085d0 2024-06-10T00:26:04.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.886+0000 7f0d277fe700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609] conn(0x7f0d100adf90 0x7f0d100b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:04.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.886+0000 7f0d277fe700 1 -- 172.21.15.121:0/4037049006 --> [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d100b0b00 con 0x7f0d100adf90 2024-06-10T00:26:04.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.886+0000 7f0d277fe700 1 -- 172.21.15.121:0/4037049006 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f0d180d1390 con 0x7f0d281085d0 2024-06-10T00:26:04.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.886+0000 7f0d2d9d2700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609] conn(0x7f0d100adf90 0x7f0d100b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:04.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.886+0000 7f0d2d9d2700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609] conn(0x7f0d100adf90 0x7f0d100b0430 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:04.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.887+0000 7f0d2e1d3700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0d100a2cb0 0x7f0d100a5170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0d20005ea0 tx=0x7f0d20005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:04.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.888+0000 7f0d277fe700 1 -- 172.21.15.121:0/4037049006 <== osd.11 v2:172.21.15.193:6824/3765048609 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d100b0b00 con 0x7f0d100adf90 2024-06-10T00:26:04.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.915+0000 7f0d30437700 1 -- 172.21.15.121:0/4037049006 --> [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0d28103c40 con 0x7f0d100adf90 2024-06-10T00:26:04.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.915+0000 7f0d277fe700 1 -- 172.21.15.121:0/4037049006 <== osd.11 v2:172.21.15.193:6824/3765048609 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0d28103c40 con 0x7f0d100adf90 2024-06-10T00:26:04.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.916+0000 7f0d30437700 1 -- 172.21.15.121:0/4037049006 >> [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609] conn(0x7f0d100adf90 msgr2=0x7f0d100b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.916+0000 7f0d30437700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609] conn(0x7f0d100adf90 0x7f0d100b0430 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.916+0000 7f0d30437700 1 -- 172.21.15.121:0/4037049006 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0d100a2cb0 msgr2=0x7f0d100a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.916+0000 7f0d30437700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0d100a2cb0 0x7f0d100a5170 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0d20005ea0 tx=0x7f0d20005e30 comp rx=0 tx=0).stop 2024-06-10T00:26:04.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.916+0000 7f0d30437700 1 -- 172.21.15.121:0/4037049006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d281085d0 msgr2=0x7f0d28198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:04.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.916+0000 7f0d30437700 1 --2- 172.21.15.121:0/4037049006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d281085d0 0x7f0d28198950 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0d18006070 tx=0x7f0d18004ef0 comp rx=0 tx=0).stop 2024-06-10T00:26:04.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.916+0000 7f0d2e9d4700 1 -- 172.21.15.121:0/4037049006 reap_dead start 2024-06-10T00:26:04.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.916+0000 7f0d30437700 1 -- 172.21.15.121:0/4037049006 shutdown_connections 2024-06-10T00:26:04.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.916+0000 7f0d30437700 1 -- 172.21.15.121:0/4037049006 >> 172.21.15.121:0/4037049006 conn(0x7f0d28079640 msgr2=0x7f0d28100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:04.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.916+0000 7f0d30437700 1 -- 172.21.15.121:0/4037049006 shutdown_connections 2024-06-10T00:26:04.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:04.916+0000 7f0d30437700 1 -- 172.21.15.121:0/4037049006 wait complete. 2024-06-10T00:26:05.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:05 smithi135 ceph-mon[36424]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:05.564 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:05 smithi121 ceph-mon[32378]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:05.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:05 smithi193 ceph-mon[36716]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.014+0000 7f710b40c700 1 -- 172.21.15.121:0/2675390540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7104101df0 msgr2=0x7f71041021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.014+0000 7f710b40c700 1 --2- 172.21.15.121:0/2675390540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7104101df0 0x7f71041021d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0099c0 tx=0x7f70fc009cd0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.014+0000 7f710b40c700 1 -- 172.21.15.121:0/2675390540 shutdown_connections 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.014+0000 7f710b40c700 1 --2- 172.21.15.121:0/2675390540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7104106750 0x7f710410e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.014+0000 7f710b40c700 1 --2- 172.21.15.121:0/2675390540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7104101df0 0x7f71041021d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.014+0000 7f710b40c700 1 --2- 172.21.15.121:0/2675390540 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f71041027a0 0x7f7104106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.014+0000 7f710b40c700 1 -- 172.21.15.121:0/2675390540 >> 172.21.15.121:0/2675390540 conn(0x7f7104077a00 msgr2=0x7f71040fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.014+0000 7f710b40c700 1 -- 172.21.15.121:0/2675390540 shutdown_connections 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.014+0000 7f710b40c700 1 -- 172.21.15.121:0/2675390540 wait complete. 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f710b40c700 1 Processor -- start 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f710b40c700 1 -- start start 2024-06-10T00:26:06.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f710b40c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7104101df0 0x7f7104199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f710b40c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71041027a0 0x7f7104199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f710b40c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7104106750 0x7f71041a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f710b40c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7104111670 con 0x7f7104101df0 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f710b40c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f71041114f0 con 0x7f7104106750 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f710b40c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f71041117f0 con 0x7f71041027a0 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f71091a8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7104101df0 0x7f7104199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f71091a8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7104101df0 0x7f7104199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46378/0 (socket says 172.21.15.121:46378) 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f71091a8700 1 -- 172.21.15.121:0/3093690717 learned_addr learned my addr 172.21.15.121:0/3093690717 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f71099a9700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7104106750 0x7f71041a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f71089a7700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71041027a0 0x7f7104199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f71091a8700 1 -- 172.21.15.121:0/3093690717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71041027a0 msgr2=0x7f7104199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f71091a8700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f71041027a0 0x7f7104199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f71091a8700 1 -- 172.21.15.121:0/3093690717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7104106750 msgr2=0x7f71041a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f71091a8700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7104106750 0x7f71041a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f71091a8700 1 -- 172.21.15.121:0/3093690717 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71041a4ff0 con 0x7f7104101df0 2024-06-10T00:26:06.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.015+0000 7f71099a9700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7104106750 0x7f71041a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:26:06.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.016+0000 7f71091a8700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7104101df0 0x7f7104199a30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0059d0 tx=0x7f70fc00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:06.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.016+0000 7f71027fc700 1 -- 172.21.15.121:0/3093690717 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc027070 con 0x7f7104101df0 2024-06-10T00:26:06.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.016+0000 7f71027fc700 1 -- 172.21.15.121:0/3093690717 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70fc003680 con 0x7f7104101df0 2024-06-10T00:26:06.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.016+0000 7f710b40c700 1 -- 172.21.15.121:0/3093690717 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71041a51c0 con 0x7f7104101df0 2024-06-10T00:26:06.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.016+0000 7f71027fc700 1 -- 172.21.15.121:0/3093690717 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc017650 con 0x7f7104101df0 2024-06-10T00:26:06.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.016+0000 7f710b40c700 1 -- 172.21.15.121:0/3093690717 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71041a5610 con 0x7f7104101df0 2024-06-10T00:26:06.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.016+0000 7f710b40c700 1 -- 172.21.15.121:0/3093690717 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f70f0001030 con 0x7f7104101df0 2024-06-10T00:26:06.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.018+0000 7f71027fc700 1 -- 172.21.15.121:0/3093690717 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f70fc02a070 con 0x7f7104101df0 2024-06-10T00:26:06.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.018+0000 7f71027fc700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f70ec0a2cb0 0x7f70ec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:06.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.019+0000 7f71089a7700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f70ec0a2cb0 0x7f70ec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:06.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.019+0000 7f71027fc700 1 -- 172.21.15.121:0/3093690717 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f70fc0d0f50 con 0x7f7104101df0 2024-06-10T00:26:06.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.019+0000 7f71027fc700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067] conn(0x7f70ec0adf90 0x7f70ec0b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:06.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.019+0000 7f71027fc700 1 -- 172.21.15.121:0/3093690717 --> [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70ec0b0b00 con 0x7f70ec0adf90 2024-06-10T00:26:06.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.019+0000 7f71027fc700 1 -- 172.21.15.121:0/3093690717 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f70fc0d13a0 con 0x7f7104101df0 2024-06-10T00:26:06.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.019+0000 7f71099a9700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067] conn(0x7f70ec0adf90 0x7f70ec0b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:06.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.019+0000 7f71089a7700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f70ec0a2cb0 0x7f70ec0a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7104103e30 tx=0x7f70f8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:06.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.019+0000 7f71099a9700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067] conn(0x7f70ec0adf90 0x7f70ec0b0430 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:06.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.020+0000 7f71027fc700 1 -- 172.21.15.121:0/3093690717 <== osd.3 v2:172.21.15.121:6810/1510255067 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70ec0b0b00 con 0x7f70ec0adf90 2024-06-10T00:26:06.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.036+0000 7f710b40c700 1 -- 172.21.15.121:0/3093690717 --> [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f70f0002e60 con 0x7f70ec0adf90 2024-06-10T00:26:06.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.036+0000 7f71027fc700 1 -- 172.21.15.121:0/3093690717 <== osd.3 v2:172.21.15.121:6810/1510255067 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f70f0002e60 con 0x7f70ec0adf90 2024-06-10T00:26:06.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.037+0000 7f710b40c700 1 -- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067] conn(0x7f70ec0adf90 msgr2=0x7f70ec0b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.037+0000 7f710b40c700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067] conn(0x7f70ec0adf90 0x7f70ec0b0430 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.037+0000 7f710b40c700 1 -- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f70ec0a2cb0 msgr2=0x7f70ec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.037+0000 7f710b40c700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f70ec0a2cb0 0x7f70ec0a5170 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7104103e30 tx=0x7f70f8009260 comp rx=0 tx=0).stop 2024-06-10T00:26:06.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.037+0000 7f710b40c700 1 -- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7104101df0 msgr2=0x7f7104199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.037+0000 7f710b40c700 1 --2- 172.21.15.121:0/3093690717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7104101df0 0x7f7104199a30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0059d0 tx=0x7f70fc00fab0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.037+0000 7f71099a9700 1 -- 172.21.15.121:0/3093690717 reap_dead start 2024-06-10T00:26:06.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.037+0000 7f710b40c700 1 -- 172.21.15.121:0/3093690717 shutdown_connections 2024-06-10T00:26:06.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.037+0000 7f710b40c700 1 -- 172.21.15.121:0/3093690717 >> 172.21.15.121:0/3093690717 conn(0x7f7104077a00 msgr2=0x7f7104104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:06.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.037+0000 7f710b40c700 1 -- 172.21.15.121:0/3093690717 shutdown_connections 2024-06-10T00:26:06.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.037+0000 7f710b40c700 1 -- 172.21.15.121:0/3093690717 wait complete. 2024-06-10T00:26:06.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.198+0000 7f44735d7700 1 -- 172.21.15.121:0/4175156853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f446c06ab60 msgr2=0x7f446c06afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.198+0000 7f44735d7700 1 --2- 172.21.15.121:0/4175156853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f446c06ab60 0x7f446c06afe0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f445c009990 tx=0x7f445c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.198+0000 7f44735d7700 1 -- 172.21.15.121:0/4175156853 shutdown_connections 2024-06-10T00:26:06.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.198+0000 7f44735d7700 1 --2- 172.21.15.121:0/4175156853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f446c06b520 0x7f446c10f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.198+0000 7f44735d7700 1 --2- 172.21.15.121:0/4175156853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f446c108530 0x7f446c108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.198+0000 7f44735d7700 1 --2- 172.21.15.121:0/4175156853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f446c06ab60 0x7f446c06afe0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.198+0000 7f44735d7700 1 -- 172.21.15.121:0/4175156853 >> 172.21.15.121:0/4175156853 conn(0x7f446c0783d0 msgr2=0x7f446c0787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.198+0000 7f44735d7700 1 -- 172.21.15.121:0/4175156853 shutdown_connections 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.198+0000 7f44735d7700 1 -- 172.21.15.121:0/4175156853 wait complete. 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f44735d7700 1 Processor -- start 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f44735d7700 1 -- start start 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f44735d7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f446c06ab60 0x7f446c101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f44735d7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f446c06b520 0x7f446c1019d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f44735d7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f446c108530 0x7f446c101f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f44735d7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f446c112810 con 0x7f446c108530 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f44735d7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f446c112690 con 0x7f446c06ab60 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f44735d7700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f446c112990 con 0x7f446c06b520 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f4471b74700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f446c108530 0x7f446c101f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f4471373700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f446c06ab60 0x7f446c101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:06.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f4471b74700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f446c108530 0x7f446c101f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46398/0 (socket says 172.21.15.121:46398) 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.199+0000 7f4471b74700 1 -- 172.21.15.121:0/3847831458 learned_addr learned my addr 172.21.15.121:0/3847831458 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f4470b72700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f446c06b520 0x7f446c1019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f4471b74700 1 -- 172.21.15.121:0/3847831458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f446c06b520 msgr2=0x7f446c1019d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f4471b74700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f446c06b520 0x7f446c1019d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f4471b74700 1 -- 172.21.15.121:0/3847831458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f446c06ab60 msgr2=0x7f446c101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f4471b74700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f446c06ab60 0x7f446c101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f4471b74700 1 -- 172.21.15.121:0/3847831458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446c1a71f0 con 0x7f446c108530 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f4471373700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f446c06ab60 0x7f446c101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f4471b74700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f446c108530 0x7f446c101f10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f446400b970 tx=0x7f446400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f44627fc700 1 -- 172.21.15.121:0/3847831458 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464019070 con 0x7f446c108530 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f44627fc700 1 -- 172.21.15.121:0/3847831458 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f446400ce10 con 0x7f446c108530 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f44735d7700 1 -- 172.21.15.121:0/3847831458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f446c1a74e0 con 0x7f446c108530 2024-06-10T00:26:06.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f44627fc700 1 -- 172.21.15.121:0/3847831458 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464007370 con 0x7f446c108530 2024-06-10T00:26:06.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.200+0000 7f44735d7700 1 -- 172.21.15.121:0/3847831458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f446c1a79d0 con 0x7f446c108530 2024-06-10T00:26:06.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.201+0000 7f44735d7700 1 -- 172.21.15.121:0/3847831458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f446c062ba0 con 0x7f446c108530 2024-06-10T00:26:06.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.202+0000 7f44627fc700 1 -- 172.21.15.121:0/3847831458 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f446401c070 con 0x7f446c108530 2024-06-10T00:26:06.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.203+0000 7f44627fc700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44580a2cb0 0x7f44580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:06.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.203+0000 7f4471373700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44580a2cb0 0x7f44580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:06.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.203+0000 7f44627fc700 1 -- 172.21.15.121:0/3847831458 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f44640ceff0 con 0x7f446c108530 2024-06-10T00:26:06.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.203+0000 7f44627fc700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323] conn(0x7f44580adf90 0x7f44580b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:06.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.203+0000 7f44627fc700 1 -- 172.21.15.121:0/3847831458 --> [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44580b0b00 con 0x7f44580adf90 2024-06-10T00:26:06.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.203+0000 7f44627fc700 1 -- 172.21.15.121:0/3847831458 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f44640cf440 con 0x7f446c108530 2024-06-10T00:26:06.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.203+0000 7f4471373700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44580a2cb0 0x7f44580a5170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4468005950 tx=0x7f446800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:06.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.204+0000 7f4470b72700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323] conn(0x7f44580adf90 0x7f44580b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:06.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.204+0000 7f4470b72700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323] conn(0x7f44580adf90 0x7f44580b0430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:06.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.205+0000 7f44627fc700 1 -- 172.21.15.121:0/3847831458 <== osd.1 v2:172.21.15.135:6800/1153602323 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44580b0b00 con 0x7f44580adf90 2024-06-10T00:26:06.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.222+0000 7f44735d7700 1 -- 172.21.15.121:0/3847831458 --> [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f446c063020 con 0x7f44580adf90 2024-06-10T00:26:06.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f44627fc700 1 -- 172.21.15.121:0/3847831458 <== osd.1 v2:172.21.15.135:6800/1153602323 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f446c063020 con 0x7f44580adf90 2024-06-10T00:26:06.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f44735d7700 1 -- 172.21.15.121:0/3847831458 >> [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323] conn(0x7f44580adf90 msgr2=0x7f44580b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f44735d7700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323] conn(0x7f44580adf90 0x7f44580b0430 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:06.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f44735d7700 1 -- 172.21.15.121:0/3847831458 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44580a2cb0 msgr2=0x7f44580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f44735d7700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44580a2cb0 0x7f44580a5170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4468005950 tx=0x7f446800b410 comp rx=0 tx=0).stop 2024-06-10T00:26:06.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f44735d7700 1 -- 172.21.15.121:0/3847831458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f446c108530 msgr2=0x7f446c101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:06.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f44735d7700 1 --2- 172.21.15.121:0/3847831458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f446c108530 0x7f446c101f10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f446400b970 tx=0x7f446400bc80 comp rx=0 tx=0).stop 2024-06-10T00:26:06.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f4471b74700 1 -- 172.21.15.121:0/3847831458 reap_dead start 2024-06-10T00:26:06.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f44735d7700 1 -- 172.21.15.121:0/3847831458 shutdown_connections 2024-06-10T00:26:06.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f44735d7700 1 -- 172.21.15.121:0/3847831458 >> 172.21.15.121:0/3847831458 conn(0x7f446c0783d0 msgr2=0x7f446c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:06.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f44735d7700 1 -- 172.21.15.121:0/3847831458 shutdown_connections 2024-06-10T00:26:06.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:06.223+0000 7f44735d7700 1 -- 172.21.15.121:0/3847831458 wait complete. 2024-06-10T00:26:07.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:07 smithi121 ceph-mon[32378]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:07.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:07 smithi135 ceph-mon[36424]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:07 smithi193 ceph-mon[36716]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:07.955 INFO:teuthology.orchestra.run.smithi121.stdout:111669149778 2024-06-10T00:26:07.955 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-10T00:26:08.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.151+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/1339014642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b41030f0 msgr2=0x7fd2b4107100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:08.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.151+0000 7fd2bb7f6700 1 --2- 172.21.15.121:0/1339014642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b41030f0 0x7fd2b4107100 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40099c0 tx=0x7fd2a4009cd0 comp rx=0 tx=0).stop 2024-06-10T00:26:08.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.151+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/1339014642 shutdown_connections 2024-06-10T00:26:08.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.151+0000 7fd2bb7f6700 1 --2- 172.21.15.121:0/1339014642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2b4107830 0x7fd2b410f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:08.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.151+0000 7fd2bb7f6700 1 --2- 172.21.15.121:0/1339014642 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2b4102740 0x7fd2b4102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:08.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.151+0000 7fd2bb7f6700 1 --2- 172.21.15.121:0/1339014642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b41030f0 0x7fd2b4107100 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:08.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.151+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/1339014642 >> 172.21.15.121:0/1339014642 conn(0x7fd2b40795b0 msgr2=0x7fd2b40799c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:08.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.151+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/1339014642 shutdown_connections 2024-06-10T00:26:08.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.151+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/1339014642 wait complete. 2024-06-10T00:26:08.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.151+0000 7fd2bb7f6700 1 Processor -- start 2024-06-10T00:26:08.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2bb7f6700 1 -- start start 2024-06-10T00:26:08.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2bb7f6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b4102740 0x7fd2b419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:08.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2bb7f6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2b41030f0 0x7fd2b419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:08.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2bb7f6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2b4107830 0x7fd2b4198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:08.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2bb7f6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b4112840 con 0x7fd2b4102740 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2bb7f6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41126c0 con 0x7fd2b4107830 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2bb7f6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b41129c0 con 0x7fd2b41030f0 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2b9592700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b4102740 0x7fd2b419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2b9592700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b4102740 0x7fd2b419e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46418/0 (socket says 172.21.15.121:46418) 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2b9592700 1 -- 172.21.15.121:0/984932943 learned_addr learned my addr 172.21.15.121:0/984932943 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2b8d91700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2b41030f0 0x7fd2b419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2b9592700 1 -- 172.21.15.121:0/984932943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2b41030f0 msgr2=0x7fd2b419ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2b9592700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2b41030f0 0x7fd2b419ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2b9592700 1 -- 172.21.15.121:0/984932943 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2b4107830 msgr2=0x7fd2b4198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2b9592700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2b4107830 0x7fd2b4198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.152+0000 7fd2b9592700 1 -- 172.21.15.121:0/984932943 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b4199240 con 0x7fd2b4102740 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.153+0000 7fd2b9592700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b4102740 0x7fd2b419e920 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd2b000b530 tx=0x7fd2b000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:08.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.153+0000 7fd2aa7fc700 1 -- 172.21.15.121:0/984932943 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0015020 con 0x7fd2b4102740 2024-06-10T00:26:08.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.153+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/984932943 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b4199530 con 0x7fd2b4102740 2024-06-10T00:26:08.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.153+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/984932943 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b41a5870 con 0x7fd2b4102740 2024-06-10T00:26:08.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.153+0000 7fd2aa7fc700 1 -- 172.21.15.121:0/984932943 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2b0011d10 con 0x7fd2b4102740 2024-06-10T00:26:08.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.153+0000 7fd2aa7fc700 1 -- 172.21.15.121:0/984932943 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b001ae20 con 0x7fd2b4102740 2024-06-10T00:26:08.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.154+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/984932943 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd298001030 con 0x7fd2b4102740 2024-06-10T00:26:08.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.155+0000 7fd2aa7fc700 1 -- 172.21.15.121:0/984932943 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fd2b0018020 con 0x7fd2b4102740 2024-06-10T00:26:08.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.156+0000 7fd2aa7fc700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2a00a2cb0 0x7fd2a00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:08.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.156+0000 7fd2aa7fc700 1 -- 172.21.15.121:0/984932943 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fd2b00cf2a0 con 0x7fd2b4102740 2024-06-10T00:26:08.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.156+0000 7fd2b8d91700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2a00a2cb0 0x7fd2a00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:08.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.156+0000 7fd2aa7fc700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559] conn(0x7fd2a00adf90 0x7fd2a00b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:08.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.156+0000 7fd2aa7fc700 1 -- 172.21.15.121:0/984932943 --> [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2b0013ec0 con 0x7fd2a00adf90 2024-06-10T00:26:08.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.156+0000 7fd2aa7fc700 1 -- 172.21.15.121:0/984932943 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7fd2b00cf6f0 con 0x7fd2b4102740 2024-06-10T00:26:08.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.157+0000 7fd2b8d91700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2a00a2cb0 0x7fd2a00a5170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4102050 tx=0x7fd2a4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:08.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.157+0000 7fd2b9d93700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559] conn(0x7fd2a00adf90 0x7fd2a00b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:08.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.157+0000 7fd2b9d93700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559] conn(0x7fd2a00adf90 0x7fd2a00b0430 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:08.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.158+0000 7fd2aa7fc700 1 -- 172.21.15.121:0/984932943 <== osd.7 v2:172.21.15.135:6816/3426631559 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2b0013ec0 con 0x7fd2a00adf90 2024-06-10T00:26:08.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.185+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/984932943 --> [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd298002e60 con 0x7fd2a00adf90 2024-06-10T00:26:08.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.185+0000 7fd2aa7fc700 1 -- 172.21.15.121:0/984932943 <== osd.7 v2:172.21.15.135:6816/3426631559 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd298002e60 con 0x7fd2a00adf90 2024-06-10T00:26:08.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.186+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/984932943 >> [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559] conn(0x7fd2a00adf90 msgr2=0x7fd2a00b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:08.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.186+0000 7fd2bb7f6700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559] conn(0x7fd2a00adf90 0x7fd2a00b0430 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:08.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.186+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/984932943 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2a00a2cb0 msgr2=0x7fd2a00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:08.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.186+0000 7fd2bb7f6700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd2a00a2cb0 0x7fd2a00a5170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4102050 tx=0x7fd2a4005fd0 comp rx=0 tx=0).stop 2024-06-10T00:26:08.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.186+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/984932943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b4102740 msgr2=0x7fd2b419e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:08.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.186+0000 7fd2bb7f6700 1 --2- 172.21.15.121:0/984932943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b4102740 0x7fd2b419e920 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd2b000b530 tx=0x7fd2b000b8f0 comp rx=0 tx=0).stop 2024-06-10T00:26:08.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.186+0000 7fd2b9d93700 1 -- 172.21.15.121:0/984932943 reap_dead start 2024-06-10T00:26:08.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.186+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/984932943 shutdown_connections 2024-06-10T00:26:08.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.186+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/984932943 >> 172.21.15.121:0/984932943 conn(0x7fd2b40795b0 msgr2=0x7fd2b41006a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:08.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.186+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/984932943 shutdown_connections 2024-06-10T00:26:08.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:08.186+0000 7fd2bb7f6700 1 -- 172.21.15.121:0/984932943 wait complete. 2024-06-10T00:26:08.879 INFO:teuthology.orchestra.run.smithi121.stdout:55834574941 2024-06-10T00:26:08.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-10T00:26:09.331 INFO:teuthology.orchestra.run.smithi121.stdout:34359738464 2024-06-10T00:26:09.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-10T00:26:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:09 smithi135 ceph-mon[36424]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:09.544 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:09 smithi121 ceph-mon[32378]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:09.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:09 smithi193 ceph-mon[36716]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:09.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.545+0000 7f2d894e5700 1 -- 172.21.15.121:0/2295424566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d84108510 msgr2=0x7f2d841088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:09.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.545+0000 7f2d894e5700 1 --2- 172.21.15.121:0/2295424566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d84108510 0x7f2d841088f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c009990 tx=0x7f2d6c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:26:09.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.545+0000 7f2d894e5700 1 -- 172.21.15.121:0/2295424566 shutdown_connections 2024-06-10T00:26:09.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.545+0000 7f2d894e5700 1 --2- 172.21.15.121:0/2295424566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d84101440 0x7f2d8410f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:09.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.545+0000 7f2d894e5700 1 --2- 172.21.15.121:0/2295424566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d84108510 0x7f2d841088f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:09.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.545+0000 7f2d894e5700 1 --2- 172.21.15.121:0/2295424566 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d84100a80 0x7f2d84100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:09.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.545+0000 7f2d894e5700 1 -- 172.21.15.121:0/2295424566 >> 172.21.15.121:0/2295424566 conn(0x7f2d84077a00 msgr2=0x7f2d840fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.545+0000 7f2d894e5700 1 -- 172.21.15.121:0/2295424566 shutdown_connections 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.545+0000 7f2d894e5700 1 -- 172.21.15.121:0/2295424566 wait complete. 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.545+0000 7f2d894e5700 1 Processor -- start 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d894e5700 1 -- start start 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d894e5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d84100a80 0x7f2d84198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d894e5700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d84101440 0x7f2d84199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d894e5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d84108510 0x7f2d8419d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d894e5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2d841128c0 con 0x7f2d84108510 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d894e5700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2d84112740 con 0x7f2d84101440 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d894e5700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2d84112a40 con 0x7f2d84100a80 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d837fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d84108510 0x7f2d8419d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d837fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d84108510 0x7f2d8419d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34870/0 (socket says 172.21.15.121:34870) 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d837fe700 1 -- 172.21.15.121:0/500776832 learned_addr learned my addr 172.21.15.121:0/500776832 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:09.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d827fc700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d84101440 0x7f2d84199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d82ffd700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d84100a80 0x7f2d84198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d837fe700 1 -- 172.21.15.121:0/500776832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d84100a80 msgr2=0x7f2d84198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d837fe700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d84100a80 0x7f2d84198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d837fe700 1 -- 172.21.15.121:0/500776832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d84101440 msgr2=0x7f2d84199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d837fe700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2d84101440 0x7f2d84199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d837fe700 1 -- 172.21.15.121:0/500776832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8419de20 con 0x7f2d84108510 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.546+0000 7f2d837fe700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d84108510 0x7f2d8419d6e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00b970 tx=0x7f2d7c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.547+0000 7f2d7bfff700 1 -- 172.21.15.121:0/500776832 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c019070 con 0x7f2d84108510 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.547+0000 7f2d7bfff700 1 -- 172.21.15.121:0/500776832 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d7c014440 con 0x7f2d84108510 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.547+0000 7f2d894e5700 1 -- 172.21.15.121:0/500776832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8419e110 con 0x7f2d84108510 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.547+0000 7f2d7bfff700 1 -- 172.21.15.121:0/500776832 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c0075d0 con 0x7f2d84108510 2024-06-10T00:26:09.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.547+0000 7f2d894e5700 1 -- 172.21.15.121:0/500776832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d841a5990 con 0x7f2d84108510 2024-06-10T00:26:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.547+0000 7f2d894e5700 1 -- 172.21.15.121:0/500776832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d84062c20 con 0x7f2d84108510 2024-06-10T00:26:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.549+0000 7f2d7bfff700 1 -- 172.21.15.121:0/500776832 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f2d7c01c070 con 0x7f2d84108510 2024-06-10T00:26:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.549+0000 7f2d7bfff700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2d700a2cb0 0x7f2d700a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.550+0000 7f2d82ffd700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2d700a2cb0 0x7f2d700a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.550+0000 7f2d7bfff700 1 -- 172.21.15.121:0/500776832 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f2d7c0cf110 con 0x7f2d84108510 2024-06-10T00:26:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.550+0000 7f2d7bfff700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206] conn(0x7f2d700adf90 0x7f2d700b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.550+0000 7f2d7bfff700 1 -- 172.21.15.121:0/500776832 --> [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d7c004860 con 0x7f2d700adf90 2024-06-10T00:26:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.550+0000 7f2d7bfff700 1 -- 172.21.15.121:0/500776832 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f2d7c0cf560 con 0x7f2d84108510 2024-06-10T00:26:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.550+0000 7f2d827fc700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206] conn(0x7f2d700adf90 0x7f2d700b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:09.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.550+0000 7f2d82ffd700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2d700a2cb0 0x7f2d700a5170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c005f50 tx=0x7f2d6c005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.551+0000 7f2d827fc700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206] conn(0x7f2d700adf90 0x7f2d700b0430 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.552+0000 7f2d7bfff700 1 -- 172.21.15.121:0/500776832 <== osd.5 v2:172.21.15.193:6808/3919206206 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2d7c004860 con 0x7f2d700adf90 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.568+0000 7f2d894e5700 1 -- 172.21.15.121:0/500776832 --> [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2d840630a0 con 0x7f2d700adf90 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d7bfff700 1 -- 172.21.15.121:0/500776832 <== osd.5 v2:172.21.15.193:6808/3919206206 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2d840630a0 con 0x7f2d700adf90 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d894e5700 1 -- 172.21.15.121:0/500776832 >> [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206] conn(0x7f2d700adf90 msgr2=0x7f2d700b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d894e5700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206] conn(0x7f2d700adf90 0x7f2d700b0430 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d894e5700 1 -- 172.21.15.121:0/500776832 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2d700a2cb0 msgr2=0x7f2d700a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d894e5700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f2d700a2cb0 0x7f2d700a5170 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c005f50 tx=0x7f2d6c005e80 comp rx=0 tx=0).stop 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d894e5700 1 -- 172.21.15.121:0/500776832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d84108510 msgr2=0x7f2d8419d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d894e5700 1 --2- 172.21.15.121:0/500776832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d84108510 0x7f2d8419d6e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00b970 tx=0x7f2d7c00bd30 comp rx=0 tx=0).stop 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d837fe700 1 -- 172.21.15.121:0/500776832 reap_dead start 2024-06-10T00:26:09.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d894e5700 1 -- 172.21.15.121:0/500776832 shutdown_connections 2024-06-10T00:26:09.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d894e5700 1 -- 172.21.15.121:0/500776832 >> 172.21.15.121:0/500776832 conn(0x7f2d84077a00 msgr2=0x7f2d840ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:09.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d894e5700 1 -- 172.21.15.121:0/500776832 shutdown_connections 2024-06-10T00:26:09.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:09.569+0000 7f2d894e5700 1 -- 172.21.15.121:0/500776832 wait complete. 2024-06-10T00:26:10.468 INFO:teuthology.orchestra.run.smithi121.stdout:347892351018 2024-06-10T00:26:10.469 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-06-10T00:26:11.375 INFO:teuthology.orchestra.run.smithi121.stdout:223338299455 2024-06-10T00:26:11.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-10T00:26:11.473 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:11 smithi121 ceph-mon[32378]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:11.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:11 smithi135 ceph-mon[36424]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:11.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.580+0000 7f9c07852700 1 -- 172.21.15.121:0/2231571281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c00108500 msgr2=0x7f9c001088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:11.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.580+0000 7f9c07852700 1 --2- 172.21.15.121:0/2231571281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c00108500 0x7f9c001088e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc00b0a0 tx=0x7f9bfc00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:26:11.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.580+0000 7f9c07852700 1 -- 172.21.15.121:0/2231571281 shutdown_connections 2024-06-10T00:26:11.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.580+0000 7f9c07852700 1 --2- 172.21.15.121:0/2231571281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c0006ab30 0x7f9c0010f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.580+0000 7f9c07852700 1 --2- 172.21.15.121:0/2231571281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c00108500 0x7f9c001088e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.580+0000 7f9c07852700 1 --2- 172.21.15.121:0/2231571281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c0006a170 0x7f9c0006a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.580+0000 7f9c07852700 1 -- 172.21.15.121:0/2231571281 >> 172.21.15.121:0/2231571281 conn(0x7f9c00078340 msgr2=0x7f9c00078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.580+0000 7f9c07852700 1 -- 172.21.15.121:0/2231571281 shutdown_connections 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.580+0000 7f9c07852700 1 -- 172.21.15.121:0/2231571281 wait complete. 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c07852700 1 Processor -- start 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c07852700 1 -- start start 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c07852700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c0006a170 0x7f9c0019e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c07852700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c0006ab30 0x7f9c0019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c07852700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c00108500 0x7f9c00198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c07852700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9c001127e0 con 0x7f9c0006a170 2024-06-10T00:26:11.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c07852700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f9c00112660 con 0x7f9c00108500 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c07852700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9c00112960 con 0x7f9c0006ab30 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c055ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c0006a170 0x7f9c0019e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c055ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c0006a170 0x7f9c0019e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34894/0 (socket says 172.21.15.121:34894) 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c055ee700 1 -- 172.21.15.121:0/3252089245 learned_addr learned my addr 172.21.15.121:0/3252089245 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c04ded700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c0006ab30 0x7f9c0019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c05def700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c00108500 0x7f9c00198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c055ee700 1 -- 172.21.15.121:0/3252089245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c0006ab30 msgr2=0x7f9c0019ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c055ee700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c0006ab30 0x7f9c0019ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c055ee700 1 -- 172.21.15.121:0/3252089245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c00108500 msgr2=0x7f9c00198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.581+0000 7f9c055ee700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c00108500 0x7f9c00198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.582+0000 7f9c055ee700 1 -- 172.21.15.121:0/3252089245 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c00199200 con 0x7f9c0006a170 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.582+0000 7f9c055ee700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c0006a170 0x7f9c0019e8e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc000f80 tx=0x7f9bfc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:11.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.582+0000 7f9bf67fc700 1 -- 172.21.15.121:0/3252089245 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc021070 con 0x7f9c0006a170 2024-06-10T00:26:11.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.582+0000 7f9c07852700 1 -- 172.21.15.121:0/3252089245 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c00199490 con 0x7f9c0006a170 2024-06-10T00:26:11.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.582+0000 7f9bf67fc700 1 -- 172.21.15.121:0/3252089245 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bfc007db0 con 0x7f9c0006a170 2024-06-10T00:26:11.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.582+0000 7f9bf67fc700 1 -- 172.21.15.121:0/3252089245 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc005550 con 0x7f9c0006a170 2024-06-10T00:26:11.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.582+0000 7f9c07852700 1 -- 172.21.15.121:0/3252089245 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c001a57b0 con 0x7f9c0006a170 2024-06-10T00:26:11.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.583+0000 7f9c07852700 1 -- 172.21.15.121:0/3252089245 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9c00062c20 con 0x7f9c0006a170 2024-06-10T00:26:11.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.584+0000 7f9bf67fc700 1 -- 172.21.15.121:0/3252089245 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f9bfc024070 con 0x7f9c0006a170 2024-06-10T00:26:11.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.585+0000 7f9bf67fc700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f9bec0a2cb0 0x7f9bec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:11.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.585+0000 7f9c04ded700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f9bec0a2cb0 0x7f9bec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:11.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.585+0000 7f9bf67fc700 1 -- 172.21.15.121:0/3252089245 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f9bfc00f020 con 0x7f9c0006a170 2024-06-10T00:26:11.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.585+0000 7f9bf67fc700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459] conn(0x7f9bec0adf90 0x7f9bec0b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:11.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.585+0000 7f9bf67fc700 1 -- 172.21.15.121:0/3252089245 --> [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9bfc02fea0 con 0x7f9bec0adf90 2024-06-10T00:26:11.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.585+0000 7f9bf67fc700 1 -- 172.21.15.121:0/3252089245 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7f9bfc0d1430 con 0x7f9c0006a170 2024-06-10T00:26:11.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.585+0000 7f9c04ded700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f9bec0a2cb0 0x7f9bec0a5170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0005fd0 tx=0x7f9bf000a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:11.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.585+0000 7f9c05def700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459] conn(0x7f9bec0adf90 0x7f9bec0b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:11.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.586+0000 7f9c05def700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459] conn(0x7f9bec0adf90 0x7f9bec0b0430 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:11.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.587+0000 7f9bf67fc700 1 -- 172.21.15.121:0/3252089245 <== osd.4 v2:172.21.15.135:6808/759537459 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9bfc02fea0 con 0x7f9bec0adf90 2024-06-10T00:26:11.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.604+0000 7f9c07852700 1 -- 172.21.15.121:0/3252089245 --> [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9c00063020 con 0x7f9bec0adf90 2024-06-10T00:26:11.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.605+0000 7f9bf67fc700 1 -- 172.21.15.121:0/3252089245 <== osd.4 v2:172.21.15.135:6808/759537459 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9c00063020 con 0x7f9bec0adf90 2024-06-10T00:26:11.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.605+0000 7f9c07852700 1 -- 172.21.15.121:0/3252089245 >> [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459] conn(0x7f9bec0adf90 msgr2=0x7f9bec0b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:11.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.605+0000 7f9c07852700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459] conn(0x7f9bec0adf90 0x7f9bec0b0430 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:11.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.606+0000 7f9c07852700 1 -- 172.21.15.121:0/3252089245 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f9bec0a2cb0 msgr2=0x7f9bec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:11.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.606+0000 7f9c07852700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f9bec0a2cb0 0x7f9bec0a5170 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9bf0005fd0 tx=0x7f9bf000a560 comp rx=0 tx=0).stop 2024-06-10T00:26:11.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.606+0000 7f9c07852700 1 -- 172.21.15.121:0/3252089245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c0006a170 msgr2=0x7f9c0019e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:11.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.606+0000 7f9c07852700 1 --2- 172.21.15.121:0/3252089245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c0006a170 0x7f9c0019e8e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc000f80 tx=0x7f9bfc00bf10 comp rx=0 tx=0).stop 2024-06-10T00:26:11.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.606+0000 7f9c05def700 1 -- 172.21.15.121:0/3252089245 reap_dead start 2024-06-10T00:26:11.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.606+0000 7f9c07852700 1 -- 172.21.15.121:0/3252089245 shutdown_connections 2024-06-10T00:26:11.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.606+0000 7f9c07852700 1 -- 172.21.15.121:0/3252089245 >> 172.21.15.121:0/3252089245 conn(0x7f9c00078340 msgr2=0x7f9c000fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:11.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.606+0000 7f9c07852700 1 -- 172.21.15.121:0/3252089245 shutdown_connections 2024-06-10T00:26:11.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:11.606+0000 7f9c07852700 1 -- 172.21.15.121:0/3252089245 wait complete. 2024-06-10T00:26:11.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:11 smithi193 ceph-mon[36716]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:12.034 INFO:teuthology.orchestra.run.smithi121.stdout:167503724617 2024-06-10T00:26:12.034 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-10T00:26:12.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.542+0000 7fa1c4905700 1 -- 172.21.15.121:0/1030061814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c0100b50 msgr2=0x7fa1c0100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.542+0000 7fa1c4905700 1 --2- 172.21.15.121:0/1030061814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c0100b50 0x7fa1c0100fd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa1a80099c0 tx=0x7fa1a8009cd0 comp rx=0 tx=0).stop 2024-06-10T00:26:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.542+0000 7fa1c4905700 1 -- 172.21.15.121:0/1030061814 shutdown_connections 2024-06-10T00:26:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.542+0000 7fa1c4905700 1 --2- 172.21.15.121:0/1030061814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1c0101510 0x7fa1c010d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.542+0000 7fa1c4905700 1 --2- 172.21.15.121:0/1030061814 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1c0106c00 0x7fa1c0106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.542+0000 7fa1c4905700 1 --2- 172.21.15.121:0/1030061814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c0100b50 0x7fa1c0100fd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.542+0000 7fa1c4905700 1 -- 172.21.15.121:0/1030061814 >> 172.21.15.121:0/1030061814 conn(0x7fa1c0077990 msgr2=0x7fa1c00fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.542+0000 7fa1c4905700 1 -- 172.21.15.121:0/1030061814 shutdown_connections 2024-06-10T00:26:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.542+0000 7fa1c4905700 1 -- 172.21.15.121:0/1030061814 wait complete. 2024-06-10T00:26:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.542+0000 7fa1c4905700 1 Processor -- start 2024-06-10T00:26:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1c4905700 1 -- start start 2024-06-10T00:26:12.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1c4905700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1c0100b50 0x7fa1c019e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1c4905700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1c0101510 0x7fa1c019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1c4905700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c0106c00 0x7fa1c0198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1c4905700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c01105e0 con 0x7fa1c0106c00 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1c4905700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c0110460 con 0x7fa1c0101510 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1c4905700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c0110760 con 0x7fa1c0100b50 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1bed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c0106c00 0x7fa1c0198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1bed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c0106c00 0x7fa1c0198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34912/0 (socket says 172.21.15.121:34912) 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1bed9d700 1 -- 172.21.15.121:0/102698017 learned_addr learned my addr 172.21.15.121:0/102698017 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1b5d9b700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1c0101510 0x7fa1c019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1be59c700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1c0100b50 0x7fa1c019e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1bed9d700 1 -- 172.21.15.121:0/102698017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1c0100b50 msgr2=0x7fa1c019e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1bed9d700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1c0100b50 0x7fa1c019e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:12.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1bed9d700 1 -- 172.21.15.121:0/102698017 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1c0101510 msgr2=0x7fa1c019ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1bed9d700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa1c0101510 0x7fa1c019ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1bed9d700 1 -- 172.21.15.121:0/102698017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c01991e0 con 0x7fa1c0106c00 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.543+0000 7fa1be59c700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1c0100b50 0x7fa1c019e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.544+0000 7fa1bed9d700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c0106c00 0x7fa1c0198a80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8007f10 tx=0x7fa1b800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.544+0000 7fa1b7fff700 1 -- 172.21.15.121:0/102698017 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8009e20 con 0x7fa1c0106c00 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.544+0000 7fa1c4905700 1 -- 172.21.15.121:0/102698017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c01994d0 con 0x7fa1c0106c00 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.544+0000 7fa1b7fff700 1 -- 172.21.15.121:0/102698017 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1b800dbd0 con 0x7fa1c0106c00 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.544+0000 7fa1b7fff700 1 -- 172.21.15.121:0/102698017 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8005a60 con 0x7fa1c0106c00 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.544+0000 7fa1c4905700 1 -- 172.21.15.121:0/102698017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c01a5790 con 0x7fa1c0106c00 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.545+0000 7fa1c4905700 1 -- 172.21.15.121:0/102698017 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa1c0062c20 con 0x7fa1c0106c00 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.546+0000 7fa1b7fff700 1 -- 172.21.15.121:0/102698017 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fa1b801a020 con 0x7fa1c0106c00 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.546+0000 7fa1b7fff700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa1ac0a2cb0 0x7fa1ac0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.547+0000 7fa1be59c700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa1ac0a2cb0 0x7fa1ac0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:12.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.547+0000 7fa1b7fff700 1 -- 172.21.15.121:0/102698017 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fa1b80cf350 con 0x7fa1c0106c00 2024-06-10T00:26:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.547+0000 7fa1b7fff700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346] conn(0x7fa1ac0adf90 0x7fa1ac0b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.547+0000 7fa1b7fff700 1 -- 172.21.15.121:0/102698017 --> [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa1ac0b0b00 con 0x7fa1ac0adf90 2024-06-10T00:26:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.547+0000 7fa1b7fff700 1 -- 172.21.15.121:0/102698017 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7fa1b80cf7a0 con 0x7fa1c0106c00 2024-06-10T00:26:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.547+0000 7fa1be59c700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa1ac0a2cb0 0x7fa1ac0a5170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0005e10 tx=0x7fa1b0005d60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.547+0000 7fa1b5d9b700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346] conn(0x7fa1ac0adf90 0x7fa1ac0b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.547+0000 7fa1b5d9b700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346] conn(0x7fa1ac0adf90 0x7fa1ac0b0430 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:12.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.548+0000 7fa1b7fff700 1 -- 172.21.15.121:0/102698017 <== osd.2 v2:172.21.15.193:6800/2430648346 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa1ac0b0b00 con 0x7fa1ac0adf90 2024-06-10T00:26:12.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.565+0000 7fa1c4905700 1 -- 172.21.15.121:0/102698017 --> [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa1c0063020 con 0x7fa1ac0adf90 2024-06-10T00:26:12.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.565+0000 7fa1b7fff700 1 -- 172.21.15.121:0/102698017 <== osd.2 v2:172.21.15.193:6800/2430648346 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa1c0063020 con 0x7fa1ac0adf90 2024-06-10T00:26:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.565+0000 7fa1c4905700 1 -- 172.21.15.121:0/102698017 >> [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346] conn(0x7fa1ac0adf90 msgr2=0x7fa1ac0b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.565+0000 7fa1c4905700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346] conn(0x7fa1ac0adf90 0x7fa1ac0b0430 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.566+0000 7fa1c4905700 1 -- 172.21.15.121:0/102698017 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa1ac0a2cb0 msgr2=0x7fa1ac0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.566+0000 7fa1c4905700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa1ac0a2cb0 0x7fa1ac0a5170 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0005e10 tx=0x7fa1b0005d60 comp rx=0 tx=0).stop 2024-06-10T00:26:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.566+0000 7fa1c4905700 1 -- 172.21.15.121:0/102698017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c0106c00 msgr2=0x7fa1c0198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.566+0000 7fa1c4905700 1 --2- 172.21.15.121:0/102698017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1c0106c00 0x7fa1c0198a80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8007f10 tx=0x7fa1b800d3b0 comp rx=0 tx=0).stop 2024-06-10T00:26:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.566+0000 7fa1bed9d700 1 -- 172.21.15.121:0/102698017 reap_dead start 2024-06-10T00:26:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.566+0000 7fa1c4905700 1 -- 172.21.15.121:0/102698017 shutdown_connections 2024-06-10T00:26:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.566+0000 7fa1c4905700 1 -- 172.21.15.121:0/102698017 >> 172.21.15.121:0/102698017 conn(0x7fa1c0077990 msgr2=0x7fa1c00ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.566+0000 7fa1c4905700 1 -- 172.21.15.121:0/102698017 shutdown_connections 2024-06-10T00:26:12.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:12.566+0000 7fa1c4905700 1 -- 172.21.15.121:0/102698017 wait complete. 2024-06-10T00:26:13.466 INFO:teuthology.orchestra.run.smithi121.stdout:77309411417 2024-06-10T00:26:13.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-10T00:26:13.488 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:13 smithi121 ceph-mon[32378]: pgmap v375: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:13.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:13 smithi135 ceph-mon[36424]: pgmap v375: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:13 smithi193 ceph-mon[36716]: pgmap v375: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:14.069 INFO:teuthology.orchestra.run.smithi121.stdout:146028888142 2024-06-10T00:26:14.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-10T00:26:14.320 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:14.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.574+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/1211037973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6b8101d80 msgr2=0x7ff6b810e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:14.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.574+0000 7ff6bf1b8700 1 --2- 172.21.15.121:0/1211037973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6b8101d80 0x7ff6b810e240 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000b0a0 tx=0x7ff6b000b3b0 comp rx=0 tx=0).stop 2024-06-10T00:26:14.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.574+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/1211037973 shutdown_connections 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.574+0000 7ff6bf1b8700 1 --2- 172.21.15.121:0/1211037973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6b8101d80 0x7ff6b810e240 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.574+0000 7ff6bf1b8700 1 --2- 172.21.15.121:0/1211037973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6b8107440 0x7ff6b8107820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.574+0000 7ff6bf1b8700 1 --2- 172.21.15.121:0/1211037973 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6b81013c0 0x7ff6b8101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.574+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/1211037973 >> 172.21.15.121:0/1211037973 conn(0x7ff6b8077400 msgr2=0x7ff6b8077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.574+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/1211037973 shutdown_connections 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.574+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/1211037973 wait complete. 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6bf1b8700 1 Processor -- start 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6bf1b8700 1 -- start start 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6bf1b8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6b81013c0 0x7ff6b819ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6bf1b8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6b8101d80 0x7ff6b819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6bf1b8700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6b8107440 0x7ff6b819f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6bf1b8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b81116e0 con 0x7ff6b8101d80 2024-06-10T00:26:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6bf1b8700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b8111560 con 0x7ff6b81013c0 2024-06-10T00:26:14.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6bf1b8700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b8111860 con 0x7ff6b8107440 2024-06-10T00:26:14.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6b7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6b8101d80 0x7ff6b819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:14.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6b7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6b8101d80 0x7ff6b819b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34918/0 (socket says 172.21.15.121:34918) 2024-06-10T00:26:14.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6b7fff700 1 -- 172.21.15.121:0/2556596574 learned_addr learned my addr 172.21.15.121:0/2556596574 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:14.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6bcf54700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6b81013c0 0x7ff6b819ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:14.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6bd755700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6b8107440 0x7ff6b819f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:14.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6b7fff700 1 -- 172.21.15.121:0/2556596574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6b8107440 msgr2=0x7ff6b819f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:14.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6b7fff700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6b8107440 0x7ff6b819f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:14.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6b7fff700 1 -- 172.21.15.121:0/2556596574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6b81013c0 msgr2=0x7ff6b819ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:14.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6b7fff700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6b81013c0 0x7ff6b819ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:14.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6b7fff700 1 -- 172.21.15.121:0/2556596574 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b819fdb0 con 0x7ff6b8101d80 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.575+0000 7ff6bcf54700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6b81013c0 0x7ff6b819ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.576+0000 7ff6b7fff700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6b8101d80 0x7ff6b819b2e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400c9b0 tx=0x7ff6a400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.576+0000 7ff6b5ffb700 1 -- 172.21.15.121:0/2556596574 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4007b00 con 0x7ff6b8101d80 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.576+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/2556596574 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b81a00a0 con 0x7ff6b8101d80 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.576+0000 7ff6b5ffb700 1 -- 172.21.15.121:0/2556596574 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6a4007ca0 con 0x7ff6b8101d80 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.576+0000 7ff6b5ffb700 1 -- 172.21.15.121:0/2556596574 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4018780 con 0x7ff6b8101d80 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.576+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/2556596574 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b81a7920 con 0x7ff6b8101d80 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.577+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/2556596574 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff6b8100f00 con 0x7ff6b8101d80 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.578+0000 7ff6b5ffb700 1 -- 172.21.15.121:0/2556596574 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff6a400f450 con 0x7ff6b8101d80 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.578+0000 7ff6b5ffb700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff6a80a2cb0 0x7ff6a80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.579+0000 7ff6bcf54700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff6a80a2cb0 0x7ff6a80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.579+0000 7ff6b5ffb700 1 -- 172.21.15.121:0/2556596574 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7ff6a40cf5f0 con 0x7ff6b8101d80 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.579+0000 7ff6b5ffb700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801] conn(0x7ff6a80adf90 0x7ff6a80b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.579+0000 7ff6b5ffb700 1 -- 172.21.15.121:0/2556596574 --> [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6a4012ec0 con 0x7ff6a80adf90 2024-06-10T00:26:14.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.579+0000 7ff6b5ffb700 1 -- 172.21.15.121:0/2556596574 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=100) ==== 24+0+0 (secure 0 0 0) 0x7ff6a40d6050 con 0x7ff6b8101d80 2024-06-10T00:26:14.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.579+0000 7ff6bcf54700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff6a80a2cb0 0x7ff6a80a5170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac0099d0 tx=0x7ff6ac008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:14.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.579+0000 7ff6bd755700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801] conn(0x7ff6a80adf90 0x7ff6a80b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:14.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.579+0000 7ff6bd755700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801] conn(0x7ff6a80adf90 0x7ff6a80b0430 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:14.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.580+0000 7ff6b5ffb700 1 -- 172.21.15.121:0/2556596574 <== osd.9 v2:172.21.15.121:6826/3736625801 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff6a4012ec0 con 0x7ff6a80adf90 2024-06-10T00:26:14.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.598+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/2556596574 --> [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff6b8102ab0 con 0x7ff6a80adf90 2024-06-10T00:26:14.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.598+0000 7ff6b5ffb700 1 -- 172.21.15.121:0/2556596574 <== osd.9 v2:172.21.15.121:6826/3736625801 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff6b8102ab0 con 0x7ff6a80adf90 2024-06-10T00:26:14.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.599+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801] conn(0x7ff6a80adf90 msgr2=0x7ff6a80b0430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:14.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.599+0000 7ff6bf1b8700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801] conn(0x7ff6a80adf90 0x7ff6a80b0430 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:14.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.599+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff6a80a2cb0 msgr2=0x7ff6a80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:14.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.599+0000 7ff6bf1b8700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff6a80a2cb0 0x7ff6a80a5170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac0099d0 tx=0x7ff6ac008040 comp rx=0 tx=0).stop 2024-06-10T00:26:14.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.599+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6b8101d80 msgr2=0x7ff6b819b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:14.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.599+0000 7ff6bf1b8700 1 --2- 172.21.15.121:0/2556596574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6b8101d80 0x7ff6b819b2e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400c9b0 tx=0x7ff6a400cd70 comp rx=0 tx=0).stop 2024-06-10T00:26:14.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.599+0000 7ff6bd755700 1 -- 172.21.15.121:0/2556596574 reap_dead start 2024-06-10T00:26:14.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.599+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/2556596574 shutdown_connections 2024-06-10T00:26:14.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.599+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/2556596574 >> 172.21.15.121:0/2556596574 conn(0x7ff6b8077400 msgr2=0x7ff6b81004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:14.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.599+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/2556596574 shutdown_connections 2024-06-10T00:26:14.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:14.599+0000 7ff6bf1b8700 1 -- 172.21.15.121:0/2556596574 wait complete. 2024-06-10T00:26:14.609 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:14.621 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:14.625 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:14.832 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:14.847 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:15.484 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:15 smithi121 ceph-mon[32378]: pgmap v376: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-06-10T00:26:15.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:15 smithi135 ceph-mon[36424]: pgmap v376: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-06-10T00:26:15.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:15 smithi193 ceph-mon[36716]: pgmap v376: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-06-10T00:26:16.715 INFO:teuthology.orchestra.run.smithi121.stdout:292057776183 2024-06-10T00:26:16.715 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-06-10T00:26:17.434 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:17 smithi121 ceph-mon[32378]: pgmap v377: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:17 smithi135 ceph-mon[36424]: pgmap v377: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:17 smithi193 ceph-mon[36716]: pgmap v377: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:18.229 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:19.360 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:19 smithi121 ceph-mon[32378]: pgmap v378: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:19.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:19 smithi135 ceph-mon[36424]: pgmap v378: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:19.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:19 smithi193 ceph-mon[36716]: pgmap v378: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:21 smithi121 ceph-mon[32378]: pgmap v379: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:21.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:21 smithi135 ceph-mon[36424]: pgmap v379: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:21.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.515+0000 7fbea0f89700 1 -- 172.21.15.121:0/4287570059 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe9c101620 msgr2=0x7fbe9c101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:21.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.515+0000 7fbea0f89700 1 --2- 172.21.15.121:0/4287570059 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe9c101620 0x7fbe9c101a00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0099c0 tx=0x7fbe8c009cd0 comp rx=0 tx=0).stop 2024-06-10T00:26:21.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.515+0000 7fbea0f89700 1 -- 172.21.15.121:0/4287570059 shutdown_connections 2024-06-10T00:26:21.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.515+0000 7fbea0f89700 1 --2- 172.21.15.121:0/4287570059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe9c1066b0 0x7fbe9c10e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:21.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.515+0000 7fbea0f89700 1 --2- 172.21.15.121:0/4287570059 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe9c101620 0x7fbe9c101a00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:21.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.515+0000 7fbea0f89700 1 --2- 172.21.15.121:0/4287570059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe9c101fd0 0x7fbe9c105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:21.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.515+0000 7fbea0f89700 1 -- 172.21.15.121:0/4287570059 >> 172.21.15.121:0/4287570059 conn(0x7fbe9c078360 msgr2=0x7fbe9c078770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:21.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.515+0000 7fbea0f89700 1 -- 172.21.15.121:0/4287570059 shutdown_connections 2024-06-10T00:26:21.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.515+0000 7fbea0f89700 1 -- 172.21.15.121:0/4287570059 wait complete. 2024-06-10T00:26:21.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbea0f89700 1 Processor -- start 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbea0f89700 1 -- start start 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbea0f89700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe9c101620 0x7fbe9c19d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbea0f89700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe9c101fd0 0x7fbe9c19dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbea0f89700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe9c1066b0 0x7fbe9c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbea0f89700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c111710 con 0x7fbe9c101620 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbea0f89700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c111590 con 0x7fbe9c101fd0 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbea0f89700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9c111890 con 0x7fbe9c1066b0 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbe9a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe9c101620 0x7fbe9c19d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbe9a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe9c101620 0x7fbe9c19d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44740/0 (socket says 172.21.15.121:44740) 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbe9a59c700 1 -- 172.21.15.121:0/3286493499 learned_addr learned my addr 172.21.15.121:0/3286493499 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbe9ad9d700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe9c1066b0 0x7fbe9c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:21.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbe9a59c700 1 -- 172.21.15.121:0/3286493499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe9c1066b0 msgr2=0x7fbe9c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:21.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbe9a59c700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe9c1066b0 0x7fbe9c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:21.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbe9a59c700 1 -- 172.21.15.121:0/3286493499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe9c101fd0 msgr2=0x7fbe9c19dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:21.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbe9a59c700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe9c101fd0 0x7fbe9c19dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:21.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.516+0000 7fbe9a59c700 1 -- 172.21.15.121:0/3286493499 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9c1981b0 con 0x7fbe9c101620 2024-06-10T00:26:21.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.517+0000 7fbe9a59c700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe9c101620 0x7fbe9c19d840 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009fd0 tx=0x7fbe8c003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:21.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.517+0000 7fbe8b7fe700 1 -- 172.21.15.121:0/3286493499 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c027070 con 0x7fbe9c101620 2024-06-10T00:26:21.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.517+0000 7fbe8b7fe700 1 -- 172.21.15.121:0/3286493499 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe8c004150 con 0x7fbe9c101620 2024-06-10T00:26:21.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.517+0000 7fbe8b7fe700 1 -- 172.21.15.121:0/3286493499 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c00f460 con 0x7fbe9c101620 2024-06-10T00:26:21.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.517+0000 7fbea0f89700 1 -- 172.21.15.121:0/3286493499 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe9c198440 con 0x7fbe9c101620 2024-06-10T00:26:21.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.517+0000 7fbea0f89700 1 -- 172.21.15.121:0/3286493499 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe9c1a4710 con 0x7fbe9c101620 2024-06-10T00:26:21.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.518+0000 7fbea0f89700 1 -- 172.21.15.121:0/3286493499 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe9c102ab0 con 0x7fbe9c101620 2024-06-10T00:26:21.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.520+0000 7fbe8b7fe700 1 -- 172.21.15.121:0/3286493499 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fbe8c02a070 con 0x7fbe9c101620 2024-06-10T00:26:21.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.520+0000 7fbe8b7fe700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbe840a2cb0 0x7fbe840a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:21.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.520+0000 7fbe99d9b700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbe840a2cb0 0x7fbe840a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:21.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.520+0000 7fbe8b7fe700 1 -- 172.21.15.121:0/3286493499 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fbe8c0d0cb0 con 0x7fbe9c101620 2024-06-10T00:26:21.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.521+0000 7fbe99d9b700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbe840a2cb0 0x7fbe840a5170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c100f90 tx=0x7fbe90006d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:21.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.523+0000 7fbe8b7fe700 1 -- 172.21.15.121:0/3286493499 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbe8c031070 con 0x7fbe9c101620 2024-06-10T00:26:21.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:21 smithi193 ceph-mon[36716]: pgmap v379: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:21.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.730+0000 7fbea0f89700 1 -- 172.21.15.121:0/3286493499 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fbe9c02dca0 con 0x7fbe9c101620 2024-06-10T00:26:21.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.730+0000 7fbe8b7fe700 1 -- 172.21.15.121:0/3286493499 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbe8c013070 con 0x7fbe9c101620 2024-06-10T00:26:21.730 INFO:teuthology.orchestra.run.smithi121.stdout:257698037820 2024-06-10T00:26:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.732+0000 7fbea0f89700 1 -- 172.21.15.121:0/3286493499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbe840a2cb0 msgr2=0x7fbe840a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.732+0000 7fbea0f89700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbe840a2cb0 0x7fbe840a5170 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c100f90 tx=0x7fbe90006d20 comp rx=0 tx=0).stop 2024-06-10T00:26:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.732+0000 7fbea0f89700 1 -- 172.21.15.121:0/3286493499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe9c101620 msgr2=0x7fbe9c19d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.732+0000 7fbea0f89700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe9c101620 0x7fbe9c19d840 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009fd0 tx=0x7fbe8c003730 comp rx=0 tx=0).stop 2024-06-10T00:26:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.732+0000 7fbea0f89700 1 -- 172.21.15.121:0/3286493499 shutdown_connections 2024-06-10T00:26:21.732 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.732+0000 7fbea0f89700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fbe840a2cb0 0x7fbe840a5170 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:21.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.732+0000 7fbea0f89700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe9c1066b0 0x7fbe9c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:21.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.732+0000 7fbea0f89700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe9c101620 0x7fbe9c19d840 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:21.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.732+0000 7fbea0f89700 1 --2- 172.21.15.121:0/3286493499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe9c101fd0 0x7fbe9c19dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:21.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.732+0000 7fbea0f89700 1 -- 172.21.15.121:0/3286493499 >> 172.21.15.121:0/3286493499 conn(0x7fbe9c078360 msgr2=0x7fbe9c0ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:21.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.733+0000 7fbea0f89700 1 -- 172.21.15.121:0/3286493499 shutdown_connections 2024-06-10T00:26:21.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:21.733+0000 7fbea0f89700 1 -- 172.21.15.121:0/3286493499 wait complete. 2024-06-10T00:26:22.368 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:22 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3286493499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-10T00:26:22.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:22 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3286493499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-10T00:26:22.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:22 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3286493499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-06-10T00:26:23.433 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:23.433 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:23.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:23 smithi135 ceph-mon[36424]: pgmap v380: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:23.537 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:23 smithi121 ceph-mon[32378]: pgmap v380: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:23.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:23 smithi193 ceph-mon[36716]: pgmap v380: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:23.885 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037815 got 257698037820 for osd.8 2024-06-10T00:26:23.885 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:24.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.850+0000 7f900652b700 1 -- 172.21.15.121:0/308218525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90001013c0 msgr2=0x7f9000101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:24.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.850+0000 7f900652b700 1 --2- 172.21.15.121:0/308218525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90001013c0 0x7f9000101840 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8009990 tx=0x7f8fe8009ca0 comp rx=0 tx=0).stop 2024-06-10T00:26:24.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.850+0000 7f900652b700 1 -- 172.21.15.121:0/308218525 shutdown_connections 2024-06-10T00:26:24.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.850+0000 7f900652b700 1 --2- 172.21.15.121:0/308218525 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9000101d80 0x7f900010e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:24.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.850+0000 7f900652b700 1 --2- 172.21.15.121:0/308218525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9000107430 0x7f9000107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:24.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.850+0000 7f900652b700 1 --2- 172.21.15.121:0/308218525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90001013c0 0x7f9000101840 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:24.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.850+0000 7f900652b700 1 -- 172.21.15.121:0/308218525 >> 172.21.15.121:0/308218525 conn(0x7f90000783d0 msgr2=0x7f90000787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:24.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.850+0000 7f900652b700 1 -- 172.21.15.121:0/308218525 shutdown_connections 2024-06-10T00:26:24.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.850+0000 7f900652b700 1 -- 172.21.15.121:0/308218525 wait complete. 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f900652b700 1 Processor -- start 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f900652b700 1 -- start start 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f900652b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90001013c0 0x7f900019e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f900652b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9000101d80 0x7f900019edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f900652b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9000107430 0x7f9000198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f900652b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9000111720 con 0x7f90001013c0 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f900652b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f90001115a0 con 0x7f9000107430 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f900652b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f90001118a0 con 0x7f9000101d80 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f8ffffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90001013c0 0x7f900019e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f8ffffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90001013c0 0x7f900019e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44758/0 (socket says 172.21.15.121:44758) 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f8ffffff700 1 -- 172.21.15.121:0/190385181 learned_addr learned my addr 172.21.15.121:0/190385181 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f9004ac8700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9000107430 0x7f9000198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:24.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f8ffffff700 1 -- 172.21.15.121:0/190385181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9000101d80 msgr2=0x7f900019edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f8ffffff700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9000101d80 0x7f900019edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f8ffffff700 1 -- 172.21.15.121:0/190385181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9000107430 msgr2=0x7f9000198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f8ffffff700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9000107430 0x7f9000198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f8ffffff700 1 -- 172.21.15.121:0/190385181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90001991d0 con 0x7f90001013c0 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.851+0000 7f9004ac8700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9000107430 0x7f9000198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.852+0000 7f8ffffff700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90001013c0 0x7f900019e8b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000c9b0 tx=0x7f8ff000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.852+0000 7f8ffdffb700 1 -- 172.21.15.121:0/190385181 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0007b00 con 0x7f90001013c0 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.852+0000 7f8ffdffb700 1 -- 172.21.15.121:0/190385181 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ff0007ca0 con 0x7f90001013c0 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.852+0000 7f900652b700 1 -- 172.21.15.121:0/190385181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90001994c0 con 0x7f90001013c0 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.852+0000 7f8ffdffb700 1 -- 172.21.15.121:0/190385181 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff0018780 con 0x7f90001013c0 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.852+0000 7f900652b700 1 -- 172.21.15.121:0/190385181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90001a5780 con 0x7f90001013c0 2024-06-10T00:26:24.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.853+0000 7f900652b700 1 -- 172.21.15.121:0/190385181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9000102ae0 con 0x7f90001013c0 2024-06-10T00:26:24.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.854+0000 7f8ffdffb700 1 -- 172.21.15.121:0/190385181 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f8ff000f450 con 0x7f90001013c0 2024-06-10T00:26:24.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.854+0000 7f8ffdffb700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fec0a2cb0 0x7f8fec0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:24.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.855+0000 7f8ff7fff700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fec0a2cb0 0x7f8fec0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:24.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.855+0000 7f8ffdffb700 1 -- 172.21.15.121:0/190385181 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f8ff00cf5f0 con 0x7f90001013c0 2024-06-10T00:26:24.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.855+0000 7f8ff7fff700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fec0a2cb0 0x7f8fec0a5170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800b5c0 tx=0x7f8fe8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:24.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:24.860+0000 7f8ffdffb700 1 -- 172.21.15.121:0/190385181 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8ff0014030 con 0x7f90001013c0 2024-06-10T00:26:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.063+0000 7f900652b700 1 -- 172.21.15.121:0/190385181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f90001a5a70 con 0x7f90001013c0 2024-06-10T00:26:25.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.063+0000 7f8ffdffb700 1 -- 172.21.15.121:0/190385181 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8ff009c040 con 0x7f90001013c0 2024-06-10T00:26:25.064 INFO:teuthology.orchestra.run.smithi121.stdout:223338299458 2024-06-10T00:26:25.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.065+0000 7f900652b700 1 -- 172.21.15.121:0/190385181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fec0a2cb0 msgr2=0x7f8fec0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:25.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.065+0000 7f900652b700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fec0a2cb0 0x7f8fec0a5170 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800b5c0 tx=0x7f8fe8005fb0 comp rx=0 tx=0).stop 2024-06-10T00:26:25.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.065+0000 7f900652b700 1 -- 172.21.15.121:0/190385181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90001013c0 msgr2=0x7f900019e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:25.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.065+0000 7f900652b700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90001013c0 0x7f900019e8b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8ff000c9b0 tx=0x7f8ff000cd70 comp rx=0 tx=0).stop 2024-06-10T00:26:25.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.065+0000 7f900652b700 1 -- 172.21.15.121:0/190385181 shutdown_connections 2024-06-10T00:26:25.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.065+0000 7f900652b700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fec0a2cb0 0x7f8fec0a5170 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:25.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.065+0000 7f900652b700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9000107430 0x7f9000198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:25.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.065+0000 7f900652b700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90001013c0 0x7f900019e8b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:25.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.065+0000 7f900652b700 1 --2- 172.21.15.121:0/190385181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9000101d80 0x7f900019edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:25.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.066+0000 7f900652b700 1 -- 172.21.15.121:0/190385181 >> 172.21.15.121:0/190385181 conn(0x7f90000783d0 msgr2=0x7f9000100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:25.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.066+0000 7f900652b700 1 -- 172.21.15.121:0/190385181 shutdown_connections 2024-06-10T00:26:25.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:25.066+0000 7f900652b700 1 -- 172.21.15.121:0/190385181 wait complete. 2024-06-10T00:26:25.521 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:25 smithi121 ceph-mon[32378]: pgmap v381: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:25.521 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:25 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/190385181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-10T00:26:25.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:25 smithi193 ceph-mon[36716]: pgmap v381: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:25.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:25 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/190385181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-10T00:26:25.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:25 smithi135 ceph-mon[36424]: pgmap v381: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:25.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:25 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/190385181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-10T00:26:26.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.375+0000 7f44f526b700 1 -- 172.21.15.121:0/577123599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f006a170 msgr2=0x7f44f006a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:26.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.375+0000 7f44f526b700 1 --2- 172.21.15.121:0/577123599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f006a170 0x7f44f006a5f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009990 tx=0x7f44e4009ca0 comp rx=0 tx=0).stop 2024-06-10T00:26:26.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.376+0000 7f44f526b700 1 -- 172.21.15.121:0/577123599 shutdown_connections 2024-06-10T00:26:26.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.376+0000 7f44f526b700 1 --2- 172.21.15.121:0/577123599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44f006ab30 0x7f44f010f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:26.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.376+0000 7f44f526b700 1 --2- 172.21.15.121:0/577123599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f0108500 0x7f44f01088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:26.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.376+0000 7f44f526b700 1 --2- 172.21.15.121:0/577123599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f006a170 0x7f44f006a5f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:26.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.376+0000 7f44f526b700 1 -- 172.21.15.121:0/577123599 >> 172.21.15.121:0/577123599 conn(0x7f44f0078340 msgr2=0x7f44f0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:26.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.376+0000 7f44f526b700 1 -- 172.21.15.121:0/577123599 shutdown_connections 2024-06-10T00:26:26.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.376+0000 7f44f526b700 1 -- 172.21.15.121:0/577123599 wait complete. 2024-06-10T00:26:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.376+0000 7f44f526b700 1 Processor -- start 2024-06-10T00:26:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.376+0000 7f44f526b700 1 -- start start 2024-06-10T00:26:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44f526b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f006a170 0x7f44f019e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44f526b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44f006ab30 0x7f44f019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44f526b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f0108500 0x7f44f0198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44f526b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f44f01127e0 con 0x7f44f0108500 2024-06-10T00:26:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44f526b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f44f0112660 con 0x7f44f006ab30 2024-06-10T00:26:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44f526b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f44f0112960 con 0x7f44f006a170 2024-06-10T00:26:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44ef7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f0108500 0x7f44f0198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44ef7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f0108500 0x7f44f0198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44774/0 (socket says 172.21.15.121:44774) 2024-06-10T00:26:26.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44ef7fe700 1 -- 172.21.15.121:0/3081897222 learned_addr learned my addr 172.21.15.121:0/3081897222 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44ee7fc700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44f006ab30 0x7f44f019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44eeffd700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f006a170 0x7f44f019e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44ef7fe700 1 -- 172.21.15.121:0/3081897222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f006a170 msgr2=0x7f44f019e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44ef7fe700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f006a170 0x7f44f019e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44ef7fe700 1 -- 172.21.15.121:0/3081897222 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44f006ab30 msgr2=0x7f44f019ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44ef7fe700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44f006ab30 0x7f44f019ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44ef7fe700 1 -- 172.21.15.121:0/3081897222 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f0199200 con 0x7f44f0108500 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.377+0000 7f44ef7fe700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f0108500 0x7f44f0198aa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f44e800b970 tx=0x7f44e800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.378+0000 7f44d7fff700 1 -- 172.21.15.121:0/3081897222 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e800d860 con 0x7f44f0108500 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.378+0000 7f44f526b700 1 -- 172.21.15.121:0/3081897222 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f01994f0 con 0x7f44f0108500 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.378+0000 7f44d7fff700 1 -- 172.21.15.121:0/3081897222 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44e800e430 con 0x7f44f0108500 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.378+0000 7f44d7fff700 1 -- 172.21.15.121:0/3081897222 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8017660 con 0x7f44f0108500 2024-06-10T00:26:26.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.378+0000 7f44f526b700 1 -- 172.21.15.121:0/3081897222 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f01a57b0 con 0x7f44f0108500 2024-06-10T00:26:26.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.379+0000 7f44f526b700 1 -- 172.21.15.121:0/3081897222 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f0069180 con 0x7f44f0108500 2024-06-10T00:26:26.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.380+0000 7f44d7fff700 1 -- 172.21.15.121:0/3081897222 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f44e800e5e0 con 0x7f44f0108500 2024-06-10T00:26:26.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.380+0000 7f44d7fff700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44d80a2cb0 0x7f44d80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:26.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.381+0000 7f44eeffd700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44d80a2cb0 0x7f44d80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:26.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.381+0000 7f44d7fff700 1 -- 172.21.15.121:0/3081897222 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f44e80cf350 con 0x7f44f0108500 2024-06-10T00:26:26.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.381+0000 7f44eeffd700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44d80a2cb0 0x7f44d80a5170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f44e0005950 tx=0x7f44e000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:26.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.384+0000 7f44d7fff700 1 -- 172.21.15.121:0/3081897222 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f44e8013030 con 0x7f44f0108500 2024-06-10T00:26:26.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.588+0000 7f44f526b700 1 -- 172.21.15.121:0/3081897222 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f44f002dca0 con 0x7f44f0108500 2024-06-10T00:26:26.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.588+0000 7f44d7fff700 1 -- 172.21.15.121:0/3081897222 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f44e809bda0 con 0x7f44f0108500 2024-06-10T00:26:26.589 INFO:teuthology.orchestra.run.smithi121.stdout:167503724620 2024-06-10T00:26:26.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 -- 172.21.15.121:0/3081897222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44d80a2cb0 msgr2=0x7f44d80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:26.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44d80a2cb0 0x7f44d80a5170 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f44e0005950 tx=0x7f44e000a300 comp rx=0 tx=0).stop 2024-06-10T00:26:26.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 -- 172.21.15.121:0/3081897222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f0108500 msgr2=0x7f44f0198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:26.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f0108500 0x7f44f0198aa0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f44e800b970 tx=0x7f44e800bc80 comp rx=0 tx=0).stop 2024-06-10T00:26:26.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 -- 172.21.15.121:0/3081897222 shutdown_connections 2024-06-10T00:26:26.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f44d80a2cb0 0x7f44d80a5170 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:26.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f0108500 0x7f44f0198aa0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f006a170 0x7f44f019e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 --2- 172.21.15.121:0/3081897222 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f44f006ab30 0x7f44f019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 -- 172.21.15.121:0/3081897222 >> 172.21.15.121:0/3081897222 conn(0x7f44f0078340 msgr2=0x7f44f00fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 -- 172.21.15.121:0/3081897222 shutdown_connections 2024-06-10T00:26:26.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:26.591+0000 7f44f526b700 1 -- 172.21.15.121:0/3081897222 wait complete. 2024-06-10T00:26:27.513 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:27 smithi121 ceph-mon[32378]: pgmap v382: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:27.513 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:27 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3081897222' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-10T00:26:27.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.540+0000 7fedb523c700 1 -- 172.21.15.121:0/3140539385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb006b3c0 msgr2=0x7fedb010f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:27.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.540+0000 7fedb523c700 1 --2- 172.21.15.121:0/3140539385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb006b3c0 0x7fedb010f310 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7feda800b0a0 tx=0x7feda800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:26:27.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.540+0000 7fedb523c700 1 -- 172.21.15.121:0/3140539385 shutdown_connections 2024-06-10T00:26:27.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.540+0000 7fedb523c700 1 --2- 172.21.15.121:0/3140539385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb006b3c0 0x7fedb010f310 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:27.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.540+0000 7fedb523c700 1 --2- 172.21.15.121:0/3140539385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fedb01084f0 0x7fedb01088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:27.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.540+0000 7fedb523c700 1 --2- 172.21.15.121:0/3140539385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedb006aa00 0x7fedb006ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:27.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.540+0000 7fedb523c700 1 -- 172.21.15.121:0/3140539385 >> 172.21.15.121:0/3140539385 conn(0x7fedb0078300 msgr2=0x7fedb0078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:27.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.540+0000 7fedb523c700 1 -- 172.21.15.121:0/3140539385 shutdown_connections 2024-06-10T00:26:27.540 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.540+0000 7fedb523c700 1 -- 172.21.15.121:0/3140539385 wait complete. 2024-06-10T00:26:27.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedb523c700 1 Processor -- start 2024-06-10T00:26:27.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedb523c700 1 -- start start 2024-06-10T00:26:27.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedb523c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedb006aa00 0x7fedb019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:27.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedb523c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fedb006b3c0 0x7fedb019ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:27.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedb523c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb01084f0 0x7fedb01988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:27.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedb523c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fedb0112750 con 0x7fedb01084f0 2024-06-10T00:26:27.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedb523c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fedb01125d0 con 0x7fedb006aa00 2024-06-10T00:26:27.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedb523c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fedb01128d0 con 0x7fedb006b3c0 2024-06-10T00:26:27.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedaf59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb01084f0 0x7fedb01988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:27.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedaf59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb01084f0 0x7fedb01988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44790/0 (socket says 172.21.15.121:44790) 2024-06-10T00:26:27.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedaf59e700 1 -- 172.21.15.121:0/3296466158 learned_addr learned my addr 172.21.15.121:0/3296466158 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:27.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7fedaed9d700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedb006aa00 0x7fedb019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:27.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.541+0000 7feda659c700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fedb006b3c0 0x7fedb019ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:27.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.542+0000 7fedaf59e700 1 -- 172.21.15.121:0/3296466158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fedb006b3c0 msgr2=0x7fedb019ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:27.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.542+0000 7fedaf59e700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fedb006b3c0 0x7fedb019ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:27.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.542+0000 7fedaf59e700 1 -- 172.21.15.121:0/3296466158 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedb006aa00 msgr2=0x7fedb019e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:27.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.542+0000 7fedaf59e700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedb006aa00 0x7fedb019e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:27.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.542+0000 7fedaf59e700 1 -- 172.21.15.121:0/3296466158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedb0199050 con 0x7fedb01084f0 2024-06-10T00:26:27.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.542+0000 7fedaf59e700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb01084f0 0x7fedb01988f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feda80062a0 tx=0x7feda8004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:27.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.542+0000 7fedacd99700 1 -- 172.21.15.121:0/3296466158 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda8021070 con 0x7fedb01084f0 2024-06-10T00:26:27.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.542+0000 7fedb523c700 1 -- 172.21.15.121:0/3296466158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedb01992e0 con 0x7fedb01084f0 2024-06-10T00:26:27.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.542+0000 7fedacd99700 1 -- 172.21.15.121:0/3296466158 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feda8007c00 con 0x7fedb01084f0 2024-06-10T00:26:27.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.542+0000 7fedacd99700 1 -- 172.21.15.121:0/3296466158 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda801d450 con 0x7fedb01084f0 2024-06-10T00:26:27.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.542+0000 7fedb523c700 1 -- 172.21.15.121:0/3296466158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedb01a5600 con 0x7fedb01084f0 2024-06-10T00:26:27.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.543+0000 7fedb523c700 1 -- 172.21.15.121:0/3296466158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedb0069180 con 0x7fedb01084f0 2024-06-10T00:26:27.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.544+0000 7fedacd99700 1 -- 172.21.15.121:0/3296466158 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7feda8004000 con 0x7fedb01084f0 2024-06-10T00:26:27.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.545+0000 7fedacd99700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fed9c0a2cb0 0x7fed9c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:27.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.545+0000 7fedaed9d700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fed9c0a2cb0 0x7fed9c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:27.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.545+0000 7fedacd99700 1 -- 172.21.15.121:0/3296466158 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7feda800f020 con 0x7fedb01084f0 2024-06-10T00:26:27.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.545+0000 7fedaed9d700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fed9c0a2cb0 0x7fed9c0a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fedb0199a30 tx=0x7feda000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:27.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.548+0000 7fedacd99700 1 -- 172.21.15.121:0/3296466158 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7feda802a020 con 0x7fedb01084f0 2024-06-10T00:26:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:27 smithi193 ceph-mon[36716]: pgmap v382: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:27 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3081897222' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-10T00:26:27.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.752+0000 7fedb523c700 1 -- 172.21.15.121:0/3296466158 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fedb002dca0 con 0x7fedb01084f0 2024-06-10T00:26:27.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.752+0000 7fedacd99700 1 -- 172.21.15.121:0/3296466158 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7feda809d590 con 0x7fedb01084f0 2024-06-10T00:26:27.753 INFO:teuthology.orchestra.run.smithi121.stdout:77309411420 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 -- 172.21.15.121:0/3296466158 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fed9c0a2cb0 msgr2=0x7fed9c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fed9c0a2cb0 0x7fed9c0a5170 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fedb0199a30 tx=0x7feda000b410 comp rx=0 tx=0).stop 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 -- 172.21.15.121:0/3296466158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb01084f0 msgr2=0x7fedb01988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb01084f0 0x7fedb01988f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7feda80062a0 tx=0x7feda8004d70 comp rx=0 tx=0).stop 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 -- 172.21.15.121:0/3296466158 shutdown_connections 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fed9c0a2cb0 0x7fed9c0a5170 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb01084f0 0x7fedb01988f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fedb006aa00 0x7fedb019e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 --2- 172.21.15.121:0/3296466158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fedb006b3c0 0x7fedb019ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 -- 172.21.15.121:0/3296466158 >> 172.21.15.121:0/3296466158 conn(0x7fedb0078300 msgr2=0x7fedb00ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 -- 172.21.15.121:0/3296466158 shutdown_connections 2024-06-10T00:26:27.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:27.755+0000 7fedb523c700 1 -- 172.21.15.121:0/3296466158 wait complete. 2024-06-10T00:26:27.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:27 smithi135 ceph-mon[36424]: pgmap v382: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:27.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:27 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3081897222' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-10T00:26:28.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:28 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3296466158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-10T00:26:28.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:28 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3296466158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-10T00:26:28.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:28 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3296466158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-10T00:26:28.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.915+0000 7fdf82774700 1 -- 172.21.15.121:0/2761109773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf7c102740 msgr2=0x7fdf7c102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:28.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.915+0000 7fdf82774700 1 --2- 172.21.15.121:0/2761109773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf7c102740 0x7fdf7c102b20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0099c0 tx=0x7fdf6c009cd0 comp rx=0 tx=0).stop 2024-06-10T00:26:28.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.916+0000 7fdf82774700 1 -- 172.21.15.121:0/2761109773 shutdown_connections 2024-06-10T00:26:28.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.916+0000 7fdf82774700 1 --2- 172.21.15.121:0/2761109773 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf7c107840 0x7fdf7c10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:28.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.916+0000 7fdf82774700 1 --2- 172.21.15.121:0/2761109773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf7c102740 0x7fdf7c102b20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:28.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.916+0000 7fdf82774700 1 --2- 172.21.15.121:0/2761109773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c1030f0 0x7fdf7c107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:28.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.916+0000 7fdf82774700 1 -- 172.21.15.121:0/2761109773 >> 172.21.15.121:0/2761109773 conn(0x7fdf7c0795f0 msgr2=0x7fdf7c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:28.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.916+0000 7fdf82774700 1 -- 172.21.15.121:0/2761109773 shutdown_connections 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.916+0000 7fdf82774700 1 -- 172.21.15.121:0/2761109773 wait complete. 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.916+0000 7fdf82774700 1 Processor -- start 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.916+0000 7fdf82774700 1 -- start start 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf82774700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf7c102740 0x7fdf7c19e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf82774700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c1030f0 0x7fdf7c19eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf82774700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf7c107840 0x7fdf7c198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf82774700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c112760 con 0x7fdf7c107840 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf82774700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c1125e0 con 0x7fdf7c102740 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf82774700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c1128e0 con 0x7fdf7c1030f0 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf80d11700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf7c107840 0x7fdf7c198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf80d11700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf7c107840 0x7fdf7c198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44818/0 (socket says 172.21.15.121:44818) 2024-06-10T00:26:28.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf80d11700 1 -- 172.21.15.121:0/761604974 learned_addr learned my addr 172.21.15.121:0/761604974 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf7bfff700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf7c102740 0x7fdf7c19e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf7b7fe700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c1030f0 0x7fdf7c19eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf80d11700 1 -- 172.21.15.121:0/761604974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c1030f0 msgr2=0x7fdf7c19eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf80d11700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c1030f0 0x7fdf7c19eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf80d11700 1 -- 172.21.15.121:0/761604974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf7c102740 msgr2=0x7fdf7c19e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf80d11700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf7c102740 0x7fdf7c19e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf80d11700 1 -- 172.21.15.121:0/761604974 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c199080 con 0x7fdf7c107840 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf80d11700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf7c107840 0x7fdf7c198920 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdf74007ee0 tx=0x7fdf7400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf797fa700 1 -- 172.21.15.121:0/761604974 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74004bb0 con 0x7fdf7c107840 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf797fa700 1 -- 172.21.15.121:0/761604974 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf74004d50 con 0x7fdf7c107840 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf82774700 1 -- 172.21.15.121:0/761604974 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7c199370 con 0x7fdf7c107840 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf797fa700 1 -- 172.21.15.121:0/761604974 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf74005960 con 0x7fdf7c107840 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.917+0000 7fdf82774700 1 -- 172.21.15.121:0/761604974 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf7c1a5630 con 0x7fdf7c107840 2024-06-10T00:26:28.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.918+0000 7fdf82774700 1 -- 172.21.15.121:0/761604974 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf7c10aa80 con 0x7fdf7c107840 2024-06-10T00:26:28.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.920+0000 7fdf797fa700 1 -- 172.21.15.121:0/761604974 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fdf7401a020 con 0x7fdf7c107840 2024-06-10T00:26:28.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.920+0000 7fdf797fa700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdf640a2cb0 0x7fdf640a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:28.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.920+0000 7fdf7bfff700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdf640a2cb0 0x7fdf640a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:28.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.920+0000 7fdf797fa700 1 -- 172.21.15.121:0/761604974 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fdf740cf300 con 0x7fdf7c107840 2024-06-10T00:26:28.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.921+0000 7fdf7bfff700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdf640a2cb0 0x7fdf640a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c000c00 tx=0x7fdf6c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:28.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:28.924+0000 7fdf797fa700 1 -- 172.21.15.121:0/761604974 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdf74010070 con 0x7fdf7c107840 2024-06-10T00:26:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.129+0000 7fdf82774700 1 -- 172.21.15.121:0/761604974 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fdf7c10a460 con 0x7fdf7c107840 2024-06-10T00:26:29.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.129+0000 7fdf797fa700 1 -- 172.21.15.121:0/761604974 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fdf7409bd50 con 0x7fdf7c107840 2024-06-10T00:26:29.129 INFO:teuthology.orchestra.run.smithi121.stdout:347892351023 2024-06-10T00:26:29.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 -- 172.21.15.121:0/761604974 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdf640a2cb0 msgr2=0x7fdf640a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdf640a2cb0 0x7fdf640a5170 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c000c00 tx=0x7fdf6c005fb0 comp rx=0 tx=0).stop 2024-06-10T00:26:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 -- 172.21.15.121:0/761604974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf7c107840 msgr2=0x7fdf7c198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf7c107840 0x7fdf7c198920 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdf74007ee0 tx=0x7fdf7400d3b0 comp rx=0 tx=0).stop 2024-06-10T00:26:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 -- 172.21.15.121:0/761604974 shutdown_connections 2024-06-10T00:26:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdf640a2cb0 0x7fdf640a5170 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf7c107840 0x7fdf7c198920 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf7c102740 0x7fdf7c19e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 --2- 172.21.15.121:0/761604974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf7c1030f0 0x7fdf7c19eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 -- 172.21.15.121:0/761604974 >> 172.21.15.121:0/761604974 conn(0x7fdf7c0795f0 msgr2=0x7fdf7c1004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 -- 172.21.15.121:0/761604974 shutdown_connections 2024-06-10T00:26:29.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.132+0000 7fdf82774700 1 -- 172.21.15.121:0/761604974 wait complete. 2024-06-10T00:26:29.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:29 smithi121 ceph-mon[32378]: pgmap v383: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:26:29.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:29 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/761604974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-10T00:26:29.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:29 smithi193 ceph-mon[36716]: pgmap v383: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:26:29.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:29 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/761604974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-10T00:26:29.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:29 smithi135 ceph-mon[36424]: pgmap v383: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:26:29.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:29 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/761604974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-06-10T00:26:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:30 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1440443140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-10T00:26:30.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.861+0000 7ff5266bf700 1 -- 172.21.15.121:0/3271996969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff520102740 msgr2=0x7ff520102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:30.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.861+0000 7ff5266bf700 1 --2- 172.21.15.121:0/3271996969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff520102740 0x7ff520102b20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff5100099c0 tx=0x7ff510009cd0 comp rx=0 tx=0).stop 2024-06-10T00:26:30.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.862+0000 7ff5266bf700 1 -- 172.21.15.121:0/3271996969 shutdown_connections 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.862+0000 7ff5266bf700 1 --2- 172.21.15.121:0/3271996969 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff520107840 0x7ff52010f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.862+0000 7ff5266bf700 1 --2- 172.21.15.121:0/3271996969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff520102740 0x7ff520102b20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.862+0000 7ff5266bf700 1 --2- 172.21.15.121:0/3271996969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5201030f0 0x7ff520107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.862+0000 7ff5266bf700 1 -- 172.21.15.121:0/3271996969 >> 172.21.15.121:0/3271996969 conn(0x7ff5200795f0 msgr2=0x7ff520079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.862+0000 7ff5266bf700 1 -- 172.21.15.121:0/3271996969 shutdown_connections 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.862+0000 7ff5266bf700 1 -- 172.21.15.121:0/3271996969 wait complete. 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.862+0000 7ff5266bf700 1 Processor -- start 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.862+0000 7ff5266bf700 1 -- start start 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff5266bf700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff520102740 0x7ff52019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff5266bf700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5201030f0 0x7ff52019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff5266bf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff520107840 0x7ff520198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:30.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff5266bf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff520112800 con 0x7ff520107840 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff5266bf700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff520112680 con 0x7ff520102740 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff5266bf700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff520112980 con 0x7ff5201030f0 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff524c5c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff520107840 0x7ff520198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff524c5c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff520107840 0x7ff520198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36134/0 (socket says 172.21.15.121:36134) 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff524c5c700 1 -- 172.21.15.121:0/1440443140 learned_addr learned my addr 172.21.15.121:0/1440443140 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff51f7fe700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5201030f0 0x7ff52019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff51ffff700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff520102740 0x7ff52019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff524c5c700 1 -- 172.21.15.121:0/1440443140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5201030f0 msgr2=0x7ff52019eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff524c5c700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5201030f0 0x7ff52019eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff524c5c700 1 -- 172.21.15.121:0/1440443140 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff520102740 msgr2=0x7ff52019e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff524c5c700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff520102740 0x7ff52019e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:30.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff524c5c700 1 -- 172.21.15.121:0/1440443140 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff520199180 con 0x7ff520107840 2024-06-10T00:26:30.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff524c5c700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff520107840 0x7ff520198a20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff51800b970 tx=0x7ff51800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:30.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff51d7fa700 1 -- 172.21.15.121:0/1440443140 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff518019070 con 0x7ff520107840 2024-06-10T00:26:30.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff5266bf700 1 -- 172.21.15.121:0/1440443140 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff520199470 con 0x7ff520107840 2024-06-10T00:26:30.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff51d7fa700 1 -- 172.21.15.121:0/1440443140 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff518014440 con 0x7ff520107840 2024-06-10T00:26:30.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff51d7fa700 1 -- 172.21.15.121:0/1440443140 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5180075d0 con 0x7ff520107840 2024-06-10T00:26:30.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.863+0000 7ff5266bf700 1 -- 172.21.15.121:0/1440443140 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5201a5730 con 0x7ff520107840 2024-06-10T00:26:30.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.864+0000 7ff5266bf700 1 -- 172.21.15.121:0/1440443140 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff52010aa80 con 0x7ff520107840 2024-06-10T00:26:30.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.866+0000 7ff51d7fa700 1 -- 172.21.15.121:0/1440443140 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff51801c070 con 0x7ff520107840 2024-06-10T00:26:30.500 INFO:teuthology.orchestra.run.smithi121.stdout:313532612659 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.866+0000 7ff51d7fa700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff5080a2cb0 0x7ff5080a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.867+0000 7ff51d7fa700 1 -- 172.21.15.121:0/1440443140 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7ff5180ceda0 con 0x7ff520107840 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.867+0000 7ff51ffff700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff5080a2cb0 0x7ff5080a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.867+0000 7ff51ffff700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff5080a2cb0 0x7ff5080a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff510000c00 tx=0x7ff510005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:29.870+0000 7ff51d7fa700 1 -- 172.21.15.121:0/1440443140 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff51809b900 con 0x7ff520107840 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.069+0000 7ff5266bf700 1 -- 172.21.15.121:0/1440443140 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7ff52010a460 con 0x7ff520107840 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.069+0000 7ff51d7fa700 1 -- 172.21.15.121:0/1440443140 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff51809b200 con 0x7ff520107840 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 -- 172.21.15.121:0/1440443140 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff5080a2cb0 msgr2=0x7ff5080a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff5080a2cb0 0x7ff5080a5170 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff510000c00 tx=0x7ff510005fd0 comp rx=0 tx=0).stop 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 -- 172.21.15.121:0/1440443140 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff520107840 msgr2=0x7ff520198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff520107840 0x7ff520198a20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff51800b970 tx=0x7ff51800bd30 comp rx=0 tx=0).stop 2024-06-10T00:26:30.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 -- 172.21.15.121:0/1440443140 shutdown_connections 2024-06-10T00:26:30.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff5080a2cb0 0x7ff5080a5170 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:30.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff520107840 0x7ff520198a20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:30.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff520102740 0x7ff52019e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:30.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 --2- 172.21.15.121:0/1440443140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5201030f0 0x7ff52019eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:30.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 -- 172.21.15.121:0/1440443140 >> 172.21.15.121:0/1440443140 conn(0x7ff5200795f0 msgr2=0x7ff520101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:30.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 -- 172.21.15.121:0/1440443140 shutdown_connections 2024-06-10T00:26:30.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:30.071+0000 7ff5266bf700 1 -- 172.21.15.121:0/1440443140 wait complete. 2024-06-10T00:26:30.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:30 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1440443140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-10T00:26:30.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:30 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1440443140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-06-10T00:26:31.124 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612654 got 313532612659 for osd.10 2024-06-10T00:26:31.124 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:31.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:31 smithi121 ceph-mon[32378]: pgmap v384: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:31.486 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724617 got 167503724620 for osd.5 2024-06-10T00:26:31.486 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:31.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:31 smithi193 ceph-mon[36716]: pgmap v384: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:31.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:31 smithi135 ceph-mon[36424]: pgmap v384: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:32.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.679+0000 7f5d7ca95700 1 -- 172.21.15.121:0/112519049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d781010b0 msgr2=0x7f5d78101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:32.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.679+0000 7f5d7ca95700 1 --2- 172.21.15.121:0/112519049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d781010b0 0x7f5d78101490 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800aaa0 tx=0x7f5d6800adb0 comp rx=0 tx=0).stop 2024-06-10T00:26:32.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.680+0000 7f5d7ca95700 1 -- 172.21.15.121:0/112519049 shutdown_connections 2024-06-10T00:26:32.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.680+0000 7f5d7ca95700 1 --2- 172.21.15.121:0/112519049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d78105f50 0x7f5d7810da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:32.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.680+0000 7f5d7ca95700 1 --2- 172.21.15.121:0/112519049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d781010b0 0x7f5d78101490 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.680+0000 7f5d7ca95700 1 --2- 172.21.15.121:0/112519049 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d78101a60 0x7f5d78105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.680+0000 7f5d7ca95700 1 -- 172.21.15.121:0/112519049 >> 172.21.15.121:0/112519049 conn(0x7f5d780766b0 msgr2=0x7f5d78076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.680+0000 7f5d7ca95700 1 -- 172.21.15.121:0/112519049 shutdown_connections 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.680+0000 7f5d7ca95700 1 -- 172.21.15.121:0/112519049 wait complete. 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d7ca95700 1 Processor -- start 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d7ca95700 1 -- start start 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d7ca95700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d781010b0 0x7f5d7819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d7ca95700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d78101a60 0x7f5d7819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d7ca95700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d78105f50 0x7f5d78196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d7ca95700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78110670 con 0x7f5d78105f50 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d7ca95700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5d781104f0 con 0x7f5d78101a60 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d7ca95700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5d781107f0 con 0x7f5d781010b0 2024-06-10T00:26:32.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d76d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d78105f50 0x7f5d78196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:32.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d76d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d78105f50 0x7f5d78196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36148/0 (socket says 172.21.15.121:36148) 2024-06-10T00:26:32.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d76d9d700 1 -- 172.21.15.121:0/1110503328 learned_addr learned my addr 172.21.15.121:0/1110503328 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:32.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d7659c700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d781010b0 0x7f5d7819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:32.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d75d9b700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d78101a60 0x7f5d7819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:32.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d76d9d700 1 -- 172.21.15.121:0/1110503328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d781010b0 msgr2=0x7f5d7819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:32.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d76d9d700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d781010b0 0x7f5d7819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:32.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d76d9d700 1 -- 172.21.15.121:0/1110503328 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d78101a60 msgr2=0x7f5d7819cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:32.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d76d9d700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d78101a60 0x7f5d7819cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:32.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d76d9d700 1 -- 172.21.15.121:0/1110503328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d78196ff0 con 0x7f5d78105f50 2024-06-10T00:26:32.307 INFO:teuthology.orchestra.run.smithi121.stdout:146028888146 2024-06-10T00:26:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.681+0000 7f5d76d9d700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d78105f50 0x7f5d78196890 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000b970 tx=0x7f5d7000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.682+0000 7f5d677fe700 1 -- 172.21.15.121:0/1110503328 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d70019070 con 0x7f5d78105f50 2024-06-10T00:26:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.682+0000 7f5d677fe700 1 -- 172.21.15.121:0/1110503328 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d70014440 con 0x7f5d78105f50 2024-06-10T00:26:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.682+0000 7f5d677fe700 1 -- 172.21.15.121:0/1110503328 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d700075d0 con 0x7f5d78105f50 2024-06-10T00:26:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.682+0000 7f5d7ca95700 1 -- 172.21.15.121:0/1110503328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d781972e0 con 0x7f5d78105f50 2024-06-10T00:26:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.682+0000 7f5d7ca95700 1 -- 172.21.15.121:0/1110503328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d781a35a0 con 0x7f5d78105f50 2024-06-10T00:26:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.683+0000 7f5d7ca95700 1 -- 172.21.15.121:0/1110503328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d78069180 con 0x7f5d78105f50 2024-06-10T00:26:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.684+0000 7f5d677fe700 1 -- 172.21.15.121:0/1110503328 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f5d7001c070 con 0x7f5d78105f50 2024-06-10T00:26:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.684+0000 7f5d677fe700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5d600a2d30 0x7f5d600a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:32.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.685+0000 7f5d7659c700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5d600a2d30 0x7f5d600a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.685+0000 7f5d677fe700 1 -- 172.21.15.121:0/1110503328 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f5d700cece0 con 0x7f5d78105f50 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.685+0000 7f5d7659c700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5d600a2d30 0x7f5d600a51f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800c6d0 tx=0x7f5d68005ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.688+0000 7f5d677fe700 1 -- 172.21.15.121:0/1110503328 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5d7009b840 con 0x7f5d78105f50 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.889+0000 7f5d7ca95700 1 -- 172.21.15.121:0/1110503328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f5d78197dc0 con 0x7f5d78105f50 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.890+0000 7f5d677fe700 1 -- 172.21.15.121:0/1110503328 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5d7009b140 con 0x7f5d78105f50 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 -- 172.21.15.121:0/1110503328 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5d600a2d30 msgr2=0x7f5d600a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5d600a2d30 0x7f5d600a51f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5d6800c6d0 tx=0x7f5d68005ab0 comp rx=0 tx=0).stop 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 -- 172.21.15.121:0/1110503328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d78105f50 msgr2=0x7f5d78196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d78105f50 0x7f5d78196890 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000b970 tx=0x7f5d7000bd30 comp rx=0 tx=0).stop 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 -- 172.21.15.121:0/1110503328 shutdown_connections 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5d600a2d30 0x7f5d600a51f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d78105f50 0x7f5d78196890 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:32.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5d781010b0 0x7f5d7819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:32.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 --2- 172.21.15.121:0/1110503328 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5d78101a60 0x7f5d7819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:32.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 -- 172.21.15.121:0/1110503328 >> 172.21.15.121:0/1110503328 conn(0x7f5d780766b0 msgr2=0x7f5d780fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:32.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 -- 172.21.15.121:0/1110503328 shutdown_connections 2024-06-10T00:26:32.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:31.892+0000 7f5d7ca95700 1 -- 172.21.15.121:0/1110503328 wait complete. 2024-06-10T00:26:32.582 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:32 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1110503328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-10T00:26:32.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:32 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1110503328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-10T00:26:32.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:32 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1110503328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-10T00:26:33.028 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888142 got 146028888146 for osd.4 2024-06-10T00:26:33.028 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:33.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:33 smithi193 ceph-mon[36716]: pgmap v385: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:33.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:33 smithi121 ceph-mon[32378]: pgmap v385: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:33.745 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411417 got 77309411420 for osd.2 2024-06-10T00:26:33.746 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:33.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:33 smithi135 ceph-mon[36424]: pgmap v385: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:34.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.069+0000 7f727b5b9700 1 -- 172.21.15.121:0/3959322905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7274102550 msgr2=0x7f727410e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:34.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.069+0000 7f727b5b9700 1 --2- 172.21.15.121:0/3959322905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7274102550 0x7f727410e210 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f726c00b0a0 tx=0x7f726c00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:26:34.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.069+0000 7f727b5b9700 1 -- 172.21.15.121:0/3959322905 shutdown_connections 2024-06-10T00:26:34.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.069+0000 7f727b5b9700 1 --2- 172.21.15.121:0/3959322905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7274102550 0x7f727410e210 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:34.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.069+0000 7f727b5b9700 1 --2- 172.21.15.121:0/3959322905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7274106370 0x7f7274106750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:34.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.069+0000 7f727b5b9700 1 --2- 172.21.15.121:0/3959322905 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7274101b90 0x7f7274102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:34.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.069+0000 7f727b5b9700 1 -- 172.21.15.121:0/3959322905 >> 172.21.15.121:0/3959322905 conn(0x7f7274077a00 msgr2=0x7f72740fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:34.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.069+0000 7f727b5b9700 1 -- 172.21.15.121:0/3959322905 shutdown_connections 2024-06-10T00:26:34.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.069+0000 7f727b5b9700 1 -- 172.21.15.121:0/3959322905 wait complete. 2024-06-10T00:26:34.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f727b5b9700 1 Processor -- start 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f727b5b9700 1 -- start start 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f727b5b9700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7274101b90 0x7f727419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f727b5b9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7274102550 0x7f727419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f727b5b9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7274106370 0x7f72741987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f727b5b9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7274111640 con 0x7f7274106370 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f727b5b9700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f72741114c0 con 0x7f7274102550 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f727b5b9700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f72741117c0 con 0x7f7274101b90 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f7279b56700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7274106370 0x7f72741987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f7279b56700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7274106370 0x7f72741987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36170/0 (socket says 172.21.15.121:36170) 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f7279b56700 1 -- 172.21.15.121:0/2661668693 learned_addr learned my addr 172.21.15.121:0/2661668693 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f7278b54700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7274102550 0x7f727419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:34.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f7279355700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7274101b90 0x7f727419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f7279b56700 1 -- 172.21.15.121:0/2661668693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7274101b90 msgr2=0x7f727419e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f7279b56700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7274101b90 0x7f727419e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f7279b56700 1 -- 172.21.15.121:0/2661668693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7274102550 msgr2=0x7f727419ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f7279b56700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7274102550 0x7f727419ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.070+0000 7f7279b56700 1 -- 172.21.15.121:0/2661668693 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7274198fd0 con 0x7f7274106370 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.071+0000 7f7279b56700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7274106370 0x7f72741987f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f726c0062a0 tx=0x7f726c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.071+0000 7f726a7fc700 1 -- 172.21.15.121:0/2661668693 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c021070 con 0x7f7274106370 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.071+0000 7f726a7fc700 1 -- 172.21.15.121:0/2661668693 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f726c007c80 con 0x7f7274106370 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.071+0000 7f727b5b9700 1 -- 172.21.15.121:0/2661668693 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7274199260 con 0x7f7274106370 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.071+0000 7f726a7fc700 1 -- 172.21.15.121:0/2661668693 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c005420 con 0x7f7274106370 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.071+0000 7f727b5b9700 1 -- 172.21.15.121:0/2661668693 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72741a54d0 con 0x7f7274106370 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.072+0000 7f727b5b9700 1 -- 172.21.15.121:0/2661668693 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7274102ba0 con 0x7f7274106370 2024-06-10T00:26:34.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.073+0000 7f726a7fc700 1 -- 172.21.15.121:0/2661668693 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f726c024070 con 0x7f7274106370 2024-06-10T00:26:34.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.073+0000 7f726a7fc700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f72600a2d30 0x7f72600a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:34.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.074+0000 7f7279355700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f72600a2d30 0x7f72600a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:34.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.074+0000 7f726a7fc700 1 -- 172.21.15.121:0/2661668693 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f726c00f020 con 0x7f7274106370 2024-06-10T00:26:34.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.074+0000 7f7279355700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f72600a2d30 0x7f72600a51f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f727000ac50 tx=0x7f727000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:34.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.078+0000 7f726a7fc700 1 -- 172.21.15.121:0/2661668693 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f726c02b030 con 0x7f7274106370 2024-06-10T00:26:34.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.292+0000 7f727b5b9700 1 -- 172.21.15.121:0/2661668693 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f72741a56a0 con 0x7f7274106370 2024-06-10T00:26:34.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.293+0000 7f726a7fc700 1 -- 172.21.15.121:0/2661668693 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f726c09d300 con 0x7f7274106370 2024-06-10T00:26:34.293 INFO:teuthology.orchestra.run.smithi121.stdout:55834574946 2024-06-10T00:26:34.294 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:34.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 -- 172.21.15.121:0/2661668693 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f72600a2d30 msgr2=0x7f72600a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:34.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f72600a2d30 0x7f72600a51f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f727000ac50 tx=0x7f727000a380 comp rx=0 tx=0).stop 2024-06-10T00:26:34.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 -- 172.21.15.121:0/2661668693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7274106370 msgr2=0x7f72741987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:34.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7274106370 0x7f72741987f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f726c0062a0 tx=0x7f726c00be60 comp rx=0 tx=0).stop 2024-06-10T00:26:34.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 -- 172.21.15.121:0/2661668693 shutdown_connections 2024-06-10T00:26:34.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f72600a2d30 0x7f72600a51f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:34.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7274106370 0x7f72741987f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:34.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7274101b90 0x7f727419e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:34.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 --2- 172.21.15.121:0/2661668693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7274102550 0x7f727419ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:34.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 -- 172.21.15.121:0/2661668693 >> 172.21.15.121:0/2661668693 conn(0x7f7274077a00 msgr2=0x7f7274100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:34.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 -- 172.21.15.121:0/2661668693 shutdown_connections 2024-06-10T00:26:34.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:34.297+0000 7f727b5b9700 1 -- 172.21.15.121:0/2661668693 wait complete. 2024-06-10T00:26:34.780 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299455 got 223338299458 for osd.7 2024-06-10T00:26:34.780 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:35.345 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:35 smithi121 ceph-mon[32378]: pgmap v386: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:35.346 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:35 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2661668693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-10T00:26:35.378 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574941 got 55834574946 for osd.1 2024-06-10T00:26:35.378 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:35.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:35 smithi193 ceph-mon[36716]: pgmap v386: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:35 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2661668693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-10T00:26:35.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:35 smithi135 ceph-mon[36424]: pgmap v386: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:26:35.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:35 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2661668693' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-10T00:26:35.815 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351018 got 347892351023 for osd.11 2024-06-10T00:26:35.815 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:36.319 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:37.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:37 smithi121 ceph-mon[32378]: pgmap v387: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:37.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.609+0000 7f5203735700 1 -- 172.21.15.121:0/939497695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51fc074fe0 msgr2=0x7f51fc10e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:37.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.609+0000 7f5203735700 1 --2- 172.21.15.121:0/939497695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51fc074fe0 0x7f51fc10e980 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f51ec009990 tx=0x7f51ec009ca0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.609+0000 7f5203735700 1 -- 172.21.15.121:0/939497695 shutdown_connections 2024-06-10T00:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.609+0000 7f5203735700 1 --2- 172.21.15.121:0/939497695 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51fc10eec0 0x7f51fc1112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.609+0000 7f5203735700 1 --2- 172.21.15.121:0/939497695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51fc0746c0 0x7f51fc074aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.609+0000 7f5203735700 1 --2- 172.21.15.121:0/939497695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51fc074fe0 0x7f51fc10e980 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.609+0000 7f5203735700 1 -- 172.21.15.121:0/939497695 >> 172.21.15.121:0/939497695 conn(0x7f51fc079600 msgr2=0x7f51fc079a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.609+0000 7f5203735700 1 -- 172.21.15.121:0/939497695 shutdown_connections 2024-06-10T00:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.609+0000 7f5203735700 1 -- 172.21.15.121:0/939497695 wait complete. 2024-06-10T00:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5203735700 1 Processor -- start 2024-06-10T00:26:37.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5203735700 1 -- start start 2024-06-10T00:26:37.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5203735700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51fc0746c0 0x7f51fc19ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:37.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5203735700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51fc074fe0 0x7f51fc19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:37.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5203735700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51fc10eec0 0x7f51fc19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:37.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5203735700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc1149e0 con 0x7f51fc10eec0 2024-06-10T00:26:37.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5203735700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc114860 con 0x7f51fc074fe0 2024-06-10T00:26:37.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5203735700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc114b60 con 0x7f51fc0746c0 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5201cd2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51fc10eec0 0x7f51fc19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5200cd0700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51fc074fe0 0x7f51fc19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5201cd2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51fc10eec0 0x7f51fc19f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36196/0 (socket says 172.21.15.121:36196) 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f5201cd2700 1 -- 172.21.15.121:0/1839136277 learned_addr learned my addr 172.21.15.121:0/1839136277 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.610+0000 7f52014d1700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51fc0746c0 0x7f51fc19ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f5201cd2700 1 -- 172.21.15.121:0/1839136277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51fc0746c0 msgr2=0x7f51fc19ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f5201cd2700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51fc0746c0 0x7f51fc19ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f5201cd2700 1 -- 172.21.15.121:0/1839136277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51fc074fe0 msgr2=0x7f51fc19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f5201cd2700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51fc074fe0 0x7f51fc19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f5201cd2700 1 -- 172.21.15.121:0/1839136277 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51fc19fe10 con 0x7f51fc10eec0 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f5200cd0700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51fc074fe0 0x7f51fc19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f5201cd2700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51fc10eec0 0x7f51fc19f6d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f51f40094f0 tx=0x7f51f400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f51f27fc700 1 -- 172.21.15.121:0/1839136277 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4004db0 con 0x7f51fc10eec0 2024-06-10T00:26:37.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f5203735700 1 -- 172.21.15.121:0/1839136277 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51fc1a0100 con 0x7f51fc10eec0 2024-06-10T00:26:37.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f51f27fc700 1 -- 172.21.15.121:0/1839136277 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f4009e20 con 0x7f51fc10eec0 2024-06-10T00:26:37.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f51f27fc700 1 -- 172.21.15.121:0/1839136277 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4015400 con 0x7f51fc10eec0 2024-06-10T00:26:37.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.611+0000 7f5203735700 1 -- 172.21.15.121:0/1839136277 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51fc1a7980 con 0x7f51fc10eec0 2024-06-10T00:26:37.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.612+0000 7f5203735700 1 -- 172.21.15.121:0/1839136277 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51fc075db0 con 0x7f51fc10eec0 2024-06-10T00:26:37.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.613+0000 7f51f27fc700 1 -- 172.21.15.121:0/1839136277 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f51f400db20 con 0x7f51fc10eec0 2024-06-10T00:26:37.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.614+0000 7f51f27fc700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f51e80a2cb0 0x7f51e80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:37.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.614+0000 7f52014d1700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f51e80a2cb0 0x7f51e80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:37.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.614+0000 7f51f27fc700 1 -- 172.21.15.121:0/1839136277 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f51f40cefd0 con 0x7f51fc10eec0 2024-06-10T00:26:37.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.614+0000 7f52014d1700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f51e80a2cb0 0x7f51e80a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f51f8004630 tx=0x7f51f800b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:37.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.618+0000 7f51f27fc700 1 -- 172.21.15.121:0/1839136277 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f51f4010070 con 0x7f51fc10eec0 2024-06-10T00:26:37.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:37 smithi193 ceph-mon[36716]: pgmap v387: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:37.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:37 smithi135 ceph-mon[36424]: pgmap v387: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:26:37.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.819+0000 7f5203735700 1 -- 172.21.15.121:0/1839136277 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f51fc19c160 con 0x7f51fc10eec0 2024-06-10T00:26:37.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.819+0000 7f51f27fc700 1 -- 172.21.15.121:0/1839136277 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f51f409ba20 con 0x7f51fc10eec0 2024-06-10T00:26:37.820 INFO:teuthology.orchestra.run.smithi121.stdout:201863462985 2024-06-10T00:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.821+0000 7f5203735700 1 -- 172.21.15.121:0/1839136277 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f51e80a2cb0 msgr2=0x7f51e80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.821+0000 7f5203735700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f51e80a2cb0 0x7f51e80a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f51f8004630 tx=0x7f51f800b3c0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.821+0000 7f5203735700 1 -- 172.21.15.121:0/1839136277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51fc10eec0 msgr2=0x7f51fc19f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:37.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.821+0000 7f5203735700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51fc10eec0 0x7f51fc19f6d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f51f40094f0 tx=0x7f51f400d3b0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.821+0000 7f5203735700 1 -- 172.21.15.121:0/1839136277 shutdown_connections 2024-06-10T00:26:37.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.821+0000 7f5203735700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f51e80a2cb0 0x7f51e80a5170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.821+0000 7f5203735700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51fc10eec0 0x7f51fc19f6d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.821+0000 7f5203735700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51fc0746c0 0x7f51fc19ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.821+0000 7f5203735700 1 --2- 172.21.15.121:0/1839136277 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51fc074fe0 0x7f51fc19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:37.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.821+0000 7f5203735700 1 -- 172.21.15.121:0/1839136277 >> 172.21.15.121:0/1839136277 conn(0x7f51fc079600 msgr2=0x7f51fc0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:37.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.822+0000 7f5203735700 1 -- 172.21.15.121:0/1839136277 shutdown_connections 2024-06-10T00:26:37.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:37.822+0000 7f5203735700 1 -- 172.21.15.121:0/1839136277 wait complete. 2024-06-10T00:26:38.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:38 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1839136277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-10T00:26:38.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:38 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1839136277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-10T00:26:38.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:38 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1839136277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-10T00:26:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:39 smithi121 ceph-mon[32378]: pgmap v388: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:26:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:39 smithi193 ceph-mon[36716]: pgmap v388: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:26:39.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:39 smithi135 ceph-mon[36424]: pgmap v388: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:26:40.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.158+0000 7fe4305be700 1 -- 172.21.15.121:0/386805501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe428108510 msgr2=0x7fe4281088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:40.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.158+0000 7fe4305be700 1 --2- 172.21.15.121:0/386805501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe428108510 0x7fe4281088f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe424009990 tx=0x7fe424009ca0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.158+0000 7fe4305be700 1 -- 172.21.15.121:0/386805501 shutdown_connections 2024-06-10T00:26:40.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.158+0000 7fe4305be700 1 --2- 172.21.15.121:0/386805501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe428101440 0x7fe42810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.158+0000 7fe4305be700 1 --2- 172.21.15.121:0/386805501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe428108510 0x7fe4281088f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.158+0000 7fe4305be700 1 --2- 172.21.15.121:0/386805501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe428100a80 0x7fe428100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.158+0000 7fe4305be700 1 -- 172.21.15.121:0/386805501 >> 172.21.15.121:0/386805501 conn(0x7fe428077a00 msgr2=0x7fe4280fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.158+0000 7fe4305be700 1 -- 172.21.15.121:0/386805501 shutdown_connections 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.158+0000 7fe4305be700 1 -- 172.21.15.121:0/386805501 wait complete. 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe4305be700 1 Processor -- start 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe4305be700 1 -- start start 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe4305be700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe428100a80 0x7fe42819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe4305be700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe428101440 0x7fe42819edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe4305be700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe428108510 0x7fe428198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe4305be700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe428112790 con 0x7fe428100a80 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe4305be700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fe428112610 con 0x7fe428101440 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe4305be700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe428112910 con 0x7fe428108510 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe42e35a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe428100a80 0x7fe42819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe42e35a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe428100a80 0x7fe42819e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48882/0 (socket says 172.21.15.121:48882) 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe42e35a700 1 -- 172.21.15.121:0/1356010817 learned_addr learned my addr 172.21.15.121:0/1356010817 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:40.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.159+0000 7fe42db59700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe428101440 0x7fe42819edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe42eb5b700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe428108510 0x7fe428198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe42e35a700 1 -- 172.21.15.121:0/1356010817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe428108510 msgr2=0x7fe428198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe42e35a700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe428108510 0x7fe428198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe42e35a700 1 -- 172.21.15.121:0/1356010817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe428101440 msgr2=0x7fe42819edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe42e35a700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe428101440 0x7fe42819edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe42e35a700 1 -- 172.21.15.121:0/1356010817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe428199190 con 0x7fe428100a80 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe42e35a700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe428100a80 0x7fe42819e870 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe424000c00 tx=0x7fe4240047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe41f7fe700 1 -- 172.21.15.121:0/1356010817 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe424027070 con 0x7fe428100a80 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe4305be700 1 -- 172.21.15.121:0/1356010817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe428199420 con 0x7fe428100a80 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe41f7fe700 1 -- 172.21.15.121:0/1356010817 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe424003780 con 0x7fe428100a80 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe41f7fe700 1 -- 172.21.15.121:0/1356010817 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe424022dd0 con 0x7fe428100a80 2024-06-10T00:26:40.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.160+0000 7fe4305be700 1 -- 172.21.15.121:0/1356010817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4281a5740 con 0x7fe428100a80 2024-06-10T00:26:40.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.161+0000 7fe4305be700 1 -- 172.21.15.121:0/1356010817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe42810a9f0 con 0x7fe428100a80 2024-06-10T00:26:40.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.162+0000 7fe41f7fe700 1 -- 172.21.15.121:0/1356010817 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fe42402a070 con 0x7fe428100a80 2024-06-10T00:26:40.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.163+0000 7fe41f7fe700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe4140a2cb0 0x7fe4140a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:40.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.163+0000 7fe42db59700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe4140a2cb0 0x7fe4140a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:40.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.163+0000 7fe41f7fe700 1 -- 172.21.15.121:0/1356010817 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fe4240d0600 con 0x7fe428100a80 2024-06-10T00:26:40.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.164+0000 7fe42db59700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe4140a2cb0 0x7fe4140a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe428107ba0 tx=0x7fe41800b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:40.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.171+0000 7fe41f7fe700 1 -- 172.21.15.121:0/1356010817 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe424013070 con 0x7fe428100a80 2024-06-10T00:26:40.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.382+0000 7fe4305be700 1 -- 172.21.15.121:0/1356010817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fe428199f60 con 0x7fe428100a80 2024-06-10T00:26:40.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.382+0000 7fe41f7fe700 1 -- 172.21.15.121:0/1356010817 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe42409d170 con 0x7fe428100a80 2024-06-10T00:26:40.382 INFO:teuthology.orchestra.run.smithi121.stdout:292057776188 2024-06-10T00:26:40.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.384+0000 7fe4305be700 1 -- 172.21.15.121:0/1356010817 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe4140a2cb0 msgr2=0x7fe4140a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:40.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.384+0000 7fe4305be700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe4140a2cb0 0x7fe4140a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe428107ba0 tx=0x7fe41800b3c0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.384+0000 7fe4305be700 1 -- 172.21.15.121:0/1356010817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe428100a80 msgr2=0x7fe42819e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:40.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.384+0000 7fe4305be700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe428100a80 0x7fe42819e870 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe424000c00 tx=0x7fe4240047b0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.385+0000 7fe4305be700 1 -- 172.21.15.121:0/1356010817 shutdown_connections 2024-06-10T00:26:40.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.385+0000 7fe4305be700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fe4140a2cb0 0x7fe4140a5170 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.385+0000 7fe4305be700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe428108510 0x7fe428198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.385+0000 7fe4305be700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe428100a80 0x7fe42819e870 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.385+0000 7fe4305be700 1 --2- 172.21.15.121:0/1356010817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe428101440 0x7fe42819edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:40.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.385+0000 7fe4305be700 1 -- 172.21.15.121:0/1356010817 >> 172.21.15.121:0/1356010817 conn(0x7fe428077a00 msgr2=0x7fe4280ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:40.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.385+0000 7fe4305be700 1 -- 172.21.15.121:0/1356010817 shutdown_connections 2024-06-10T00:26:40.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.385+0000 7fe4305be700 1 -- 172.21.15.121:0/1356010817 wait complete. 2024-06-10T00:26:40.404 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:40 smithi121 ceph-mon[32378]: pgmap v389: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:26:40.405 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:40 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1544485650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-10T00:26:40.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:40 smithi193 ceph-mon[36716]: pgmap v389: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:26:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:40 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1544485650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-10T00:26:40.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:40 smithi135 ceph-mon[36424]: pgmap v389: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:26:40.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:40 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1544485650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-10T00:26:41.067 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462978 got 201863462985 for osd.6 2024-06-10T00:26:41.068 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:41.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.260+0000 7f2287666700 1 -- 172.21.15.121:0/2038660792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2280101fd0 msgr2=0x7f2280105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:41.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.260+0000 7f2287666700 1 --2- 172.21.15.121:0/2038660792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2280101fd0 0x7f2280105fd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f22740099c0 tx=0x7f2274009cd0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.260+0000 7f2287666700 1 -- 172.21.15.121:0/2038660792 shutdown_connections 2024-06-10T00:26:41.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.260+0000 7f2287666700 1 --2- 172.21.15.121:0/2038660792 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22801066b0 0x7f228010e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.260+0000 7f2287666700 1 --2- 172.21.15.121:0/2038660792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2280101620 0x7f2280101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.260+0000 7f2287666700 1 --2- 172.21.15.121:0/2038660792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2280101fd0 0x7f2280105fd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.260+0000 7f2287666700 1 -- 172.21.15.121:0/2038660792 >> 172.21.15.121:0/2038660792 conn(0x7f2280078340 msgr2=0x7f2280078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:41.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.260+0000 7f2287666700 1 -- 172.21.15.121:0/2038660792 shutdown_connections 2024-06-10T00:26:41.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.260+0000 7f2287666700 1 -- 172.21.15.121:0/2038660792 wait complete. 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2287666700 1 Processor -- start 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2287666700 1 -- start start 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2287666700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2280101620 0x7f228019e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2287666700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2280101fd0 0x7f228019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2287666700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22801066b0 0x7f2280198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2287666700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2280111690 con 0x7f2280101fd0 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2287666700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2280111510 con 0x7f2280101620 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2287666700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2280111810 con 0x7f22801066b0 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2284c01700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2280101fd0 0x7f228019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2284c01700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2280101fd0 0x7f228019ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48900/0 (socket says 172.21.15.121:48900) 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2284c01700 1 -- 172.21.15.121:0/2540121782 learned_addr learned my addr 172.21.15.121:0/2540121782 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2285402700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2280101620 0x7f228019e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:41.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.261+0000 7f2285c03700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22801066b0 0x7f2280198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f2284c01700 1 -- 172.21.15.121:0/2540121782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22801066b0 msgr2=0x7f2280198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f2284c01700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22801066b0 0x7f2280198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f2284c01700 1 -- 172.21.15.121:0/2540121782 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2280101620 msgr2=0x7f228019e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f2284c01700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2280101620 0x7f228019e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f2284c01700 1 -- 172.21.15.121:0/2540121782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2280199200 con 0x7f2280101fd0 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f2285c03700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22801066b0 0x7f2280198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f2284c01700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2280101fd0 0x7f228019ee20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2274000c00 tx=0x7f2274004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f22727fc700 1 -- 172.21.15.121:0/2540121782 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274027070 con 0x7f2280101fd0 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f2287666700 1 -- 172.21.15.121:0/2540121782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2280199490 con 0x7f2280101fd0 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f22727fc700 1 -- 172.21.15.121:0/2540121782 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22740040b0 con 0x7f2280101fd0 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f22727fc700 1 -- 172.21.15.121:0/2540121782 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227400f460 con 0x7f2280101fd0 2024-06-10T00:26:41.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.262+0000 7f2287666700 1 -- 172.21.15.121:0/2540121782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22801a57b0 con 0x7f2280101fd0 2024-06-10T00:26:41.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.264+0000 7f2287666700 1 -- 172.21.15.121:0/2540121782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2280069180 con 0x7f2280101fd0 2024-06-10T00:26:41.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.265+0000 7f22727fc700 1 -- 172.21.15.121:0/2540121782 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f227402a070 con 0x7f2280101fd0 2024-06-10T00:26:41.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.266+0000 7f22727fc700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f226c0a2cb0 0x7f226c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:41.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.266+0000 7f2285402700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f226c0a2cb0 0x7f226c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:41.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.266+0000 7f22727fc700 1 -- 172.21.15.121:0/2540121782 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f22740d02c0 con 0x7f2280101fd0 2024-06-10T00:26:41.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.266+0000 7f2285402700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f226c0a2cb0 0x7f226c0a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f227c005950 tx=0x7f227c00a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:41.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.271+0000 7f22727fc700 1 -- 172.21.15.121:0/2540121782 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2274013070 con 0x7f2280101fd0 2024-06-10T00:26:41.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.047+0000 7fd7b13ad700 1 -- 172.21.15.121:0/902369502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac101440 msgr2=0x7fd7ac10f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:41.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.047+0000 7fd7b13ad700 1 --2- 172.21.15.121:0/902369502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac101440 0x7fd7ac10f330 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd7a400b0a0 tx=0x7fd7a400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.047+0000 7fd7b13ad700 1 -- 172.21.15.121:0/902369502 shutdown_connections 2024-06-10T00:26:41.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.047+0000 7fd7b13ad700 1 --2- 172.21.15.121:0/902369502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac101440 0x7fd7ac10f330 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.047+0000 7fd7b13ad700 1 --2- 172.21.15.121:0/902369502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac108510 0x7fd7ac1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.047+0000 7fd7b13ad700 1 --2- 172.21.15.121:0/902369502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ac100a80 0x7fd7ac100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.047+0000 7fd7b13ad700 1 -- 172.21.15.121:0/902369502 >> 172.21.15.121:0/902369502 conn(0x7fd7ac077a00 msgr2=0x7fd7ac0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:41.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.047+0000 7fd7b13ad700 1 -- 172.21.15.121:0/902369502 shutdown_connections 2024-06-10T00:26:41.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.047+0000 7fd7b13ad700 1 -- 172.21.15.121:0/902369502 wait complete. 2024-06-10T00:26:41.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.048+0000 7fd7b13ad700 1 Processor -- start 2024-06-10T00:26:41.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.048+0000 7fd7b13ad700 1 -- start start 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.048+0000 7fd7b13ad700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac100a80 0x7fd7ac19e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.048+0000 7fd7b13ad700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ac101440 0x7fd7ac19ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.048+0000 7fd7b13ad700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac108510 0x7fd7ac198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.048+0000 7fd7b13ad700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac112790 con 0x7fd7ac100a80 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.048+0000 7fd7b13ad700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac112610 con 0x7fd7ac101440 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.048+0000 7fd7b13ad700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac112910 con 0x7fd7ac108510 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7aaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac100a80 0x7fd7ac19e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7aaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac100a80 0x7fd7ac19e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48856/0 (socket says 172.21.15.121:48856) 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7aaffd700 1 -- 172.21.15.121:0/1544485650 learned_addr learned my addr 172.21.15.121:0/1544485650 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7ab7fe700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac108510 0x7fd7ac198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7a3fff700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ac101440 0x7fd7ac19ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7aaffd700 1 -- 172.21.15.121:0/1544485650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac108510 msgr2=0x7fd7ac198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:41.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7aaffd700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac108510 0x7fd7ac198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.409 INFO:teuthology.orchestra.run.smithi121.stdout:111669149785 2024-06-10T00:26:41.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7aaffd700 1 -- 172.21.15.121:0/1544485650 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ac101440 msgr2=0x7fd7ac19ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:41.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7aaffd700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ac101440 0x7fd7ac19ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7aaffd700 1 -- 172.21.15.121:0/1544485650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac1990b0 con 0x7fd7ac100a80 2024-06-10T00:26:41.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7a3fff700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ac101440 0x7fd7ac19ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:26:41.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7aaffd700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac100a80 0x7fd7ac19e790 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00d980 tx=0x7fd79c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:41.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7a8ff9700 1 -- 172.21.15.121:0/1544485650 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c017070 con 0x7fd7ac100a80 2024-06-10T00:26:41.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7a8ff9700 1 -- 172.21.15.121:0/1544485650 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd79c005020 con 0x7fd7ac100a80 2024-06-10T00:26:41.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7b13ad700 1 -- 172.21.15.121:0/1544485650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ac1993a0 con 0x7fd7ac100a80 2024-06-10T00:26:41.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7a8ff9700 1 -- 172.21.15.121:0/1544485650 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c009bc0 con 0x7fd7ac100a80 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.049+0000 7fd7b13ad700 1 -- 172.21.15.121:0/1544485650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ac1a5660 con 0x7fd7ac100a80 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.050+0000 7fd7b13ad700 1 -- 172.21.15.121:0/1544485650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ac069180 con 0x7fd7ac100a80 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.052+0000 7fd7a8ff9700 1 -- 172.21.15.121:0/1544485650 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fd79c0051d0 con 0x7fd7ac100a80 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.052+0000 7fd7a8ff9700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7980a2cb0 0x7fd7980a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.052+0000 7fd7a3fff700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7980a2cb0 0x7fd7980a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.052+0000 7fd7a8ff9700 1 -- 172.21.15.121:0/1544485650 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fd79c0cf820 con 0x7fd7ac100a80 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.053+0000 7fd7a3fff700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7980a2cb0 0x7fd7980a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd794005fd0 tx=0x7fd794005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.056+0000 7fd7a8ff9700 1 -- 172.21.15.121:0/1544485650 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd79c020020 con 0x7fd7ac100a80 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.266+0000 7fd7b13ad700 1 -- 172.21.15.121:0/1544485650 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd7ac1a5830 con 0x7fd7ac100a80 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.266+0000 7fd7a8ff9700 1 -- 172.21.15.121:0/1544485650 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd79c09c270 con 0x7fd7ac100a80 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.268+0000 7fd7b13ad700 1 -- 172.21.15.121:0/1544485650 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7980a2cb0 msgr2=0x7fd7980a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.268+0000 7fd7b13ad700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7980a2cb0 0x7fd7980a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd794005fd0 tx=0x7fd794005e20 comp rx=0 tx=0).stop 2024-06-10T00:26:41.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.268+0000 7fd7b13ad700 1 -- 172.21.15.121:0/1544485650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac100a80 msgr2=0x7fd7ac19e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:41.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.268+0000 7fd7b13ad700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac100a80 0x7fd7ac19e790 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00d980 tx=0x7fd79c00dc90 comp rx=0 tx=0).stop 2024-06-10T00:26:41.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.268+0000 7fd7b13ad700 1 -- 172.21.15.121:0/1544485650 shutdown_connections 2024-06-10T00:26:41.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.268+0000 7fd7b13ad700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7980a2cb0 0x7fd7980a5170 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.268+0000 7fd7b13ad700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7ac108510 0x7fd7ac198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.268+0000 7fd7b13ad700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7ac100a80 0x7fd7ac19e790 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.268+0000 7fd7b13ad700 1 --2- 172.21.15.121:0/1544485650 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd7ac101440 0x7fd7ac19ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.268+0000 7fd7b13ad700 1 -- 172.21.15.121:0/1544485650 >> 172.21.15.121:0/1544485650 conn(0x7fd7ac077a00 msgr2=0x7fd7ac0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:41.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.268+0000 7fd7b13ad700 1 -- 172.21.15.121:0/1544485650 shutdown_connections 2024-06-10T00:26:41.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:40.269+0000 7fd7b13ad700 1 -- 172.21.15.121:0/1544485650 wait complete. 2024-06-10T00:26:41.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.472+0000 7f2287666700 1 -- 172.21.15.121:0/2540121782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f228002dca0 con 0x7f2280101fd0 2024-06-10T00:26:41.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.472+0000 7f22727fc700 1 -- 172.21.15.121:0/2540121782 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f227409cda0 con 0x7f2280101fd0 2024-06-10T00:26:41.472 INFO:teuthology.orchestra.run.smithi121.stdout:34359738471 2024-06-10T00:26:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 -- 172.21.15.121:0/2540121782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f226c0a2cb0 msgr2=0x7f226c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f226c0a2cb0 0x7f226c0a5170 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f227c005950 tx=0x7f227c00a300 comp rx=0 tx=0).stop 2024-06-10T00:26:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 -- 172.21.15.121:0/2540121782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2280101fd0 msgr2=0x7f228019ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2280101fd0 0x7f228019ee20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2274000c00 tx=0x7f2274004580 comp rx=0 tx=0).stop 2024-06-10T00:26:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 -- 172.21.15.121:0/2540121782 shutdown_connections 2024-06-10T00:26:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f226c0a2cb0 0x7f226c0a5170 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22801066b0 0x7f2280198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2280101620 0x7f228019e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 --2- 172.21.15.121:0/2540121782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2280101fd0 0x7f228019ee20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:41.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 -- 172.21.15.121:0/2540121782 >> 172.21.15.121:0/2540121782 conn(0x7f2280078340 msgr2=0x7f228006aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:41.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 -- 172.21.15.121:0/2540121782 shutdown_connections 2024-06-10T00:26:41.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:41.475+0000 7f2287666700 1 -- 172.21.15.121:0/2540121782 wait complete. 2024-06-10T00:26:41.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:41 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1356010817' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-10T00:26:41.694 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:41 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1356010817' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-10T00:26:41.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:41 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1356010817' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-06-10T00:26:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:42 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2540121782' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-10T00:26:42.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:42 smithi121 ceph-mon[32378]: pgmap v390: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:42.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:42 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2540121782' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-10T00:26:42.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:42 smithi193 ceph-mon[36716]: pgmap v390: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:42.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:42 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2540121782' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-10T00:26:42.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:42 smithi135 ceph-mon[36424]: pgmap v390: 129 pgs: 129 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:26:42.802 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738464 got 34359738471 for osd.0 2024-06-10T00:26:42.802 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:43.193 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776183 got 292057776188 for osd.9 2024-06-10T00:26:43.194 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:43.276 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149778 got 111669149785 for osd.3 2024-06-10T00:26:43.277 DEBUG:teuthology.parallel:result is None 2024-06-10T00:26:43.277 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-10T00:26:43.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-10T00:26:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:43 smithi121 ceph-mon[32378]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-06-10T00:26:43.550 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:43 smithi193 ceph-mon[36716]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-06-10T00:26:43.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:43 smithi135 ceph-mon[36424]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-06-10T00:26:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:26:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:26:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:26:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:26:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:26:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:44 smithi193 ceph-mon[36716]: pgmap v391: 129 pgs: 129 active+clean; 642 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-06-10T00:26:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:26:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:26:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:26:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:26:44.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:26:44.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:44 smithi121 ceph-mon[32378]: pgmap v391: 129 pgs: 129 active+clean; 642 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-06-10T00:26:44.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:26:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:26:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:26:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:26:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:26:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:44 smithi135 ceph-mon[36424]: pgmap v391: 129 pgs: 129 active+clean; 642 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-06-10T00:26:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.422+0000 7f7151caa700 1 -- 172.21.15.121:0/3770375050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c101fc0 msgr2=0x7f714c106010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.422+0000 7f7151caa700 1 --2- 172.21.15.121:0/3770375050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c101fc0 0x7f714c106010 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f71340099c0 tx=0x7f7134009cd0 comp rx=0 tx=0).stop 2024-06-10T00:26:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.422+0000 7f7151caa700 1 -- 172.21.15.121:0/3770375050 shutdown_connections 2024-06-10T00:26:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.422+0000 7f7151caa700 1 --2- 172.21.15.121:0/3770375050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f714c1066f0 0x7f714c10e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.422+0000 7f7151caa700 1 --2- 172.21.15.121:0/3770375050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f714c101610 0x7f714c1019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:45.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.422+0000 7f7151caa700 1 --2- 172.21.15.121:0/3770375050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c101fc0 0x7f714c106010 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.422+0000 7f7151caa700 1 -- 172.21.15.121:0/3770375050 >> 172.21.15.121:0/3770375050 conn(0x7f714c078430 msgr2=0x7f714c078840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.422+0000 7f7151caa700 1 -- 172.21.15.121:0/3770375050 shutdown_connections 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.422+0000 7f7151caa700 1 -- 172.21.15.121:0/3770375050 wait complete. 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f7151caa700 1 Processor -- start 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f7151caa700 1 -- start start 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f7151caa700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f714c101610 0x7f714c19ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f7151caa700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f714c101fc0 0x7f714c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f7151caa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1066f0 0x7f714c198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f7151caa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f714c112830 con 0x7f714c1066f0 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f7151caa700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f714c1126b0 con 0x7f714c101610 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f7151caa700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f714c1129b0 con 0x7f714c101fc0 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f714bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1066f0 0x7f714c198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:45.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f714bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1066f0 0x7f714c198b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48914/0 (socket says 172.21.15.121:48914) 2024-06-10T00:26:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f714bfff700 1 -- 172.21.15.121:0/348165641 learned_addr learned my addr 172.21.15.121:0/348165641 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.423+0000 7f714b7fe700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f714c101610 0x7f714c19ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.424+0000 7f714bfff700 1 -- 172.21.15.121:0/348165641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f714c101fc0 msgr2=0x7f714c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.424+0000 7f714bfff700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f714c101fc0 0x7f714c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.424+0000 7f714bfff700 1 -- 172.21.15.121:0/348165641 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f714c101610 msgr2=0x7f714c19ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.424+0000 7f714bfff700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f714c101610 0x7f714c19ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.424+0000 7f714bfff700 1 -- 172.21.15.121:0/348165641 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714c1993f0 con 0x7f714c1066f0 2024-06-10T00:26:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.424+0000 7f714bfff700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1066f0 0x7f714c198b10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7144007ee0 tx=0x7f714400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.424+0000 7f71497fa700 1 -- 172.21.15.121:0/348165641 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714400dbd0 con 0x7f714c1066f0 2024-06-10T00:26:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.424+0000 7f71497fa700 1 -- 172.21.15.121:0/348165641 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f714400dd70 con 0x7f714c1066f0 2024-06-10T00:26:45.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.424+0000 7f7151caa700 1 -- 172.21.15.121:0/348165641 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f714c1996e0 con 0x7f714c1066f0 2024-06-10T00:26:45.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.424+0000 7f71497fa700 1 -- 172.21.15.121:0/348165641 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144005a60 con 0x7f714c1066f0 2024-06-10T00:26:45.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.424+0000 7f7151caa700 1 -- 172.21.15.121:0/348165641 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f714c1a59c0 con 0x7f714c1066f0 2024-06-10T00:26:45.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.425+0000 7f7151caa700 1 -- 172.21.15.121:0/348165641 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f714c102af0 con 0x7f714c1066f0 2024-06-10T00:26:45.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.426+0000 7f71497fa700 1 -- 172.21.15.121:0/348165641 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f714401a020 con 0x7f714c1066f0 2024-06-10T00:26:45.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.427+0000 7f71497fa700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f71380a2ca0 0x7f71380a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:45.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.427+0000 7f714b7fe700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f71380a2ca0 0x7f71380a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:45.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.427+0000 7f71497fa700 1 -- 172.21.15.121:0/348165641 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f71440cf130 con 0x7f714c1066f0 2024-06-10T00:26:45.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.427+0000 7f714b7fe700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f71380a2ca0 0x7f71380a5160 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f713c009a10 tx=0x7f713c009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:45.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.430+0000 7f71497fa700 1 -- 172.21.15.121:0/348165641 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7144010070 con 0x7f714c1066f0 2024-06-10T00:26:45.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.624+0000 7f7151caa700 1 -- 172.21.15.121:0/348165641 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f714c199fd0 con 0x7f71380a2ca0 2024-06-10T00:26:45.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.634+0000 7f71497fa700 1 -- 172.21.15.121:0/348165641 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340942 (secure 0 0 0) 0x7f714c199fd0 con 0x7f71380a2ca0 2024-06-10T00:26:45.639 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:26:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 -- 172.21.15.121:0/348165641 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f71380a2ca0 msgr2=0x7f71380a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f71380a2ca0 0x7f71380a5160 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f713c009a10 tx=0x7f713c009190 comp rx=0 tx=0).stop 2024-06-10T00:26:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 -- 172.21.15.121:0/348165641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1066f0 msgr2=0x7f714c198b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1066f0 0x7f714c198b10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7144007ee0 tx=0x7f714400d3b0 comp rx=0 tx=0).stop 2024-06-10T00:26:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 -- 172.21.15.121:0/348165641 shutdown_connections 2024-06-10T00:26:45.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f71380a2ca0 0x7f71380a5160 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:45.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1066f0 0x7f714c198b10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:45.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f714c101610 0x7f714c19ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:45.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 --2- 172.21.15.121:0/348165641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f714c101fc0 0x7f714c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:45.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 -- 172.21.15.121:0/348165641 >> 172.21.15.121:0/348165641 conn(0x7f714c078430 msgr2=0x7f714c0ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:45.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 -- 172.21.15.121:0/348165641 shutdown_connections 2024-06-10T00:26:45.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:45.637+0000 7f7151caa700 1 -- 172.21.15.121:0/348165641 wait complete. 2024-06-10T00:26:45.643 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-06-10T00:26:46.411 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-06-10T00:26:44.138613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657506,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":753,"num_read_kb":501,"num_write":356,"num_write_kb":3062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":539,"ondisk_log_size":539,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":362584,"kb_used_data":13412,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124579240,"statfs":{"total":1151940427776,"available":1151569141760,"internally_reserved":0,"allocated":13733888,"data_stored":8823317,"data_compressed":494712,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":46,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003398"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.190101+0000","last_change":"2024-06-10T00:23:57.564073+0000","last_active":"2024-06-10T00:25:44.190101+0000","last_peered":"2024-06-10T00:25:44.190101+0000","last_clean":"2024-06-10T00:25:44.190101+0000","last_became_active":"2024-06-10T00:23:57.563618+0000","last_became_peered":"2024-06-10T00:23:57.563618+0000","last_unstale":"2024-06-10T00:25:44.190101+0000","last_undegraded":"2024-06-10T00:25:44.190101+0000","last_fullsized":"2024-06-10T00:25:44.190101+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:26:25.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081600+0000","last_change":"2024-06-10T00:23:57.560148+0000","last_active":"2024-06-10T00:25:44.081600+0000","last_peered":"2024-06-10T00:25:44.081600+0000","last_clean":"2024-06-10T00:25:44.081600+0000","last_became_active":"2024-06-10T00:23:57.559794+0000","last_became_peered":"2024-06-10T00:23:57.559794+0000","last_unstale":"2024-06-10T00:25:44.081600+0000","last_undegraded":"2024-06-10T00:25:44.081600+0000","last_fullsized":"2024-06-10T00:25:44.081600+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:59:50.624542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082320+0000","last_change":"2024-06-10T00:23:57.563292+0000","last_active":"2024-06-10T00:25:44.082320+0000","last_peered":"2024-06-10T00:25:44.082320+0000","last_clean":"2024-06-10T00:25:44.082320+0000","last_became_active":"2024-06-10T00:23:57.563007+0000","last_became_peered":"2024-06-10T00:23:57.563007+0000","last_unstale":"2024-06-10T00:25:44.082320+0000","last_undegraded":"2024-06-10T00:25:44.082320+0000","last_fullsized":"2024-06-10T00:25:44.082320+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:27:44.897770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345090+0000","last_change":"2024-06-10T00:23:57.558789+0000","last_active":"2024-06-10T00:25:44.345090+0000","last_peered":"2024-06-10T00:25:44.345090+0000","last_clean":"2024-06-10T00:25:44.345090+0000","last_became_active":"2024-06-10T00:23:57.557943+0000","last_became_peered":"2024-06-10T00:23:57.557943+0000","last_unstale":"2024-06-10T00:25:44.345090+0000","last_undegraded":"2024-06-10T00:25:44.345090+0000","last_fullsized":"2024-06-10T00:25:44.345090+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:10:40.081996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"100'4","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.120346+0000","last_change":"2024-06-10T00:23:57.550806+0000","last_active":"2024-06-10T00:26:42.120346+0000","last_peered":"2024-06-10T00:26:42.120346+0000","last_clean":"2024-06-10T00:26:42.120346+0000","last_became_active":"2024-06-10T00:23:57.550453+0000","last_became_peered":"2024-06-10T00:23:57.550453+0000","last_unstale":"2024-06-10T00:26:42.120346+0000","last_undegraded":"2024-06-10T00:26:42.120346+0000","last_fullsized":"2024-06-10T00:26:42.120346+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:11:58.765099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.122945+0000","last_change":"2024-06-10T00:23:57.567307+0000","last_active":"2024-06-10T00:26:42.122945+0000","last_peered":"2024-06-10T00:26:42.122945+0000","last_clean":"2024-06-10T00:26:42.122945+0000","last_became_active":"2024-06-10T00:23:57.566973+0000","last_became_peered":"2024-06-10T00:23:57.566973+0000","last_unstale":"2024-06-10T00:26:42.122945+0000","last_undegraded":"2024-06-10T00:26:42.122945+0000","last_fullsized":"2024-06-10T00:26:42.122945+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:27:25.135046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"100'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.130655+0000","last_change":"2024-06-10T00:23:57.547998+0000","last_active":"2024-06-10T00:26:42.130655+0000","last_peered":"2024-06-10T00:26:42.130655+0000","last_clean":"2024-06-10T00:26:42.130655+0000","last_became_active":"2024-06-10T00:23:57.547593+0000","last_became_peered":"2024-06-10T00:23:57.547593+0000","last_unstale":"2024-06-10T00:26:42.130655+0000","last_undegraded":"2024-06-10T00:26:42.130655+0000","last_fullsized":"2024-06-10T00:26:42.130655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:34:04.219399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"100'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.129642+0000","last_change":"2024-06-10T00:23:57.560252+0000","last_active":"2024-06-10T00:26:42.129642+0000","last_peered":"2024-06-10T00:26:42.129642+0000","last_clean":"2024-06-10T00:26:42.129642+0000","last_became_active":"2024-06-10T00:23:57.559924+0000","last_became_peered":"2024-06-10T00:23:57.559924+0000","last_unstale":"2024-06-10T00:26:42.129642+0000","last_undegraded":"2024-06-10T00:26:42.129642+0000","last_fullsized":"2024-06-10T00:26:42.129642+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:01:43.684262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.124734+0000","last_change":"2024-06-10T00:23:57.564031+0000","last_active":"2024-06-10T00:25:44.124734+0000","last_peered":"2024-06-10T00:25:44.124734+0000","last_clean":"2024-06-10T00:25:44.124734+0000","last_became_active":"2024-06-10T00:23:57.563642+0000","last_became_peered":"2024-06-10T00:23:57.563642+0000","last_unstale":"2024-06-10T00:25:44.124734+0000","last_undegraded":"2024-06-10T00:25:44.124734+0000","last_fullsized":"2024-06-10T00:25:44.124734+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:16:23.003152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"100'16","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.122373+0000","last_change":"2024-06-10T00:23:57.566877+0000","last_active":"2024-06-10T00:26:42.122373+0000","last_peered":"2024-06-10T00:26:42.122373+0000","last_clean":"2024-06-10T00:26:42.122373+0000","last_became_active":"2024-06-10T00:23:57.566529+0000","last_became_peered":"2024-06-10T00:23:57.566529+0000","last_unstale":"2024-06-10T00:26:42.122373+0000","last_undegraded":"2024-06-10T00:26:42.122373+0000","last_fullsized":"2024-06-10T00:26:42.122373+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:30:43.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.140791+0000","last_change":"2024-06-10T00:23:57.564979+0000","last_active":"2024-06-10T00:26:42.140791+0000","last_peered":"2024-06-10T00:26:42.140791+0000","last_clean":"2024-06-10T00:26:42.140791+0000","last_became_active":"2024-06-10T00:23:57.564578+0000","last_became_peered":"2024-06-10T00:23:57.564578+0000","last_unstale":"2024-06-10T00:26:42.140791+0000","last_undegraded":"2024-06-10T00:26:42.140791+0000","last_fullsized":"2024-06-10T00:26:42.140791+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:40:53.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345085+0000","last_change":"2024-06-10T00:23:57.559126+0000","last_active":"2024-06-10T00:25:44.345085+0000","last_peered":"2024-06-10T00:25:44.345085+0000","last_clean":"2024-06-10T00:25:44.345085+0000","last_became_active":"2024-06-10T00:23:57.558401+0000","last_became_peered":"2024-06-10T00:23:57.558401+0000","last_unstale":"2024-06-10T00:25:44.345085+0000","last_undegraded":"2024-06-10T00:25:44.345085+0000","last_fullsized":"2024-06-10T00:25:44.345085+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:57.894137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"100'10","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.126736+0000","last_change":"2024-06-10T00:23:57.559272+0000","last_active":"2024-06-10T00:26:42.126736+0000","last_peered":"2024-06-10T00:26:42.126736+0000","last_clean":"2024-06-10T00:26:42.126736+0000","last_became_active":"2024-06-10T00:23:57.558941+0000","last_became_peered":"2024-06-10T00:23:57.558941+0000","last_unstale":"2024-06-10T00:26:42.126736+0000","last_undegraded":"2024-06-10T00:26:42.126736+0000","last_fullsized":"2024-06-10T00:26:42.126736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:36:48.088347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081547+0000","last_change":"2024-06-10T00:23:57.555875+0000","last_active":"2024-06-10T00:25:44.081547+0000","last_peered":"2024-06-10T00:25:44.081547+0000","last_clean":"2024-06-10T00:25:44.081547+0000","last_became_active":"2024-06-10T00:23:57.555492+0000","last_became_peered":"2024-06-10T00:23:57.555492+0000","last_unstale":"2024-06-10T00:25:44.081547+0000","last_undegraded":"2024-06-10T00:25:44.081547+0000","last_fullsized":"2024-06-10T00:25:44.081547+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:49:10.827458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081427+0000","last_change":"2024-06-10T00:23:57.555316+0000","last_active":"2024-06-10T00:25:44.081427+0000","last_peered":"2024-06-10T00:25:44.081427+0000","last_clean":"2024-06-10T00:25:44.081427+0000","last_became_active":"2024-06-10T00:23:57.554952+0000","last_became_peered":"2024-06-10T00:23:57.554952+0000","last_unstale":"2024-06-10T00:25:44.081427+0000","last_undegraded":"2024-06-10T00:25:44.081427+0000","last_fullsized":"2024-06-10T00:25:44.081427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:03:57.557330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.124138+0000","last_change":"2024-06-10T00:23:57.562442+0000","last_active":"2024-06-10T00:25:44.124138+0000","last_peered":"2024-06-10T00:25:44.124138+0000","last_clean":"2024-06-10T00:25:44.124138+0000","last_became_active":"2024-06-10T00:23:57.561965+0000","last_became_peered":"2024-06-10T00:23:57.561965+0000","last_unstale":"2024-06-10T00:25:44.124138+0000","last_undegraded":"2024-06-10T00:25:44.124138+0000","last_fullsized":"2024-06-10T00:25:44.124138+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:40:16.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189984+0000","last_change":"2024-06-10T00:23:57.564422+0000","last_active":"2024-06-10T00:25:44.189984+0000","last_peered":"2024-06-10T00:25:44.189984+0000","last_clean":"2024-06-10T00:25:44.189984+0000","last_became_active":"2024-06-10T00:23:57.564069+0000","last_became_peered":"2024-06-10T00:23:57.564069+0000","last_unstale":"2024-06-10T00:25:44.189984+0000","last_undegraded":"2024-06-10T00:25:44.189984+0000","last_fullsized":"2024-06-10T00:25:44.189984+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:07.662465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082916+0000","last_change":"2024-06-10T00:23:57.561284+0000","last_active":"2024-06-10T00:25:44.082916+0000","last_peered":"2024-06-10T00:25:44.082916+0000","last_clean":"2024-06-10T00:25:44.082916+0000","last_became_active":"2024-06-10T00:23:57.560933+0000","last_became_peered":"2024-06-10T00:23:57.560933+0000","last_unstale":"2024-06-10T00:25:44.082916+0000","last_undegraded":"2024-06-10T00:25:44.082916+0000","last_fullsized":"2024-06-10T00:25:44.082916+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:58:48.511774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.128126+0000","last_change":"2024-06-10T00:23:57.557752+0000","last_active":"2024-06-10T00:26:42.128126+0000","last_peered":"2024-06-10T00:26:42.128126+0000","last_clean":"2024-06-10T00:26:42.128126+0000","last_became_active":"2024-06-10T00:23:57.557419+0000","last_became_peered":"2024-06-10T00:23:57.557419+0000","last_unstale":"2024-06-10T00:26:42.128126+0000","last_undegraded":"2024-06-10T00:26:42.128126+0000","last_fullsized":"2024-06-10T00:26:42.128126+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:48.234181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081440+0000","last_change":"2024-06-10T00:24:09.177183+0000","last_active":"2024-06-10T00:25:44.081440+0000","last_peered":"2024-06-10T00:25:44.081440+0000","last_clean":"2024-06-10T00:25:44.081440+0000","last_became_active":"2024-06-10T00:24:09.176680+0000","last_became_peered":"2024-06-10T00:24:09.176680+0000","last_unstale":"2024-06-10T00:25:44.081440+0000","last_undegraded":"2024-06-10T00:25:44.081440+0000","last_fullsized":"2024-06-10T00:25:44.081440+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:17:06.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"98'9","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082765+0000","last_change":"2024-06-10T00:23:54.426364+0000","last_active":"2024-06-10T00:25:44.082765+0000","last_peered":"2024-06-10T00:25:44.082765+0000","last_clean":"2024-06-10T00:25:44.082765+0000","last_became_active":"2024-06-10T00:23:54.426054+0000","last_became_peered":"2024-06-10T00:23:54.426054+0000","last_unstale":"2024-06-10T00:25:44.082765+0000","last_undegraded":"2024-06-10T00:25:44.082765+0000","last_fullsized":"2024-06-10T00:25:44.082765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:47.613402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"100'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.127015+0000","last_change":"2024-06-10T00:23:57.557206+0000","last_active":"2024-06-10T00:26:42.127015+0000","last_peered":"2024-06-10T00:26:42.127015+0000","last_clean":"2024-06-10T00:26:42.127015+0000","last_became_active":"2024-06-10T00:23:57.556853+0000","last_became_peered":"2024-06-10T00:23:57.556853+0000","last_unstale":"2024-06-10T00:26:42.127015+0000","last_undegraded":"2024-06-10T00:26:42.127015+0000","last_fullsized":"2024-06-10T00:26:42.127015+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:09.686334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345231+0000","last_change":"2024-06-10T00:24:09.171047+0000","last_active":"2024-06-10T00:25:44.345231+0000","last_peered":"2024-06-10T00:25:44.345231+0000","last_clean":"2024-06-10T00:25:44.345231+0000","last_became_active":"2024-06-10T00:24:09.170501+0000","last_became_peered":"2024-06-10T00:24:09.170501+0000","last_unstale":"2024-06-10T00:25:44.345231+0000","last_undegraded":"2024-06-10T00:25:44.345231+0000","last_fullsized":"2024-06-10T00:25:44.345231+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:47:27.690439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"98'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733915+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:25:44.733915+0000","last_peered":"2024-06-10T00:25:44.733915+0000","last_clean":"2024-06-10T00:25:44.733915+0000","last_became_active":"2024-06-10T00:23:54.822447+0000","last_became_peered":"2024-06-10T00:23:54.822447+0000","last_unstale":"2024-06-10T00:25:44.733915+0000","last_undegraded":"2024-06-10T00:25:44.733915+0000","last_fullsized":"2024-06-10T00:25:44.733915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:45:14.956540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082428+0000","last_change":"2024-06-10T00:23:57.554958+0000","last_active":"2024-06-10T00:25:44.082428+0000","last_peered":"2024-06-10T00:25:44.082428+0000","last_clean":"2024-06-10T00:25:44.082428+0000","last_became_active":"2024-06-10T00:23:57.554477+0000","last_became_peered":"2024-06-10T00:23:57.554477+0000","last_unstale":"2024-06-10T00:25:44.082428+0000","last_undegraded":"2024-06-10T00:25:44.082428+0000","last_fullsized":"2024-06-10T00:25:44.082428+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:40:06.066967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344744+0000","last_change":"2024-06-10T00:24:09.169642+0000","last_active":"2024-06-10T00:25:44.344744+0000","last_peered":"2024-06-10T00:25:44.344744+0000","last_clean":"2024-06-10T00:25:44.344744+0000","last_became_active":"2024-06-10T00:24:09.168071+0000","last_became_peered":"2024-06-10T00:24:09.168071+0000","last_unstale":"2024-06-10T00:25:44.344744+0000","last_undegraded":"2024-06-10T00:25:44.344744+0000","last_fullsized":"2024-06-10T00:25:44.344744+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:44.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"98'4","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733966+0000","last_change":"2024-06-10T00:23:54.430543+0000","last_active":"2024-06-10T00:25:44.733966+0000","last_peered":"2024-06-10T00:25:44.733966+0000","last_clean":"2024-06-10T00:25:44.733966+0000","last_became_active":"2024-06-10T00:23:54.430138+0000","last_became_peered":"2024-06-10T00:23:54.430138+0000","last_unstale":"2024-06-10T00:25:44.733966+0000","last_undegraded":"2024-06-10T00:25:44.733966+0000","last_fullsized":"2024-06-10T00:25:44.733966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:39:53.212607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.124322+0000","last_change":"2024-06-10T00:23:57.562251+0000","last_active":"2024-06-10T00:25:44.124322+0000","last_peered":"2024-06-10T00:25:44.124322+0000","last_clean":"2024-06-10T00:25:44.124322+0000","last_became_active":"2024-06-10T00:23:57.561903+0000","last_became_peered":"2024-06-10T00:23:57.561903+0000","last_unstale":"2024-06-10T00:25:44.124322+0000","last_undegraded":"2024-06-10T00:25:44.124322+0000","last_fullsized":"2024-06-10T00:25:44.124322+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:59.489475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344593+0000","last_change":"2024-06-10T00:24:09.165894+0000","last_active":"2024-06-10T00:25:44.344593+0000","last_peered":"2024-06-10T00:25:44.344593+0000","last_clean":"2024-06-10T00:25:44.344593+0000","last_became_active":"2024-06-10T00:24:09.165435+0000","last_became_peered":"2024-06-10T00:24:09.165435+0000","last_unstale":"2024-06-10T00:25:44.344593+0000","last_undegraded":"2024-06-10T00:25:44.344593+0000","last_fullsized":"2024-06-10T00:25:44.344593+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:13:36.058375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.734024+0000","last_change":"2024-06-10T00:23:54.427868+0000","last_active":"2024-06-10T00:25:44.734024+0000","last_peered":"2024-06-10T00:25:44.734024+0000","last_clean":"2024-06-10T00:25:44.734024+0000","last_became_active":"2024-06-10T00:23:54.427366+0000","last_became_peered":"2024-06-10T00:23:54.427366+0000","last_unstale":"2024-06-10T00:25:44.734024+0000","last_undegraded":"2024-06-10T00:25:44.734024+0000","last_fullsized":"2024-06-10T00:25:44.734024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:15.499292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345665+0000","last_change":"2024-06-10T00:24:43.768556+0000","last_active":"2024-06-10T00:25:44.345665+0000","last_peered":"2024-06-10T00:25:44.345665+0000","last_clean":"2024-06-10T00:25:44.345665+0000","last_became_active":"2024-06-10T00:24:43.768090+0000","last_became_peered":"2024-06-10T00:24:43.768090+0000","last_unstale":"2024-06-10T00:25:44.345665+0000","last_undegraded":"2024-06-10T00:25:44.345665+0000","last_fullsized":"2024-06-10T00:25:44.345665+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:05:41.604184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081837+0000","last_change":"2024-06-10T00:24:09.170514+0000","last_active":"2024-06-10T00:25:44.081837+0000","last_peered":"2024-06-10T00:25:44.081837+0000","last_clean":"2024-06-10T00:25:44.081837+0000","last_became_active":"2024-06-10T00:24:09.168902+0000","last_became_peered":"2024-06-10T00:24:09.168902+0000","last_unstale":"2024-06-10T00:25:44.081837+0000","last_undegraded":"2024-06-10T00:25:44.081837+0000","last_fullsized":"2024-06-10T00:25:44.081837+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:33:10.534657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"98'6","reported_seq":150,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:39.428700+0000","last_change":"2024-06-10T00:23:54.426581+0000","last_active":"2024-06-10T00:26:39.428700+0000","last_peered":"2024-06-10T00:26:39.428700+0000","last_clean":"2024-06-10T00:26:39.428700+0000","last_became_active":"2024-06-10T00:23:54.426211+0000","last_became_peered":"2024-06-10T00:23:54.426211+0000","last_unstale":"2024-06-10T00:26:39.428700+0000","last_undegraded":"2024-06-10T00:26:39.428700+0000","last_fullsized":"2024-06-10T00:26:39.428700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:49:05.360742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.120352+0000","last_change":"2024-06-10T00:23:57.552308+0000","last_active":"2024-06-10T00:26:42.120352+0000","last_peered":"2024-06-10T00:26:42.120352+0000","last_clean":"2024-06-10T00:26:42.120352+0000","last_became_active":"2024-06-10T00:23:57.551762+0000","last_became_peered":"2024-06-10T00:23:57.551762+0000","last_unstale":"2024-06-10T00:26:42.120352+0000","last_undegraded":"2024-06-10T00:26:42.120352+0000","last_fullsized":"2024-06-10T00:26:42.120352+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:22:28.476019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081833+0000","last_change":"2024-06-10T00:24:09.171456+0000","last_active":"2024-06-10T00:25:44.081833+0000","last_peered":"2024-06-10T00:25:44.081833+0000","last_clean":"2024-06-10T00:25:44.081833+0000","last_became_active":"2024-06-10T00:24:09.170969+0000","last_became_peered":"2024-06-10T00:24:09.170969+0000","last_unstale":"2024-06-10T00:25:44.081833+0000","last_undegraded":"2024-06-10T00:25:44.081833+0000","last_fullsized":"2024-06-10T00:25:44.081833+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:22:46.023897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"98'3","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078975+0000","last_change":"2024-06-10T00:23:54.428384+0000","last_active":"2024-06-10T00:25:44.078975+0000","last_peered":"2024-06-10T00:25:44.078975+0000","last_clean":"2024-06-10T00:25:44.078975+0000","last_became_active":"2024-06-10T00:23:54.427967+0000","last_became_peered":"2024-06-10T00:23:54.427967+0000","last_unstale":"2024-06-10T00:25:44.078975+0000","last_undegraded":"2024-06-10T00:25:44.078975+0000","last_fullsized":"2024-06-10T00:25:44.078975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:54:30.408684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345595+0000","last_change":"2024-06-10T00:24:43.262593+0000","last_active":"2024-06-10T00:25:44.345595+0000","last_peered":"2024-06-10T00:25:44.345595+0000","last_clean":"2024-06-10T00:25:44.345595+0000","last_became_active":"2024-06-10T00:24:43.262212+0000","last_became_peered":"2024-06-10T00:24:43.262212+0000","last_unstale":"2024-06-10T00:25:44.345595+0000","last_undegraded":"2024-06-10T00:25:44.345595+0000","last_fullsized":"2024-06-10T00:25:44.345595+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:40:24.442712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":34,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189955+0000","last_change":"2024-06-10T00:24:43.752805+0000","last_active":"2024-06-10T00:25:44.189955+0000","last_peered":"2024-06-10T00:25:44.189955+0000","last_clean":"2024-06-10T00:25:44.189955+0000","last_became_active":"2024-06-10T00:24:43.752251+0000","last_became_peered":"2024-06-10T00:24:43.752251+0000","last_unstale":"2024-06-10T00:25:44.189955+0000","last_undegraded":"2024-06-10T00:25:44.189955+0000","last_fullsized":"2024-06-10T00:25:44.189955+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:27:57.617443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"98'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078007+0000","last_change":"2024-06-10T00:23:54.421493+0000","last_active":"2024-06-10T00:25:44.078007+0000","last_peered":"2024-06-10T00:25:44.078007+0000","last_clean":"2024-06-10T00:25:44.078007+0000","last_became_active":"2024-06-10T00:23:54.421151+0000","last_became_peered":"2024-06-10T00:23:54.421151+0000","last_unstale":"2024-06-10T00:25:44.078007+0000","last_undegraded":"2024-06-10T00:25:44.078007+0000","last_fullsized":"2024-06-10T00:25:44.078007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:03.995244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"100'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.130337+0000","last_change":"2024-06-10T00:23:57.554020+0000","last_active":"2024-06-10T00:26:42.130337+0000","last_peered":"2024-06-10T00:26:42.130337+0000","last_clean":"2024-06-10T00:26:42.130337+0000","last_became_active":"2024-06-10T00:23:57.553690+0000","last_became_peered":"2024-06-10T00:23:57.553690+0000","last_unstale":"2024-06-10T00:26:42.130337+0000","last_undegraded":"2024-06-10T00:26:42.130337+0000","last_fullsized":"2024-06-10T00:26:42.130337+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:48.097074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081507+0000","last_change":"2024-06-10T00:24:09.166426+0000","last_active":"2024-06-10T00:25:44.081507+0000","last_peered":"2024-06-10T00:25:44.081507+0000","last_clean":"2024-06-10T00:25:44.081507+0000","last_became_active":"2024-06-10T00:24:09.166047+0000","last_became_peered":"2024-06-10T00:24:09.166047+0000","last_unstale":"2024-06-10T00:25:44.081507+0000","last_undegraded":"2024-06-10T00:25:44.081507+0000","last_fullsized":"2024-06-10T00:25:44.081507+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:56:11.608061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"98'1","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345414+0000","last_change":"2024-06-10T00:23:54.426409+0000","last_active":"2024-06-10T00:25:44.345414+0000","last_peered":"2024-06-10T00:25:44.345414+0000","last_clean":"2024-06-10T00:25:44.345414+0000","last_became_active":"2024-06-10T00:23:54.425385+0000","last_became_peered":"2024-06-10T00:23:54.425385+0000","last_unstale":"2024-06-10T00:25:44.345414+0000","last_undegraded":"2024-06-10T00:25:44.345414+0000","last_fullsized":"2024-06-10T00:25:44.345414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:27:00.379189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081986+0000","last_change":"2024-06-10T00:23:57.560132+0000","last_active":"2024-06-10T00:25:44.081986+0000","last_peered":"2024-06-10T00:25:44.081986+0000","last_clean":"2024-06-10T00:25:44.081986+0000","last_became_active":"2024-06-10T00:23:57.559803+0000","last_became_peered":"2024-06-10T00:23:57.559803+0000","last_unstale":"2024-06-10T00:25:44.081986+0000","last_undegraded":"2024-06-10T00:25:44.081986+0000","last_fullsized":"2024-06-10T00:25:44.081986+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:13:16.004233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733402+0000","last_change":"2024-06-10T00:24:09.170962+0000","last_active":"2024-06-10T00:25:44.733402+0000","last_peered":"2024-06-10T00:25:44.733402+0000","last_clean":"2024-06-10T00:25:44.733402+0000","last_became_active":"2024-06-10T00:24:09.170090+0000","last_became_peered":"2024-06-10T00:24:09.170090+0000","last_unstale":"2024-06-10T00:25:44.733402+0000","last_undegraded":"2024-06-10T00:25:44.733402+0000","last_fullsized":"2024-06-10T00:25:44.733402+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:07:33.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'4","reported_seq":148,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.432251+0000","last_change":"2024-06-10T00:23:54.421472+0000","last_active":"2024-06-10T00:26:42.432251+0000","last_peered":"2024-06-10T00:26:42.432251+0000","last_clean":"2024-06-10T00:26:42.432251+0000","last_became_active":"2024-06-10T00:23:54.421019+0000","last_became_peered":"2024-06-10T00:23:54.421019+0000","last_unstale":"2024-06-10T00:26:42.432251+0000","last_undegraded":"2024-06-10T00:26:42.432251+0000","last_fullsized":"2024-06-10T00:26:42.432251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:28:18.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.124578+0000","last_change":"2024-06-10T00:23:57.563010+0000","last_active":"2024-06-10T00:25:44.124578+0000","last_peered":"2024-06-10T00:25:44.124578+0000","last_clean":"2024-06-10T00:25:44.124578+0000","last_became_active":"2024-06-10T00:23:57.561986+0000","last_became_peered":"2024-06-10T00:23:57.561986+0000","last_unstale":"2024-06-10T00:25:44.124578+0000","last_undegraded":"2024-06-10T00:25:44.124578+0000","last_fullsized":"2024-06-10T00:25:44.124578+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:31:17.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082277+0000","last_change":"2024-06-10T00:24:09.171746+0000","last_active":"2024-06-10T00:25:44.082277+0000","last_peered":"2024-06-10T00:25:44.082277+0000","last_clean":"2024-06-10T00:25:44.082277+0000","last_became_active":"2024-06-10T00:24:09.171030+0000","last_became_peered":"2024-06-10T00:24:09.171030+0000","last_unstale":"2024-06-10T00:25:44.082277+0000","last_undegraded":"2024-06-10T00:25:44.082277+0000","last_fullsized":"2024-06-10T00:25:44.082277+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:41.669471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"98'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.734164+0000","last_change":"2024-06-10T00:23:54.428917+0000","last_active":"2024-06-10T00:25:44.734164+0000","last_peered":"2024-06-10T00:25:44.734164+0000","last_clean":"2024-06-10T00:25:44.734164+0000","last_became_active":"2024-06-10T00:23:54.428585+0000","last_became_peered":"2024-06-10T00:23:54.428585+0000","last_unstale":"2024-06-10T00:25:44.734164+0000","last_undegraded":"2024-06-10T00:25:44.734164+0000","last_fullsized":"2024-06-10T00:25:44.734164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:21:58.235848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082601+0000","last_change":"2024-06-10T00:23:57.551778+0000","last_active":"2024-06-10T00:25:44.082601+0000","last_peered":"2024-06-10T00:25:44.082601+0000","last_clean":"2024-06-10T00:25:44.082601+0000","last_became_active":"2024-06-10T00:23:57.551436+0000","last_became_peered":"2024-06-10T00:23:57.551436+0000","last_unstale":"2024-06-10T00:25:44.082601+0000","last_undegraded":"2024-06-10T00:25:44.082601+0000","last_fullsized":"2024-06-10T00:25:44.082601+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:30.059783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344998+0000","last_change":"2024-06-10T00:24:09.167547+0000","last_active":"2024-06-10T00:25:44.344998+0000","last_peered":"2024-06-10T00:25:44.344998+0000","last_clean":"2024-06-10T00:25:44.344998+0000","last_became_active":"2024-06-10T00:24:09.167132+0000","last_became_peered":"2024-06-10T00:24:09.167132+0000","last_unstale":"2024-06-10T00:25:44.344998+0000","last_undegraded":"2024-06-10T00:25:44.344998+0000","last_fullsized":"2024-06-10T00:25:44.344998+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:11.520330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"98'4","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345502+0000","last_change":"2024-06-10T00:23:54.823754+0000","last_active":"2024-06-10T00:25:44.345502+0000","last_peered":"2024-06-10T00:25:44.345502+0000","last_clean":"2024-06-10T00:25:44.345502+0000","last_became_active":"2024-06-10T00:23:54.823362+0000","last_became_peered":"2024-06-10T00:23:54.823362+0000","last_unstale":"2024-06-10T00:25:44.345502+0000","last_undegraded":"2024-06-10T00:25:44.345502+0000","last_fullsized":"2024-06-10T00:25:44.345502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:48:50.915439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.124604+0000","last_change":"2024-06-10T00:23:57.563313+0000","last_active":"2024-06-10T00:25:44.124604+0000","last_peered":"2024-06-10T00:25:44.124604+0000","last_clean":"2024-06-10T00:25:44.124604+0000","last_became_active":"2024-06-10T00:23:57.562307+0000","last_became_peered":"2024-06-10T00:23:57.562307+0000","last_unstale":"2024-06-10T00:25:44.124604+0000","last_undegraded":"2024-06-10T00:25:44.124604+0000","last_fullsized":"2024-06-10T00:25:44.124604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:23.815267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081609+0000","last_change":"2024-06-10T00:24:09.174660+0000","last_active":"2024-06-10T00:25:44.081609+0000","last_peered":"2024-06-10T00:25:44.081609+0000","last_clean":"2024-06-10T00:25:44.081609+0000","last_became_active":"2024-06-10T00:24:09.174283+0000","last_became_peered":"2024-06-10T00:24:09.174283+0000","last_unstale":"2024-06-10T00:25:44.081609+0000","last_undegraded":"2024-06-10T00:25:44.081609+0000","last_fullsized":"2024-06-10T00:25:44.081609+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:15.236146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"100'36","reported_seq":289,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:43.434262+0000","last_change":"2024-06-10T00:23:54.426621+0000","last_active":"2024-06-10T00:26:43.434262+0000","last_peered":"2024-06-10T00:26:43.434262+0000","last_clean":"2024-06-10T00:26:43.434262+0000","last_became_active":"2024-06-10T00:23:54.426131+0000","last_became_peered":"2024-06-10T00:23:54.426131+0000","last_unstale":"2024-06-10T00:26:43.434262+0000","last_undegraded":"2024-06-10T00:26:43.434262+0000","last_fullsized":"2024-06-10T00:26:43.434262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:56:14.819454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62904,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":208,"num_read_kb":166,"num_write":37,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.132208+0000","last_change":"2024-06-10T00:23:57.552606+0000","last_active":"2024-06-10T00:26:42.132208+0000","last_peered":"2024-06-10T00:26:42.132208+0000","last_clean":"2024-06-10T00:26:42.132208+0000","last_became_active":"2024-06-10T00:23:57.552246+0000","last_became_peered":"2024-06-10T00:23:57.552246+0000","last_unstale":"2024-06-10T00:26:42.132208+0000","last_undegraded":"2024-06-10T00:26:42.132208+0000","last_fullsized":"2024-06-10T00:26:42.132208+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:19:21.508721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.346039+0000","last_change":"2024-06-10T00:24:09.174862+0000","last_active":"2024-06-10T00:25:44.346039+0000","last_peered":"2024-06-10T00:25:44.346039+0000","last_clean":"2024-06-10T00:25:44.346039+0000","last_became_active":"2024-06-10T00:24:09.173550+0000","last_became_peered":"2024-06-10T00:24:09.173550+0000","last_unstale":"2024-06-10T00:25:44.346039+0000","last_undegraded":"2024-06-10T00:25:44.346039+0000","last_fullsized":"2024-06-10T00:25:44.346039+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:49:25.549084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"98'6","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081771+0000","last_change":"2024-06-10T00:23:54.426074+0000","last_active":"2024-06-10T00:25:44.081771+0000","last_peered":"2024-06-10T00:25:44.081771+0000","last_clean":"2024-06-10T00:25:44.081771+0000","last_became_active":"2024-06-10T00:23:54.425707+0000","last_became_peered":"2024-06-10T00:23:54.425707+0000","last_unstale":"2024-06-10T00:25:44.081771+0000","last_undegraded":"2024-06-10T00:25:44.081771+0000","last_fullsized":"2024-06-10T00:25:44.081771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:11:10.563595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"100'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.128427+0000","last_change":"2024-06-10T00:23:57.549784+0000","last_active":"2024-06-10T00:26:42.128427+0000","last_peered":"2024-06-10T00:26:42.128427+0000","last_clean":"2024-06-10T00:26:42.128427+0000","last_became_active":"2024-06-10T00:23:57.549457+0000","last_became_peered":"2024-06-10T00:23:57.549457+0000","last_unstale":"2024-06-10T00:26:42.128427+0000","last_undegraded":"2024-06-10T00:26:42.128427+0000","last_fullsized":"2024-06-10T00:26:42.128427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:59:27.352927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189697+0000","last_change":"2024-06-10T00:24:09.171028+0000","last_active":"2024-06-10T00:25:44.189697+0000","last_peered":"2024-06-10T00:25:44.189697+0000","last_clean":"2024-06-10T00:25:44.189697+0000","last_became_active":"2024-06-10T00:24:09.170543+0000","last_became_peered":"2024-06-10T00:24:09.170543+0000","last_unstale":"2024-06-10T00:25:44.189697+0000","last_undegraded":"2024-06-10T00:25:44.189697+0000","last_fullsized":"2024-06-10T00:25:44.189697+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:57:40.816585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"98'3","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345649+0000","last_change":"2024-06-10T00:23:54.423332+0000","last_active":"2024-06-10T00:25:44.345649+0000","last_peered":"2024-06-10T00:25:44.345649+0000","last_clean":"2024-06-10T00:25:44.345649+0000","last_became_active":"2024-06-10T00:23:54.422792+0000","last_became_peered":"2024-06-10T00:23:54.422792+0000","last_unstale":"2024-06-10T00:25:44.345649+0000","last_undegraded":"2024-06-10T00:25:44.345649+0000","last_fullsized":"2024-06-10T00:25:44.345649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:50:52.655641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"100'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.139586+0000","last_change":"2024-06-10T00:23:57.551411+0000","last_active":"2024-06-10T00:26:42.139586+0000","last_peered":"2024-06-10T00:26:42.139586+0000","last_clean":"2024-06-10T00:26:42.139586+0000","last_became_active":"2024-06-10T00:23:57.551053+0000","last_became_peered":"2024-06-10T00:23:57.551053+0000","last_unstale":"2024-06-10T00:26:42.139586+0000","last_undegraded":"2024-06-10T00:26:42.139586+0000","last_fullsized":"2024-06-10T00:26:42.139586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:16:18.082777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.079218+0000","last_change":"2024-06-10T00:24:43.648767+0000","last_active":"2024-06-10T00:25:44.079218+0000","last_peered":"2024-06-10T00:25:44.079218+0000","last_clean":"2024-06-10T00:25:44.079218+0000","last_became_active":"2024-06-10T00:24:43.648267+0000","last_became_peered":"2024-06-10T00:24:43.648267+0000","last_unstale":"2024-06-10T00:25:44.079218+0000","last_undegraded":"2024-06-10T00:25:44.079218+0000","last_fullsized":"2024-06-10T00:25:44.079218+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:41:09.005202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"98'3","reported_seq":146,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:40.429198+0000","last_change":"2024-06-10T00:23:54.425963+0000","last_active":"2024-06-10T00:26:40.429198+0000","last_peered":"2024-06-10T00:26:40.429198+0000","last_clean":"2024-06-10T00:26:40.429198+0000","last_became_active":"2024-06-10T00:23:54.425581+0000","last_became_peered":"2024-06-10T00:23:54.425581+0000","last_unstale":"2024-06-10T00:26:40.429198+0000","last_undegraded":"2024-06-10T00:26:40.429198+0000","last_fullsized":"2024-06-10T00:26:40.429198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:54.760610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.129666+0000","last_change":"2024-06-10T00:23:57.557464+0000","last_active":"2024-06-10T00:26:42.129666+0000","last_peered":"2024-06-10T00:26:42.129666+0000","last_clean":"2024-06-10T00:26:42.129666+0000","last_became_active":"2024-06-10T00:23:57.557130+0000","last_became_peered":"2024-06-10T00:23:57.557130+0000","last_unstale":"2024-06-10T00:26:42.129666+0000","last_undegraded":"2024-06-10T00:26:42.129666+0000","last_fullsized":"2024-06-10T00:26:42.129666+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:16:36.341210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":36,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078404+0000","last_change":"2024-06-10T00:24:43.753291+0000","last_active":"2024-06-10T00:25:44.078404+0000","last_peered":"2024-06-10T00:25:44.078404+0000","last_clean":"2024-06-10T00:25:44.078404+0000","last_became_active":"2024-06-10T00:24:43.752765+0000","last_became_peered":"2024-06-10T00:24:43.752765+0000","last_unstale":"2024-06-10T00:25:44.078404+0000","last_undegraded":"2024-06-10T00:25:44.078404+0000","last_fullsized":"2024-06-10T00:25:44.078404+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:55:06.349564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"98'15","reported_seq":61,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733740+0000","last_change":"2024-06-10T00:23:54.823489+0000","last_active":"2024-06-10T00:25:44.733740+0000","last_peered":"2024-06-10T00:25:44.733740+0000","last_clean":"2024-06-10T00:25:44.733740+0000","last_became_active":"2024-06-10T00:23:54.823073+0000","last_became_peered":"2024-06-10T00:23:54.823073+0000","last_unstale":"2024-06-10T00:25:44.733740+0000","last_undegraded":"2024-06-10T00:25:44.733740+0000","last_fullsized":"2024-06-10T00:25:44.733740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:07:52.563009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345321+0000","last_change":"2024-06-10T00:23:57.563388+0000","last_active":"2024-06-10T00:25:44.345321+0000","last_peered":"2024-06-10T00:25:44.345321+0000","last_clean":"2024-06-10T00:25:44.345321+0000","last_became_active":"2024-06-10T00:23:57.563004+0000","last_became_peered":"2024-06-10T00:23:57.563004+0000","last_unstale":"2024-06-10T00:25:44.345321+0000","last_undegraded":"2024-06-10T00:25:44.345321+0000","last_fullsized":"2024-06-10T00:25:44.345321+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:51:18.805662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345323+0000","last_change":"2024-06-10T00:24:09.168447+0000","last_active":"2024-06-10T00:25:44.345323+0000","last_peered":"2024-06-10T00:25:44.345323+0000","last_clean":"2024-06-10T00:25:44.345323+0000","last_became_active":"2024-06-10T00:24:09.167439+0000","last_became_peered":"2024-06-10T00:24:09.167439+0000","last_unstale":"2024-06-10T00:25:44.345323+0000","last_undegraded":"2024-06-10T00:25:44.345323+0000","last_fullsized":"2024-06-10T00:25:44.345323+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:52.319395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"98'2","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.734275+0000","last_change":"2024-06-10T00:23:54.427666+0000","last_active":"2024-06-10T00:25:44.734275+0000","last_peered":"2024-06-10T00:25:44.734275+0000","last_clean":"2024-06-10T00:25:44.734275+0000","last_became_active":"2024-06-10T00:23:54.427285+0000","last_became_peered":"2024-06-10T00:23:54.427285+0000","last_unstale":"2024-06-10T00:25:44.734275+0000","last_undegraded":"2024-06-10T00:25:44.734275+0000","last_fullsized":"2024-06-10T00:25:44.734275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:28:20.196168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189931+0000","last_change":"2024-06-10T00:23:57.558787+0000","last_active":"2024-06-10T00:25:44.189931+0000","last_peered":"2024-06-10T00:25:44.189931+0000","last_clean":"2024-06-10T00:25:44.189931+0000","last_became_active":"2024-06-10T00:23:57.558470+0000","last_became_peered":"2024-06-10T00:23:57.558470+0000","last_unstale":"2024-06-10T00:25:44.189931+0000","last_undegraded":"2024-06-10T00:25:44.189931+0000","last_fullsized":"2024-06-10T00:25:44.189931+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:02:14.758802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733802+0000","last_change":"2024-06-10T00:24:09.264952+0000","last_active":"2024-06-10T00:25:44.733802+0000","last_peered":"2024-06-10T00:25:44.733802+0000","last_clean":"2024-06-10T00:25:44.733802+0000","last_became_active":"2024-06-10T00:24:09.264379+0000","last_became_peered":"2024-06-10T00:24:09.264379+0000","last_unstale":"2024-06-10T00:25:44.733802+0000","last_undegraded":"2024-06-10T00:25:44.733802+0000","last_fullsized":"2024-06-10T00:25:44.733802+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:25:40.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082876+0000","last_change":"2024-06-10T00:23:54.823298+0000","last_active":"2024-06-10T00:25:44.082876+0000","last_peered":"2024-06-10T00:25:44.082876+0000","last_clean":"2024-06-10T00:25:44.082876+0000","last_became_active":"2024-06-10T00:23:54.822917+0000","last_became_peered":"2024-06-10T00:23:54.822917+0000","last_unstale":"2024-06-10T00:25:44.082876+0000","last_undegraded":"2024-06-10T00:25:44.082876+0000","last_fullsized":"2024-06-10T00:25:44.082876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:07:52.500641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"100'4","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.126638+0000","last_change":"2024-06-10T00:23:57.559202+0000","last_active":"2024-06-10T00:26:42.126638+0000","last_peered":"2024-06-10T00:26:42.126638+0000","last_clean":"2024-06-10T00:26:42.126638+0000","last_became_active":"2024-06-10T00:23:57.558800+0000","last_became_peered":"2024-06-10T00:23:57.558800+0000","last_unstale":"2024-06-10T00:26:42.126638+0000","last_undegraded":"2024-06-10T00:26:42.126638+0000","last_fullsized":"2024-06-10T00:26:42.126638+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:49:51.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078291+0000","last_change":"2024-06-10T00:24:09.168603+0000","last_active":"2024-06-10T00:25:44.078291+0000","last_peered":"2024-06-10T00:25:44.078291+0000","last_clean":"2024-06-10T00:25:44.078291+0000","last_became_active":"2024-06-10T00:24:09.168233+0000","last_became_peered":"2024-06-10T00:24:09.168233+0000","last_unstale":"2024-06-10T00:25:44.078291+0000","last_undegraded":"2024-06-10T00:25:44.078291+0000","last_fullsized":"2024-06-10T00:25:44.078291+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:59.585023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'1","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733804+0000","last_change":"2024-06-10T00:23:54.425505+0000","last_active":"2024-06-10T00:25:44.733804+0000","last_peered":"2024-06-10T00:25:44.733804+0000","last_clean":"2024-06-10T00:25:44.733804+0000","last_became_active":"2024-06-10T00:23:54.425058+0000","last_became_peered":"2024-06-10T00:23:54.425058+0000","last_unstale":"2024-06-10T00:25:44.733804+0000","last_undegraded":"2024-06-10T00:25:44.733804+0000","last_fullsized":"2024-06-10T00:25:44.733804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:48:05.605947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.125899+0000","last_change":"2024-06-10T00:23:57.566060+0000","last_active":"2024-06-10T00:26:42.125899+0000","last_peered":"2024-06-10T00:26:42.125899+0000","last_clean":"2024-06-10T00:26:42.125899+0000","last_became_active":"2024-06-10T00:23:57.565099+0000","last_became_peered":"2024-06-10T00:23:57.565099+0000","last_unstale":"2024-06-10T00:26:42.125899+0000","last_undegraded":"2024-06-10T00:26:42.125899+0000","last_fullsized":"2024-06-10T00:26:42.125899+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:24:54.204399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078462+0000","last_change":"2024-06-10T00:24:09.264741+0000","last_active":"2024-06-10T00:25:44.078462+0000","last_peered":"2024-06-10T00:25:44.078462+0000","last_clean":"2024-06-10T00:25:44.078462+0000","last_became_active":"2024-06-10T00:24:09.264267+0000","last_became_peered":"2024-06-10T00:24:09.264267+0000","last_unstale":"2024-06-10T00:25:44.078462+0000","last_undegraded":"2024-06-10T00:25:44.078462+0000","last_fullsized":"2024-06-10T00:25:44.078462+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:27.627391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.346070+0000","last_change":"2024-06-10T00:25:44.346070+0000","last_active":"2024-06-10T00:25:44.346070+0000","last_peered":"2024-06-10T00:25:44.346070+0000","last_clean":"2024-06-10T00:25:44.346070+0000","last_became_active":"2024-06-10T00:25:44.345798+0000","last_became_peered":"2024-06-10T00:25:44.345798+0000","last_unstale":"2024-06-10T00:25:44.346070+0000","last_undegraded":"2024-06-10T00:25:44.346070+0000","last_fullsized":"2024-06-10T00:25:44.346070+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:04.506535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"98'4","reported_seq":147,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:41.429991+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:26:41.429991+0000","last_peered":"2024-06-10T00:26:41.429991+0000","last_clean":"2024-06-10T00:26:41.429991+0000","last_became_active":"2024-06-10T00:23:54.822442+0000","last_became_peered":"2024-06-10T00:23:54.822442+0000","last_unstale":"2024-06-10T00:26:41.429991+0000","last_undegraded":"2024-06-10T00:26:41.429991+0000","last_fullsized":"2024-06-10T00:26:41.429991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:51:10.278257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"98'1","reported_seq":30,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.734162+0000","last_change":"2024-06-10T00:24:09.168073+0000","last_active":"2024-06-10T00:25:44.734162+0000","last_peered":"2024-06-10T00:25:44.734162+0000","last_clean":"2024-06-10T00:25:44.734162+0000","last_became_active":"2024-06-10T00:24:09.167657+0000","last_became_peered":"2024-06-10T00:24:09.167657+0000","last_unstale":"2024-06-10T00:25:44.734162+0000","last_undegraded":"2024-06-10T00:25:44.734162+0000","last_fullsized":"2024-06-10T00:25:44.734162+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:20:23.481685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.346629+0000","last_change":"2024-06-10T00:25:44.346629+0000","last_active":"2024-06-10T00:25:44.346629+0000","last_peered":"2024-06-10T00:25:44.346629+0000","last_clean":"2024-06-10T00:25:44.346629+0000","last_became_active":"2024-06-10T00:25:44.346388+0000","last_became_peered":"2024-06-10T00:25:44.346388+0000","last_unstale":"2024-06-10T00:25:44.346629+0000","last_undegraded":"2024-06-10T00:25:44.346629+0000","last_fullsized":"2024-06-10T00:25:44.346629+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:03:19.228632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"98'3","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344835+0000","last_change":"2024-06-10T00:23:54.423507+0000","last_active":"2024-06-10T00:25:44.344835+0000","last_peered":"2024-06-10T00:25:44.344835+0000","last_clean":"2024-06-10T00:25:44.344835+0000","last_became_active":"2024-06-10T00:23:54.423092+0000","last_became_peered":"2024-06-10T00:23:54.423092+0000","last_unstale":"2024-06-10T00:25:44.344835+0000","last_undegraded":"2024-06-10T00:25:44.344835+0000","last_fullsized":"2024-06-10T00:25:44.344835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:06:15.924827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":377,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733568+0000","last_change":"2024-06-10T00:22:28.025243+0000","last_active":"2024-06-10T00:25:44.733568+0000","last_peered":"2024-06-10T00:25:44.733568+0000","last_clean":"2024-06-10T00:25:44.733568+0000","last_became_active":"2024-06-10T00:22:28.024751+0000","last_became_peered":"2024-06-10T00:22:28.024751+0000","last_unstale":"2024-06-10T00:25:44.733568+0000","last_undegraded":"2024-06-10T00:25:44.733568+0000","last_fullsized":"2024-06-10T00:25:44.733568+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_clean_scrub_stamp":"2024-06-10T00:19:05.039631+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:02:36.557931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345832+0000","last_change":"2024-06-10T00:24:09.265084+0000","last_active":"2024-06-10T00:25:44.345832+0000","last_peered":"2024-06-10T00:25:44.345832+0000","last_clean":"2024-06-10T00:25:44.345832+0000","last_became_active":"2024-06-10T00:24:09.264425+0000","last_became_peered":"2024-06-10T00:24:09.264425+0000","last_unstale":"2024-06-10T00:25:44.345832+0000","last_undegraded":"2024-06-10T00:25:44.345832+0000","last_fullsized":"2024-06-10T00:25:44.345832+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:37:31.577688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"100'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.122763+0000","last_change":"2024-06-10T00:23:57.558540+0000","last_active":"2024-06-10T00:26:42.122763+0000","last_peered":"2024-06-10T00:26:42.122763+0000","last_clean":"2024-06-10T00:26:42.122763+0000","last_became_active":"2024-06-10T00:23:57.558197+0000","last_became_peered":"2024-06-10T00:23:57.558197+0000","last_unstale":"2024-06-10T00:26:42.122763+0000","last_undegraded":"2024-06-10T00:26:42.122763+0000","last_fullsized":"2024-06-10T00:26:42.122763+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:04:25.566135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"98'3","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733939+0000","last_change":"2024-06-10T00:23:54.428126+0000","last_active":"2024-06-10T00:25:44.733939+0000","last_peered":"2024-06-10T00:25:44.733939+0000","last_clean":"2024-06-10T00:25:44.733939+0000","last_became_active":"2024-06-10T00:23:54.427630+0000","last_became_peered":"2024-06-10T00:23:54.427630+0000","last_unstale":"2024-06-10T00:25:44.733939+0000","last_undegraded":"2024-06-10T00:25:44.733939+0000","last_fullsized":"2024-06-10T00:25:44.733939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:05:23.679704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189679+0000","last_change":"2024-06-10T00:24:09.169984+0000","last_active":"2024-06-10T00:25:44.189679+0000","last_peered":"2024-06-10T00:25:44.189679+0000","last_clean":"2024-06-10T00:25:44.189679+0000","last_became_active":"2024-06-10T00:24:09.169556+0000","last_became_peered":"2024-06-10T00:24:09.169556+0000","last_unstale":"2024-06-10T00:25:44.189679+0000","last_undegraded":"2024-06-10T00:25:44.189679+0000","last_fullsized":"2024-06-10T00:25:44.189679+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:54:13.575207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"98'1","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189843+0000","last_change":"2024-06-10T00:24:43.767901+0000","last_active":"2024-06-10T00:25:44.189843+0000","last_peered":"2024-06-10T00:25:44.189843+0000","last_clean":"2024-06-10T00:25:44.189843+0000","last_became_active":"2024-06-10T00:24:43.767562+0000","last_became_peered":"2024-06-10T00:24:43.767562+0000","last_unstale":"2024-06-10T00:25:44.189843+0000","last_undegraded":"2024-06-10T00:25:44.189843+0000","last_fullsized":"2024-06-10T00:25:44.189843+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:46:36.099750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"100'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.116650+0000","last_change":"2024-06-10T00:23:57.554906+0000","last_active":"2024-06-10T00:26:42.116650+0000","last_peered":"2024-06-10T00:26:42.116650+0000","last_clean":"2024-06-10T00:26:42.116650+0000","last_became_active":"2024-06-10T00:23:57.554608+0000","last_became_peered":"2024-06-10T00:23:57.554608+0000","last_unstale":"2024-06-10T00:26:42.116650+0000","last_undegraded":"2024-06-10T00:26:42.116650+0000","last_fullsized":"2024-06-10T00:26:42.116650+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:24.690594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733319+0000","last_change":"2024-06-10T00:24:09.264676+0000","last_active":"2024-06-10T00:25:44.733319+0000","last_peered":"2024-06-10T00:25:44.733319+0000","last_clean":"2024-06-10T00:25:44.733319+0000","last_became_active":"2024-06-10T00:24:09.264259+0000","last_became_peered":"2024-06-10T00:24:09.264259+0000","last_unstale":"2024-06-10T00:25:44.733319+0000","last_undegraded":"2024-06-10T00:25:44.733319+0000","last_fullsized":"2024-06-10T00:25:44.733319+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:37:53.787039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"98'12","reported_seq":57,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082806+0000","last_change":"2024-06-10T00:23:54.424201+0000","last_active":"2024-06-10T00:25:44.082806+0000","last_peered":"2024-06-10T00:25:44.082806+0000","last_clean":"2024-06-10T00:25:44.082806+0000","last_became_active":"2024-06-10T00:23:54.423853+0000","last_became_peered":"2024-06-10T00:23:54.423853+0000","last_unstale":"2024-06-10T00:25:44.082806+0000","last_undegraded":"2024-06-10T00:25:44.082806+0000","last_fullsized":"2024-06-10T00:25:44.082806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:00.884041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"100'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.132292+0000","last_change":"2024-06-10T00:23:57.554276+0000","last_active":"2024-06-10T00:26:42.132292+0000","last_peered":"2024-06-10T00:26:42.132292+0000","last_clean":"2024-06-10T00:26:42.132292+0000","last_became_active":"2024-06-10T00:23:57.553964+0000","last_became_peered":"2024-06-10T00:23:57.553964+0000","last_unstale":"2024-06-10T00:26:42.132292+0000","last_undegraded":"2024-06-10T00:26:42.132292+0000","last_fullsized":"2024-06-10T00:26:42.132292+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:39:51.974770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081427+0000","last_change":"2024-06-10T00:24:09.170249+0000","last_active":"2024-06-10T00:25:44.081427+0000","last_peered":"2024-06-10T00:25:44.081427+0000","last_clean":"2024-06-10T00:25:44.081427+0000","last_became_active":"2024-06-10T00:24:09.169830+0000","last_became_peered":"2024-06-10T00:24:09.169830+0000","last_unstale":"2024-06-10T00:25:44.081427+0000","last_undegraded":"2024-06-10T00:25:44.081427+0000","last_fullsized":"2024-06-10T00:25:44.081427+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.320609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"98'4","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345351+0000","last_change":"2024-06-10T00:23:54.473943+0000","last_active":"2024-06-10T00:25:44.345351+0000","last_peered":"2024-06-10T00:25:44.345351+0000","last_clean":"2024-06-10T00:25:44.345351+0000","last_became_active":"2024-06-10T00:23:54.473652+0000","last_became_peered":"2024-06-10T00:23:54.473652+0000","last_unstale":"2024-06-10T00:25:44.345351+0000","last_undegraded":"2024-06-10T00:25:44.345351+0000","last_fullsized":"2024-06-10T00:25:44.345351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:52:06.773684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.735414+0000","last_change":"2024-06-10T00:25:44.735050+0000","last_active":"2024-06-10T00:25:44.735414+0000","last_peered":"2024-06-10T00:25:44.735414+0000","last_clean":"2024-06-10T00:25:44.735414+0000","last_became_active":"2024-06-10T00:25:44.734756+0000","last_became_peered":"2024-06-10T00:25:44.734756+0000","last_unstale":"2024-06-10T00:25:44.735414+0000","last_undegraded":"2024-06-10T00:25:44.735414+0000","last_fullsized":"2024-06-10T00:25:44.735414+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:16:03.924238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":34,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344802+0000","last_change":"2024-06-10T00:24:43.768837+0000","last_active":"2024-06-10T00:25:44.344802+0000","last_peered":"2024-06-10T00:25:44.344802+0000","last_clean":"2024-06-10T00:25:44.344802+0000","last_became_active":"2024-06-10T00:24:43.768295+0000","last_became_peered":"2024-06-10T00:24:43.768295+0000","last_unstale":"2024-06-10T00:25:44.344802+0000","last_undegraded":"2024-06-10T00:25:44.344802+0000","last_fullsized":"2024-06-10T00:25:44.344802+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:59.600864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"98'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.124404+0000","last_change":"2024-06-10T00:23:54.823448+0000","last_active":"2024-06-10T00:25:44.124404+0000","last_peered":"2024-06-10T00:25:44.124404+0000","last_clean":"2024-06-10T00:25:44.124404+0000","last_became_active":"2024-06-10T00:23:54.823107+0000","last_became_peered":"2024-06-10T00:23:54.823107+0000","last_unstale":"2024-06-10T00:25:44.124404+0000","last_undegraded":"2024-06-10T00:25:44.124404+0000","last_fullsized":"2024-06-10T00:25:44.124404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:50.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"100'2","reported_seq":41,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.092265+0000","last_change":"2024-06-10T00:23:57.556733+0000","last_active":"2024-06-10T00:26:42.092265+0000","last_peered":"2024-06-10T00:26:42.092265+0000","last_clean":"2024-06-10T00:26:42.092265+0000","last_became_active":"2024-06-10T00:23:57.556386+0000","last_became_peered":"2024-06-10T00:23:57.556386+0000","last_unstale":"2024-06-10T00:26:42.092265+0000","last_undegraded":"2024-06-10T00:26:42.092265+0000","last_fullsized":"2024-06-10T00:26:42.092265+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:39:00.572311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082653+0000","last_change":"2024-06-10T00:24:09.174600+0000","last_active":"2024-06-10T00:25:44.082653+0000","last_peered":"2024-06-10T00:25:44.082653+0000","last_clean":"2024-06-10T00:25:44.082653+0000","last_became_active":"2024-06-10T00:24:09.174236+0000","last_became_peered":"2024-06-10T00:24:09.174236+0000","last_unstale":"2024-06-10T00:25:44.082653+0000","last_undegraded":"2024-06-10T00:25:44.082653+0000","last_fullsized":"2024-06-10T00:25:44.082653+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:58:18.447768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"98'3","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733487+0000","last_change":"2024-06-10T00:23:54.425111+0000","last_active":"2024-06-10T00:25:44.733487+0000","last_peered":"2024-06-10T00:25:44.733487+0000","last_clean":"2024-06-10T00:25:44.733487+0000","last_became_active":"2024-06-10T00:23:54.424724+0000","last_became_peered":"2024-06-10T00:23:54.424724+0000","last_unstale":"2024-06-10T00:25:44.733487+0000","last_undegraded":"2024-06-10T00:25:44.733487+0000","last_fullsized":"2024-06-10T00:25:44.733487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:54.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189649+0000","last_change":"2024-06-10T00:23:57.556355+0000","last_active":"2024-06-10T00:25:44.189649+0000","last_peered":"2024-06-10T00:25:44.189649+0000","last_clean":"2024-06-10T00:25:44.189649+0000","last_became_active":"2024-06-10T00:23:57.555479+0000","last_became_peered":"2024-06-10T00:23:57.555479+0000","last_unstale":"2024-06-10T00:25:44.189649+0000","last_undegraded":"2024-06-10T00:25:44.189649+0000","last_fullsized":"2024-06-10T00:25:44.189649+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:08.817381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"98'1","reported_seq":28,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344830+0000","last_change":"2024-06-10T00:24:09.169589+0000","last_active":"2024-06-10T00:25:44.344830+0000","last_peered":"2024-06-10T00:25:44.344830+0000","last_clean":"2024-06-10T00:25:44.344830+0000","last_became_active":"2024-06-10T00:24:09.168436+0000","last_became_peered":"2024-06-10T00:24:09.168436+0000","last_unstale":"2024-06-10T00:25:44.344830+0000","last_undegraded":"2024-06-10T00:25:44.344830+0000","last_fullsized":"2024-06-10T00:25:44.344830+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:17:32.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"98'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189512+0000","last_change":"2024-06-10T00:24:43.751399+0000","last_active":"2024-06-10T00:25:44.189512+0000","last_peered":"2024-06-10T00:25:44.189512+0000","last_clean":"2024-06-10T00:25:44.189512+0000","last_became_active":"2024-06-10T00:24:43.750987+0000","last_became_peered":"2024-06-10T00:24:43.750987+0000","last_unstale":"2024-06-10T00:25:44.189512+0000","last_undegraded":"2024-06-10T00:25:44.189512+0000","last_fullsized":"2024-06-10T00:25:44.189512+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:14:47.902017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"100'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.129730+0000","last_change":"2024-06-10T00:23:57.560213+0000","last_active":"2024-06-10T00:26:42.129730+0000","last_peered":"2024-06-10T00:26:42.129730+0000","last_clean":"2024-06-10T00:26:42.129730+0000","last_became_active":"2024-06-10T00:23:57.559925+0000","last_became_peered":"2024-06-10T00:23:57.559925+0000","last_unstale":"2024-06-10T00:26:42.129730+0000","last_undegraded":"2024-06-10T00:26:42.129730+0000","last_fullsized":"2024-06-10T00:26:42.129730+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:00:37.403087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344834+0000","last_change":"2024-06-10T00:24:09.264758+0000","last_active":"2024-06-10T00:25:44.344834+0000","last_peered":"2024-06-10T00:25:44.344834+0000","last_clean":"2024-06-10T00:25:44.344834+0000","last_became_active":"2024-06-10T00:24:09.263928+0000","last_became_peered":"2024-06-10T00:24:09.263928+0000","last_unstale":"2024-06-10T00:25:44.344834+0000","last_undegraded":"2024-06-10T00:25:44.344834+0000","last_fullsized":"2024-06-10T00:25:44.344834+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:23:41.083091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"98'4","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.734116+0000","last_change":"2024-06-10T00:23:54.428553+0000","last_active":"2024-06-10T00:25:44.734116+0000","last_peered":"2024-06-10T00:25:44.734116+0000","last_clean":"2024-06-10T00:25:44.734116+0000","last_became_active":"2024-06-10T00:23:54.428108+0000","last_became_peered":"2024-06-10T00:23:54.428108+0000","last_unstale":"2024-06-10T00:25:44.734116+0000","last_undegraded":"2024-06-10T00:25:44.734116+0000","last_fullsized":"2024-06-10T00:25:44.734116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:57:26.207915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"100'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.135760+0000","last_change":"2024-06-10T00:23:57.560686+0000","last_active":"2024-06-10T00:26:42.135760+0000","last_peered":"2024-06-10T00:26:42.135760+0000","last_clean":"2024-06-10T00:26:42.135760+0000","last_became_active":"2024-06-10T00:23:57.560326+0000","last_became_peered":"2024-06-10T00:23:57.560326+0000","last_unstale":"2024-06-10T00:26:42.135760+0000","last_undegraded":"2024-06-10T00:26:42.135760+0000","last_fullsized":"2024-06-10T00:26:42.135760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:08:39.506567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.124235+0000","last_change":"2024-06-10T00:24:09.166947+0000","last_active":"2024-06-10T00:25:44.124235+0000","last_peered":"2024-06-10T00:25:44.124235+0000","last_clean":"2024-06-10T00:25:44.124235+0000","last_became_active":"2024-06-10T00:24:09.166503+0000","last_became_peered":"2024-06-10T00:24:09.166503+0000","last_unstale":"2024-06-10T00:25:44.124235+0000","last_undegraded":"2024-06-10T00:25:44.124235+0000","last_fullsized":"2024-06-10T00:25:44.124235+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:16:45.094642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"98'3","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081865+0000","last_change":"2024-06-10T00:23:54.823522+0000","last_active":"2024-06-10T00:25:44.081865+0000","last_peered":"2024-06-10T00:25:44.081865+0000","last_clean":"2024-06-10T00:25:44.081865+0000","last_became_active":"2024-06-10T00:23:54.823142+0000","last_became_peered":"2024-06-10T00:23:54.823142+0000","last_unstale":"2024-06-10T00:25:44.081865+0000","last_undegraded":"2024-06-10T00:25:44.081865+0000","last_fullsized":"2024-06-10T00:25:44.081865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:03.836850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.141160+0000","last_change":"2024-06-10T00:23:57.563710+0000","last_active":"2024-06-10T00:26:42.141160+0000","last_peered":"2024-06-10T00:26:42.141160+0000","last_clean":"2024-06-10T00:26:42.141160+0000","last_became_active":"2024-06-10T00:23:57.563331+0000","last_became_peered":"2024-06-10T00:23:57.563331+0000","last_unstale":"2024-06-10T00:26:42.141160+0000","last_undegraded":"2024-06-10T00:26:42.141160+0000","last_fullsized":"2024-06-10T00:26:42.141160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:08:48.061885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081657+0000","last_change":"2024-06-10T00:24:09.171203+0000","last_active":"2024-06-10T00:25:44.081657+0000","last_peered":"2024-06-10T00:25:44.081657+0000","last_clean":"2024-06-10T00:25:44.081657+0000","last_became_active":"2024-06-10T00:24:09.170824+0000","last_became_peered":"2024-06-10T00:24:09.170824+0000","last_unstale":"2024-06-10T00:25:44.081657+0000","last_undegraded":"2024-06-10T00:25:44.081657+0000","last_fullsized":"2024-06-10T00:25:44.081657+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:47:28.045602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345877+0000","last_change":"2024-06-10T00:23:54.421307+0000","last_active":"2024-06-10T00:25:44.345877+0000","last_peered":"2024-06-10T00:25:44.345877+0000","last_clean":"2024-06-10T00:25:44.345877+0000","last_became_active":"2024-06-10T00:23:54.420928+0000","last_became_peered":"2024-06-10T00:23:54.420928+0000","last_unstale":"2024-06-10T00:25:44.345877+0000","last_undegraded":"2024-06-10T00:25:44.345877+0000","last_fullsized":"2024-06-10T00:25:44.345877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:59:41.433605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.133494+0000","last_change":"2024-06-10T00:23:57.556494+0000","last_active":"2024-06-10T00:26:42.133494+0000","last_peered":"2024-06-10T00:26:42.133494+0000","last_clean":"2024-06-10T00:26:42.133494+0000","last_became_active":"2024-06-10T00:23:57.556088+0000","last_became_peered":"2024-06-10T00:23:57.556088+0000","last_unstale":"2024-06-10T00:26:42.133494+0000","last_undegraded":"2024-06-10T00:26:42.133494+0000","last_fullsized":"2024-06-10T00:26:42.133494+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:06:23.873281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344954+0000","last_change":"2024-06-10T00:24:09.166734+0000","last_active":"2024-06-10T00:25:44.344954+0000","last_peered":"2024-06-10T00:25:44.344954+0000","last_clean":"2024-06-10T00:25:44.344954+0000","last_became_active":"2024-06-10T00:24:09.166270+0000","last_became_peered":"2024-06-10T00:24:09.166270+0000","last_unstale":"2024-06-10T00:25:44.344954+0000","last_undegraded":"2024-06-10T00:25:44.344954+0000","last_fullsized":"2024-06-10T00:25:44.344954+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:38.039633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078379+0000","last_change":"2024-06-10T00:23:54.426226+0000","last_active":"2024-06-10T00:25:44.078379+0000","last_peered":"2024-06-10T00:25:44.078379+0000","last_clean":"2024-06-10T00:25:44.078379+0000","last_became_active":"2024-06-10T00:23:54.425879+0000","last_became_peered":"2024-06-10T00:23:54.425879+0000","last_unstale":"2024-06-10T00:25:44.078379+0000","last_undegraded":"2024-06-10T00:25:44.078379+0000","last_fullsized":"2024-06-10T00:25:44.078379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:15:32.936556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"100'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.136571+0000","last_change":"2024-06-10T00:23:57.553814+0000","last_active":"2024-06-10T00:26:42.136571+0000","last_peered":"2024-06-10T00:26:42.136571+0000","last_clean":"2024-06-10T00:26:42.136571+0000","last_became_active":"2024-06-10T00:23:57.553478+0000","last_became_peered":"2024-06-10T00:23:57.553478+0000","last_unstale":"2024-06-10T00:26:42.136571+0000","last_undegraded":"2024-06-10T00:26:42.136571+0000","last_fullsized":"2024-06-10T00:26:42.136571+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:54:39.264526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189599+0000","last_change":"2024-06-10T00:23:57.556676+0000","last_active":"2024-06-10T00:25:44.189599+0000","last_peered":"2024-06-10T00:25:44.189599+0000","last_clean":"2024-06-10T00:25:44.189599+0000","last_became_active":"2024-06-10T00:23:57.555946+0000","last_became_peered":"2024-06-10T00:23:57.555946+0000","last_unstale":"2024-06-10T00:25:44.189599+0000","last_undegraded":"2024-06-10T00:25:44.189599+0000","last_fullsized":"2024-06-10T00:25:44.189599+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:10:10.868799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"100'12","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.116481+0000","last_change":"2024-06-10T00:23:57.565423+0000","last_active":"2024-06-10T00:26:42.116481+0000","last_peered":"2024-06-10T00:26:42.116481+0000","last_clean":"2024-06-10T00:26:42.116481+0000","last_became_active":"2024-06-10T00:23:57.564830+0000","last_became_peered":"2024-06-10T00:23:57.564830+0000","last_unstale":"2024-06-10T00:26:42.116481+0000","last_undegraded":"2024-06-10T00:26:42.116481+0000","last_fullsized":"2024-06-10T00:26:42.116481+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:11:06.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"100'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.127160+0000","last_change":"2024-06-10T00:23:57.558971+0000","last_active":"2024-06-10T00:26:42.127160+0000","last_peered":"2024-06-10T00:26:42.127160+0000","last_clean":"2024-06-10T00:26:42.127160+0000","last_became_active":"2024-06-10T00:23:57.558620+0000","last_became_peered":"2024-06-10T00:23:57.558620+0000","last_unstale":"2024-06-10T00:26:42.127160+0000","last_undegraded":"2024-06-10T00:26:42.127160+0000","last_fullsized":"2024-06-10T00:26:42.127160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:45:43.308008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081924+0000","last_change":"2024-06-10T00:23:57.552341+0000","last_active":"2024-06-10T00:25:44.081924+0000","last_peered":"2024-06-10T00:25:44.081924+0000","last_clean":"2024-06-10T00:25:44.081924+0000","last_became_active":"2024-06-10T00:23:57.551464+0000","last_became_peered":"2024-06-10T00:23:57.551464+0000","last_unstale":"2024-06-10T00:25:44.081924+0000","last_undegraded":"2024-06-10T00:25:44.081924+0000","last_fullsized":"2024-06-10T00:25:44.081924+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:44:41.063516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"100'4","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.129662+0000","last_change":"2024-06-10T00:24:43.768961+0000","last_active":"2024-06-10T00:26:42.129662+0000","last_peered":"2024-06-10T00:26:42.129662+0000","last_clean":"2024-06-10T00:26:42.129662+0000","last_became_active":"2024-06-10T00:24:43.768542+0000","last_became_peered":"2024-06-10T00:24:43.768542+0000","last_unstale":"2024-06-10T00:26:42.129662+0000","last_undegraded":"2024-06-10T00:26:42.129662+0000","last_fullsized":"2024-06-10T00:26:42.129662+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:23:58.412054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189867+0000","last_change":"2024-06-10T00:23:57.554924+0000","last_active":"2024-06-10T00:25:44.189867+0000","last_peered":"2024-06-10T00:25:44.189867+0000","last_clean":"2024-06-10T00:25:44.189867+0000","last_became_active":"2024-06-10T00:23:57.554515+0000","last_became_peered":"2024-06-10T00:23:57.554515+0000","last_unstale":"2024-06-10T00:25:44.189867+0000","last_undegraded":"2024-06-10T00:25:44.189867+0000","last_fullsized":"2024-06-10T00:25:44.189867+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:58:15.624775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"100'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.134946+0000","last_change":"2024-06-10T00:23:57.561940+0000","last_active":"2024-06-10T00:26:42.134946+0000","last_peered":"2024-06-10T00:26:42.134946+0000","last_clean":"2024-06-10T00:26:42.134946+0000","last_became_active":"2024-06-10T00:23:57.561512+0000","last_became_peered":"2024-06-10T00:23:57.561512+0000","last_unstale":"2024-06-10T00:26:42.134946+0000","last_undegraded":"2024-06-10T00:26:42.134946+0000","last_fullsized":"2024-06-10T00:26:42.134946+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:57:57.147311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"100'2","reported_seq":41,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.126595+0000","last_change":"2024-06-10T00:23:57.557229+0000","last_active":"2024-06-10T00:26:42.126595+0000","last_peered":"2024-06-10T00:26:42.126595+0000","last_clean":"2024-06-10T00:26:42.126595+0000","last_became_active":"2024-06-10T00:23:57.556905+0000","last_became_peered":"2024-06-10T00:23:57.556905+0000","last_unstale":"2024-06-10T00:26:42.126595+0000","last_undegraded":"2024-06-10T00:26:42.126595+0000","last_fullsized":"2024-06-10T00:26:42.126595+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:30.513517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"100'6","reported_seq":21,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.132491+0000","last_change":"2024-06-10T00:24:43.768509+0000","last_active":"2024-06-10T00:26:42.132491+0000","last_peered":"2024-06-10T00:26:42.132491+0000","last_clean":"2024-06-10T00:26:42.132491+0000","last_became_active":"2024-06-10T00:24:43.768038+0000","last_became_peered":"2024-06-10T00:24:43.768038+0000","last_unstale":"2024-06-10T00:26:42.132491+0000","last_undegraded":"2024-06-10T00:26:42.132491+0000","last_fullsized":"2024-06-10T00:26:42.132491+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:38.344375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"100'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.128321+0000","last_change":"2024-06-10T00:23:57.557830+0000","last_active":"2024-06-10T00:26:42.128321+0000","last_peered":"2024-06-10T00:26:42.128321+0000","last_clean":"2024-06-10T00:26:42.128321+0000","last_became_active":"2024-06-10T00:23:57.556986+0000","last_became_peered":"2024-06-10T00:23:57.556986+0000","last_unstale":"2024-06-10T00:26:42.128321+0000","last_undegraded":"2024-06-10T00:26:42.128321+0000","last_fullsized":"2024-06-10T00:26:42.128321+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:53.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":39,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.124805+0000","last_change":"2024-06-10T00:23:57.565602+0000","last_active":"2024-06-10T00:25:44.124805+0000","last_peered":"2024-06-10T00:25:44.124805+0000","last_clean":"2024-06-10T00:25:44.124805+0000","last_became_active":"2024-06-10T00:23:57.565264+0000","last_became_peered":"2024-06-10T00:23:57.565264+0000","last_unstale":"2024-06-10T00:25:44.124805+0000","last_undegraded":"2024-06-10T00:25:44.124805+0000","last_fullsized":"2024-06-10T00:25:44.124805+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:09:12.773087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"100'11","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.138705+0000","last_change":"2024-06-10T00:23:57.548516+0000","last_active":"2024-06-10T00:26:42.138705+0000","last_peered":"2024-06-10T00:26:42.138705+0000","last_clean":"2024-06-10T00:26:42.138705+0000","last_became_active":"2024-06-10T00:23:57.548202+0000","last_became_peered":"2024-06-10T00:23:57.548202+0000","last_unstale":"2024-06-10T00:26:42.138705+0000","last_undegraded":"2024-06-10T00:26:42.138705+0000","last_fullsized":"2024-06-10T00:26:42.138705+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:13.901478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":38,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189839+0000","last_change":"2024-06-10T00:23:57.565900+0000","last_active":"2024-06-10T00:25:44.189839+0000","last_peered":"2024-06-10T00:25:44.189839+0000","last_clean":"2024-06-10T00:25:44.189839+0000","last_became_active":"2024-06-10T00:23:57.565573+0000","last_became_peered":"2024-06-10T00:23:57.565573+0000","last_unstale":"2024-06-10T00:25:44.189839+0000","last_undegraded":"2024-06-10T00:25:44.189839+0000","last_fullsized":"2024-06-10T00:25:44.189839+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:08.255905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280680,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66904,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":617,"num_read_kb":382,"num_write":173,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":200745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351026,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1105920,"data_stored":572925,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612663,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32396,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712756,"statfs":{"total":95995035648,"available":95961862144,"internally_reserved":0,"allocated":1249280,"data_stored":1223075,"data_compressed":68001,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776188,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":577235,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037825,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":573037,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299462,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":1169108,"data_compressed":68001,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462987,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":568315,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1167360,"data_stored":630880,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738472,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32272,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712880,"statfs":{"total":95995035648,"available":95961989120,"internally_reserved":0,"allocated":1122304,"data_stored":569703,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36500,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708652,"statfs":{"total":95995035648,"available":95957659648,"internally_reserved":0,"allocated":1191936,"data_stored":633010,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":568509,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888149,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1110016,"data_stored":572965,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724624,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32328,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712824,"statfs":{"total":95995035648,"available":95961931776,"internally_reserved":0,"allocated":1175552,"data_stored":1164555,"data_compressed":68001,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-10T00:26:46.417 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-10T00:26:46.655 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:47 smithi121 ceph-mon[32378]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:26:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:47 smithi121 ceph-mon[32378]: pgmap v392: 129 pgs: 129 active+clean; 642 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-06-10T00:26:47.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:47 smithi135 ceph-mon[36424]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:26:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:47 smithi135 ceph-mon[36424]: pgmap v392: 129 pgs: 129 active+clean; 642 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-06-10T00:26:47.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:47 smithi193 ceph-mon[36716]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:26:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:47 smithi193 ceph-mon[36716]: pgmap v392: 129 pgs: 129 active+clean; 642 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-06-10T00:26:48.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.308+0000 7f5f73b5c700 1 -- 172.21.15.121:0/3309964663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c101880 msgr2=0x7f5f6c10f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:48.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.308+0000 7f5f73b5c700 1 --2- 172.21.15.121:0/3309964663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c101880 0x7f5f6c10f380 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5f6400b0a0 tx=0x7f5f6400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:26:48.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.308+0000 7f5f73b5c700 1 -- 172.21.15.121:0/3309964663 shutdown_connections 2024-06-10T00:26:48.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.308+0000 7f5f73b5c700 1 --2- 172.21.15.121:0/3309964663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c101880 0x7f5f6c10f380 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.308+0000 7f5f73b5c700 1 --2- 172.21.15.121:0/3309964663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f6c108670 0x7f5f6c108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.308+0000 7f5f73b5c700 1 --2- 172.21.15.121:0/3309964663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f6c100ec0 0x7f5f6c101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.308+0000 7f5f73b5c700 1 -- 172.21.15.121:0/3309964663 >> 172.21.15.121:0/3309964663 conn(0x7f5f6c076600 msgr2=0x7f5f6c076a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.308+0000 7f5f73b5c700 1 -- 172.21.15.121:0/3309964663 shutdown_connections 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.308+0000 7f5f73b5c700 1 -- 172.21.15.121:0/3309964663 wait complete. 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f73b5c700 1 Processor -- start 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f73b5c700 1 -- start start 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f73b5c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f6c100ec0 0x7f5f6c19e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f73b5c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f6c101880 0x7f5f6c19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f73b5c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c108670 0x7f5f6c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f73b5c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c112870 con 0x7f5f6c108670 2024-06-10T00:26:48.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f73b5c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1126f0 con 0x7f5f6c101880 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f73b5c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c1129f0 con 0x7f5f6c100ec0 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f720f9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c108670 0x7f5f6c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f720f9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c108670 0x7f5f6c198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48928/0 (socket says 172.21.15.121:48928) 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f720f9700 1 -- 172.21.15.121:0/802028478 learned_addr learned my addr 172.21.15.121:0/802028478 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f718f8700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f6c100ec0 0x7f5f6c19e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f710f7700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f6c101880 0x7f5f6c19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f720f9700 1 -- 172.21.15.121:0/802028478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f6c100ec0 msgr2=0x7f5f6c19e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f720f9700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f6c100ec0 0x7f5f6c19e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f720f9700 1 -- 172.21.15.121:0/802028478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f6c101880 msgr2=0x7f5f6c19edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f720f9700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f6c101880 0x7f5f6c19edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f720f9700 1 -- 172.21.15.121:0/802028478 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c1991b0 con 0x7f5f6c108670 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f720f9700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c108670 0x7f5f6c198a50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5f640062a0 tx=0x7f5f6400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:48.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f62ffd700 1 -- 172.21.15.121:0/802028478 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64021070 con 0x7f5f6c108670 2024-06-10T00:26:48.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f62ffd700 1 -- 172.21.15.121:0/802028478 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f64007d00 con 0x7f5f6c108670 2024-06-10T00:26:48.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.309+0000 7f5f62ffd700 1 -- 172.21.15.121:0/802028478 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64005460 con 0x7f5f6c108670 2024-06-10T00:26:48.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.310+0000 7f5f73b5c700 1 -- 172.21.15.121:0/802028478 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6c199440 con 0x7f5f6c108670 2024-06-10T00:26:48.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.310+0000 7f5f73b5c700 1 -- 172.21.15.121:0/802028478 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6c1a5760 con 0x7f5f6c108670 2024-06-10T00:26:48.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.310+0000 7f5f73b5c700 1 -- 172.21.15.121:0/802028478 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f6c069180 con 0x7f5f6c108670 2024-06-10T00:26:48.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.312+0000 7f5f62ffd700 1 -- 172.21.15.121:0/802028478 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f5f64024070 con 0x7f5f6c108670 2024-06-10T00:26:48.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.312+0000 7f5f62ffd700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5f580a2c20 0x7f5f580a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:48.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.312+0000 7f5f718f8700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5f580a2c20 0x7f5f580a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:48.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.312+0000 7f5f62ffd700 1 -- 172.21.15.121:0/802028478 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f5f6400f020 con 0x7f5f6c108670 2024-06-10T00:26:48.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.313+0000 7f5f718f8700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5f580a2c20 0x7f5f580a50e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c199b90 tx=0x7f5f5c009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:48.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.316+0000 7f5f62ffd700 1 -- 172.21.15.121:0/802028478 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5f6402b020 con 0x7f5f6c108670 2024-06-10T00:26:48.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.510+0000 7f5f73b5c700 1 -- 172.21.15.121:0/802028478 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5f6c100020 con 0x7f5f580a2c20 2024-06-10T00:26:48.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.520+0000 7f5f62ffd700 1 -- 172.21.15.121:0/802028478 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+341056 (secure 0 0 0) 0x7f5f6c100020 con 0x7f5f580a2c20 2024-06-10T00:26:48.524 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:26:48.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 -- 172.21.15.121:0/802028478 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5f580a2c20 msgr2=0x7f5f580a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:48.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5f580a2c20 0x7f5f580a50e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c199b90 tx=0x7f5f5c009190 comp rx=0 tx=0).stop 2024-06-10T00:26:48.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 -- 172.21.15.121:0/802028478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c108670 msgr2=0x7f5f6c198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:48.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c108670 0x7f5f6c198a50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5f640062a0 tx=0x7f5f6400be60 comp rx=0 tx=0).stop 2024-06-10T00:26:48.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 -- 172.21.15.121:0/802028478 shutdown_connections 2024-06-10T00:26:48.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f5f580a2c20 0x7f5f580a50e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:48.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f6c108670 0x7f5f6c198a50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:48.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f6c100ec0 0x7f5f6c19e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:48.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 --2- 172.21.15.121:0/802028478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5f6c101880 0x7f5f6c19edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:48.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 -- 172.21.15.121:0/802028478 >> 172.21.15.121:0/802028478 conn(0x7f5f6c076600 msgr2=0x7f5f6c0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:48.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 -- 172.21.15.121:0/802028478 shutdown_connections 2024-06-10T00:26:48.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:48.523+0000 7f5f73b5c700 1 -- 172.21.15.121:0/802028478 wait complete. 2024-06-10T00:26:48.528 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-06-10T00:26:49.350 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-06-10T00:26:48.139573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657647,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":769,"num_read_kb":507,"num_write":359,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":366724,"kb_used_data":13412,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575100,"statfs":{"total":1151940427776,"available":1151564902400,"internally_reserved":0,"allocated":13733888,"data_stored":8823707,"data_compressed":494712,"data_compressed_allocated":675840,"data_compressed_original":2850816,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":77,"num_read_kb":37,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003108"},"pg_stats":[{"pgid":"3.3f","version":"100'8","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.135437+0000","last_change":"2024-06-10T00:23:57.564073+0000","last_active":"2024-06-10T00:26:42.135437+0000","last_peered":"2024-06-10T00:26:42.135437+0000","last_clean":"2024-06-10T00:26:42.135437+0000","last_became_active":"2024-06-10T00:23:57.563618+0000","last_became_peered":"2024-06-10T00:23:57.563618+0000","last_unstale":"2024-06-10T00:26:42.135437+0000","last_undegraded":"2024-06-10T00:26:42.135437+0000","last_fullsized":"2024-06-10T00:26:42.135437+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:26:25.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"100'4","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.126275+0000","last_change":"2024-06-10T00:23:57.560148+0000","last_active":"2024-06-10T00:26:42.126275+0000","last_peered":"2024-06-10T00:26:42.126275+0000","last_clean":"2024-06-10T00:26:42.126275+0000","last_became_active":"2024-06-10T00:23:57.559794+0000","last_became_peered":"2024-06-10T00:23:57.559794+0000","last_unstale":"2024-06-10T00:26:42.126275+0000","last_undegraded":"2024-06-10T00:26:42.126275+0000","last_fullsized":"2024-06-10T00:26:42.126275+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:59:50.624542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.133679+0000","last_change":"2024-06-10T00:23:57.563292+0000","last_active":"2024-06-10T00:26:42.133679+0000","last_peered":"2024-06-10T00:26:42.133679+0000","last_clean":"2024-06-10T00:26:42.133679+0000","last_became_active":"2024-06-10T00:23:57.563007+0000","last_became_peered":"2024-06-10T00:23:57.563007+0000","last_unstale":"2024-06-10T00:26:42.133679+0000","last_undegraded":"2024-06-10T00:26:42.133679+0000","last_fullsized":"2024-06-10T00:26:42.133679+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:27:44.897770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"100'8","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.123939+0000","last_change":"2024-06-10T00:23:57.558789+0000","last_active":"2024-06-10T00:26:42.123939+0000","last_peered":"2024-06-10T00:26:42.123939+0000","last_clean":"2024-06-10T00:26:42.123939+0000","last_became_active":"2024-06-10T00:23:57.557943+0000","last_became_peered":"2024-06-10T00:23:57.557943+0000","last_unstale":"2024-06-10T00:26:42.123939+0000","last_undegraded":"2024-06-10T00:26:42.123939+0000","last_fullsized":"2024-06-10T00:26:42.123939+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:10:40.081996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"100'4","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.120346+0000","last_change":"2024-06-10T00:23:57.550806+0000","last_active":"2024-06-10T00:26:42.120346+0000","last_peered":"2024-06-10T00:26:42.120346+0000","last_clean":"2024-06-10T00:26:42.120346+0000","last_became_active":"2024-06-10T00:23:57.550453+0000","last_became_peered":"2024-06-10T00:23:57.550453+0000","last_unstale":"2024-06-10T00:26:42.120346+0000","last_undegraded":"2024-06-10T00:26:42.120346+0000","last_fullsized":"2024-06-10T00:26:42.120346+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:11:58.765099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.122945+0000","last_change":"2024-06-10T00:23:57.567307+0000","last_active":"2024-06-10T00:26:42.122945+0000","last_peered":"2024-06-10T00:26:42.122945+0000","last_clean":"2024-06-10T00:26:42.122945+0000","last_became_active":"2024-06-10T00:23:57.566973+0000","last_became_peered":"2024-06-10T00:23:57.566973+0000","last_unstale":"2024-06-10T00:26:42.122945+0000","last_undegraded":"2024-06-10T00:26:42.122945+0000","last_fullsized":"2024-06-10T00:26:42.122945+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:27:25.135046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"100'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.130655+0000","last_change":"2024-06-10T00:23:57.547998+0000","last_active":"2024-06-10T00:26:42.130655+0000","last_peered":"2024-06-10T00:26:42.130655+0000","last_clean":"2024-06-10T00:26:42.130655+0000","last_became_active":"2024-06-10T00:23:57.547593+0000","last_became_peered":"2024-06-10T00:23:57.547593+0000","last_unstale":"2024-06-10T00:26:42.130655+0000","last_undegraded":"2024-06-10T00:26:42.130655+0000","last_fullsized":"2024-06-10T00:26:42.130655+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:34:04.219399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"100'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.129642+0000","last_change":"2024-06-10T00:23:57.560252+0000","last_active":"2024-06-10T00:26:42.129642+0000","last_peered":"2024-06-10T00:26:42.129642+0000","last_clean":"2024-06-10T00:26:42.129642+0000","last_became_active":"2024-06-10T00:23:57.559924+0000","last_became_peered":"2024-06-10T00:23:57.559924+0000","last_unstale":"2024-06-10T00:26:42.129642+0000","last_undegraded":"2024-06-10T00:26:42.129642+0000","last_fullsized":"2024-06-10T00:26:42.129642+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:01:43.684262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"100'11","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.139604+0000","last_change":"2024-06-10T00:23:57.564031+0000","last_active":"2024-06-10T00:26:42.139604+0000","last_peered":"2024-06-10T00:26:42.139604+0000","last_clean":"2024-06-10T00:26:42.139604+0000","last_became_active":"2024-06-10T00:23:57.563642+0000","last_became_peered":"2024-06-10T00:23:57.563642+0000","last_unstale":"2024-06-10T00:26:42.139604+0000","last_undegraded":"2024-06-10T00:26:42.139604+0000","last_fullsized":"2024-06-10T00:26:42.139604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:16:23.003152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"100'16","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.122373+0000","last_change":"2024-06-10T00:23:57.566877+0000","last_active":"2024-06-10T00:26:42.122373+0000","last_peered":"2024-06-10T00:26:42.122373+0000","last_clean":"2024-06-10T00:26:42.122373+0000","last_became_active":"2024-06-10T00:23:57.566529+0000","last_became_peered":"2024-06-10T00:23:57.566529+0000","last_unstale":"2024-06-10T00:26:42.122373+0000","last_undegraded":"2024-06-10T00:26:42.122373+0000","last_fullsized":"2024-06-10T00:26:42.122373+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:30:43.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.140791+0000","last_change":"2024-06-10T00:23:57.564979+0000","last_active":"2024-06-10T00:26:42.140791+0000","last_peered":"2024-06-10T00:26:42.140791+0000","last_clean":"2024-06-10T00:26:42.140791+0000","last_became_active":"2024-06-10T00:23:57.564578+0000","last_became_peered":"2024-06-10T00:23:57.564578+0000","last_unstale":"2024-06-10T00:26:42.140791+0000","last_undegraded":"2024-06-10T00:26:42.140791+0000","last_fullsized":"2024-06-10T00:26:42.140791+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:40:53.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"100'7","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.123562+0000","last_change":"2024-06-10T00:23:57.559126+0000","last_active":"2024-06-10T00:26:42.123562+0000","last_peered":"2024-06-10T00:26:42.123562+0000","last_clean":"2024-06-10T00:26:42.123562+0000","last_became_active":"2024-06-10T00:23:57.558401+0000","last_became_peered":"2024-06-10T00:23:57.558401+0000","last_unstale":"2024-06-10T00:26:42.123562+0000","last_undegraded":"2024-06-10T00:26:42.123562+0000","last_fullsized":"2024-06-10T00:26:42.123562+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:57.894137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"100'10","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.126736+0000","last_change":"2024-06-10T00:23:57.559272+0000","last_active":"2024-06-10T00:26:42.126736+0000","last_peered":"2024-06-10T00:26:42.126736+0000","last_clean":"2024-06-10T00:26:42.126736+0000","last_became_active":"2024-06-10T00:23:57.558941+0000","last_became_peered":"2024-06-10T00:23:57.558941+0000","last_unstale":"2024-06-10T00:26:42.126736+0000","last_undegraded":"2024-06-10T00:26:42.126736+0000","last_fullsized":"2024-06-10T00:26:42.126736+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:36:48.088347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.129241+0000","last_change":"2024-06-10T00:23:57.555875+0000","last_active":"2024-06-10T00:26:42.129241+0000","last_peered":"2024-06-10T00:26:42.129241+0000","last_clean":"2024-06-10T00:26:42.129241+0000","last_became_active":"2024-06-10T00:23:57.555492+0000","last_became_peered":"2024-06-10T00:23:57.555492+0000","last_unstale":"2024-06-10T00:26:42.129241+0000","last_undegraded":"2024-06-10T00:26:42.129241+0000","last_fullsized":"2024-06-10T00:26:42.129241+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:49:10.827458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.128483+0000","last_change":"2024-06-10T00:23:57.555316+0000","last_active":"2024-06-10T00:26:42.128483+0000","last_peered":"2024-06-10T00:26:42.128483+0000","last_clean":"2024-06-10T00:26:42.128483+0000","last_became_active":"2024-06-10T00:23:57.554952+0000","last_became_peered":"2024-06-10T00:23:57.554952+0000","last_unstale":"2024-06-10T00:26:42.128483+0000","last_undegraded":"2024-06-10T00:26:42.128483+0000","last_fullsized":"2024-06-10T00:26:42.128483+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:03:57.557330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"100'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.140012+0000","last_change":"2024-06-10T00:23:57.562442+0000","last_active":"2024-06-10T00:26:42.140012+0000","last_peered":"2024-06-10T00:26:42.140012+0000","last_clean":"2024-06-10T00:26:42.140012+0000","last_became_active":"2024-06-10T00:23:57.561965+0000","last_became_peered":"2024-06-10T00:23:57.561965+0000","last_unstale":"2024-06-10T00:26:42.140012+0000","last_undegraded":"2024-06-10T00:26:42.140012+0000","last_fullsized":"2024-06-10T00:26:42.140012+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:40:16.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"100'8","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.135468+0000","last_change":"2024-06-10T00:23:57.564422+0000","last_active":"2024-06-10T00:26:42.135468+0000","last_peered":"2024-06-10T00:26:42.135468+0000","last_clean":"2024-06-10T00:26:42.135468+0000","last_became_active":"2024-06-10T00:23:57.564069+0000","last_became_peered":"2024-06-10T00:23:57.564069+0000","last_unstale":"2024-06-10T00:26:42.135468+0000","last_undegraded":"2024-06-10T00:26:42.135468+0000","last_fullsized":"2024-06-10T00:26:42.135468+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:07.662465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"100'10","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.135008+0000","last_change":"2024-06-10T00:23:57.561284+0000","last_active":"2024-06-10T00:26:42.135008+0000","last_peered":"2024-06-10T00:26:42.135008+0000","last_clean":"2024-06-10T00:26:42.135008+0000","last_became_active":"2024-06-10T00:23:57.560933+0000","last_became_peered":"2024-06-10T00:23:57.560933+0000","last_unstale":"2024-06-10T00:26:42.135008+0000","last_undegraded":"2024-06-10T00:26:42.135008+0000","last_fullsized":"2024-06-10T00:26:42.135008+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:58:48.511774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.128126+0000","last_change":"2024-06-10T00:23:57.557752+0000","last_active":"2024-06-10T00:26:42.128126+0000","last_peered":"2024-06-10T00:26:42.128126+0000","last_clean":"2024-06-10T00:26:42.128126+0000","last_became_active":"2024-06-10T00:23:57.557419+0000","last_became_peered":"2024-06-10T00:23:57.557419+0000","last_unstale":"2024-06-10T00:26:42.128126+0000","last_undegraded":"2024-06-10T00:26:42.128126+0000","last_fullsized":"2024-06-10T00:26:42.128126+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:48.234181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081440+0000","last_change":"2024-06-10T00:24:09.177183+0000","last_active":"2024-06-10T00:25:44.081440+0000","last_peered":"2024-06-10T00:25:44.081440+0000","last_clean":"2024-06-10T00:25:44.081440+0000","last_became_active":"2024-06-10T00:24:09.176680+0000","last_became_peered":"2024-06-10T00:24:09.176680+0000","last_unstale":"2024-06-10T00:25:44.081440+0000","last_undegraded":"2024-06-10T00:25:44.081440+0000","last_fullsized":"2024-06-10T00:25:44.081440+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:17:06.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"98'9","reported_seq":55,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082765+0000","last_change":"2024-06-10T00:23:54.426364+0000","last_active":"2024-06-10T00:25:44.082765+0000","last_peered":"2024-06-10T00:25:44.082765+0000","last_clean":"2024-06-10T00:25:44.082765+0000","last_became_active":"2024-06-10T00:23:54.426054+0000","last_became_peered":"2024-06-10T00:23:54.426054+0000","last_unstale":"2024-06-10T00:25:44.082765+0000","last_undegraded":"2024-06-10T00:25:44.082765+0000","last_fullsized":"2024-06-10T00:25:44.082765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:47.613402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"100'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.127015+0000","last_change":"2024-06-10T00:23:57.557206+0000","last_active":"2024-06-10T00:26:42.127015+0000","last_peered":"2024-06-10T00:26:42.127015+0000","last_clean":"2024-06-10T00:26:42.127015+0000","last_became_active":"2024-06-10T00:23:57.556853+0000","last_became_peered":"2024-06-10T00:23:57.556853+0000","last_unstale":"2024-06-10T00:26:42.127015+0000","last_undegraded":"2024-06-10T00:26:42.127015+0000","last_fullsized":"2024-06-10T00:26:42.127015+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:09.686334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345231+0000","last_change":"2024-06-10T00:24:09.171047+0000","last_active":"2024-06-10T00:25:44.345231+0000","last_peered":"2024-06-10T00:25:44.345231+0000","last_clean":"2024-06-10T00:25:44.345231+0000","last_became_active":"2024-06-10T00:24:09.170501+0000","last_became_peered":"2024-06-10T00:24:09.170501+0000","last_unstale":"2024-06-10T00:25:44.345231+0000","last_undegraded":"2024-06-10T00:25:44.345231+0000","last_fullsized":"2024-06-10T00:25:44.345231+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:47:27.690439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"98'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733915+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:25:44.733915+0000","last_peered":"2024-06-10T00:25:44.733915+0000","last_clean":"2024-06-10T00:25:44.733915+0000","last_became_active":"2024-06-10T00:23:54.822447+0000","last_became_peered":"2024-06-10T00:23:54.822447+0000","last_unstale":"2024-06-10T00:25:44.733915+0000","last_undegraded":"2024-06-10T00:25:44.733915+0000","last_fullsized":"2024-06-10T00:25:44.733915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:45:14.956540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.133398+0000","last_change":"2024-06-10T00:23:57.554958+0000","last_active":"2024-06-10T00:26:42.133398+0000","last_peered":"2024-06-10T00:26:42.133398+0000","last_clean":"2024-06-10T00:26:42.133398+0000","last_became_active":"2024-06-10T00:23:57.554477+0000","last_became_peered":"2024-06-10T00:23:57.554477+0000","last_unstale":"2024-06-10T00:26:42.133398+0000","last_undegraded":"2024-06-10T00:26:42.133398+0000","last_fullsized":"2024-06-10T00:26:42.133398+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:40:06.066967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344744+0000","last_change":"2024-06-10T00:24:09.169642+0000","last_active":"2024-06-10T00:25:44.344744+0000","last_peered":"2024-06-10T00:25:44.344744+0000","last_clean":"2024-06-10T00:25:44.344744+0000","last_became_active":"2024-06-10T00:24:09.168071+0000","last_became_peered":"2024-06-10T00:24:09.168071+0000","last_unstale":"2024-06-10T00:25:44.344744+0000","last_undegraded":"2024-06-10T00:25:44.344744+0000","last_fullsized":"2024-06-10T00:25:44.344744+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:44.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"98'4","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733966+0000","last_change":"2024-06-10T00:23:54.430543+0000","last_active":"2024-06-10T00:25:44.733966+0000","last_peered":"2024-06-10T00:25:44.733966+0000","last_clean":"2024-06-10T00:25:44.733966+0000","last_became_active":"2024-06-10T00:23:54.430138+0000","last_became_peered":"2024-06-10T00:23:54.430138+0000","last_unstale":"2024-06-10T00:25:44.733966+0000","last_undegraded":"2024-06-10T00:25:44.733966+0000","last_fullsized":"2024-06-10T00:25:44.733966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:39:53.212607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"100'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.138587+0000","last_change":"2024-06-10T00:23:57.562251+0000","last_active":"2024-06-10T00:26:42.138587+0000","last_peered":"2024-06-10T00:26:42.138587+0000","last_clean":"2024-06-10T00:26:42.138587+0000","last_became_active":"2024-06-10T00:23:57.561903+0000","last_became_peered":"2024-06-10T00:23:57.561903+0000","last_unstale":"2024-06-10T00:26:42.138587+0000","last_undegraded":"2024-06-10T00:26:42.138587+0000","last_fullsized":"2024-06-10T00:26:42.138587+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:59.489475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344593+0000","last_change":"2024-06-10T00:24:09.165894+0000","last_active":"2024-06-10T00:25:44.344593+0000","last_peered":"2024-06-10T00:25:44.344593+0000","last_clean":"2024-06-10T00:25:44.344593+0000","last_became_active":"2024-06-10T00:24:09.165435+0000","last_became_peered":"2024-06-10T00:24:09.165435+0000","last_unstale":"2024-06-10T00:25:44.344593+0000","last_undegraded":"2024-06-10T00:25:44.344593+0000","last_fullsized":"2024-06-10T00:25:44.344593+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:13:36.058375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.734024+0000","last_change":"2024-06-10T00:23:54.427868+0000","last_active":"2024-06-10T00:25:44.734024+0000","last_peered":"2024-06-10T00:25:44.734024+0000","last_clean":"2024-06-10T00:25:44.734024+0000","last_became_active":"2024-06-10T00:23:54.427366+0000","last_became_peered":"2024-06-10T00:23:54.427366+0000","last_unstale":"2024-06-10T00:25:44.734024+0000","last_undegraded":"2024-06-10T00:25:44.734024+0000","last_fullsized":"2024-06-10T00:25:44.734024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:15.499292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"100'15","reported_seq":60,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.134116+0000","last_change":"2024-06-10T00:24:43.768556+0000","last_active":"2024-06-10T00:26:42.134116+0000","last_peered":"2024-06-10T00:26:42.134116+0000","last_clean":"2024-06-10T00:26:42.134116+0000","last_became_active":"2024-06-10T00:24:43.768090+0000","last_became_peered":"2024-06-10T00:24:43.768090+0000","last_unstale":"2024-06-10T00:26:42.134116+0000","last_undegraded":"2024-06-10T00:26:42.134116+0000","last_fullsized":"2024-06-10T00:26:42.134116+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:05:41.604184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081837+0000","last_change":"2024-06-10T00:24:09.170514+0000","last_active":"2024-06-10T00:25:44.081837+0000","last_peered":"2024-06-10T00:25:44.081837+0000","last_clean":"2024-06-10T00:25:44.081837+0000","last_became_active":"2024-06-10T00:24:09.168902+0000","last_became_peered":"2024-06-10T00:24:09.168902+0000","last_unstale":"2024-06-10T00:25:44.081837+0000","last_undegraded":"2024-06-10T00:25:44.081837+0000","last_fullsized":"2024-06-10T00:25:44.081837+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:33:10.534657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"100'7","reported_seq":157,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:44.436627+0000","last_change":"2024-06-10T00:23:54.426581+0000","last_active":"2024-06-10T00:26:44.436627+0000","last_peered":"2024-06-10T00:26:44.436627+0000","last_clean":"2024-06-10T00:26:44.436627+0000","last_became_active":"2024-06-10T00:23:54.426211+0000","last_became_peered":"2024-06-10T00:23:54.426211+0000","last_unstale":"2024-06-10T00:26:44.436627+0000","last_undegraded":"2024-06-10T00:26:44.436627+0000","last_fullsized":"2024-06-10T00:26:44.436627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:49:05.360742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1153,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.120352+0000","last_change":"2024-06-10T00:23:57.552308+0000","last_active":"2024-06-10T00:26:42.120352+0000","last_peered":"2024-06-10T00:26:42.120352+0000","last_clean":"2024-06-10T00:26:42.120352+0000","last_became_active":"2024-06-10T00:23:57.551762+0000","last_became_peered":"2024-06-10T00:23:57.551762+0000","last_unstale":"2024-06-10T00:26:42.120352+0000","last_undegraded":"2024-06-10T00:26:42.120352+0000","last_fullsized":"2024-06-10T00:26:42.120352+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:22:28.476019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081833+0000","last_change":"2024-06-10T00:24:09.171456+0000","last_active":"2024-06-10T00:25:44.081833+0000","last_peered":"2024-06-10T00:25:44.081833+0000","last_clean":"2024-06-10T00:25:44.081833+0000","last_became_active":"2024-06-10T00:24:09.170969+0000","last_became_peered":"2024-06-10T00:24:09.170969+0000","last_unstale":"2024-06-10T00:25:44.081833+0000","last_undegraded":"2024-06-10T00:25:44.081833+0000","last_fullsized":"2024-06-10T00:25:44.081833+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:22:46.023897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"98'3","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078975+0000","last_change":"2024-06-10T00:23:54.428384+0000","last_active":"2024-06-10T00:25:44.078975+0000","last_peered":"2024-06-10T00:25:44.078975+0000","last_clean":"2024-06-10T00:25:44.078975+0000","last_became_active":"2024-06-10T00:23:54.427967+0000","last_became_peered":"2024-06-10T00:23:54.427967+0000","last_unstale":"2024-06-10T00:25:44.078975+0000","last_undegraded":"2024-06-10T00:25:44.078975+0000","last_fullsized":"2024-06-10T00:25:44.078975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:54:30.408684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"100'8","reported_seq":53,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.118226+0000","last_change":"2024-06-10T00:24:43.262593+0000","last_active":"2024-06-10T00:26:42.118226+0000","last_peered":"2024-06-10T00:26:42.118226+0000","last_clean":"2024-06-10T00:26:42.118226+0000","last_became_active":"2024-06-10T00:24:43.262212+0000","last_became_peered":"2024-06-10T00:24:43.262212+0000","last_unstale":"2024-06-10T00:26:42.118226+0000","last_undegraded":"2024-06-10T00:26:42.118226+0000","last_fullsized":"2024-06-10T00:26:42.118226+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:40:24.442712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":34,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189955+0000","last_change":"2024-06-10T00:24:43.752805+0000","last_active":"2024-06-10T00:25:44.189955+0000","last_peered":"2024-06-10T00:25:44.189955+0000","last_clean":"2024-06-10T00:25:44.189955+0000","last_became_active":"2024-06-10T00:24:43.752251+0000","last_became_peered":"2024-06-10T00:24:43.752251+0000","last_unstale":"2024-06-10T00:25:44.189955+0000","last_undegraded":"2024-06-10T00:25:44.189955+0000","last_fullsized":"2024-06-10T00:25:44.189955+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:27:57.617443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"98'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078007+0000","last_change":"2024-06-10T00:23:54.421493+0000","last_active":"2024-06-10T00:25:44.078007+0000","last_peered":"2024-06-10T00:25:44.078007+0000","last_clean":"2024-06-10T00:25:44.078007+0000","last_became_active":"2024-06-10T00:23:54.421151+0000","last_became_peered":"2024-06-10T00:23:54.421151+0000","last_unstale":"2024-06-10T00:25:44.078007+0000","last_undegraded":"2024-06-10T00:25:44.078007+0000","last_fullsized":"2024-06-10T00:25:44.078007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:03.995244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"100'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.130337+0000","last_change":"2024-06-10T00:23:57.554020+0000","last_active":"2024-06-10T00:26:42.130337+0000","last_peered":"2024-06-10T00:26:42.130337+0000","last_clean":"2024-06-10T00:26:42.130337+0000","last_became_active":"2024-06-10T00:23:57.553690+0000","last_became_peered":"2024-06-10T00:23:57.553690+0000","last_unstale":"2024-06-10T00:26:42.130337+0000","last_undegraded":"2024-06-10T00:26:42.130337+0000","last_fullsized":"2024-06-10T00:26:42.130337+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:48.097074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081507+0000","last_change":"2024-06-10T00:24:09.166426+0000","last_active":"2024-06-10T00:25:44.081507+0000","last_peered":"2024-06-10T00:25:44.081507+0000","last_clean":"2024-06-10T00:25:44.081507+0000","last_became_active":"2024-06-10T00:24:09.166047+0000","last_became_peered":"2024-06-10T00:24:09.166047+0000","last_unstale":"2024-06-10T00:25:44.081507+0000","last_undegraded":"2024-06-10T00:25:44.081507+0000","last_fullsized":"2024-06-10T00:25:44.081507+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:56:11.608061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"98'1","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345414+0000","last_change":"2024-06-10T00:23:54.426409+0000","last_active":"2024-06-10T00:25:44.345414+0000","last_peered":"2024-06-10T00:25:44.345414+0000","last_clean":"2024-06-10T00:25:44.345414+0000","last_became_active":"2024-06-10T00:23:54.425385+0000","last_became_peered":"2024-06-10T00:23:54.425385+0000","last_unstale":"2024-06-10T00:25:44.345414+0000","last_undegraded":"2024-06-10T00:25:44.345414+0000","last_fullsized":"2024-06-10T00:25:44.345414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:27:00.379189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"100'11","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.119925+0000","last_change":"2024-06-10T00:23:57.560132+0000","last_active":"2024-06-10T00:26:42.119925+0000","last_peered":"2024-06-10T00:26:42.119925+0000","last_clean":"2024-06-10T00:26:42.119925+0000","last_became_active":"2024-06-10T00:23:57.559803+0000","last_became_peered":"2024-06-10T00:23:57.559803+0000","last_unstale":"2024-06-10T00:26:42.119925+0000","last_undegraded":"2024-06-10T00:26:42.119925+0000","last_fullsized":"2024-06-10T00:26:42.119925+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:13:16.004233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733402+0000","last_change":"2024-06-10T00:24:09.170962+0000","last_active":"2024-06-10T00:25:44.733402+0000","last_peered":"2024-06-10T00:25:44.733402+0000","last_clean":"2024-06-10T00:25:44.733402+0000","last_became_active":"2024-06-10T00:24:09.170090+0000","last_became_peered":"2024-06-10T00:24:09.170090+0000","last_unstale":"2024-06-10T00:25:44.733402+0000","last_undegraded":"2024-06-10T00:25:44.733402+0000","last_fullsized":"2024-06-10T00:25:44.733402+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:07:33.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'4","reported_seq":148,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.432251+0000","last_change":"2024-06-10T00:23:54.421472+0000","last_active":"2024-06-10T00:26:42.432251+0000","last_peered":"2024-06-10T00:26:42.432251+0000","last_clean":"2024-06-10T00:26:42.432251+0000","last_became_active":"2024-06-10T00:23:54.421019+0000","last_became_peered":"2024-06-10T00:23:54.421019+0000","last_unstale":"2024-06-10T00:26:42.432251+0000","last_undegraded":"2024-06-10T00:26:42.432251+0000","last_fullsized":"2024-06-10T00:26:42.432251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:28:18.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"100'10","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.133651+0000","last_change":"2024-06-10T00:23:57.563010+0000","last_active":"2024-06-10T00:26:42.133651+0000","last_peered":"2024-06-10T00:26:42.133651+0000","last_clean":"2024-06-10T00:26:42.133651+0000","last_became_active":"2024-06-10T00:23:57.561986+0000","last_became_peered":"2024-06-10T00:23:57.561986+0000","last_unstale":"2024-06-10T00:26:42.133651+0000","last_undegraded":"2024-06-10T00:26:42.133651+0000","last_fullsized":"2024-06-10T00:26:42.133651+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:31:17.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082277+0000","last_change":"2024-06-10T00:24:09.171746+0000","last_active":"2024-06-10T00:25:44.082277+0000","last_peered":"2024-06-10T00:25:44.082277+0000","last_clean":"2024-06-10T00:25:44.082277+0000","last_became_active":"2024-06-10T00:24:09.171030+0000","last_became_peered":"2024-06-10T00:24:09.171030+0000","last_unstale":"2024-06-10T00:25:44.082277+0000","last_undegraded":"2024-06-10T00:25:44.082277+0000","last_fullsized":"2024-06-10T00:25:44.082277+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:41.669471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"98'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.734164+0000","last_change":"2024-06-10T00:23:54.428917+0000","last_active":"2024-06-10T00:25:44.734164+0000","last_peered":"2024-06-10T00:25:44.734164+0000","last_clean":"2024-06-10T00:25:44.734164+0000","last_became_active":"2024-06-10T00:23:54.428585+0000","last_became_peered":"2024-06-10T00:23:54.428585+0000","last_unstale":"2024-06-10T00:25:44.734164+0000","last_undegraded":"2024-06-10T00:25:44.734164+0000","last_fullsized":"2024-06-10T00:25:44.734164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:21:58.235848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"100'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.127382+0000","last_change":"2024-06-10T00:23:57.551778+0000","last_active":"2024-06-10T00:26:42.127382+0000","last_peered":"2024-06-10T00:26:42.127382+0000","last_clean":"2024-06-10T00:26:42.127382+0000","last_became_active":"2024-06-10T00:23:57.551436+0000","last_became_peered":"2024-06-10T00:23:57.551436+0000","last_unstale":"2024-06-10T00:26:42.127382+0000","last_undegraded":"2024-06-10T00:26:42.127382+0000","last_fullsized":"2024-06-10T00:26:42.127382+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:30.059783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344998+0000","last_change":"2024-06-10T00:24:09.167547+0000","last_active":"2024-06-10T00:25:44.344998+0000","last_peered":"2024-06-10T00:25:44.344998+0000","last_clean":"2024-06-10T00:25:44.344998+0000","last_became_active":"2024-06-10T00:24:09.167132+0000","last_became_peered":"2024-06-10T00:24:09.167132+0000","last_unstale":"2024-06-10T00:25:44.344998+0000","last_undegraded":"2024-06-10T00:25:44.344998+0000","last_fullsized":"2024-06-10T00:25:44.344998+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:11.520330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"98'4","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345502+0000","last_change":"2024-06-10T00:23:54.823754+0000","last_active":"2024-06-10T00:25:44.345502+0000","last_peered":"2024-06-10T00:25:44.345502+0000","last_clean":"2024-06-10T00:25:44.345502+0000","last_became_active":"2024-06-10T00:23:54.823362+0000","last_became_peered":"2024-06-10T00:23:54.823362+0000","last_unstale":"2024-06-10T00:25:44.345502+0000","last_undegraded":"2024-06-10T00:25:44.345502+0000","last_fullsized":"2024-06-10T00:25:44.345502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:48:50.915439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"100'11","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.140613+0000","last_change":"2024-06-10T00:23:57.563313+0000","last_active":"2024-06-10T00:26:42.140613+0000","last_peered":"2024-06-10T00:26:42.140613+0000","last_clean":"2024-06-10T00:26:42.140613+0000","last_became_active":"2024-06-10T00:23:57.562307+0000","last_became_peered":"2024-06-10T00:23:57.562307+0000","last_unstale":"2024-06-10T00:26:42.140613+0000","last_undegraded":"2024-06-10T00:26:42.140613+0000","last_fullsized":"2024-06-10T00:26:42.140613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:23.815267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081609+0000","last_change":"2024-06-10T00:24:09.174660+0000","last_active":"2024-06-10T00:25:44.081609+0000","last_peered":"2024-06-10T00:25:44.081609+0000","last_clean":"2024-06-10T00:25:44.081609+0000","last_became_active":"2024-06-10T00:24:09.174283+0000","last_became_peered":"2024-06-10T00:24:09.174283+0000","last_unstale":"2024-06-10T00:25:44.081609+0000","last_undegraded":"2024-06-10T00:25:44.081609+0000","last_fullsized":"2024-06-10T00:25:44.081609+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:15.236146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"100'36","reported_seq":289,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:43.434262+0000","last_change":"2024-06-10T00:23:54.426621+0000","last_active":"2024-06-10T00:26:43.434262+0000","last_peered":"2024-06-10T00:26:43.434262+0000","last_clean":"2024-06-10T00:26:43.434262+0000","last_became_active":"2024-06-10T00:23:54.426131+0000","last_became_peered":"2024-06-10T00:23:54.426131+0000","last_unstale":"2024-06-10T00:26:43.434262+0000","last_undegraded":"2024-06-10T00:26:43.434262+0000","last_fullsized":"2024-06-10T00:26:43.434262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:56:14.819454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62904,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":208,"num_read_kb":166,"num_write":37,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.132208+0000","last_change":"2024-06-10T00:23:57.552606+0000","last_active":"2024-06-10T00:26:42.132208+0000","last_peered":"2024-06-10T00:26:42.132208+0000","last_clean":"2024-06-10T00:26:42.132208+0000","last_became_active":"2024-06-10T00:23:57.552246+0000","last_became_peered":"2024-06-10T00:23:57.552246+0000","last_unstale":"2024-06-10T00:26:42.132208+0000","last_undegraded":"2024-06-10T00:26:42.132208+0000","last_fullsized":"2024-06-10T00:26:42.132208+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:19:21.508721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.346039+0000","last_change":"2024-06-10T00:24:09.174862+0000","last_active":"2024-06-10T00:25:44.346039+0000","last_peered":"2024-06-10T00:25:44.346039+0000","last_clean":"2024-06-10T00:25:44.346039+0000","last_became_active":"2024-06-10T00:24:09.173550+0000","last_became_peered":"2024-06-10T00:24:09.173550+0000","last_unstale":"2024-06-10T00:25:44.346039+0000","last_undegraded":"2024-06-10T00:25:44.346039+0000","last_fullsized":"2024-06-10T00:25:44.346039+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:49:25.549084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"98'6","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081771+0000","last_change":"2024-06-10T00:23:54.426074+0000","last_active":"2024-06-10T00:25:44.081771+0000","last_peered":"2024-06-10T00:25:44.081771+0000","last_clean":"2024-06-10T00:25:44.081771+0000","last_became_active":"2024-06-10T00:23:54.425707+0000","last_became_peered":"2024-06-10T00:23:54.425707+0000","last_unstale":"2024-06-10T00:25:44.081771+0000","last_undegraded":"2024-06-10T00:25:44.081771+0000","last_fullsized":"2024-06-10T00:25:44.081771+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:11:10.563595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"100'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.128427+0000","last_change":"2024-06-10T00:23:57.549784+0000","last_active":"2024-06-10T00:26:42.128427+0000","last_peered":"2024-06-10T00:26:42.128427+0000","last_clean":"2024-06-10T00:26:42.128427+0000","last_became_active":"2024-06-10T00:23:57.549457+0000","last_became_peered":"2024-06-10T00:23:57.549457+0000","last_unstale":"2024-06-10T00:26:42.128427+0000","last_undegraded":"2024-06-10T00:26:42.128427+0000","last_fullsized":"2024-06-10T00:26:42.128427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:59:27.352927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189697+0000","last_change":"2024-06-10T00:24:09.171028+0000","last_active":"2024-06-10T00:25:44.189697+0000","last_peered":"2024-06-10T00:25:44.189697+0000","last_clean":"2024-06-10T00:25:44.189697+0000","last_became_active":"2024-06-10T00:24:09.170543+0000","last_became_peered":"2024-06-10T00:24:09.170543+0000","last_unstale":"2024-06-10T00:25:44.189697+0000","last_undegraded":"2024-06-10T00:25:44.189697+0000","last_fullsized":"2024-06-10T00:25:44.189697+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:57:40.816585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"98'3","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345649+0000","last_change":"2024-06-10T00:23:54.423332+0000","last_active":"2024-06-10T00:25:44.345649+0000","last_peered":"2024-06-10T00:25:44.345649+0000","last_clean":"2024-06-10T00:25:44.345649+0000","last_became_active":"2024-06-10T00:23:54.422792+0000","last_became_peered":"2024-06-10T00:23:54.422792+0000","last_unstale":"2024-06-10T00:25:44.345649+0000","last_undegraded":"2024-06-10T00:25:44.345649+0000","last_fullsized":"2024-06-10T00:25:44.345649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:50:52.655641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"100'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.139586+0000","last_change":"2024-06-10T00:23:57.551411+0000","last_active":"2024-06-10T00:26:42.139586+0000","last_peered":"2024-06-10T00:26:42.139586+0000","last_clean":"2024-06-10T00:26:42.139586+0000","last_became_active":"2024-06-10T00:23:57.551053+0000","last_became_peered":"2024-06-10T00:23:57.551053+0000","last_unstale":"2024-06-10T00:26:42.139586+0000","last_undegraded":"2024-06-10T00:26:42.139586+0000","last_fullsized":"2024-06-10T00:26:42.139586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:16:18.082777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.079218+0000","last_change":"2024-06-10T00:24:43.648767+0000","last_active":"2024-06-10T00:25:44.079218+0000","last_peered":"2024-06-10T00:25:44.079218+0000","last_clean":"2024-06-10T00:25:44.079218+0000","last_became_active":"2024-06-10T00:24:43.648267+0000","last_became_peered":"2024-06-10T00:24:43.648267+0000","last_unstale":"2024-06-10T00:25:44.079218+0000","last_undegraded":"2024-06-10T00:25:44.079218+0000","last_fullsized":"2024-06-10T00:25:44.079218+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:41:09.005202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"98'3","reported_seq":151,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:45.436024+0000","last_change":"2024-06-10T00:23:54.425963+0000","last_active":"2024-06-10T00:26:45.436024+0000","last_peered":"2024-06-10T00:26:45.436024+0000","last_clean":"2024-06-10T00:26:45.436024+0000","last_became_active":"2024-06-10T00:23:54.425581+0000","last_became_peered":"2024-06-10T00:23:54.425581+0000","last_unstale":"2024-06-10T00:26:45.436024+0000","last_undegraded":"2024-06-10T00:26:45.436024+0000","last_fullsized":"2024-06-10T00:26:45.436024+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:54.760610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.129666+0000","last_change":"2024-06-10T00:23:57.557464+0000","last_active":"2024-06-10T00:26:42.129666+0000","last_peered":"2024-06-10T00:26:42.129666+0000","last_clean":"2024-06-10T00:26:42.129666+0000","last_became_active":"2024-06-10T00:23:57.557130+0000","last_became_peered":"2024-06-10T00:23:57.557130+0000","last_unstale":"2024-06-10T00:26:42.129666+0000","last_undegraded":"2024-06-10T00:26:42.129666+0000","last_fullsized":"2024-06-10T00:26:42.129666+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:16:36.341210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":36,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078404+0000","last_change":"2024-06-10T00:24:43.753291+0000","last_active":"2024-06-10T00:25:44.078404+0000","last_peered":"2024-06-10T00:25:44.078404+0000","last_clean":"2024-06-10T00:25:44.078404+0000","last_became_active":"2024-06-10T00:24:43.752765+0000","last_became_peered":"2024-06-10T00:24:43.752765+0000","last_unstale":"2024-06-10T00:25:44.078404+0000","last_undegraded":"2024-06-10T00:25:44.078404+0000","last_fullsized":"2024-06-10T00:25:44.078404+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:55:06.349564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"98'15","reported_seq":61,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733740+0000","last_change":"2024-06-10T00:23:54.823489+0000","last_active":"2024-06-10T00:25:44.733740+0000","last_peered":"2024-06-10T00:25:44.733740+0000","last_clean":"2024-06-10T00:25:44.733740+0000","last_became_active":"2024-06-10T00:23:54.823073+0000","last_became_peered":"2024-06-10T00:23:54.823073+0000","last_unstale":"2024-06-10T00:25:44.733740+0000","last_undegraded":"2024-06-10T00:25:44.733740+0000","last_fullsized":"2024-06-10T00:25:44.733740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:07:52.563009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.124711+0000","last_change":"2024-06-10T00:23:57.563388+0000","last_active":"2024-06-10T00:26:42.124711+0000","last_peered":"2024-06-10T00:26:42.124711+0000","last_clean":"2024-06-10T00:26:42.124711+0000","last_became_active":"2024-06-10T00:23:57.563004+0000","last_became_peered":"2024-06-10T00:23:57.563004+0000","last_unstale":"2024-06-10T00:26:42.124711+0000","last_undegraded":"2024-06-10T00:26:42.124711+0000","last_fullsized":"2024-06-10T00:26:42.124711+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:51:18.805662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345323+0000","last_change":"2024-06-10T00:24:09.168447+0000","last_active":"2024-06-10T00:25:44.345323+0000","last_peered":"2024-06-10T00:25:44.345323+0000","last_clean":"2024-06-10T00:25:44.345323+0000","last_became_active":"2024-06-10T00:24:09.167439+0000","last_became_peered":"2024-06-10T00:24:09.167439+0000","last_unstale":"2024-06-10T00:25:44.345323+0000","last_undegraded":"2024-06-10T00:25:44.345323+0000","last_fullsized":"2024-06-10T00:25:44.345323+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:52.319395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"98'2","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.734275+0000","last_change":"2024-06-10T00:23:54.427666+0000","last_active":"2024-06-10T00:25:44.734275+0000","last_peered":"2024-06-10T00:25:44.734275+0000","last_clean":"2024-06-10T00:25:44.734275+0000","last_became_active":"2024-06-10T00:23:54.427285+0000","last_became_peered":"2024-06-10T00:23:54.427285+0000","last_unstale":"2024-06-10T00:25:44.734275+0000","last_undegraded":"2024-06-10T00:25:44.734275+0000","last_fullsized":"2024-06-10T00:25:44.734275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:28:20.196168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"100'5","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.126854+0000","last_change":"2024-06-10T00:23:57.558787+0000","last_active":"2024-06-10T00:26:42.126854+0000","last_peered":"2024-06-10T00:26:42.126854+0000","last_clean":"2024-06-10T00:26:42.126854+0000","last_became_active":"2024-06-10T00:23:57.558470+0000","last_became_peered":"2024-06-10T00:23:57.558470+0000","last_unstale":"2024-06-10T00:26:42.126854+0000","last_undegraded":"2024-06-10T00:26:42.126854+0000","last_fullsized":"2024-06-10T00:26:42.126854+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:02:14.758802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733802+0000","last_change":"2024-06-10T00:24:09.264952+0000","last_active":"2024-06-10T00:25:44.733802+0000","last_peered":"2024-06-10T00:25:44.733802+0000","last_clean":"2024-06-10T00:25:44.733802+0000","last_became_active":"2024-06-10T00:24:09.264379+0000","last_became_peered":"2024-06-10T00:24:09.264379+0000","last_unstale":"2024-06-10T00:25:44.733802+0000","last_undegraded":"2024-06-10T00:25:44.733802+0000","last_fullsized":"2024-06-10T00:25:44.733802+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:25:40.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082876+0000","last_change":"2024-06-10T00:23:54.823298+0000","last_active":"2024-06-10T00:25:44.082876+0000","last_peered":"2024-06-10T00:25:44.082876+0000","last_clean":"2024-06-10T00:25:44.082876+0000","last_became_active":"2024-06-10T00:23:54.822917+0000","last_became_peered":"2024-06-10T00:23:54.822917+0000","last_unstale":"2024-06-10T00:25:44.082876+0000","last_undegraded":"2024-06-10T00:25:44.082876+0000","last_fullsized":"2024-06-10T00:25:44.082876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:07:52.500641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"100'4","reported_seq":43,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.126638+0000","last_change":"2024-06-10T00:23:57.559202+0000","last_active":"2024-06-10T00:26:42.126638+0000","last_peered":"2024-06-10T00:26:42.126638+0000","last_clean":"2024-06-10T00:26:42.126638+0000","last_became_active":"2024-06-10T00:23:57.558800+0000","last_became_peered":"2024-06-10T00:23:57.558800+0000","last_unstale":"2024-06-10T00:26:42.126638+0000","last_undegraded":"2024-06-10T00:26:42.126638+0000","last_fullsized":"2024-06-10T00:26:42.126638+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:49:51.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078291+0000","last_change":"2024-06-10T00:24:09.168603+0000","last_active":"2024-06-10T00:25:44.078291+0000","last_peered":"2024-06-10T00:25:44.078291+0000","last_clean":"2024-06-10T00:25:44.078291+0000","last_became_active":"2024-06-10T00:24:09.168233+0000","last_became_peered":"2024-06-10T00:24:09.168233+0000","last_unstale":"2024-06-10T00:25:44.078291+0000","last_undegraded":"2024-06-10T00:25:44.078291+0000","last_fullsized":"2024-06-10T00:25:44.078291+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:59.585023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'1","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733804+0000","last_change":"2024-06-10T00:23:54.425505+0000","last_active":"2024-06-10T00:25:44.733804+0000","last_peered":"2024-06-10T00:25:44.733804+0000","last_clean":"2024-06-10T00:25:44.733804+0000","last_became_active":"2024-06-10T00:23:54.425058+0000","last_became_peered":"2024-06-10T00:23:54.425058+0000","last_unstale":"2024-06-10T00:25:44.733804+0000","last_undegraded":"2024-06-10T00:25:44.733804+0000","last_fullsized":"2024-06-10T00:25:44.733804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:48:05.605947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.125899+0000","last_change":"2024-06-10T00:23:57.566060+0000","last_active":"2024-06-10T00:26:42.125899+0000","last_peered":"2024-06-10T00:26:42.125899+0000","last_clean":"2024-06-10T00:26:42.125899+0000","last_became_active":"2024-06-10T00:23:57.565099+0000","last_became_peered":"2024-06-10T00:23:57.565099+0000","last_unstale":"2024-06-10T00:26:42.125899+0000","last_undegraded":"2024-06-10T00:26:42.125899+0000","last_fullsized":"2024-06-10T00:26:42.125899+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:24:54.204399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078462+0000","last_change":"2024-06-10T00:24:09.264741+0000","last_active":"2024-06-10T00:25:44.078462+0000","last_peered":"2024-06-10T00:25:44.078462+0000","last_clean":"2024-06-10T00:25:44.078462+0000","last_became_active":"2024-06-10T00:24:09.264267+0000","last_became_peered":"2024-06-10T00:24:09.264267+0000","last_unstale":"2024-06-10T00:25:44.078462+0000","last_undegraded":"2024-06-10T00:25:44.078462+0000","last_fullsized":"2024-06-10T00:25:44.078462+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:27.627391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"100'8","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.110416+0000","last_change":"2024-06-10T00:25:44.346070+0000","last_active":"2024-06-10T00:26:42.110416+0000","last_peered":"2024-06-10T00:26:42.110416+0000","last_clean":"2024-06-10T00:26:42.110416+0000","last_became_active":"2024-06-10T00:25:44.345798+0000","last_became_peered":"2024-06-10T00:25:44.345798+0000","last_unstale":"2024-06-10T00:26:42.110416+0000","last_undegraded":"2024-06-10T00:26:42.110416+0000","last_fullsized":"2024-06-10T00:26:42.110416+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:04.506535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"98'4","reported_seq":152,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:46.436849+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:26:46.436849+0000","last_peered":"2024-06-10T00:26:46.436849+0000","last_clean":"2024-06-10T00:26:46.436849+0000","last_became_active":"2024-06-10T00:23:54.822442+0000","last_became_peered":"2024-06-10T00:23:54.822442+0000","last_unstale":"2024-06-10T00:26:46.436849+0000","last_undegraded":"2024-06-10T00:26:46.436849+0000","last_fullsized":"2024-06-10T00:26:46.436849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:51:10.278257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":103,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"98'1","reported_seq":30,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.734162+0000","last_change":"2024-06-10T00:24:09.168073+0000","last_active":"2024-06-10T00:25:44.734162+0000","last_peered":"2024-06-10T00:25:44.734162+0000","last_clean":"2024-06-10T00:25:44.734162+0000","last_became_active":"2024-06-10T00:24:09.167657+0000","last_became_peered":"2024-06-10T00:24:09.167657+0000","last_unstale":"2024-06-10T00:25:44.734162+0000","last_undegraded":"2024-06-10T00:25:44.734162+0000","last_fullsized":"2024-06-10T00:25:44.734162+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:20:23.481685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"100'8","reported_seq":19,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.128202+0000","last_change":"2024-06-10T00:25:44.346629+0000","last_active":"2024-06-10T00:26:42.128202+0000","last_peered":"2024-06-10T00:26:42.128202+0000","last_clean":"2024-06-10T00:26:42.128202+0000","last_became_active":"2024-06-10T00:25:44.346388+0000","last_became_peered":"2024-06-10T00:25:44.346388+0000","last_unstale":"2024-06-10T00:26:42.128202+0000","last_undegraded":"2024-06-10T00:26:42.128202+0000","last_fullsized":"2024-06-10T00:26:42.128202+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:03:19.228632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"98'3","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344835+0000","last_change":"2024-06-10T00:23:54.423507+0000","last_active":"2024-06-10T00:25:44.344835+0000","last_peered":"2024-06-10T00:25:44.344835+0000","last_clean":"2024-06-10T00:25:44.344835+0000","last_became_active":"2024-06-10T00:23:54.423092+0000","last_became_peered":"2024-06-10T00:23:54.423092+0000","last_unstale":"2024-06-10T00:25:44.344835+0000","last_undegraded":"2024-06-10T00:25:44.344835+0000","last_fullsized":"2024-06-10T00:25:44.344835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:06:15.924827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":377,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733568+0000","last_change":"2024-06-10T00:22:28.025243+0000","last_active":"2024-06-10T00:25:44.733568+0000","last_peered":"2024-06-10T00:25:44.733568+0000","last_clean":"2024-06-10T00:25:44.733568+0000","last_became_active":"2024-06-10T00:22:28.024751+0000","last_became_peered":"2024-06-10T00:22:28.024751+0000","last_unstale":"2024-06-10T00:25:44.733568+0000","last_undegraded":"2024-06-10T00:25:44.733568+0000","last_fullsized":"2024-06-10T00:25:44.733568+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_clean_scrub_stamp":"2024-06-10T00:19:05.039631+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:02:36.557931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345832+0000","last_change":"2024-06-10T00:24:09.265084+0000","last_active":"2024-06-10T00:25:44.345832+0000","last_peered":"2024-06-10T00:25:44.345832+0000","last_clean":"2024-06-10T00:25:44.345832+0000","last_became_active":"2024-06-10T00:24:09.264425+0000","last_became_peered":"2024-06-10T00:24:09.264425+0000","last_unstale":"2024-06-10T00:25:44.345832+0000","last_undegraded":"2024-06-10T00:25:44.345832+0000","last_fullsized":"2024-06-10T00:25:44.345832+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:37:31.577688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"100'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.122763+0000","last_change":"2024-06-10T00:23:57.558540+0000","last_active":"2024-06-10T00:26:42.122763+0000","last_peered":"2024-06-10T00:26:42.122763+0000","last_clean":"2024-06-10T00:26:42.122763+0000","last_became_active":"2024-06-10T00:23:57.558197+0000","last_became_peered":"2024-06-10T00:23:57.558197+0000","last_unstale":"2024-06-10T00:26:42.122763+0000","last_undegraded":"2024-06-10T00:26:42.122763+0000","last_fullsized":"2024-06-10T00:26:42.122763+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:04:25.566135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"98'3","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733939+0000","last_change":"2024-06-10T00:23:54.428126+0000","last_active":"2024-06-10T00:25:44.733939+0000","last_peered":"2024-06-10T00:25:44.733939+0000","last_clean":"2024-06-10T00:25:44.733939+0000","last_became_active":"2024-06-10T00:23:54.427630+0000","last_became_peered":"2024-06-10T00:23:54.427630+0000","last_unstale":"2024-06-10T00:25:44.733939+0000","last_undegraded":"2024-06-10T00:25:44.733939+0000","last_fullsized":"2024-06-10T00:25:44.733939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:05:23.679704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189679+0000","last_change":"2024-06-10T00:24:09.169984+0000","last_active":"2024-06-10T00:25:44.189679+0000","last_peered":"2024-06-10T00:25:44.189679+0000","last_clean":"2024-06-10T00:25:44.189679+0000","last_became_active":"2024-06-10T00:24:09.169556+0000","last_became_peered":"2024-06-10T00:24:09.169556+0000","last_unstale":"2024-06-10T00:25:44.189679+0000","last_undegraded":"2024-06-10T00:25:44.189679+0000","last_fullsized":"2024-06-10T00:25:44.189679+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:54:13.575207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"98'1","reported_seq":17,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189843+0000","last_change":"2024-06-10T00:24:43.767901+0000","last_active":"2024-06-10T00:25:44.189843+0000","last_peered":"2024-06-10T00:25:44.189843+0000","last_clean":"2024-06-10T00:25:44.189843+0000","last_became_active":"2024-06-10T00:24:43.767562+0000","last_became_peered":"2024-06-10T00:24:43.767562+0000","last_unstale":"2024-06-10T00:25:44.189843+0000","last_undegraded":"2024-06-10T00:25:44.189843+0000","last_fullsized":"2024-06-10T00:25:44.189843+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:46:36.099750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"100'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.116650+0000","last_change":"2024-06-10T00:23:57.554906+0000","last_active":"2024-06-10T00:26:42.116650+0000","last_peered":"2024-06-10T00:26:42.116650+0000","last_clean":"2024-06-10T00:26:42.116650+0000","last_became_active":"2024-06-10T00:23:57.554608+0000","last_became_peered":"2024-06-10T00:23:57.554608+0000","last_unstale":"2024-06-10T00:26:42.116650+0000","last_undegraded":"2024-06-10T00:26:42.116650+0000","last_fullsized":"2024-06-10T00:26:42.116650+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:24.690594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733319+0000","last_change":"2024-06-10T00:24:09.264676+0000","last_active":"2024-06-10T00:25:44.733319+0000","last_peered":"2024-06-10T00:25:44.733319+0000","last_clean":"2024-06-10T00:25:44.733319+0000","last_became_active":"2024-06-10T00:24:09.264259+0000","last_became_peered":"2024-06-10T00:24:09.264259+0000","last_unstale":"2024-06-10T00:25:44.733319+0000","last_undegraded":"2024-06-10T00:25:44.733319+0000","last_fullsized":"2024-06-10T00:25:44.733319+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:37:53.787039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"100'13","reported_seq":58,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:04.158655+0000","last_change":"2024-06-10T00:23:54.424201+0000","last_active":"2024-06-10T00:26:04.158655+0000","last_peered":"2024-06-10T00:26:04.158655+0000","last_clean":"2024-06-10T00:26:04.158655+0000","last_became_active":"2024-06-10T00:23:54.423853+0000","last_became_peered":"2024-06-10T00:23:54.423853+0000","last_unstale":"2024-06-10T00:26:04.158655+0000","last_undegraded":"2024-06-10T00:26:04.158655+0000","last_fullsized":"2024-06-10T00:26:04.158655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:00.884041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"100'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.132292+0000","last_change":"2024-06-10T00:23:57.554276+0000","last_active":"2024-06-10T00:26:42.132292+0000","last_peered":"2024-06-10T00:26:42.132292+0000","last_clean":"2024-06-10T00:26:42.132292+0000","last_became_active":"2024-06-10T00:23:57.553964+0000","last_became_peered":"2024-06-10T00:23:57.553964+0000","last_unstale":"2024-06-10T00:26:42.132292+0000","last_undegraded":"2024-06-10T00:26:42.132292+0000","last_fullsized":"2024-06-10T00:26:42.132292+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:39:51.974770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081427+0000","last_change":"2024-06-10T00:24:09.170249+0000","last_active":"2024-06-10T00:25:44.081427+0000","last_peered":"2024-06-10T00:25:44.081427+0000","last_clean":"2024-06-10T00:25:44.081427+0000","last_became_active":"2024-06-10T00:24:09.169830+0000","last_became_peered":"2024-06-10T00:24:09.169830+0000","last_unstale":"2024-06-10T00:25:44.081427+0000","last_undegraded":"2024-06-10T00:25:44.081427+0000","last_fullsized":"2024-06-10T00:25:44.081427+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.320609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"98'4","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345351+0000","last_change":"2024-06-10T00:23:54.473943+0000","last_active":"2024-06-10T00:25:44.345351+0000","last_peered":"2024-06-10T00:25:44.345351+0000","last_clean":"2024-06-10T00:25:44.345351+0000","last_became_active":"2024-06-10T00:23:54.473652+0000","last_became_peered":"2024-06-10T00:23:54.473652+0000","last_unstale":"2024-06-10T00:25:44.345351+0000","last_undegraded":"2024-06-10T00:25:44.345351+0000","last_fullsized":"2024-06-10T00:25:44.345351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:52:06.773684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"100'7","reported_seq":52,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.129247+0000","last_change":"2024-06-10T00:25:44.735050+0000","last_active":"2024-06-10T00:26:42.129247+0000","last_peered":"2024-06-10T00:26:42.129247+0000","last_clean":"2024-06-10T00:26:42.129247+0000","last_became_active":"2024-06-10T00:25:44.734756+0000","last_became_peered":"2024-06-10T00:25:44.734756+0000","last_unstale":"2024-06-10T00:26:42.129247+0000","last_undegraded":"2024-06-10T00:26:42.129247+0000","last_fullsized":"2024-06-10T00:26:42.129247+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:16:03.924238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":34,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344802+0000","last_change":"2024-06-10T00:24:43.768837+0000","last_active":"2024-06-10T00:25:44.344802+0000","last_peered":"2024-06-10T00:25:44.344802+0000","last_clean":"2024-06-10T00:25:44.344802+0000","last_became_active":"2024-06-10T00:24:43.768295+0000","last_became_peered":"2024-06-10T00:24:43.768295+0000","last_unstale":"2024-06-10T00:25:44.344802+0000","last_undegraded":"2024-06-10T00:25:44.344802+0000","last_fullsized":"2024-06-10T00:25:44.344802+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:59.600864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"98'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.124404+0000","last_change":"2024-06-10T00:23:54.823448+0000","last_active":"2024-06-10T00:25:44.124404+0000","last_peered":"2024-06-10T00:25:44.124404+0000","last_clean":"2024-06-10T00:25:44.124404+0000","last_became_active":"2024-06-10T00:23:54.823107+0000","last_became_peered":"2024-06-10T00:23:54.823107+0000","last_unstale":"2024-06-10T00:25:44.124404+0000","last_undegraded":"2024-06-10T00:25:44.124404+0000","last_fullsized":"2024-06-10T00:25:44.124404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:50.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"100'2","reported_seq":41,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.092265+0000","last_change":"2024-06-10T00:23:57.556733+0000","last_active":"2024-06-10T00:26:42.092265+0000","last_peered":"2024-06-10T00:26:42.092265+0000","last_clean":"2024-06-10T00:26:42.092265+0000","last_became_active":"2024-06-10T00:23:57.556386+0000","last_became_peered":"2024-06-10T00:23:57.556386+0000","last_unstale":"2024-06-10T00:26:42.092265+0000","last_undegraded":"2024-06-10T00:26:42.092265+0000","last_fullsized":"2024-06-10T00:26:42.092265+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:39:00.572311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.082653+0000","last_change":"2024-06-10T00:24:09.174600+0000","last_active":"2024-06-10T00:25:44.082653+0000","last_peered":"2024-06-10T00:25:44.082653+0000","last_clean":"2024-06-10T00:25:44.082653+0000","last_became_active":"2024-06-10T00:24:09.174236+0000","last_became_peered":"2024-06-10T00:24:09.174236+0000","last_unstale":"2024-06-10T00:25:44.082653+0000","last_undegraded":"2024-06-10T00:25:44.082653+0000","last_fullsized":"2024-06-10T00:25:44.082653+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:58:18.447768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"98'3","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.733487+0000","last_change":"2024-06-10T00:23:54.425111+0000","last_active":"2024-06-10T00:25:44.733487+0000","last_peered":"2024-06-10T00:25:44.733487+0000","last_clean":"2024-06-10T00:25:44.733487+0000","last_became_active":"2024-06-10T00:23:54.424724+0000","last_became_peered":"2024-06-10T00:23:54.424724+0000","last_unstale":"2024-06-10T00:25:44.733487+0000","last_undegraded":"2024-06-10T00:25:44.733487+0000","last_fullsized":"2024-06-10T00:25:44.733487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:54.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"100'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.130971+0000","last_change":"2024-06-10T00:23:57.556355+0000","last_active":"2024-06-10T00:26:42.130971+0000","last_peered":"2024-06-10T00:26:42.130971+0000","last_clean":"2024-06-10T00:26:42.130971+0000","last_became_active":"2024-06-10T00:23:57.555479+0000","last_became_peered":"2024-06-10T00:23:57.555479+0000","last_unstale":"2024-06-10T00:26:42.130971+0000","last_undegraded":"2024-06-10T00:26:42.130971+0000","last_fullsized":"2024-06-10T00:26:42.130971+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:08.817381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"98'1","reported_seq":28,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344830+0000","last_change":"2024-06-10T00:24:09.169589+0000","last_active":"2024-06-10T00:25:44.344830+0000","last_peered":"2024-06-10T00:25:44.344830+0000","last_clean":"2024-06-10T00:25:44.344830+0000","last_became_active":"2024-06-10T00:24:09.168436+0000","last_became_peered":"2024-06-10T00:24:09.168436+0000","last_unstale":"2024-06-10T00:25:44.344830+0000","last_undegraded":"2024-06-10T00:25:44.344830+0000","last_fullsized":"2024-06-10T00:25:44.344830+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:17:32.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"98'1","reported_seq":16,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.189512+0000","last_change":"2024-06-10T00:24:43.751399+0000","last_active":"2024-06-10T00:25:44.189512+0000","last_peered":"2024-06-10T00:25:44.189512+0000","last_clean":"2024-06-10T00:25:44.189512+0000","last_became_active":"2024-06-10T00:24:43.750987+0000","last_became_peered":"2024-06-10T00:24:43.750987+0000","last_unstale":"2024-06-10T00:25:44.189512+0000","last_undegraded":"2024-06-10T00:25:44.189512+0000","last_fullsized":"2024-06-10T00:25:44.189512+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:14:47.902017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"100'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.129730+0000","last_change":"2024-06-10T00:23:57.560213+0000","last_active":"2024-06-10T00:26:42.129730+0000","last_peered":"2024-06-10T00:26:42.129730+0000","last_clean":"2024-06-10T00:26:42.129730+0000","last_became_active":"2024-06-10T00:23:57.559925+0000","last_became_peered":"2024-06-10T00:23:57.559925+0000","last_unstale":"2024-06-10T00:26:42.129730+0000","last_undegraded":"2024-06-10T00:26:42.129730+0000","last_fullsized":"2024-06-10T00:26:42.129730+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:00:37.403087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344834+0000","last_change":"2024-06-10T00:24:09.264758+0000","last_active":"2024-06-10T00:25:44.344834+0000","last_peered":"2024-06-10T00:25:44.344834+0000","last_clean":"2024-06-10T00:25:44.344834+0000","last_became_active":"2024-06-10T00:24:09.263928+0000","last_became_peered":"2024-06-10T00:24:09.263928+0000","last_unstale":"2024-06-10T00:25:44.344834+0000","last_undegraded":"2024-06-10T00:25:44.344834+0000","last_fullsized":"2024-06-10T00:25:44.344834+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:23:41.083091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"98'4","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.734116+0000","last_change":"2024-06-10T00:23:54.428553+0000","last_active":"2024-06-10T00:25:44.734116+0000","last_peered":"2024-06-10T00:25:44.734116+0000","last_clean":"2024-06-10T00:25:44.734116+0000","last_became_active":"2024-06-10T00:23:54.428108+0000","last_became_peered":"2024-06-10T00:23:54.428108+0000","last_unstale":"2024-06-10T00:25:44.734116+0000","last_undegraded":"2024-06-10T00:25:44.734116+0000","last_fullsized":"2024-06-10T00:25:44.734116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:57:26.207915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"100'5","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.135760+0000","last_change":"2024-06-10T00:23:57.560686+0000","last_active":"2024-06-10T00:26:42.135760+0000","last_peered":"2024-06-10T00:26:42.135760+0000","last_clean":"2024-06-10T00:26:42.135760+0000","last_became_active":"2024-06-10T00:23:57.560326+0000","last_became_peered":"2024-06-10T00:23:57.560326+0000","last_unstale":"2024-06-10T00:26:42.135760+0000","last_undegraded":"2024-06-10T00:26:42.135760+0000","last_fullsized":"2024-06-10T00:26:42.135760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:08:39.506567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.124235+0000","last_change":"2024-06-10T00:24:09.166947+0000","last_active":"2024-06-10T00:25:44.124235+0000","last_peered":"2024-06-10T00:25:44.124235+0000","last_clean":"2024-06-10T00:25:44.124235+0000","last_became_active":"2024-06-10T00:24:09.166503+0000","last_became_peered":"2024-06-10T00:24:09.166503+0000","last_unstale":"2024-06-10T00:25:44.124235+0000","last_undegraded":"2024-06-10T00:25:44.124235+0000","last_fullsized":"2024-06-10T00:25:44.124235+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:16:45.094642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"98'3","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081865+0000","last_change":"2024-06-10T00:23:54.823522+0000","last_active":"2024-06-10T00:25:44.081865+0000","last_peered":"2024-06-10T00:25:44.081865+0000","last_clean":"2024-06-10T00:25:44.081865+0000","last_became_active":"2024-06-10T00:23:54.823142+0000","last_became_peered":"2024-06-10T00:23:54.823142+0000","last_unstale":"2024-06-10T00:25:44.081865+0000","last_undegraded":"2024-06-10T00:25:44.081865+0000","last_fullsized":"2024-06-10T00:25:44.081865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:03.836850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.141160+0000","last_change":"2024-06-10T00:23:57.563710+0000","last_active":"2024-06-10T00:26:42.141160+0000","last_peered":"2024-06-10T00:26:42.141160+0000","last_clean":"2024-06-10T00:26:42.141160+0000","last_became_active":"2024-06-10T00:23:57.563331+0000","last_became_peered":"2024-06-10T00:23:57.563331+0000","last_unstale":"2024-06-10T00:26:42.141160+0000","last_undegraded":"2024-06-10T00:26:42.141160+0000","last_fullsized":"2024-06-10T00:26:42.141160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:08:48.061885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.081657+0000","last_change":"2024-06-10T00:24:09.171203+0000","last_active":"2024-06-10T00:25:44.081657+0000","last_peered":"2024-06-10T00:25:44.081657+0000","last_clean":"2024-06-10T00:25:44.081657+0000","last_became_active":"2024-06-10T00:24:09.170824+0000","last_became_peered":"2024-06-10T00:24:09.170824+0000","last_unstale":"2024-06-10T00:25:44.081657+0000","last_undegraded":"2024-06-10T00:25:44.081657+0000","last_fullsized":"2024-06-10T00:25:44.081657+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:47:28.045602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'1","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.345877+0000","last_change":"2024-06-10T00:23:54.421307+0000","last_active":"2024-06-10T00:25:44.345877+0000","last_peered":"2024-06-10T00:25:44.345877+0000","last_clean":"2024-06-10T00:25:44.345877+0000","last_became_active":"2024-06-10T00:23:54.420928+0000","last_became_peered":"2024-06-10T00:23:54.420928+0000","last_unstale":"2024-06-10T00:25:44.345877+0000","last_undegraded":"2024-06-10T00:25:44.345877+0000","last_fullsized":"2024-06-10T00:25:44.345877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:59:41.433605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"100'6","reported_seq":45,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.133494+0000","last_change":"2024-06-10T00:23:57.556494+0000","last_active":"2024-06-10T00:26:42.133494+0000","last_peered":"2024-06-10T00:26:42.133494+0000","last_clean":"2024-06-10T00:26:42.133494+0000","last_became_active":"2024-06-10T00:23:57.556088+0000","last_became_peered":"2024-06-10T00:23:57.556088+0000","last_unstale":"2024-06-10T00:26:42.133494+0000","last_undegraded":"2024-06-10T00:26:42.133494+0000","last_fullsized":"2024-06-10T00:26:42.133494+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:06:23.873281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.344954+0000","last_change":"2024-06-10T00:24:09.166734+0000","last_active":"2024-06-10T00:25:44.344954+0000","last_peered":"2024-06-10T00:25:44.344954+0000","last_clean":"2024-06-10T00:25:44.344954+0000","last_became_active":"2024-06-10T00:24:09.166270+0000","last_became_peered":"2024-06-10T00:24:09.166270+0000","last_unstale":"2024-06-10T00:25:44.344954+0000","last_undegraded":"2024-06-10T00:25:44.344954+0000","last_fullsized":"2024-06-10T00:25:44.344954+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:38.039633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:25:44.078379+0000","last_change":"2024-06-10T00:23:54.426226+0000","last_active":"2024-06-10T00:25:44.078379+0000","last_peered":"2024-06-10T00:25:44.078379+0000","last_clean":"2024-06-10T00:25:44.078379+0000","last_became_active":"2024-06-10T00:23:54.425879+0000","last_became_peered":"2024-06-10T00:23:54.425879+0000","last_unstale":"2024-06-10T00:25:44.078379+0000","last_undegraded":"2024-06-10T00:25:44.078379+0000","last_fullsized":"2024-06-10T00:25:44.078379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:15:32.936556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"100'9","reported_seq":48,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.136571+0000","last_change":"2024-06-10T00:23:57.553814+0000","last_active":"2024-06-10T00:26:42.136571+0000","last_peered":"2024-06-10T00:26:42.136571+0000","last_clean":"2024-06-10T00:26:42.136571+0000","last_became_active":"2024-06-10T00:23:57.553478+0000","last_became_peered":"2024-06-10T00:23:57.553478+0000","last_unstale":"2024-06-10T00:26:42.136571+0000","last_undegraded":"2024-06-10T00:26:42.136571+0000","last_fullsized":"2024-06-10T00:26:42.136571+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:54:39.264526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"100'9","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.131195+0000","last_change":"2024-06-10T00:23:57.556676+0000","last_active":"2024-06-10T00:26:42.131195+0000","last_peered":"2024-06-10T00:26:42.131195+0000","last_clean":"2024-06-10T00:26:42.131195+0000","last_became_active":"2024-06-10T00:23:57.555946+0000","last_became_peered":"2024-06-10T00:23:57.555946+0000","last_unstale":"2024-06-10T00:26:42.131195+0000","last_undegraded":"2024-06-10T00:26:42.131195+0000","last_fullsized":"2024-06-10T00:26:42.131195+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:10:10.868799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"100'12","reported_seq":51,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.116481+0000","last_change":"2024-06-10T00:23:57.565423+0000","last_active":"2024-06-10T00:26:42.116481+0000","last_peered":"2024-06-10T00:26:42.116481+0000","last_clean":"2024-06-10T00:26:42.116481+0000","last_became_active":"2024-06-10T00:23:57.564830+0000","last_became_peered":"2024-06-10T00:23:57.564830+0000","last_unstale":"2024-06-10T00:26:42.116481+0000","last_undegraded":"2024-06-10T00:26:42.116481+0000","last_fullsized":"2024-06-10T00:26:42.116481+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:11:06.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"100'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.127160+0000","last_change":"2024-06-10T00:23:57.558971+0000","last_active":"2024-06-10T00:26:42.127160+0000","last_peered":"2024-06-10T00:26:42.127160+0000","last_clean":"2024-06-10T00:26:42.127160+0000","last_became_active":"2024-06-10T00:23:57.558620+0000","last_became_peered":"2024-06-10T00:23:57.558620+0000","last_unstale":"2024-06-10T00:26:42.127160+0000","last_undegraded":"2024-06-10T00:26:42.127160+0000","last_fullsized":"2024-06-10T00:26:42.127160+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:45:43.308008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"100'7","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.129626+0000","last_change":"2024-06-10T00:23:57.552341+0000","last_active":"2024-06-10T00:26:42.129626+0000","last_peered":"2024-06-10T00:26:42.129626+0000","last_clean":"2024-06-10T00:26:42.129626+0000","last_became_active":"2024-06-10T00:23:57.551464+0000","last_became_peered":"2024-06-10T00:23:57.551464+0000","last_unstale":"2024-06-10T00:26:42.129626+0000","last_undegraded":"2024-06-10T00:26:42.129626+0000","last_fullsized":"2024-06-10T00:26:42.129626+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:44:41.063516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"100'4","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.129662+0000","last_change":"2024-06-10T00:24:43.768961+0000","last_active":"2024-06-10T00:26:42.129662+0000","last_peered":"2024-06-10T00:26:42.129662+0000","last_clean":"2024-06-10T00:26:42.129662+0000","last_became_active":"2024-06-10T00:24:43.768542+0000","last_became_peered":"2024-06-10T00:24:43.768542+0000","last_unstale":"2024-06-10T00:26:42.129662+0000","last_undegraded":"2024-06-10T00:26:42.129662+0000","last_fullsized":"2024-06-10T00:26:42.129662+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:23:58.412054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"100'6","reported_seq":44,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.126072+0000","last_change":"2024-06-10T00:23:57.554924+0000","last_active":"2024-06-10T00:26:42.126072+0000","last_peered":"2024-06-10T00:26:42.126072+0000","last_clean":"2024-06-10T00:26:42.126072+0000","last_became_active":"2024-06-10T00:23:57.554515+0000","last_became_peered":"2024-06-10T00:23:57.554515+0000","last_unstale":"2024-06-10T00:26:42.126072+0000","last_undegraded":"2024-06-10T00:26:42.126072+0000","last_fullsized":"2024-06-10T00:26:42.126072+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:58:15.624775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"100'15","reported_seq":54,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.134946+0000","last_change":"2024-06-10T00:23:57.561940+0000","last_active":"2024-06-10T00:26:42.134946+0000","last_peered":"2024-06-10T00:26:42.134946+0000","last_clean":"2024-06-10T00:26:42.134946+0000","last_became_active":"2024-06-10T00:23:57.561512+0000","last_became_peered":"2024-06-10T00:23:57.561512+0000","last_unstale":"2024-06-10T00:26:42.134946+0000","last_undegraded":"2024-06-10T00:26:42.134946+0000","last_fullsized":"2024-06-10T00:26:42.134946+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:57:57.147311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"100'2","reported_seq":41,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.126595+0000","last_change":"2024-06-10T00:23:57.557229+0000","last_active":"2024-06-10T00:26:42.126595+0000","last_peered":"2024-06-10T00:26:42.126595+0000","last_clean":"2024-06-10T00:26:42.126595+0000","last_became_active":"2024-06-10T00:23:57.556905+0000","last_became_peered":"2024-06-10T00:23:57.556905+0000","last_unstale":"2024-06-10T00:26:42.126595+0000","last_undegraded":"2024-06-10T00:26:42.126595+0000","last_fullsized":"2024-06-10T00:26:42.126595+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:30.513517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"100'6","reported_seq":21,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.132491+0000","last_change":"2024-06-10T00:24:43.768509+0000","last_active":"2024-06-10T00:26:42.132491+0000","last_peered":"2024-06-10T00:26:42.132491+0000","last_clean":"2024-06-10T00:26:42.132491+0000","last_became_active":"2024-06-10T00:24:43.768038+0000","last_became_peered":"2024-06-10T00:24:43.768038+0000","last_unstale":"2024-06-10T00:26:42.132491+0000","last_undegraded":"2024-06-10T00:26:42.132491+0000","last_fullsized":"2024-06-10T00:26:42.132491+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:38.344375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"100'8","reported_seq":47,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.128321+0000","last_change":"2024-06-10T00:23:57.557830+0000","last_active":"2024-06-10T00:26:42.128321+0000","last_peered":"2024-06-10T00:26:42.128321+0000","last_clean":"2024-06-10T00:26:42.128321+0000","last_became_active":"2024-06-10T00:23:57.556986+0000","last_became_peered":"2024-06-10T00:23:57.556986+0000","last_unstale":"2024-06-10T00:26:42.128321+0000","last_undegraded":"2024-06-10T00:26:42.128321+0000","last_fullsized":"2024-06-10T00:26:42.128321+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:53.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"100'10","reported_seq":49,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.139578+0000","last_change":"2024-06-10T00:23:57.565602+0000","last_active":"2024-06-10T00:26:42.139578+0000","last_peered":"2024-06-10T00:26:42.139578+0000","last_clean":"2024-06-10T00:26:42.139578+0000","last_became_active":"2024-06-10T00:23:57.565264+0000","last_became_peered":"2024-06-10T00:23:57.565264+0000","last_unstale":"2024-06-10T00:26:42.139578+0000","last_undegraded":"2024-06-10T00:26:42.139578+0000","last_fullsized":"2024-06-10T00:26:42.139578+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:09:12.773087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"100'11","reported_seq":50,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.138705+0000","last_change":"2024-06-10T00:23:57.548516+0000","last_active":"2024-06-10T00:26:42.138705+0000","last_peered":"2024-06-10T00:26:42.138705+0000","last_clean":"2024-06-10T00:26:42.138705+0000","last_became_active":"2024-06-10T00:23:57.548202+0000","last_became_peered":"2024-06-10T00:23:57.548202+0000","last_unstale":"2024-06-10T00:26:42.138705+0000","last_undegraded":"2024-06-10T00:26:42.138705+0000","last_fullsized":"2024-06-10T00:26:42.138705+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:13.901478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"100'8","reported_seq":46,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-06-10T00:26:42.127831+0000","last_change":"2024-06-10T00:23:57.565900+0000","last_active":"2024-06-10T00:26:42.127831+0000","last_peered":"2024-06-10T00:26:42.127831+0000","last_clean":"2024-06-10T00:26:42.127831+0000","last_became_active":"2024-06-10T00:23:57.565573+0000","last_became_peered":"2024-06-10T00:23:57.565573+0000","last_unstale":"2024-06-10T00:26:42.127831+0000","last_undegraded":"2024-06-10T00:26:42.127831+0000","last_fullsized":"2024-06-10T00:26:42.127831+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:08.255905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280680,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67045,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":633,"num_read_kb":388,"num_write":176,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":201135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351027,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":572925,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612663,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32396,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712756,"statfs":{"total":95995035648,"available":95961862144,"internally_reserved":0,"allocated":1249280,"data_stored":1223075,"data_compressed":68001,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776189,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":577235,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037825,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":573037,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299463,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":1169108,"data_compressed":68001,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462988,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":568315,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":631129,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":569703,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36500,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708652,"statfs":{"total":95995035648,"available":95957659648,"internally_reserved":0,"allocated":1191936,"data_stored":633010,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":568509,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888150,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":572965,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724625,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1175552,"data_stored":1164696,"data_compressed":68001,"data_compressed_allocated":90112,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34060,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-10T00:26:49.355 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-10T00:26:49.355 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-10T00:26:49.355 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-10T00:26:49.356 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-10T00:26:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:49 smithi121 ceph-mon[32378]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-06-10T00:26:49.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:49 smithi135 ceph-mon[36424]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-06-10T00:26:49.598 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:49.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:49 smithi193 ceph-mon[36716]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-06-10T00:26:50.444 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:50 smithi121 ceph-mon[32378]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:26:50.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:50 smithi135 ceph-mon[36424]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:26:50.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:50 smithi193 ceph-mon[36716]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:26:51.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.385+0000 7f0bb7610700 1 -- 172.21.15.121:0/413375675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bb01013c0 msgr2=0x7f0bb0101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:51.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.385+0000 7f0bb7610700 1 --2- 172.21.15.121:0/413375675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bb01013c0 0x7f0bb0101840 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009990 tx=0x7f0ba0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.385+0000 7f0bb7610700 1 -- 172.21.15.121:0/413375675 shutdown_connections 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.385+0000 7f0bb7610700 1 --2- 172.21.15.121:0/413375675 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb0101d80 0x7f0bb010e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.385+0000 7f0bb7610700 1 --2- 172.21.15.121:0/413375675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb01032e0 0x7f0bb01036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.385+0000 7f0bb7610700 1 --2- 172.21.15.121:0/413375675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bb01013c0 0x7f0bb0101840 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.385+0000 7f0bb7610700 1 -- 172.21.15.121:0/413375675 >> 172.21.15.121:0/413375675 conn(0x7f0bb0077400 msgr2=0x7f0bb0077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.385+0000 7f0bb7610700 1 -- 172.21.15.121:0/413375675 shutdown_connections 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.385+0000 7f0bb7610700 1 -- 172.21.15.121:0/413375675 wait complete. 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb7610700 1 Processor -- start 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb7610700 1 -- start start 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb7610700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb01013c0 0x7f0bb0104180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb7610700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bb0101d80 0x7f0bb01046c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb7610700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb01032e0 0x7f0bb0104c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:51.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb7610700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb0111870 con 0x7f0bb0101d80 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb7610700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb01116f0 con 0x7f0bb01013c0 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb7610700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb01119f0 con 0x7f0bb01032e0 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb4bab700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bb0101d80 0x7f0bb01046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb4bab700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bb0101d80 0x7f0bb01046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46028/0 (socket says 172.21.15.121:46028) 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb4bab700 1 -- 172.21.15.121:0/3928066236 learned_addr learned my addr 172.21.15.121:0/3928066236 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb53ac700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb01013c0 0x7f0bb0104180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb5bad700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb01032e0 0x7f0bb0104c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb4bab700 1 -- 172.21.15.121:0/3928066236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb01032e0 msgr2=0x7f0bb0104c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb4bab700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb01032e0 0x7f0bb0104c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb4bab700 1 -- 172.21.15.121:0/3928066236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb01013c0 msgr2=0x7f0bb0104180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb4bab700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb01013c0 0x7f0bb0104180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.386+0000 7f0bb4bab700 1 -- 172.21.15.121:0/3928066236 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb01a72e0 con 0x7f0bb0101d80 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.387+0000 7f0bb4bab700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bb0101d80 0x7f0bb01046c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0000c00 tx=0x7f0ba0003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.387+0000 7f0ba67fc700 1 -- 172.21.15.121:0/3928066236 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0027070 con 0x7f0bb0101d80 2024-06-10T00:26:51.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.387+0000 7f0bb7610700 1 -- 172.21.15.121:0/3928066236 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bb01a7570 con 0x7f0bb0101d80 2024-06-10T00:26:51.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.387+0000 7f0ba67fc700 1 -- 172.21.15.121:0/3928066236 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ba0004050 con 0x7f0bb0101d80 2024-06-10T00:26:51.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.387+0000 7f0bb7610700 1 -- 172.21.15.121:0/3928066236 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bb01a7aa0 con 0x7f0bb0101d80 2024-06-10T00:26:51.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.387+0000 7f0ba67fc700 1 -- 172.21.15.121:0/3928066236 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba000f460 con 0x7f0bb0101d80 2024-06-10T00:26:51.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.388+0000 7f0bb7610700 1 -- 172.21.15.121:0/3928066236 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bb0069180 con 0x7f0bb0101d80 2024-06-10T00:26:51.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.389+0000 7f0ba67fc700 1 -- 172.21.15.121:0/3928066236 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0ba002a070 con 0x7f0bb0101d80 2024-06-10T00:26:51.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.389+0000 7f0ba67fc700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0b9c0a2ca0 0x7f0b9c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:51.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.390+0000 7f0bb53ac700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0b9c0a2ca0 0x7f0b9c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:51.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.390+0000 7f0ba67fc700 1 -- 172.21.15.121:0/3928066236 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f0ba00d0360 con 0x7f0bb0101d80 2024-06-10T00:26:51.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.390+0000 7f0bb53ac700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0b9c0a2ca0 0x7f0b9c0a5160 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00a400 tx=0x7f0bac00d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:51.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.394+0000 7f0ba67fc700 1 -- 172.21.15.121:0/3928066236 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0ba005e850 con 0x7f0bb0101d80 2024-06-10T00:26:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:51 smithi121 ceph-mon[32378]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-06-10T00:26:51.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:51 smithi135 ceph-mon[36424]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-06-10T00:26:51.532 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:51 smithi193 ceph-mon[36716]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 6 op/s 2024-06-10T00:26:51.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.635+0000 7f0bb7610700 1 -- 172.21.15.121:0/3928066236 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0bb01056b0 con 0x7f0bb0101d80 2024-06-10T00:26:51.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.635+0000 7f0ba67fc700 1 -- 172.21.15.121:0/3928066236 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f0ba0013070 con 0x7f0bb0101d80 2024-06-10T00:26:51.635 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:26:51.635 INFO:teuthology.orchestra.run.smithi121.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-10T00:26:51.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.637+0000 7f0bb7610700 1 -- 172.21.15.121:0/3928066236 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0b9c0a2ca0 msgr2=0x7f0b9c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:51.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.637+0000 7f0bb7610700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0b9c0a2ca0 0x7f0b9c0a5160 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00a400 tx=0x7f0bac00d040 comp rx=0 tx=0).stop 2024-06-10T00:26:51.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.637+0000 7f0bb7610700 1 -- 172.21.15.121:0/3928066236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bb0101d80 msgr2=0x7f0bb01046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:51.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.637+0000 7f0bb7610700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bb0101d80 0x7f0bb01046c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0000c00 tx=0x7f0ba0003680 comp rx=0 tx=0).stop 2024-06-10T00:26:51.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.637+0000 7f0bb7610700 1 -- 172.21.15.121:0/3928066236 shutdown_connections 2024-06-10T00:26:51.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.637+0000 7f0bb7610700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0b9c0a2ca0 0x7f0b9c0a5160 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:51.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.637+0000 7f0bb7610700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bb01032e0 0x7f0bb0104c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:51.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.637+0000 7f0bb7610700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0bb01013c0 0x7f0bb0104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:51.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.637+0000 7f0bb7610700 1 --2- 172.21.15.121:0/3928066236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bb0101d80 0x7f0bb01046c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:51.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.637+0000 7f0bb7610700 1 -- 172.21.15.121:0/3928066236 >> 172.21.15.121:0/3928066236 conn(0x7f0bb0077400 msgr2=0x7f0bb006af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:51.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.638+0000 7f0bb7610700 1 -- 172.21.15.121:0/3928066236 shutdown_connections 2024-06-10T00:26:51.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:51.638+0000 7f0bb7610700 1 -- 172.21.15.121:0/3928066236 wait complete. 2024-06-10T00:26:52.448 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:52 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3928066236' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-10T00:26:52.483 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-06-10T00:26:52.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:26:52.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:52 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3928066236' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-10T00:26:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:52 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3928066236' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-10T00:26:52.731 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:53.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:53 smithi121 ceph-mon[32378]: pgmap v395: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-06-10T00:26:53.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:53 smithi135 ceph-mon[36424]: pgmap v395: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-06-10T00:26:53.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:53 smithi193 ceph-mon[36716]: pgmap v395: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-06-10T00:26:54.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.396+0000 7f0d45824700 1 -- 172.21.15.121:0/2444804395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4006a170 msgr2=0x7f0d4006a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:54.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.396+0000 7f0d45824700 1 --2- 172.21.15.121:0/2444804395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4006a170 0x7f0d4006a5f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009990 tx=0x7f0d34009ca0 comp rx=0 tx=0).stop 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.396+0000 7f0d45824700 1 -- 172.21.15.121:0/2444804395 shutdown_connections 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.396+0000 7f0d45824700 1 --2- 172.21.15.121:0/2444804395 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d4006ab30 0x7f0d4010f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.396+0000 7f0d45824700 1 --2- 172.21.15.121:0/2444804395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d40108500 0x7f0d401088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.396+0000 7f0d45824700 1 --2- 172.21.15.121:0/2444804395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4006a170 0x7f0d4006a5f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.396+0000 7f0d45824700 1 -- 172.21.15.121:0/2444804395 >> 172.21.15.121:0/2444804395 conn(0x7f0d40078340 msgr2=0x7f0d40078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.396+0000 7f0d45824700 1 -- 172.21.15.121:0/2444804395 shutdown_connections 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.396+0000 7f0d45824700 1 -- 172.21.15.121:0/2444804395 wait complete. 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.397+0000 7f0d45824700 1 Processor -- start 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.397+0000 7f0d45824700 1 -- start start 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.397+0000 7f0d45824700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4006a170 0x7f0d4019e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.397+0000 7f0d45824700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d4006ab30 0x7f0d4019ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.397+0000 7f0d45824700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d40108500 0x7f0d40198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:54.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.397+0000 7f0d45824700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0d40112850 con 0x7f0d4006a170 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.397+0000 7f0d45824700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f0d401126d0 con 0x7f0d40108500 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.397+0000 7f0d45824700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0d401129d0 con 0x7f0d4006ab30 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d3effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4006a170 0x7f0d4019e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d3effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4006a170 0x7f0d4019e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46044/0 (socket says 172.21.15.121:46044) 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d3effd700 1 -- 172.21.15.121:0/2263826482 learned_addr learned my addr 172.21.15.121:0/2263826482 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d3e7fc700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d4006ab30 0x7f0d4019ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d3f7fe700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d40108500 0x7f0d40198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d3effd700 1 -- 172.21.15.121:0/2263826482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d4006ab30 msgr2=0x7f0d4019ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d3effd700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d4006ab30 0x7f0d4019ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d3effd700 1 -- 172.21.15.121:0/2263826482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d40108500 msgr2=0x7f0d40198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d3effd700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d40108500 0x7f0d40198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d3effd700 1 -- 172.21.15.121:0/2263826482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d40199230 con 0x7f0d4006a170 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d3effd700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4006a170 0x7f0d4019e910 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000d980 tx=0x7f0d3000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:54.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d44822700 1 -- 172.21.15.121:0/2263826482 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30017070 con 0x7f0d4006a170 2024-06-10T00:26:54.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d44822700 1 -- 172.21.15.121:0/2263826482 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d30005020 con 0x7f0d4006a170 2024-06-10T00:26:54.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d45824700 1 -- 172.21.15.121:0/2263826482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d40199520 con 0x7f0d4006a170 2024-06-10T00:26:54.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d44822700 1 -- 172.21.15.121:0/2263826482 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3000fbf0 con 0x7f0d4006a170 2024-06-10T00:26:54.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.398+0000 7f0d45824700 1 -- 172.21.15.121:0/2263826482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d401a57e0 con 0x7f0d4006a170 2024-06-10T00:26:54.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.399+0000 7f0d45824700 1 -- 172.21.15.121:0/2263826482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d40069180 con 0x7f0d4006a170 2024-06-10T00:26:54.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.400+0000 7f0d44822700 1 -- 172.21.15.121:0/2263826482 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0d300051d0 con 0x7f0d4006a170 2024-06-10T00:26:54.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.401+0000 7f0d44822700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0d280a2c20 0x7f0d280a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:54.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.401+0000 7f0d3e7fc700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0d280a2c20 0x7f0d280a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:54.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.401+0000 7f0d44822700 1 -- 172.21.15.121:0/2263826482 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f0d300cf220 con 0x7f0d4006a170 2024-06-10T00:26:54.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.402+0000 7f0d3e7fc700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0d280a2c20 0x7f0d280a50e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0d400788b0 tx=0x7f0d34005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:54.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.405+0000 7f0d44822700 1 -- 172.21.15.121:0/2263826482 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0d30020020 con 0x7f0d4006a170 2024-06-10T00:26:54.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.646+0000 7f0d45824700 1 -- 172.21.15.121:0/2263826482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d4002dca0 con 0x7f0d4006a170 2024-06-10T00:26:54.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.646+0000 7f0d44822700 1 -- 172.21.15.121:0/2263826482 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11885 (secure 0 0 0) 0x7f0d3009bdd0 con 0x7f0d4006a170 2024-06-10T00:26:54.647 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:26:54.647 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:25:08.157143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 -- 172.21.15.121:0/2263826482 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0d280a2c20 msgr2=0x7f0d280a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0d280a2c20 0x7f0d280a50e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0d400788b0 tx=0x7f0d34005fd0 comp rx=0 tx=0).stop 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 -- 172.21.15.121:0/2263826482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4006a170 msgr2=0x7f0d4019e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4006a170 0x7f0d4019e910 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000d980 tx=0x7f0d3000dc90 comp rx=0 tx=0).stop 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 -- 172.21.15.121:0/2263826482 shutdown_connections 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f0d280a2c20 0x7f0d280a50e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d40108500 0x7f0d40198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4006a170 0x7f0d4019e910 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 --2- 172.21.15.121:0/2263826482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d4006ab30 0x7f0d4019ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 -- 172.21.15.121:0/2263826482 >> 172.21.15.121:0/2263826482 conn(0x7f0d40078340 msgr2=0x7f0d400fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 -- 172.21.15.121:0/2263826482 shutdown_connections 2024-06-10T00:26:54.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:54.649+0000 7f0d45824700 1 -- 172.21.15.121:0/2263826482 wait complete. 2024-06-10T00:26:54.650 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 25 2024-06-10T00:26:55.412 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:55 smithi121 ceph-mon[32378]: pgmap v396: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-06-10T00:26:55.413 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:55 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2263826482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:26:55.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:26:55.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:55 smithi135 ceph-mon[36424]: pgmap v396: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-06-10T00:26:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:55 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2263826482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:26:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:55 smithi193 ceph-mon[36716]: pgmap v396: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-06-10T00:26:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:55 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2263826482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:26:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:57.371 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:57 smithi121 ceph-mon[32378]: pgmap v397: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:26:57.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.473+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/4205421336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7cc107400 msgr2=0x7ff7cc1077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:57.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.473+0000 7ff7d1ab0700 1 --2- 172.21.15.121:0/4205421336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7cc107400 0x7ff7cc1077e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009990 tx=0x7ff7bc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:26:57.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.473+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/4205421336 shutdown_connections 2024-06-10T00:26:57.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.473+0000 7ff7d1ab0700 1 --2- 172.21.15.121:0/4205421336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7cc101d50 0x7ff7cc10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.473+0000 7ff7d1ab0700 1 --2- 172.21.15.121:0/4205421336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7cc107400 0x7ff7cc1077e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.473+0000 7ff7d1ab0700 1 --2- 172.21.15.121:0/4205421336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7cc101390 0x7ff7cc101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.473+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/4205421336 >> 172.21.15.121:0/4205421336 conn(0x7ff7cc078340 msgr2=0x7ff7cc078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.473+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/4205421336 shutdown_connections 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.473+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/4205421336 wait complete. 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.474+0000 7ff7d1ab0700 1 Processor -- start 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.474+0000 7ff7d1ab0700 1 -- start start 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.474+0000 7ff7d1ab0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7cc101390 0x7ff7cc19e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.474+0000 7ff7d1ab0700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7cc101d50 0x7ff7cc19ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.474+0000 7ff7d1ab0700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7cc107400 0x7ff7cc198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.474+0000 7ff7d1ab0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc1116f0 con 0x7ff7cc101390 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.474+0000 7ff7d1ab0700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc111570 con 0x7ff7cc107400 2024-06-10T00:26:57.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.474+0000 7ff7d1ab0700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff7cc111870 con 0x7ff7cc101d50 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.474+0000 7ff7cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7cc101390 0x7ff7cc19e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.474+0000 7ff7cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7cc101390 0x7ff7cc19e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46058/0 (socket says 172.21.15.121:46058) 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.474+0000 7ff7cb7fe700 1 -- 172.21.15.121:0/3193057411 learned_addr learned my addr 172.21.15.121:0/3193057411 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7caffd700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7cc101d50 0x7ff7cc19ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7cbfff700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7cc107400 0x7ff7cc198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7cb7fe700 1 -- 172.21.15.121:0/3193057411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7cc101d50 msgr2=0x7ff7cc19ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7cb7fe700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7cc101d50 0x7ff7cc19ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7cb7fe700 1 -- 172.21.15.121:0/3193057411 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7cc107400 msgr2=0x7ff7cc198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7cb7fe700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7cc107400 0x7ff7cc198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7cb7fe700 1 -- 172.21.15.121:0/3193057411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7cc199170 con 0x7ff7cc101390 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7cb7fe700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7cc101390 0x7ff7cc19e850 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc000c00 tx=0x7ff7bc003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7c8ff9700 1 -- 172.21.15.121:0/3193057411 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc027070 con 0x7ff7cc101390 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7c8ff9700 1 -- 172.21.15.121:0/3193057411 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7bc004190 con 0x7ff7cc101390 2024-06-10T00:26:57.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/3193057411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7cc199400 con 0x7ff7cc101390 2024-06-10T00:26:57.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7c8ff9700 1 -- 172.21.15.121:0/3193057411 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc00f460 con 0x7ff7cc101390 2024-06-10T00:26:57.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.475+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/3193057411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7cc1a5720 con 0x7ff7cc101390 2024-06-10T00:26:57.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.476+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/3193057411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7cc069180 con 0x7ff7cc101390 2024-06-10T00:26:57.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.477+0000 7ff7c8ff9700 1 -- 172.21.15.121:0/3193057411 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff7bc02a070 con 0x7ff7cc101390 2024-06-10T00:26:57.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.478+0000 7ff7c8ff9700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff7b40a2c20 0x7ff7b40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:26:57.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.478+0000 7ff7caffd700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff7b40a2c20 0x7ff7b40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:26:57.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.478+0000 7ff7c8ff9700 1 -- 172.21.15.121:0/3193057411 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7ff7bc0d0260 con 0x7ff7cc101390 2024-06-10T00:26:57.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.478+0000 7ff7caffd700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff7b40a2c20 0x7ff7b40a50e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0006fd0 tx=0x7ff7c0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:26:57.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.482+0000 7ff7c8ff9700 1 -- 172.21.15.121:0/3193057411 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff7bc013070 con 0x7ff7cc101390 2024-06-10T00:26:57.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:57 smithi135 ceph-mon[36424]: pgmap v397: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:26:57.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:57 smithi193 ceph-mon[36716]: pgmap v397: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:26:57.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.726+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/3193057411 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7cc1a58f0 con 0x7ff7cc101390 2024-06-10T00:26:57.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.727+0000 7ff7c8ff9700 1 -- 172.21.15.121:0/3193057411 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11885 (secure 0 0 0) 0x7ff7bc09cea0 con 0x7ff7cc101390 2024-06-10T00:26:57.727 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:26:57.727 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:25:08.157143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:26:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/3193057411 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff7b40a2c20 msgr2=0x7ff7b40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff7b40a2c20 0x7ff7b40a50e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0006fd0 tx=0x7ff7c0008040 comp rx=0 tx=0).stop 2024-06-10T00:26:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/3193057411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7cc101390 msgr2=0x7ff7cc19e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:26:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7cc101390 0x7ff7cc19e850 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc000c00 tx=0x7ff7bc003730 comp rx=0 tx=0).stop 2024-06-10T00:26:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/3193057411 shutdown_connections 2024-06-10T00:26:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff7b40a2c20 0x7ff7b40a50e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:57.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7cc107400 0x7ff7cc198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:57.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7cc101390 0x7ff7cc19e850 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:57.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 --2- 172.21.15.121:0/3193057411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7cc101d50 0x7ff7cc19ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:26:57.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/3193057411 >> 172.21.15.121:0/3193057411 conn(0x7ff7cc078340 msgr2=0x7ff7cc1005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:26:57.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/3193057411 shutdown_connections 2024-06-10T00:26:57.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:26:57.729+0000 7ff7d1ab0700 1 -- 172.21.15.121:0/3193057411 wait complete. 2024-06-10T00:26:57.730 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 25 2024-06-10T00:26:58.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:58 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3193057411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:26:58.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:58 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3193057411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:26:58.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:26:58.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:58 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3193057411' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:26:58.864 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:26:59.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:26:59 smithi121 ceph-mon[32378]: pgmap v398: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-06-10T00:26:59.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:26:59 smithi135 ceph-mon[36424]: pgmap v398: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-06-10T00:26:59.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:26:59 smithi193 ceph-mon[36716]: pgmap v398: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-06-10T00:27:00.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.584+0000 7fa9510d9700 1 -- 172.21.15.121:0/70125264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa94c101620 msgr2=0x7fa94c101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.584+0000 7fa9510d9700 1 --2- 172.21.15.121:0/70125264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa94c101620 0x7fa94c101a00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa93c0099c0 tx=0x7fa93c009cd0 comp rx=0 tx=0).stop 2024-06-10T00:27:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.584+0000 7fa9510d9700 1 -- 172.21.15.121:0/70125264 shutdown_connections 2024-06-10T00:27:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.584+0000 7fa9510d9700 1 --2- 172.21.15.121:0/70125264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa94c1066b0 0x7fa94c10e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:00.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.584+0000 7fa9510d9700 1 --2- 172.21.15.121:0/70125264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa94c101620 0x7fa94c101a00 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.584+0000 7fa9510d9700 1 --2- 172.21.15.121:0/70125264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa94c101fd0 0x7fa94c105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.584+0000 7fa9510d9700 1 -- 172.21.15.121:0/70125264 >> 172.21.15.121:0/70125264 conn(0x7fa94c078340 msgr2=0x7fa94c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.584+0000 7fa9510d9700 1 -- 172.21.15.121:0/70125264 shutdown_connections 2024-06-10T00:27:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.584+0000 7fa9510d9700 1 -- 172.21.15.121:0/70125264 wait complete. 2024-06-10T00:27:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.585+0000 7fa9510d9700 1 Processor -- start 2024-06-10T00:27:00.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.585+0000 7fa9510d9700 1 -- start start 2024-06-10T00:27:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.585+0000 7fa9510d9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa94c101620 0x7fa94c19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.585+0000 7fa9510d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa94c101fd0 0x7fa94c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.585+0000 7fa9510d9700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa94c1066b0 0x7fa94c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.585+0000 7fa9510d9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c111710 con 0x7fa94c101fd0 2024-06-10T00:27:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.585+0000 7fa9510d9700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c111590 con 0x7fa94c101620 2024-06-10T00:27:00.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.585+0000 7fa9510d9700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c111890 con 0x7fa94c1066b0 2024-06-10T00:27:00.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.585+0000 7fa94a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa94c101fd0 0x7fa94c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:00.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa94a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa94c101fd0 0x7fa94c19ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53424/0 (socket says 172.21.15.121:53424) 2024-06-10T00:27:00.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa94a59c700 1 -- 172.21.15.121:0/4028384222 learned_addr learned my addr 172.21.15.121:0/4028384222 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:00.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa94b59e700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa94c1066b0 0x7fa94c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:00.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa94ad9d700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa94c101620 0x7fa94c19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:00.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa94a59c700 1 -- 172.21.15.121:0/4028384222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa94c1066b0 msgr2=0x7fa94c198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:00.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa94a59c700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa94c1066b0 0x7fa94c198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:00.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa94a59c700 1 -- 172.21.15.121:0/4028384222 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa94c101620 msgr2=0x7fa94c19e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:00.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa94a59c700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa94c101620 0x7fa94c19e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:00.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa94a59c700 1 -- 172.21.15.121:0/4028384222 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c199250 con 0x7fa94c101fd0 2024-06-10T00:27:00.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa94ad9d700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa94c101620 0x7fa94c19e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:00.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa94a59c700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa94c101fd0 0x7fa94c19ee20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa94000d980 tx=0x7fa94000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:00.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa933fff700 1 -- 172.21.15.121:0/4028384222 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa940017070 con 0x7fa94c101fd0 2024-06-10T00:27:00.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa933fff700 1 -- 172.21.15.121:0/4028384222 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa940009e10 con 0x7fa94c101fd0 2024-06-10T00:27:00.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa9510d9700 1 -- 172.21.15.121:0/4028384222 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c199540 con 0x7fa94c101fd0 2024-06-10T00:27:00.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa933fff700 1 -- 172.21.15.121:0/4028384222 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa94000f620 con 0x7fa94c101fd0 2024-06-10T00:27:00.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.586+0000 7fa9510d9700 1 -- 172.21.15.121:0/4028384222 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c1a57b0 con 0x7fa94c101fd0 2024-06-10T00:27:00.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.587+0000 7fa9510d9700 1 -- 172.21.15.121:0/4028384222 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa94c069180 con 0x7fa94c101fd0 2024-06-10T00:27:00.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.589+0000 7fa933fff700 1 -- 172.21.15.121:0/4028384222 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa940010460 con 0x7fa94c101fd0 2024-06-10T00:27:00.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.589+0000 7fa933fff700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9340a2c20 0x7fa9340a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:00.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.589+0000 7fa94ad9d700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9340a2c20 0x7fa9340a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:00.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.589+0000 7fa933fff700 1 -- 172.21.15.121:0/4028384222 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fa9400cef00 con 0x7fa94c101fd0 2024-06-10T00:27:00.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.590+0000 7fa94ad9d700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9340a2c20 0x7fa9340a50e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa93c000c00 tx=0x7fa93c005940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:00.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.593+0000 7fa933fff700 1 -- 172.21.15.121:0/4028384222 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa94009bab0 con 0x7fa94c101fd0 2024-06-10T00:27:00.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.790+0000 7fa9510d9700 1 -- 172.21.15.121:0/4028384222 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa94c1a5aa0 con 0x7fa94c101fd0 2024-06-10T00:27:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.791+0000 7fa933fff700 1 -- 172.21.15.121:0/4028384222 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+24919 (secure 0 0 0) 0x7fa94009b3b0 con 0x7fa94c101fd0 2024-06-10T00:27:00.792 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:27:00.792 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":100,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:25:44.074803+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:27:00.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.793+0000 7fa9510d9700 1 -- 172.21.15.121:0/4028384222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9340a2c20 msgr2=0x7fa9340a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.793+0000 7fa9510d9700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9340a2c20 0x7fa9340a50e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa93c000c00 tx=0x7fa93c005940 comp rx=0 tx=0).stop 2024-06-10T00:27:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.793+0000 7fa9510d9700 1 -- 172.21.15.121:0/4028384222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa94c101fd0 msgr2=0x7fa94c19ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.793+0000 7fa9510d9700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa94c101fd0 0x7fa94c19ee20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa94000d980 tx=0x7fa94000dc90 comp rx=0 tx=0).stop 2024-06-10T00:27:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.794+0000 7fa9510d9700 1 -- 172.21.15.121:0/4028384222 shutdown_connections 2024-06-10T00:27:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.794+0000 7fa9510d9700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fa9340a2c20 0x7fa9340a50e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.794+0000 7fa9510d9700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa94c1066b0 0x7fa94c198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.794+0000 7fa9510d9700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa94c101620 0x7fa94c19e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.794+0000 7fa9510d9700 1 --2- 172.21.15.121:0/4028384222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa94c101fd0 0x7fa94c19ee20 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.794+0000 7fa9510d9700 1 -- 172.21.15.121:0/4028384222 >> 172.21.15.121:0/4028384222 conn(0x7fa94c078340 msgr2=0x7fa94c104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.794+0000 7fa9510d9700 1 -- 172.21.15.121:0/4028384222 shutdown_connections 2024-06-10T00:27:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:00.794+0000 7fa9510d9700 1 -- 172.21.15.121:0/4028384222 wait complete. 2024-06-10T00:27:01.420 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:01 smithi121 ceph-mon[32378]: pgmap v399: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:01.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:01 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/4028384222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:27:01.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:27:01.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:01 smithi135 ceph-mon[36424]: pgmap v399: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:01.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:01 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/4028384222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:27:01.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:01 smithi193 ceph-mon[36716]: pgmap v399: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:01.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:01 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/4028384222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:27:01.690 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:03.358 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:03 smithi121 ceph-mon[32378]: pgmap v400: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:03.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.359+0000 7f1f00e0d700 1 -- 172.21.15.121:0/783755213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1efc101090 msgr2=0x7f1efc101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.359+0000 7f1f00e0d700 1 --2- 172.21.15.121:0/783755213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1efc101090 0x7f1efc101470 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4009990 tx=0x7f1ee4009ca0 comp rx=0 tx=0).stop 2024-06-10T00:27:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.360+0000 7f1f00e0d700 1 -- 172.21.15.121:0/783755213 shutdown_connections 2024-06-10T00:27:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.360+0000 7f1f00e0d700 1 --2- 172.21.15.121:0/783755213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1efc105f30 0x7f1efc10da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.360+0000 7f1f00e0d700 1 --2- 172.21.15.121:0/783755213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1efc101090 0x7f1efc101470 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.360+0000 7f1f00e0d700 1 --2- 172.21.15.121:0/783755213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1efc101a40 0x7f1efc105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.360+0000 7f1f00e0d700 1 -- 172.21.15.121:0/783755213 >> 172.21.15.121:0/783755213 conn(0x7f1efc0766b0 msgr2=0x7f1efc076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.360+0000 7f1f00e0d700 1 -- 172.21.15.121:0/783755213 shutdown_connections 2024-06-10T00:27:03.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.360+0000 7f1f00e0d700 1 -- 172.21.15.121:0/783755213 wait complete. 2024-06-10T00:27:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.360+0000 7f1f00e0d700 1 Processor -- start 2024-06-10T00:27:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.360+0000 7f1f00e0d700 1 -- start start 2024-06-10T00:27:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1f00e0d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1efc101090 0x7f1efc19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1f00e0d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1efc101a40 0x7f1efc19cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1f00e0d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1efc105f30 0x7f1efc196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1f00e0d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc110640 con 0x7f1efc105f30 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1f00e0d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1104c0 con 0x7f1efc101a40 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1f00e0d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1107c0 con 0x7f1efc101090 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1efad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1efc105f30 0x7f1efc196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1efad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1efc105f30 0x7f1efc196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53440/0 (socket says 172.21.15.121:53440) 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1efad9d700 1 -- 172.21.15.121:0/3045202050 learned_addr learned my addr 172.21.15.121:0/3045202050 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1efa59c700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1efc101090 0x7f1efc19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1ef9d9b700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1efc101a40 0x7f1efc19cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1efad9d700 1 -- 172.21.15.121:0/3045202050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1efc101090 msgr2=0x7f1efc19c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1efad9d700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1efc101090 0x7f1efc19c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1efad9d700 1 -- 172.21.15.121:0/3045202050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1efc101a40 msgr2=0x7f1efc19cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1efad9d700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1efc101a40 0x7f1efc19cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1efad9d700 1 -- 172.21.15.121:0/3045202050 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1efc196fc0 con 0x7f1efc105f30 2024-06-10T00:27:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1efad9d700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1efc105f30 0x7f1efc196860 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40111f0 tx=0x7f1ef4011500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1ef37fe700 1 -- 172.21.15.121:0/3045202050 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4004db0 con 0x7f1efc105f30 2024-06-10T00:27:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1ef37fe700 1 -- 172.21.15.121:0/3045202050 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef4005500 con 0x7f1efc105f30 2024-06-10T00:27:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.361+0000 7f1f00e0d700 1 -- 172.21.15.121:0/3045202050 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1efc1972b0 con 0x7f1efc105f30 2024-06-10T00:27:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.362+0000 7f1ef37fe700 1 -- 172.21.15.121:0/3045202050 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4019840 con 0x7f1efc105f30 2024-06-10T00:27:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.362+0000 7f1f00e0d700 1 -- 172.21.15.121:0/3045202050 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1efc1a3570 con 0x7f1efc105f30 2024-06-10T00:27:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.362+0000 7f1f00e0d700 1 -- 172.21.15.121:0/3045202050 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1efc1090c0 con 0x7f1efc105f30 2024-06-10T00:27:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.364+0000 7f1ef37fe700 1 -- 172.21.15.121:0/3045202050 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1ef40056b0 con 0x7f1efc105f30 2024-06-10T00:27:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.364+0000 7f1ef37fe700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1ee80a2c20 0x7f1ee80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.364+0000 7f1efa59c700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1ee80a2c20 0x7f1ee80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.364+0000 7f1ef37fe700 1 -- 172.21.15.121:0/3045202050 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f1ef40d3bf0 con 0x7f1efc105f30 2024-06-10T00:27:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.365+0000 7f1efa59c700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1ee80a2c20 0x7f1ee80a50e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400b5c0 tx=0x7f1ee4005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:03.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.368+0000 7f1ef37fe700 1 -- 172.21.15.121:0/3045202050 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1ef4014070 con 0x7f1efc105f30 2024-06-10T00:27:03.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:03 smithi135 ceph-mon[36424]: pgmap v400: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:03.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.606+0000 7f1f00e0d700 1 -- 172.21.15.121:0/3045202050 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1efc102270 con 0x7f1efc105f30 2024-06-10T00:27:03.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.607+0000 7f1ef37fe700 1 -- 172.21.15.121:0/3045202050 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11885 (secure 0 0 0) 0x7f1ef40a07a0 con 0x7f1efc105f30 2024-06-10T00:27:03.607 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:27:03.607 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:25:08.157143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:27:03.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.609+0000 7f1f00e0d700 1 -- 172.21.15.121:0/3045202050 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1ee80a2c20 msgr2=0x7f1ee80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:03.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.609+0000 7f1f00e0d700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1ee80a2c20 0x7f1ee80a50e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400b5c0 tx=0x7f1ee4005960 comp rx=0 tx=0).stop 2024-06-10T00:27:03.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.609+0000 7f1f00e0d700 1 -- 172.21.15.121:0/3045202050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1efc105f30 msgr2=0x7f1efc196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:03.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.609+0000 7f1f00e0d700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1efc105f30 0x7f1efc196860 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40111f0 tx=0x7f1ef4011500 comp rx=0 tx=0).stop 2024-06-10T00:27:03.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.609+0000 7f1f00e0d700 1 -- 172.21.15.121:0/3045202050 shutdown_connections 2024-06-10T00:27:03.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.609+0000 7f1f00e0d700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f1ee80a2c20 0x7f1ee80a50e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:03.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.609+0000 7f1f00e0d700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1efc105f30 0x7f1efc196860 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:03.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.609+0000 7f1f00e0d700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1efc101090 0x7f1efc19c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:03.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.609+0000 7f1f00e0d700 1 --2- 172.21.15.121:0/3045202050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1efc101a40 0x7f1efc19cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:03.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.609+0000 7f1f00e0d700 1 -- 172.21.15.121:0/3045202050 >> 172.21.15.121:0/3045202050 conn(0x7f1efc0766b0 msgr2=0x7f1efc0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:03.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.609+0000 7f1f00e0d700 1 -- 172.21.15.121:0/3045202050 shutdown_connections 2024-06-10T00:27:03.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:03.610+0000 7f1f00e0d700 1 -- 172.21.15.121:0/3045202050 wait complete. 2024-06-10T00:27:03.611 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 25 2024-06-10T00:27:03.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:03 smithi193 ceph-mon[36716]: pgmap v400: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:04.292 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-06-10T00:23:59.392036+0000', 'modified': '2024-06-10T00:25:08.157143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24479, 'mds_1': 24472, 'mds_2': 24455}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.135:6835/284701222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6834', 'nonce': 284701222}, {'type': 'v1', 'addr': '172.21.15.135:6835', 'nonce': 284701222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/3023450024', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 3023450024}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 3023450024}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24448': {'gid': 24448, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/2863018656', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 2863018656}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 2863018656}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.135:6837/1549198529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6836', 'nonce': 1549198529}, {'type': 'v1', 'addr': '172.21.15.135:6837', 'nonce': 1549198529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.193:6839/2677954983', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 2677954983}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 2677954983}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24479': {'gid': 24479, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.135:6839/1599631648', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6838', 'nonce': 1599631648}, {'type': 'v1', 'addr': '172.21.15.135:6839', 'nonce': 1599631648}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24479, 'qdb_cluster': [24455, 24479, 24472]} 2024-06-10T00:27:04.292 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-06-10T00:27:04.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-06-10T00:27:04.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:04 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3045202050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:27:04.534 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:04.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:04 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3045202050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:27:04.656 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:04 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3045202050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:27:05.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:05 smithi135 ceph-mon[36424]: pgmap v401: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:05.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:05 smithi193 ceph-mon[36716]: pgmap v401: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:05.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:05 smithi121 ceph-mon[32378]: pgmap v401: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:06.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.337+0000 7fdced9fc700 1 -- 172.21.15.121:0/3849432930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdce8107430 msgr2=0x7fdce8107810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:06.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.337+0000 7fdced9fc700 1 --2- 172.21.15.121:0/3849432930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdce8107430 0x7fdce8107810 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009990 tx=0x7fdcd8009ca0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.337+0000 7fdced9fc700 1 -- 172.21.15.121:0/3849432930 shutdown_connections 2024-06-10T00:27:06.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.337+0000 7fdced9fc700 1 --2- 172.21.15.121:0/3849432930 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdce8101d80 0x7fdce810e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.337+0000 7fdced9fc700 1 --2- 172.21.15.121:0/3849432930 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdce8107430 0x7fdce8107810 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.337+0000 7fdced9fc700 1 --2- 172.21.15.121:0/3849432930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce81013c0 0x7fdce8101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.337+0000 7fdced9fc700 1 -- 172.21.15.121:0/3849432930 >> 172.21.15.121:0/3849432930 conn(0x7fdce80783d0 msgr2=0x7fdce80787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:06.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.337+0000 7fdced9fc700 1 -- 172.21.15.121:0/3849432930 shutdown_connections 2024-06-10T00:27:06.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.337+0000 7fdced9fc700 1 -- 172.21.15.121:0/3849432930 wait complete. 2024-06-10T00:27:06.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdced9fc700 1 Processor -- start 2024-06-10T00:27:06.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdced9fc700 1 -- start start 2024-06-10T00:27:06.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdced9fc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce81013c0 0x7fdce819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdced9fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdce8101d80 0x7fdce819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdced9fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdce8107430 0x7fdce8198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdced9fc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdce81116d0 con 0x7fdce8101d80 2024-06-10T00:27:06.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdced9fc700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fdce8111550 con 0x7fdce8107430 2024-06-10T00:27:06.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdced9fc700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdce8111850 con 0x7fdce81013c0 2024-06-10T00:27:06.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdce67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdce8101d80 0x7fdce819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdce67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdce8101d80 0x7fdce819edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53454/0 (socket says 172.21.15.121:53454) 2024-06-10T00:27:06.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdce67fc700 1 -- 172.21.15.121:0/700778323 learned_addr learned my addr 172.21.15.121:0/700778323 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.338+0000 7fdce6ffd700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce81013c0 0x7fdce819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdce77fe700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdce8107430 0x7fdce8198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdce67fc700 1 -- 172.21.15.121:0/700778323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce81013c0 msgr2=0x7fdce819e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdce67fc700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce81013c0 0x7fdce819e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdce67fc700 1 -- 172.21.15.121:0/700778323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdce8107430 msgr2=0x7fdce8198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdce67fc700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdce8107430 0x7fdce8198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdce67fc700 1 -- 172.21.15.121:0/700778323 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdce81991b0 con 0x7fdce8101d80 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdce67fc700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdce8101d80 0x7fdce819edd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc00d980 tx=0x7fdcdc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdcec9fa700 1 -- 172.21.15.121:0/700778323 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc017070 con 0x7fdce8101d80 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdced9fc700 1 -- 172.21.15.121:0/700778323 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdce81994a0 con 0x7fdce8101d80 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdcec9fa700 1 -- 172.21.15.121:0/700778323 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcdc005020 con 0x7fdce8101d80 2024-06-10T00:27:06.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdcec9fa700 1 -- 172.21.15.121:0/700778323 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc00fbf0 con 0x7fdce8101d80 2024-06-10T00:27:06.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.339+0000 7fdced9fc700 1 -- 172.21.15.121:0/700778323 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdce81a5760 con 0x7fdce8101d80 2024-06-10T00:27:06.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.341+0000 7fdcec9fa700 1 -- 172.21.15.121:0/700778323 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fdcdc0051d0 con 0x7fdce8101d80 2024-06-10T00:27:06.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.342+0000 7fdcec9fa700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdcd00a2c20 0x7fdcd00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.342+0000 7fdce6ffd700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdcd00a2c20 0x7fdcd00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.342+0000 7fdcec9fa700 1 -- 172.21.15.121:0/700778323 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fdcdc0cf220 con 0x7fdce8101d80 2024-06-10T00:27:06.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdce6ffd700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdcd00a2c20 0x7fdcd00a50e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdce8199b90 tx=0x7fdcd8005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:06.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdcf827fb80 1 Processor -- start 2024-06-10T00:27:06.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdcf827fb80 1 -- start start 2024-06-10T00:27:06.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a8832289d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a883222990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a8832232f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a8831747c0 con 0x55a883228f10 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55a88311c6d0 con 0x55a883222ed0 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a883170780 con 0x55a8832285f0 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdce67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a883222990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdce67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a883222990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53464/0 (socket says 172.21.15.121:53464) 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdce67fc700 1 -- 172.21.15.121:0/3547163098 learned_addr learned my addr 172.21.15.121:0/3547163098 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdce77fe700 1 --2- 172.21.15.121:0/3547163098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a8832232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdce6ffd700 1 --2- 172.21.15.121:0/3547163098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a8832289d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdce67fc700 1 -- 172.21.15.121:0/3547163098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 msgr2=0x55a8832289d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdce67fc700 1 --2- 172.21.15.121:0/3547163098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a8832289d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdce67fc700 1 -- 172.21.15.121:0/3547163098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 msgr2=0x55a8832232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdce67fc700 1 --2- 172.21.15.121:0/3547163098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a8832232f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.343+0000 7fdce67fc700 1 -- 172.21.15.121:0/3547163098 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a883112f00 con 0x55a883228f10 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.344+0000 7fdce77fe700 1 --2- 172.21.15.121:0/3547163098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a8832232f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:27:06.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.344+0000 7fdce67fc700 1 --2- 172.21.15.121:0/3547163098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a883222990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc097b30 tx=0x7fdcdc09bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe58742649051da9 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.344+0000 7fdccdffb700 1 -- 172.21.15.121:0/3547163098 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc099770 con 0x55a883228f10 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.344+0000 7fdccdffb700 1 -- 172.21.15.121:0/3547163098 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcdc0999d0 con 0x55a883228f10 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.344+0000 7fdccdffb700 1 -- 172.21.15.121:0/3547163098 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc0a3eb0 con 0x55a883228f10 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.344+0000 7fdcf827fb80 1 -- 172.21.15.121:0/3547163098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 msgr2=0x55a883222990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.344+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/3547163098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a883222990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc097b30 tx=0x7fdcdc09bec0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 -- 172.21.15.121:0/3547163098 shutdown_connections 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/3547163098 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a8832232f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/3547163098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a8832289d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/3547163098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a883222990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 -- 172.21.15.121:0/3547163098 >> 172.21.15.121:0/3547163098 conn(0x55a88314dc70 msgr2=0x55a88321d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 -- 172.21.15.121:0/3547163098 shutdown_connections 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 -- 172.21.15.121:0/3547163098 wait complete. 2024-06-10T00:27:06.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 -- start start 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 Processor -- start 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883222ed0 0x55a88322eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a88322bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883228f10 0x55a88322c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a88322f980 con 0x55a883222ed0 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55a883114ba0 con 0x55a883228f10 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.345+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a88322fb00 con 0x55a8832285f0 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdcf827fb80 10 client.0 _ll_register_callbacks cb 0x55a88310e8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883222ed0 0x55a88322eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883222ed0 0x55a88322eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53476/0 (socket says 172.21.15.121:53476) 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce6ffd700 1 -- 172.21.15.121:0/4159617872 learned_addr learned my addr 172.21.15.121:0/4159617872 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:06.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce77fe700 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883228f10 0x55a88322c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce67fc700 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a88322bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce6ffd700 1 -- 172.21.15.121:0/4159617872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 msgr2=0x55a88322bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce6ffd700 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a88322bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce6ffd700 1 -- 172.21.15.121:0/4159617872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883228f10 msgr2=0x55a88322c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce6ffd700 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883228f10 0x55a88322c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce6ffd700 1 -- 172.21.15.121:0/4159617872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8830c3880 con 0x55a883222ed0 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce67fc700 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a88322bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdce6ffd700 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883222ed0 0x55a88322eca0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8015000 tx=0x7fdcd8009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd08539b2733ef1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdcc6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/4159617872 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd8034060 con 0x55a883222ed0 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/4159617872 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fdcd802f640 con 0x55a883222ed0 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdcf827fb80 1 -- 172.21.15.121:0/4159617872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcdc0999d0 con 0x55a883222ed0 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/4159617872 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdcd803b050 con 0x55a883222ed0 2024-06-10T00:27:06.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdcf827fb80 10 client.15039 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.346+0000 7fdcf827fb80 1 -- 172.21.15.121:0/4159617872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a88323a2d0 con 0x55a883222ed0 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.347+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/4159617872 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcdc0999d0 con 0x55a883222ed0 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.347+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/4159617872 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fdcd8024850 con 0x55a883222ed0 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.347+0000 7fdcf827fb80 10 client.15039 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.347+0000 7fdcf827fb80 10 client.15039 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.347+0000 7fdcf827fb80 20 client.15039 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.347+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55a88323c960 0x55a88323ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.347+0000 7fdcf827fb80 4 client.15039 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.347+0000 7fdcf827fb80 1 -- 172.21.15.121:0/4159617872 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a883181cc0 con 0x55a88323c960 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.347+0000 7fdce67fc700 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55a88323c960 0x55a88323ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.348+0000 7fdcc6ffd700 10 client.15039 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:27:06.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.348+0000 7fdce67fc700 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55a88323c960 0x55a88323ddd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d7e43a8ab1f17b1f server_cookie=fa8273812ad4fe6 in_seq=0 out_seq=0 2024-06-10T00:27:06.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/4159617872 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a883181cc0 con 0x55a88323c960 2024-06-10T00:27:06.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcc6ffd700 10 client.15039 handle_command_reply: tid=0 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 1 client.15039 shutdown 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 1 -- 172.21.15.121:0/4159617872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883222ed0 msgr2=0x55a88322eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883222ed0 0x55a88322eca0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8015000 tx=0x7fdcd8009ca0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 1 -- 172.21.15.121:0/4159617872 shutdown_connections 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55a88323c960 0x55a88323ddd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883228f10 0x55a88322c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883222ed0 0x55a88322eca0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcc6ffd700 0 client.15039 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/4159617872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a88322bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 1 -- 172.21.15.121:0/4159617872 >> 172.21.15.121:0/4159617872 conn(0x55a88314dc70 msgr2=0x55a883221980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 1 -- 172.21.15.121:0/4159617872 shutdown_connections 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 1 -- 172.21.15.121:0/4159617872 wait complete. 2024-06-10T00:27:06.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.349+0000 7fdcf827fb80 20 client.15039 trim_cache size 0 max 16384 2024-06-10T00:27:06.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdcf827fb80 1 Processor -- start 2024-06-10T00:27:06.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdcf827fb80 1 -- start start 2024-06-10T00:27:06.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a883222990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a8832285f0 0x55a883217ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a883228f10 0x55a883247620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a88323c110 con 0x55a8832285f0 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55a88323c7e0 con 0x55a883222ed0 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a88323ceb0 con 0x55a883228f10 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdce67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a8832285f0 0x55a883217ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdce67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a8832285f0 0x55a883217ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53484/0 (socket says 172.21.15.121:53484) 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdce67fc700 1 -- 172.21.15.121:0/769566220 learned_addr learned my addr 172.21.15.121:0/769566220 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdce77fe700 1 --2- 172.21.15.121:0/769566220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a883228f10 0x55a883247620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdce67fc700 1 -- 172.21.15.121:0/769566220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a883228f10 msgr2=0x55a883247620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdce67fc700 1 --2- 172.21.15.121:0/769566220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a883228f10 0x55a883247620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdce67fc700 1 -- 172.21.15.121:0/769566220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 msgr2=0x55a883222990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdce67fc700 1 --2- 172.21.15.121:0/769566220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a883222990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdce67fc700 1 -- 172.21.15.121:0/769566220 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a88322a510 con 0x55a8832285f0 2024-06-10T00:27:06.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdce77fe700 1 --2- 172.21.15.121:0/769566220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a883228f10 0x55a883247620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:27:06.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdce67fc700 1 --2- 172.21.15.121:0/769566220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a8832285f0 0x55a883217ec0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc09fc40 tx=0x7fdcdc097aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d1dcfdd4c34429 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:06.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdccd7fa700 1 -- 172.21.15.121:0/769566220 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc0b8070 con 0x55a8832285f0 2024-06-10T00:27:06.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdccd7fa700 1 -- 172.21.15.121:0/769566220 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcdc0999d0 con 0x55a8832285f0 2024-06-10T00:27:06.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.370+0000 7fdccd7fa700 1 -- 172.21.15.121:0/769566220 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc09bb90 con 0x55a8832285f0 2024-06-10T00:27:06.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.371+0000 7fdcf827fb80 1 -- 172.21.15.121:0/769566220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a8832285f0 msgr2=0x55a883217ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.371+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/769566220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a8832285f0 0x55a883217ec0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc09fc40 tx=0x7fdcdc097aa0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.371+0000 7fdcf827fb80 1 -- 172.21.15.121:0/769566220 shutdown_connections 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.371+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/769566220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a883228f10 0x55a883247620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.371+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/769566220 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a883222990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.371+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/769566220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a8832285f0 0x55a883217ec0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.371+0000 7fdcf827fb80 1 -- 172.21.15.121:0/769566220 >> 172.21.15.121:0/769566220 conn(0x55a88314dc70 msgr2=0x55a88322a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.371+0000 7fdcf827fb80 1 -- 172.21.15.121:0/769566220 shutdown_connections 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.371+0000 7fdcf827fb80 1 -- 172.21.15.121:0/769566220 wait complete. 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.371+0000 7fdcf827fb80 1 -- start start 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdcf827fb80 1 Processor -- start 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a883246c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a883247160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdcf827fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a8832ab220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a88323dc50 con 0x55a883228f10 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55a88323d580 con 0x55a883222ed0 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdcf827fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a88323e320 con 0x55a8832285f0 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdcf827fb80 10 client.0 _ll_register_callbacks cb 0x55a88310e8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdce77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a8832ab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdce77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a8832ab220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53492/0 (socket says 172.21.15.121:53492) 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdce77fe700 1 -- 172.21.15.121:0/2250844729 learned_addr learned my addr 172.21.15.121:0/2250844729 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdce6ffd700 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a883246c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdce67fc700 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a883247160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdce77fe700 1 -- 172.21.15.121:0/2250844729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 msgr2=0x55a883247160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdce77fe700 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a883247160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdce77fe700 1 -- 172.21.15.121:0/2250844729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 msgr2=0x55a883246c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdce77fe700 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a883246c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.372+0000 7fdce77fe700 1 -- 172.21.15.121:0/2250844729 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8832ab760 con 0x55a883228f10 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdce77fe700 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a8832ab220 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdce0007a10 tx=0x7fdce0007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd322b54dcb0db6c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcc6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/2250844729 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce001d070 con 0x55a883228f10 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/2250844729 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fdce0014940 con 0x55a883228f10 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcf827fb80 1 -- 172.21.15.121:0/2250844729 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcdc0999d0 con 0x55a883228f10 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/2250844729 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdce0029050 con 0x55a883228f10 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcf827fb80 10 client.15045 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcf827fb80 1 -- 172.21.15.121:0/2250844729 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a8832ac810 con 0x55a883228f10 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/2250844729 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcdc0999d0 con 0x55a883228f10 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/2250844729 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fdce00213b0 con 0x55a883228f10 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcf827fb80 10 client.15045 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:06.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcf827fb80 10 client.15045 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcf827fb80 20 client.15045 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55a8832b1460 0x55a8832b0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcf827fb80 4 client.15045 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.373+0000 7fdcf827fb80 1 -- 172.21.15.121:0/2250844729 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "status"}) -- 0x55a88317c240 con 0x55a8832b1460 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.374+0000 7fdce6ffd700 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55a8832b1460 0x55a8832b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.374+0000 7fdcc6ffd700 10 client.15045 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.374+0000 7fdce6ffd700 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55a8832b1460 0x55a8832b0170 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c29641d5a322a04d server_cookie=c51b49d4f0aeb13e in_seq=0 out_seq=0 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.374+0000 7fdcc6ffd700 1 -- 172.21.15.121:0/2250844729 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55a88317c240 con 0x55a8832b1460 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.374+0000 7fdcc6ffd700 10 client.15045 handle_command_reply: tid=0 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.374+0000 7fdcf827fb80 1 client.15045 shutdown 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.374+0000 7fdcf827fb80 1 -- 172.21.15.121:0/2250844729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 msgr2=0x55a8832ab220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.374+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a8832ab220 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdce0007a10 tx=0x7fdce0007dd0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.374+0000 7fdcf827fb80 1 -- 172.21.15.121:0/2250844729 shutdown_connections 2024-06-10T00:27:06.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.374+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55a8832b1460 0x55a8832b0170 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a883228f10 0x55a8832ab220 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a883222ed0 0x55a883246c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdcc6ffd700 0 client.15045 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:27:06.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdcf827fb80 1 --2- 172.21.15.121:0/2250844729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8832285f0 0x55a883247160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdcf827fb80 1 -- 172.21.15.121:0/2250844729 >> 172.21.15.121:0/2250844729 conn(0x55a88314dc70 msgr2=0x55a883216db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:06.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdcf827fb80 1 -- 172.21.15.121:0/2250844729 shutdown_connections 2024-06-10T00:27:06.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdcf827fb80 1 -- 172.21.15.121:0/2250844729 wait complete. 2024-06-10T00:27:06.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdcf827fb80 20 client.15045 trim_cache size 0 max 16384 2024-06-10T00:27:06.380 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:27:06.380 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:27:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "whoami": 2, 2024-06-10T00:27:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 24455, 2024-06-10T00:27:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "want_state": "up:active", 2024-06-10T00:27:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "up:active", 2024-06-10T00:27:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs", 2024-06-10T00:27:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "rank_uptime": 123.119974489, 2024-06-10T00:27:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "mdsmap_epoch": 25, 2024-06-10T00:27:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch": 100, 2024-06-10T00:27:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch_barrier": 98, 2024-06-10T00:27:06.381 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 217.95341296000001 2024-06-10T00:27:06.381 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:27:06.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 -- 172.21.15.121:0/700778323 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdcd00a2c20 msgr2=0x7fdcd00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:06.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdcd00a2c20 0x7fdcd00a50e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdce8199b90 tx=0x7fdcd8005960 comp rx=0 tx=0).stop 2024-06-10T00:27:06.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 -- 172.21.15.121:0/700778323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdce8101d80 msgr2=0x7fdce819edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:06.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdce8101d80 0x7fdce819edd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc00d980 tx=0x7fdcdc00dc90 comp rx=0 tx=0).stop 2024-06-10T00:27:06.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 -- 172.21.15.121:0/700778323 shutdown_connections 2024-06-10T00:27:06.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdcd00a2c20 0x7fdcd00a50e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdce8107430 0x7fdce8198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdce81013c0 0x7fdce819e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 --2- 172.21.15.121:0/700778323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdce8101d80 0x7fdce819edd0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:06.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 -- 172.21.15.121:0/700778323 >> 172.21.15.121:0/700778323 conn(0x7fdce80783d0 msgr2=0x7fdce81006b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:06.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 -- 172.21.15.121:0/700778323 shutdown_connections 2024-06-10T00:27:06.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:06.375+0000 7fdced9fc700 1 -- 172.21.15.121:0/700778323 wait complete. 2024-06-10T00:27:07.184 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-06-10T00:27:07.424 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:07.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:07 smithi121 ceph-mon[32378]: pgmap v402: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:07.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:07 smithi135 ceph-mon[36424]: pgmap v402: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:07.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:07 smithi193 ceph-mon[36716]: pgmap v402: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:09.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.139+0000 7fc7108e1700 1 -- 172.21.15.121:0/1558392780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc70c101390 msgr2=0x7fc70c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:09.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.139+0000 7fc7108e1700 1 --2- 172.21.15.121:0/1558392780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc70c101390 0x7fc70c101810 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc009990 tx=0x7fc6fc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.139+0000 7fc7108e1700 1 -- 172.21.15.121:0/1558392780 shutdown_connections 2024-06-10T00:27:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.139+0000 7fc7108e1700 1 --2- 172.21.15.121:0/1558392780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc70c101d50 0x7fc70c10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.139+0000 7fc7108e1700 1 --2- 172.21.15.121:0/1558392780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc70c107400 0x7fc70c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.139+0000 7fc7108e1700 1 --2- 172.21.15.121:0/1558392780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc70c101390 0x7fc70c101810 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.139+0000 7fc7108e1700 1 -- 172.21.15.121:0/1558392780 >> 172.21.15.121:0/1558392780 conn(0x7fc70c078340 msgr2=0x7fc70c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.139+0000 7fc7108e1700 1 -- 172.21.15.121:0/1558392780 shutdown_connections 2024-06-10T00:27:09.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.139+0000 7fc7108e1700 1 -- 172.21.15.121:0/1558392780 wait complete. 2024-06-10T00:27:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.140+0000 7fc7108e1700 1 Processor -- start 2024-06-10T00:27:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.140+0000 7fc7108e1700 1 -- start start 2024-06-10T00:27:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.140+0000 7fc7108e1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc70c101390 0x7fc70c19e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.140+0000 7fc7108e1700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc70c101d50 0x7fc70c19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.140+0000 7fc7108e1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc70c107400 0x7fc70c198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.140+0000 7fc7108e1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c111720 con 0x7fc70c107400 2024-06-10T00:27:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.140+0000 7fc7108e1700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c1115a0 con 0x7fc70c101d50 2024-06-10T00:27:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.140+0000 7fc7108e1700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c1118a0 con 0x7fc70c101390 2024-06-10T00:27:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.140+0000 7fc70ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc70c107400 0x7fc70c198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc70ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc70c107400 0x7fc70c198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58484/0 (socket says 172.21.15.121:58484) 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc70ad9d700 1 -- 172.21.15.121:0/749635612 learned_addr learned my addr 172.21.15.121:0/749635612 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc70a59c700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc70c101390 0x7fc70c19e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc709d9b700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc70c101d50 0x7fc70c19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc70ad9d700 1 -- 172.21.15.121:0/749635612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc70c101390 msgr2=0x7fc70c19e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc70ad9d700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc70c101390 0x7fc70c19e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc70ad9d700 1 -- 172.21.15.121:0/749635612 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc70c101d50 msgr2=0x7fc70c19edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc70ad9d700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc70c101d50 0x7fc70c19edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc70ad9d700 1 -- 172.21.15.121:0/749635612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70c1991d0 con 0x7fc70c107400 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc70ad9d700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc70c107400 0x7fc70c198a70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc704007ee0 tx=0x7fc70400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc7037fe700 1 -- 172.21.15.121:0/749635612 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704004bb0 con 0x7fc70c107400 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc7108e1700 1 -- 172.21.15.121:0/749635612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc70c1994c0 con 0x7fc70c107400 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc7037fe700 1 -- 172.21.15.121:0/749635612 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc704004d50 con 0x7fc70c107400 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc7037fe700 1 -- 172.21.15.121:0/749635612 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704005960 con 0x7fc70c107400 2024-06-10T00:27:09.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.141+0000 7fc7108e1700 1 -- 172.21.15.121:0/749635612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc70c1a5780 con 0x7fc70c107400 2024-06-10T00:27:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.143+0000 7fc7037fe700 1 -- 172.21.15.121:0/749635612 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fc70401a020 con 0x7fc70c107400 2024-06-10T00:27:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.144+0000 7fc7037fe700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc6f80a2c20 0x7fc6f80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.144+0000 7fc70a59c700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc6f80a2c20 0x7fc6f80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.144+0000 7fc7037fe700 1 -- 172.21.15.121:0/749635612 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fc7040ceae0 con 0x7fc70c107400 2024-06-10T00:27:09.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.144+0000 7fc70a59c700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc6f80a2c20 0x7fc6f80a50e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc70c199bb0 tx=0x7fc6f4009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.145+0000 7fc71b164b80 1 Processor -- start 2024-06-10T00:27:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.145+0000 7fc71b164b80 1 -- start start 2024-06-10T00:27:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.145+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1bc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.145+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.145+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.145+0000 7fc71b164b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d36e1087c0 con 0x55d36e1b6ed0 2024-06-10T00:27:09.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.145+0000 7fc71b164b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d36e0b06d0 con 0x55d36e1bcf10 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.145+0000 7fc71b164b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d36e104780 con 0x55d36e1bc5f0 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc70ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc70ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58494/0 (socket says 172.21.15.121:58494) 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc70ad9d700 1 -- 172.21.15.121:0/3662083652 learned_addr learned my addr 172.21.15.121:0/3662083652 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc709d9b700 1 --2- 172.21.15.121:0/3662083652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc70a59c700 1 --2- 172.21.15.121:0/3662083652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1bc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc70ad9d700 1 -- 172.21.15.121:0/3662083652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 msgr2=0x55d36e1bc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc70ad9d700 1 --2- 172.21.15.121:0/3662083652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1bc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc70ad9d700 1 -- 172.21.15.121:0/3662083652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 msgr2=0x55d36e1b6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc70ad9d700 1 --2- 172.21.15.121:0/3662083652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc70ad9d700 1 -- 172.21.15.121:0/3662083652 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d36e0a6f00 con 0x55d36e1b6ed0 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc70a59c700 1 --2- 172.21.15.121:0/3662083652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1bc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:27:09.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc709d9b700 1 --2- 172.21.15.121:0/3662083652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc70ad9d700 1 --2- 172.21.15.121:0/3662083652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b72f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc70409ac90 tx=0x7fc70409ab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f98b1ad6536d36 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc700ff9700 1 -- 172.21.15.121:0/3662083652 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70409b690 con 0x55d36e1b6ed0 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc700ff9700 1 -- 172.21.15.121:0/3662083652 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc70409af90 con 0x55d36e1b6ed0 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.146+0000 7fc700ff9700 1 -- 172.21.15.121:0/3662083652 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704097600 con 0x55d36e1b6ed0 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.147+0000 7fc71b164b80 1 -- 172.21.15.121:0/3662083652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 msgr2=0x55d36e1b72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.147+0000 7fc71b164b80 1 --2- 172.21.15.121:0/3662083652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b72f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc70409ac90 tx=0x7fc70409ab50 comp rx=0 tx=0).stop 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.147+0000 7fc71b164b80 1 -- 172.21.15.121:0/3662083652 shutdown_connections 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.147+0000 7fc71b164b80 1 --2- 172.21.15.121:0/3662083652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b72f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.147+0000 7fc71b164b80 1 --2- 172.21.15.121:0/3662083652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1bc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.147+0000 7fc71b164b80 1 --2- 172.21.15.121:0/3662083652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.147+0000 7fc71b164b80 1 -- 172.21.15.121:0/3662083652 >> 172.21.15.121:0/3662083652 conn(0x55d36e0e1c70 msgr2=0x55d36e1b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.147+0000 7fc71b164b80 1 -- 172.21.15.121:0/3662083652 shutdown_connections 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.147+0000 7fc71b164b80 1 -- 172.21.15.121:0/3662083652 wait complete. 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.147+0000 7fc71b164b80 1 -- start start 2024-06-10T00:27:09.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc71b164b80 1 Processor -- start 2024-06-10T00:27:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bc5f0 0x55d36e1c0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bcf10 0x55d36e1c0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc71b164b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d36e1c3980 con 0x55d36e1b6ed0 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc71b164b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d36e0a8ba0 con 0x55d36e1bc5f0 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc71b164b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d36e1c3b00 con 0x55d36e1bcf10 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc71b164b80 10 client.0 _ll_register_callbacks cb 0x55d36e0a28d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc70a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc70a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1c2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58508/0 (socket says 172.21.15.121:58508) 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc70a59c700 1 -- 172.21.15.121:0/118328157 learned_addr learned my addr 172.21.15.121:0/118328157 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc709d9b700 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bc5f0 0x55d36e1c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc70ad9d700 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bcf10 0x55d36e1c0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc70a59c700 1 -- 172.21.15.121:0/118328157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bcf10 msgr2=0x55d36e1c0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc70a59c700 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bcf10 0x55d36e1c0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc70a59c700 1 -- 172.21.15.121:0/118328157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bc5f0 msgr2=0x55d36e1c0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc70a59c700 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bc5f0 0x55d36e1c0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc70a59c700 1 -- 172.21.15.121:0/118328157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d36e057880 con 0x55d36e1b6ed0 2024-06-10T00:27:09.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc70a59c700 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1c2d30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4006e80 tx=0x7fc6f4004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7779ecfeed86c182 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.148+0000 7fc6f1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/118328157 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4024020 con 0x55d36e1b6ed0 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/118328157 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fc6f400f040 con 0x55d36e1b6ed0 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc71b164b80 1 -- 172.21.15.121:0/118328157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc70409af90 con 0x55d36e1b6ed0 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/118328157 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc6f4015b20 con 0x55d36e1b6ed0 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc71b164b80 10 client.15057 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc71b164b80 1 -- 172.21.15.121:0/118328157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d36e1ce330 con 0x55d36e1b6ed0 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/118328157 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc70409af90 con 0x55d36e1b6ed0 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/118328157 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc6f40065b0 con 0x55d36e1b6ed0 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc71b164b80 10 client.15057 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc71b164b80 10 client.15057 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc71b164b80 20 client.15057 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc71b164b80 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55d36e1d0990 0x55d36e1d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc71b164b80 4 client.15057 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:27:09.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.149+0000 7fc71b164b80 1 -- 172.21.15.121:0/118328157 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d36e115cc0 con 0x55d36e1d0990 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.150+0000 7fc709d9b700 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55d36e1d0990 0x55d36e1d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.150+0000 7fc709d9b700 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55d36e1d0990 0x55d36e1d1e00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=76dbc844d30536d server_cookie=8488e13f5ad6acb3 in_seq=0 out_seq=0 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.150+0000 7fc6f1ffb700 10 client.15057 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/118328157 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d36e115cc0 con 0x55d36e1d0990 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc6f1ffb700 10 client.15057 handle_command_reply: tid=0 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 1 client.15057 shutdown 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 1 -- 172.21.15.121:0/118328157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 msgr2=0x55d36e1c2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1c2d30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4006e80 tx=0x7fc6f4004860 comp rx=0 tx=0).stop 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 1 -- 172.21.15.121:0/118328157 shutdown_connections 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55d36e1d0990 0x55d36e1d1e00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bcf10 0x55d36e1c0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1c2d30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 1 --2- 172.21.15.121:0/118328157 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bc5f0 0x55d36e1c0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 1 -- 172.21.15.121:0/118328157 >> 172.21.15.121:0/118328157 conn(0x55d36e0e1c70 msgr2=0x55d36e1b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:09.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc6f1ffb700 0 client.15057 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:27:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 1 -- 172.21.15.121:0/118328157 shutdown_connections 2024-06-10T00:27:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 1 -- 172.21.15.121:0/118328157 wait complete. 2024-06-10T00:27:09.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.151+0000 7fc71b164b80 20 client.15057 trim_cache size 0 max 16384 2024-06-10T00:27:09.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.172+0000 7fc71b164b80 1 Processor -- start 2024-06-10T00:27:09.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.172+0000 7fc71b164b80 1 -- start start 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc71b164b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d36e1d0140 con 0x55d36e1b6ed0 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc71b164b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d36e1d0810 con 0x55d36e1bcf10 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc71b164b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d36e1d0ee0 con 0x55d36e1bc5f0 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc70a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc70a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58520/0 (socket says 172.21.15.121:58520) 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc70a59c700 1 -- 172.21.15.121:0/894862599 learned_addr learned my addr 172.21.15.121:0/894862599 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc709d9b700 1 --2- 172.21.15.121:0/894862599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc70a59c700 1 -- 172.21.15.121:0/894862599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 msgr2=0x55d36e1d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc70a59c700 1 --2- 172.21.15.121:0/894862599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1d02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc70a59c700 1 -- 172.21.15.121:0/894862599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 msgr2=0x55d36e1b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:09.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc70a59c700 1 --2- 172.21.15.121:0/894862599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc70a59c700 1 -- 172.21.15.121:0/894862599 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d36e1be5a0 con 0x55d36e1b6ed0 2024-06-10T00:27:09.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.173+0000 7fc70a59c700 1 --2- 172.21.15.121:0/894862599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b6990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4009fd0 tx=0x7fc6f4015760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4681ee4ae06330 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:09.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc6f3fff700 1 -- 172.21.15.121:0/894862599 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4024070 con 0x55d36e1b6ed0 2024-06-10T00:27:09.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc6f3fff700 1 -- 172.21.15.121:0/894862599 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6f4006000 con 0x55d36e1b6ed0 2024-06-10T00:27:09.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc6f3fff700 1 -- 172.21.15.121:0/894862599 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4020800 con 0x55d36e1b6ed0 2024-06-10T00:27:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc71b164b80 1 -- 172.21.15.121:0/894862599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 msgr2=0x55d36e1b6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc71b164b80 1 --2- 172.21.15.121:0/894862599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b6990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4009fd0 tx=0x7fc6f4015760 comp rx=0 tx=0).stop 2024-06-10T00:27:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc71b164b80 1 -- 172.21.15.121:0/894862599 shutdown_connections 2024-06-10T00:27:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc71b164b80 1 --2- 172.21.15.121:0/894862599 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc71b164b80 1 --2- 172.21.15.121:0/894862599 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b6990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc71b164b80 1 --2- 172.21.15.121:0/894862599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc71b164b80 1 -- 172.21.15.121:0/894862599 >> 172.21.15.121:0/894862599 conn(0x55d36e0e1c70 msgr2=0x55d36e1ba070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc71b164b80 1 -- 172.21.15.121:0/894862599 shutdown_connections 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.174+0000 7fc71b164b80 1 -- 172.21.15.121:0/894862599 wait complete. 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc71b164b80 1 -- start start 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc71b164b80 1 Processor -- start 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc71b164b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc71b164b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d36e1d1c80 con 0x55d36e1b6ed0 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc71b164b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d36e1d15b0 con 0x55d36e1bcf10 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc71b164b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d36e1d2350 con 0x55d36e1bc5f0 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc71b164b80 10 client.0 _ll_register_callbacks cb 0x55d36e0a28d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc70a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc70a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58530/0 (socket says 172.21.15.121:58530) 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc70a59c700 1 -- 172.21.15.121:0/1893278714 learned_addr learned my addr 172.21.15.121:0/1893278714 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc709d9b700 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.175+0000 7fc70ad9d700 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc70a59c700 1 -- 172.21.15.121:0/1893278714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 msgr2=0x55d36e1b5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc70a59c700 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1b5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc70a59c700 1 -- 172.21.15.121:0/1893278714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 msgr2=0x55d36e1b6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc70a59c700 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc70a59c700 1 -- 172.21.15.121:0/1893278714 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d36e23f300 con 0x55d36e1b6ed0 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc70ad9d700 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc709d9b700 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc70a59c700 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b5980 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4006920 tx=0x7fc6f4009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd0e01b57129f80 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc6f1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/1893278714 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4024070 con 0x55d36e1b6ed0 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/1893278714 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fc6f4020710 con 0x55d36e1b6ed0 2024-06-10T00:27:09.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc71b164b80 1 -- 172.21.15.121:0/1893278714 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6f4006000 con 0x55d36e1b6ed0 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/1893278714 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc6f4049050 con 0x55d36e1b6ed0 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc71b164b80 10 client.15063 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc71b164b80 1 -- 172.21.15.121:0/1893278714 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d36e2405e0 con 0x55d36e1b6ed0 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.176+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/1893278714 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6f4006000 con 0x55d36e1b6ed0 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/1893278714 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc6f40338a0 con 0x55d36e1b6ed0 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc71b164b80 10 client.15063 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc71b164b80 10 client.15063 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc71b164b80 20 client.15063 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc71b164b80 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55d36e244d10 0x55d36e243a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc71b164b80 4 client.15063 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc71b164b80 1 -- 172.21.15.121:0/1893278714 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "status"}) -- 0x55d36e110240 con 0x55d36e244d10 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc709d9b700 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55d36e244d10 0x55d36e243a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc6f1ffb700 10 client.15063 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:27:09.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc709d9b700 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55d36e244d10 0x55d36e243a20 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c2ce2191cac680de server_cookie=6a0df6f8cd6c425c in_seq=0 out_seq=0 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc6f1ffb700 1 -- 172.21.15.121:0/1893278714 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55d36e110240 con 0x55d36e244d10 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc6f1ffb700 10 client.15063 handle_command_reply: tid=0 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.177+0000 7fc71b164b80 1 client.15063 shutdown 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc71b164b80 1 -- 172.21.15.121:0/1893278714 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 msgr2=0x55d36e1b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc71b164b80 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b5980 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4006920 tx=0x7fc6f4009290 comp rx=0 tx=0).stop 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc71b164b80 1 -- 172.21.15.121:0/1893278714 shutdown_connections 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc71b164b80 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55d36e244d10 0x55d36e243a20 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc71b164b80 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d36e1bcf10 0x55d36e1b6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc71b164b80 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d36e1b6ed0 0x55d36e1b5980 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc71b164b80 1 --2- 172.21.15.121:0/1893278714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d36e1bc5f0 0x55d36e1b5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc71b164b80 1 -- 172.21.15.121:0/1893278714 >> 172.21.15.121:0/1893278714 conn(0x55d36e0e1c70 msgr2=0x55d36e1aadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc6f1ffb700 0 client.15063 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc71b164b80 1 -- 172.21.15.121:0/1893278714 shutdown_connections 2024-06-10T00:27:09.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc71b164b80 1 -- 172.21.15.121:0/1893278714 wait complete. 2024-06-10T00:27:09.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc71b164b80 20 client.15063 trim_cache size 0 max 16384 2024-06-10T00:27:09.183 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:27:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:27:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "whoami": 1, 2024-06-10T00:27:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 24472, 2024-06-10T00:27:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "want_state": "up:active", 2024-06-10T00:27:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "up:active", 2024-06-10T00:27:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs", 2024-06-10T00:27:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "rank_uptime": 126.930941158, 2024-06-10T00:27:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "mdsmap_epoch": 25, 2024-06-10T00:27:09.183 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch": 100, 2024-06-10T00:27:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch_barrier": 98, 2024-06-10T00:27:09.184 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 217.86090830500001 2024-06-10T00:27:09.184 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:27:09.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc7108e1700 1 -- 172.21.15.121:0/749635612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc6f80a2c20 msgr2=0x7fc6f80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:09.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc7108e1700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc6f80a2c20 0x7fc6f80a50e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc70c199bb0 tx=0x7fc6f4009190 comp rx=0 tx=0).stop 2024-06-10T00:27:09.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc7108e1700 1 -- 172.21.15.121:0/749635612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc70c107400 msgr2=0x7fc70c198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:09.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.178+0000 7fc7108e1700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc70c107400 0x7fc70c198a70 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc704007ee0 tx=0x7fc70400d3b0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.179+0000 7fc7108e1700 1 -- 172.21.15.121:0/749635612 shutdown_connections 2024-06-10T00:27:09.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.179+0000 7fc7108e1700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc6f80a2c20 0x7fc6f80a50e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.179+0000 7fc7108e1700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc70c107400 0x7fc70c198a70 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.179+0000 7fc7108e1700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc70c101390 0x7fc70c19e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.179+0000 7fc7108e1700 1 --2- 172.21.15.121:0/749635612 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc70c101d50 0x7fc70c19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:09.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.179+0000 7fc7108e1700 1 -- 172.21.15.121:0/749635612 >> 172.21.15.121:0/749635612 conn(0x7fc70c078340 msgr2=0x7fc70c1005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:09.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.179+0000 7fc7108e1700 1 -- 172.21.15.121:0/749635612 shutdown_connections 2024-06-10T00:27:09.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:09.179+0000 7fc7108e1700 1 -- 172.21.15.121:0/749635612 wait complete. 2024-06-10T00:27:09.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:09 smithi121 ceph-mon[32378]: pgmap v403: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:09 smithi135 ceph-mon[36424]: pgmap v403: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:09.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:09 smithi193 ceph-mon[36716]: pgmap v403: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:09.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-06-10T00:27:10.184 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:11.335 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:11 smithi121 ceph-mon[32378]: pgmap v404: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:11 smithi193 ceph-mon[36716]: pgmap v404: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:11.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:11 smithi135 ceph-mon[36424]: pgmap v404: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.881+0000 7ff88e307700 1 -- 172.21.15.121:0/2691642368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff888101e80 msgr2=0x7ff88810e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.881+0000 7ff88e307700 1 --2- 172.21.15.121:0/2691642368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff888101e80 0x7ff88810e2b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff88000b0a0 tx=0x7ff88000b3b0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.881+0000 7ff88e307700 1 -- 172.21.15.121:0/2691642368 shutdown_connections 2024-06-10T00:27:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.881+0000 7ff88e307700 1 --2- 172.21.15.121:0/2691642368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff888101e80 0x7ff88810e2b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.881+0000 7ff88e307700 1 --2- 172.21.15.121:0/2691642368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff888107470 0x7ff888107850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.881+0000 7ff88e307700 1 --2- 172.21.15.121:0/2691642368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8881014c0 0x7ff888101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.881+0000 7ff88e307700 1 -- 172.21.15.121:0/2691642368 >> 172.21.15.121:0/2691642368 conn(0x7ff8880772a0 msgr2=0x7ff8880776b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.881+0000 7ff88e307700 1 -- 172.21.15.121:0/2691642368 shutdown_connections 2024-06-10T00:27:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.881+0000 7ff88e307700 1 -- 172.21.15.121:0/2691642368 wait complete. 2024-06-10T00:27:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.882+0000 7ff88e307700 1 Processor -- start 2024-06-10T00:27:11.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.882+0000 7ff88e307700 1 -- start start 2024-06-10T00:27:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.882+0000 7ff88e307700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8881014c0 0x7ff888198d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.882+0000 7ff88e307700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff888101e80 0x7ff888199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.882+0000 7ff88e307700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff888107470 0x7ff88819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.882+0000 7ff88e307700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff888111750 con 0x7ff888101e80 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.882+0000 7ff88e307700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff8881115d0 con 0x7ff888107470 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.882+0000 7ff88e307700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff8881118d0 con 0x7ff8881014c0 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.882+0000 7ff8877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff888101e80 0x7ff888199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff8877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff888101e80 0x7ff888199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58554/0 (socket says 172.21.15.121:58554) 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff8877fe700 1 -- 172.21.15.121:0/799642082 learned_addr learned my addr 172.21.15.121:0/799642082 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff88c8a4700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff888107470 0x7ff88819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff887fff700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8881014c0 0x7ff888198d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff8877fe700 1 -- 172.21.15.121:0/799642082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8881014c0 msgr2=0x7ff888198d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff8877fe700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8881014c0 0x7ff888198d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff8877fe700 1 -- 172.21.15.121:0/799642082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff888107470 msgr2=0x7ff88819d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff8877fe700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff888107470 0x7ff88819d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff8877fe700 1 -- 172.21.15.121:0/799642082 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88819dd30 con 0x7ff888101e80 2024-06-10T00:27:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff88c8a4700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff888107470 0x7ff88819d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff8877fe700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff888101e80 0x7ff888199260 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00b530 tx=0x7ff87c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff8857fa700 1 -- 172.21.15.121:0/799642082 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c015020 con 0x7ff888101e80 2024-06-10T00:27:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff88e307700 1 -- 172.21.15.121:0/799642082 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88819e020 con 0x7ff888101e80 2024-06-10T00:27:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff8857fa700 1 -- 172.21.15.121:0/799642082 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff87c011d10 con 0x7ff888101e80 2024-06-10T00:27:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff8857fa700 1 -- 172.21.15.121:0/799642082 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c010e50 con 0x7ff888101e80 2024-06-10T00:27:11.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.883+0000 7ff88e307700 1 -- 172.21.15.121:0/799642082 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8881a58a0 con 0x7ff888101e80 2024-06-10T00:27:11.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.885+0000 7ff8857fa700 1 -- 172.21.15.121:0/799642082 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff87c018020 con 0x7ff888101e80 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.886+0000 7ff8857fa700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff8700a2dc0 0x7ff8700a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.886+0000 7ff887fff700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff8700a2dc0 0x7ff8700a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.886+0000 7ff8857fa700 1 -- 172.21.15.121:0/799642082 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7ff87c0cec80 con 0x7ff888101e80 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.886+0000 7ff887fff700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff8700a2dc0 0x7ff8700a5280 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff878005950 tx=0x7ff8780058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.887+0000 7ff898b8ab80 1 Processor -- start 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.887+0000 7ff898b8ab80 1 -- start start 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.887+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccddc320 0x55eaccddc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.887+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdd6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.887+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccdd6b60 0x55eaccdd6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.887+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55eacccc4f10 con 0x55eaccddcc40 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.887+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55eaccd23800 con 0x55eaccddc320 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.887+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55eaccc8b410 con 0x55eaccdd6b60 2024-06-10T00:27:11.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.887+0000 7ff8877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdd6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.887+0000 7ff8877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdd6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58562/0 (socket says 172.21.15.121:58562) 2024-06-10T00:27:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.887+0000 7ff8877fe700 1 -- 172.21.15.121:0/1269251270 learned_addr learned my addr 172.21.15.121:0/1269251270 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff88c8a4700 1 --2- 172.21.15.121:0/1269251270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccdd6b60 0x55eaccdd6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff887fff700 1 --2- 172.21.15.121:0/1269251270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccddc320 0x55eaccddc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff8877fe700 1 -- 172.21.15.121:0/1269251270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccdd6b60 msgr2=0x55eaccdd6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff8877fe700 1 --2- 172.21.15.121:0/1269251270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccdd6b60 0x55eaccdd6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff8877fe700 1 -- 172.21.15.121:0/1269251270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccddc320 msgr2=0x55eaccddc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff8877fe700 1 --2- 172.21.15.121:0/1269251270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccddc320 0x55eaccddc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff8877fe700 1 -- 172.21.15.121:0/1269251270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eaccc72880 con 0x55eaccddcc40 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff887fff700 1 --2- 172.21.15.121:0/1269251270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccddc320 0x55eaccddc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff8877fe700 1 --2- 172.21.15.121:0/1269251270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdd6620 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff87c097590 tx=0x7ff87c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b4403db0fb742 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff86e7fc700 1 -- 172.21.15.121:0/1269251270 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c09b130 con 0x55eaccddcc40 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff86e7fc700 1 -- 172.21.15.121:0/1269251270 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff87c09f4e0 con 0x55eaccddcc40 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.888+0000 7ff86e7fc700 1 -- 172.21.15.121:0/1269251270 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0a3b70 con 0x55eaccddcc40 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.889+0000 7ff898b8ab80 1 -- 172.21.15.121:0/1269251270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 msgr2=0x55eaccdd6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.889+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/1269251270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdd6620 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff87c097590 tx=0x7ff87c004750 comp rx=0 tx=0).stop 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.889+0000 7ff898b8ab80 1 -- 172.21.15.121:0/1269251270 shutdown_connections 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.889+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/1269251270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccdd6b60 0x55eaccdd6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.889+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/1269251270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccddc320 0x55eaccddc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.889+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/1269251270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdd6620 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.889+0000 7ff898b8ab80 1 -- 172.21.15.121:0/1269251270 >> 172.21.15.121:0/1269251270 conn(0x55eacccff030 msgr2=0x55eaccdcd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.889+0000 7ff898b8ab80 1 -- 172.21.15.121:0/1269251270 shutdown_connections 2024-06-10T00:27:11.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.889+0000 7ff898b8ab80 1 -- 172.21.15.121:0/1269251270 wait complete. 2024-06-10T00:27:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.889+0000 7ff898b8ab80 1 -- start start 2024-06-10T00:27:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff898b8ab80 1 Processor -- start 2024-06-10T00:27:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccde2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddc320 0x55eaccddfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccde0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55eacccc3f70 con 0x55eaccddcc40 2024-06-10T00:27:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55eaccd1f7c0 con 0x55eaccdd6b60 2024-06-10T00:27:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55eaccde3630 con 0x55eaccddc320 2024-06-10T00:27:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff898b8ab80 10 client.0 _ll_register_callbacks cb 0x55eacccb9560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff88c8a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccde0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff88c8a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccde0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58570/0 (socket says 172.21.15.121:58570) 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff88c8a4700 1 -- 172.21.15.121:0/603814805 learned_addr learned my addr 172.21.15.121:0/603814805 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff8877fe700 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddc320 0x55eaccddfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff887fff700 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccde2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff88c8a4700 1 -- 172.21.15.121:0/603814805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddc320 msgr2=0x55eaccddfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff88c8a4700 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddc320 0x55eaccddfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff88c8a4700 1 -- 172.21.15.121:0/603814805 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 msgr2=0x55eaccde2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff88c8a4700 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccde2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.890+0000 7ff88c8a4700 1 -- 172.21.15.121:0/603814805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eaccce9700 con 0x55eaccddcc40 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.891+0000 7ff88c8a4700 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccde0280 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff88000b0a0 tx=0x7ff880004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ecb4c306ec23f5a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.891+0000 7ff8677fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.891+0000 7ff8677fe700 1 -- 172.21.15.121:0/603814805 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8800078f0 con 0x55eaccddcc40 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.891+0000 7ff8677fe700 1 -- 172.21.15.121:0/603814805 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7ff880007a90 con 0x55eaccddcc40 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.891+0000 7ff898b8ab80 1 -- 172.21.15.121:0/603814805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff87c09f4e0 con 0x55eaccddcc40 2024-06-10T00:27:11.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.891+0000 7ff8677fe700 1 -- 172.21.15.121:0/603814805 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff88002f050 con 0x55eaccddcc40 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.891+0000 7ff898b8ab80 10 client.15075 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.891+0000 7ff898b8ab80 1 -- 172.21.15.121:0/603814805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eaccdede00 con 0x55eaccddcc40 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.891+0000 7ff8677fe700 1 -- 172.21.15.121:0/603814805 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff87c09f4e0 con 0x55eaccddcc40 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.892+0000 7ff8677fe700 1 -- 172.21.15.121:0/603814805 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff880011070 con 0x55eaccddcc40 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.892+0000 7ff898b8ab80 10 client.15075 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.892+0000 7ff898b8ab80 10 client.15075 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.892+0000 7ff898b8ab80 20 client.15075 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.892+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55eaccdf0410 0x55eaccdf1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.892+0000 7ff898b8ab80 4 client.15075 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.892+0000 7ff898b8ab80 1 -- 172.21.15.121:0/603814805 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eaccd30d00 con 0x55eaccdf0410 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.892+0000 7ff887fff700 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55eaccdf0410 0x55eaccdf1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.892+0000 7ff8677fe700 10 client.15075 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.892+0000 7ff887fff700 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55eaccdf0410 0x55eaccdf1880 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=707c73aab32b6a69 server_cookie=7f3015ea5f20a9f in_seq=0 out_seq=0 2024-06-10T00:27:11.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.893+0000 7ff8677fe700 1 -- 172.21.15.121:0/603814805 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55eaccd30d00 con 0x55eaccdf0410 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.893+0000 7ff8677fe700 10 client.15075 handle_command_reply: tid=0 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.893+0000 7ff898b8ab80 1 client.15075 shutdown 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.893+0000 7ff898b8ab80 1 -- 172.21.15.121:0/603814805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 msgr2=0x55eaccde0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.893+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccde0280 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff88000b0a0 tx=0x7ff880004cf0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.893+0000 7ff898b8ab80 1 -- 172.21.15.121:0/603814805 shutdown_connections 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.893+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55eaccdf0410 0x55eaccdf1880 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.893+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccde0280 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.893+0000 7ff8677fe700 0 client.15075 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.894+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccde2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.894+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/603814805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddc320 0x55eaccddfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.894+0000 7ff898b8ab80 1 -- 172.21.15.121:0/603814805 >> 172.21.15.121:0/603814805 conn(0x55eacccff030 msgr2=0x55eaccdcc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.894+0000 7ff898b8ab80 1 -- 172.21.15.121:0/603814805 shutdown_connections 2024-06-10T00:27:11.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.894+0000 7ff898b8ab80 1 -- 172.21.15.121:0/603814805 wait complete. 2024-06-10T00:27:11.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.894+0000 7ff898b8ab80 20 client.15075 trim_cache size 0 max 16384 2024-06-10T00:27:11.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff898b8ab80 1 Processor -- start 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff898b8ab80 1 -- start start 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccddd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddc320 0x55eaccde02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddcc40 0x55eaccde2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55eaccdefbc0 con 0x55eaccddc320 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55eaccdf0290 con 0x55eaccdd6b60 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55eaccdf0960 con 0x55eaccddcc40 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff8877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddc320 0x55eaccde02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff8877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddc320 0x55eaccde02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58584/0 (socket says 172.21.15.121:58584) 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff8877fe700 1 -- 172.21.15.121:0/24352894 learned_addr learned my addr 172.21.15.121:0/24352894 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff887fff700 1 --2- 172.21.15.121:0/24352894 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccddd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff88c8a4700 1 --2- 172.21.15.121:0/24352894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddcc40 0x55eaccde2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff8877fe700 1 -- 172.21.15.121:0/24352894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddcc40 msgr2=0x55eaccde2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff8877fe700 1 --2- 172.21.15.121:0/24352894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddcc40 0x55eaccde2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff8877fe700 1 -- 172.21.15.121:0/24352894 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 msgr2=0x55eaccddd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff8877fe700 1 --2- 172.21.15.121:0/24352894 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccddd7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff8877fe700 1 -- 172.21.15.121:0/24352894 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eaccdd6870 con 0x55eaccddc320 2024-06-10T00:27:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff887fff700 1 --2- 172.21.15.121:0/24352894 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccddd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.915+0000 7ff8877fe700 1 --2- 172.21.15.121:0/24352894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddc320 0x55eaccde02b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff87c097b00 tx=0x7ff87c097500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839efcdee75d465e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.916+0000 7ff86dffb700 1 -- 172.21.15.121:0/24352894 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0b3020 con 0x55eaccddc320 2024-06-10T00:27:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.916+0000 7ff86dffb700 1 -- 172.21.15.121:0/24352894 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff87c0a3da0 con 0x55eaccddc320 2024-06-10T00:27:11.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.916+0000 7ff86dffb700 1 -- 172.21.15.121:0/24352894 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0a3910 con 0x55eaccddc320 2024-06-10T00:27:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 -- 172.21.15.121:0/24352894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddc320 msgr2=0x55eaccde02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/24352894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddc320 0x55eaccde02b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff87c097b00 tx=0x7ff87c097500 comp rx=0 tx=0).stop 2024-06-10T00:27:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 -- 172.21.15.121:0/24352894 shutdown_connections 2024-06-10T00:27:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/24352894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddcc40 0x55eaccde2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/24352894 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccddd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/24352894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddc320 0x55eaccde02b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 -- 172.21.15.121:0/24352894 >> 172.21.15.121:0/24352894 conn(0x55eacccff030 msgr2=0x55eaccddd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 -- 172.21.15.121:0/24352894 shutdown_connections 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 -- 172.21.15.121:0/24352894 wait complete. 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 -- start start 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 Processor -- start 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccdfb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddc320 0x55eaccdfb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdcc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55eaccdf1700 con 0x55eaccddcc40 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55eaccdf1030 con 0x55eaccdd6b60 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.917+0000 7ff898b8ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55eaccdf1dd0 con 0x55eaccddc320 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff898b8ab80 10 client.0 _ll_register_callbacks cb 0x55eacccb9560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff88c8a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdcc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff88c8a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdcc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58586/0 (socket says 172.21.15.121:58586) 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff88c8a4700 1 -- 172.21.15.121:0/3733995144 learned_addr learned my addr 172.21.15.121:0/3733995144 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff887fff700 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccdfb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff8877fe700 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddc320 0x55eaccdfb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff88c8a4700 1 -- 172.21.15.121:0/3733995144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddc320 msgr2=0x55eaccdfb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff88c8a4700 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddc320 0x55eaccdfb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff88c8a4700 1 -- 172.21.15.121:0/3733995144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 msgr2=0x55eaccdfb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff88c8a4700 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccdfb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff88c8a4700 1 -- 172.21.15.121:0/3733995144 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eaccde07f0 con 0x55eaccddcc40 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff887fff700 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccdfb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff88c8a4700 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdcc400 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff880005af0 tx=0x7ff880004e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ea16e38b64721f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff8677fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:11.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff8677fe700 1 -- 172.21.15.121:0/3733995144 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8800290e0 con 0x55eaccddcc40 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff898b8ab80 1 -- 172.21.15.121:0/3733995144 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff87c0a3da0 con 0x55eaccddcc40 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff8677fe700 1 -- 172.21.15.121:0/3733995144 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7ff880029280 con 0x55eaccddcc40 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff8677fe700 1 -- 172.21.15.121:0/3733995144 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff8800076f0 con 0x55eaccddcc40 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff898b8ab80 10 client.15081 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.918+0000 7ff898b8ab80 1 -- 172.21.15.121:0/3733995144 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eaccdf97e0 con 0x55eaccddcc40 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.919+0000 7ff8677fe700 1 -- 172.21.15.121:0/3733995144 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff87c0a3da0 con 0x55eaccddcc40 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.919+0000 7ff8677fe700 1 -- 172.21.15.121:0/3733995144 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff880011070 con 0x55eaccddcc40 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.919+0000 7ff898b8ab80 10 client.15081 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.919+0000 7ff898b8ab80 10 client.15081 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.919+0000 7ff898b8ab80 20 client.15081 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.919+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55eacce60310 0x55eacce5f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.919+0000 7ff898b8ab80 4 client.15081 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-10T00:27:11.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.919+0000 7ff898b8ab80 1 -- 172.21.15.121:0/3733995144 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "status"}) -- 0x55eaccd2b280 con 0x55eacce60310 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.919+0000 7ff887fff700 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55eacce60310 0x55eacce5f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.919+0000 7ff8677fe700 10 client.15081 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.919+0000 7ff887fff700 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55eacce60310 0x55eacce5f020 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dce00a33fdff0215 server_cookie=a2d2a6b15c33765e in_seq=0 out_seq=0 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff8677fe700 1 -- 172.21.15.121:0/3733995144 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x55eaccd2b280 con 0x55eacce60310 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff8677fe700 10 client.15081 handle_command_reply: tid=0 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 1 client.15081 shutdown 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 1 -- 172.21.15.121:0/3733995144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 msgr2=0x55eaccdcc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdcc400 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff880005af0 tx=0x7ff880004e10 comp rx=0 tx=0).stop 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 1 -- 172.21.15.121:0/3733995144 shutdown_connections 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55eacce60310 0x55eacce5f020 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55eaccddcc40 0x55eaccdcc400 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55eaccdd6b60 0x55eaccdfb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 1 --2- 172.21.15.121:0/3733995144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaccddc320 0x55eaccdfb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 1 -- 172.21.15.121:0/3733995144 >> 172.21.15.121:0/3733995144 conn(0x55eacccff030 msgr2=0x55eaccddaeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff8677fe700 0 client.15081 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:27:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 1 -- 172.21.15.121:0/3733995144 shutdown_connections 2024-06-10T00:27:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 1 -- 172.21.15.121:0/3733995144 wait complete. 2024-06-10T00:27:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.920+0000 7ff898b8ab80 20 client.15081 trim_cache size 0 max 16384 2024-06-10T00:27:11.926 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:27:11.926 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:27:11.926 INFO:teuthology.orchestra.run.smithi121.stdout: "whoami": 0, 2024-06-10T00:27:11.926 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 24479, 2024-06-10T00:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout: "want_state": "up:active", 2024-06-10T00:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "up:active", 2024-06-10T00:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs", 2024-06-10T00:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout: "rank_uptime": 192.37444357199999, 2024-06-10T00:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout: "mdsmap_epoch": 25, 2024-06-10T00:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch": 100, 2024-06-10T00:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch_barrier": 98, 2024-06-10T00:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 210.66843572499999 2024-06-10T00:27:11.927 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:27:11.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 -- 172.21.15.121:0/799642082 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff8700a2dc0 msgr2=0x7ff8700a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:11.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff8700a2dc0 0x7ff8700a5280 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff878005950 tx=0x7ff8780058e0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 -- 172.21.15.121:0/799642082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff888101e80 msgr2=0x7ff888199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:11.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff888101e80 0x7ff888199260 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00b530 tx=0x7ff87c00b8f0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 -- 172.21.15.121:0/799642082 shutdown_connections 2024-06-10T00:27:11.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff8700a2dc0 0x7ff8700a5280 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff888107470 0x7ff88819d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8881014c0 0x7ff888198d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 --2- 172.21.15.121:0/799642082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff888101e80 0x7ff888199260 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:11.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 -- 172.21.15.121:0/799642082 >> 172.21.15.121:0/799642082 conn(0x7ff8880772a0 msgr2=0x7ff88806afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:11.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 -- 172.21.15.121:0/799642082 shutdown_connections 2024-06-10T00:27:11.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:11.921+0000 7ff88e307700 1 -- 172.21.15.121:0/799642082 wait complete. 2024-06-10T00:27:12.657 INFO:tasks.cephadm:Setup complete, yielding 2024-06-10T00:27:12.657 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-10T00:27:12.667 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi121.front.sepia.ceph.com 2024-06-10T00:27:12.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch status' 2024-06-10T00:27:12.908 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:13.380 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:13 smithi121 ceph-mon[32378]: pgmap v405: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:13.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:13 smithi193 ceph-mon[36716]: pgmap v405: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:13.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:13 smithi135 ceph-mon[36424]: pgmap v405: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:14.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.575+0000 7f15a5bf3700 1 -- 172.21.15.121:0/1828197524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a0103740 msgr2=0x7f15a0103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:14.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.575+0000 7f15a5bf3700 1 --2- 172.21.15.121:0/1828197524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a0103740 0x7f15a0103ba0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f15940099c0 tx=0x7f1594009cd0 comp rx=0 tx=0).stop 2024-06-10T00:27:14.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.575+0000 7f15a5bf3700 1 -- 172.21.15.121:0/1828197524 shutdown_connections 2024-06-10T00:27:14.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.575+0000 7f15a5bf3700 1 --2- 172.21.15.121:0/1828197524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a01040e0 0x7f15a010a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:14.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.575+0000 7f15a5bf3700 1 --2- 172.21.15.121:0/1828197524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a0102540 0x7f15a0102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:14.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.575+0000 7f15a5bf3700 1 --2- 172.21.15.121:0/1828197524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a0103740 0x7f15a0103ba0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:14.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.575+0000 7f15a5bf3700 1 -- 172.21.15.121:0/1828197524 >> 172.21.15.121:0/1828197524 conn(0x7f15a00fdae0 msgr2=0x7f15a00fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:14.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.575+0000 7f15a5bf3700 1 -- 172.21.15.121:0/1828197524 shutdown_connections 2024-06-10T00:27:14.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.575+0000 7f15a5bf3700 1 -- 172.21.15.121:0/1828197524 wait complete. 2024-06-10T00:27:14.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.576+0000 7f15a5bf3700 1 Processor -- start 2024-06-10T00:27:14.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.576+0000 7f15a5bf3700 1 -- start start 2024-06-10T00:27:14.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.576+0000 7f15a5bf3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a0102540 0x7f15a0076230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:14.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.576+0000 7f15a5bf3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a0103740 0x7f15a0074880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:14.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.576+0000 7f15a5bf3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a01040e0 0x7f15a0074dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:14.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.576+0000 7f15a5bf3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f15a010e130 con 0x7f15a01040e0 2024-06-10T00:27:14.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.576+0000 7f15a5bf3700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f15a010dfb0 con 0x7f15a0103740 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.576+0000 7f15a5bf3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f15a010e2b0 con 0x7f15a0102540 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.576+0000 7f159ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a01040e0 0x7f15a0074dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.576+0000 7f159ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a01040e0 0x7f15a0074dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58604/0 (socket says 172.21.15.121:58604) 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.576+0000 7f159ffff700 1 -- 172.21.15.121:0/3665639118 learned_addr learned my addr 172.21.15.121:0/3665639118 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159f7fe700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a0102540 0x7f15a0076230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159effd700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a0103740 0x7f15a0074880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159ffff700 1 -- 172.21.15.121:0/3665639118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a0102540 msgr2=0x7f15a0076230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159ffff700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a0102540 0x7f15a0076230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159ffff700 1 -- 172.21.15.121:0/3665639118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a0103740 msgr2=0x7f15a0074880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159ffff700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a0103740 0x7f15a0074880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159ffff700 1 -- 172.21.15.121:0/3665639118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a0075670 con 0x7f15a01040e0 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159f7fe700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a0102540 0x7f15a0076230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:14.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159ffff700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a01040e0 0x7f15a0074dc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f159800ada0 tx=0x7f159800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:14.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159cff9700 1 -- 172.21.15.121:0/3665639118 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598017020 con 0x7f15a01040e0 2024-06-10T00:27:14.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f15a5bf3700 1 -- 172.21.15.121:0/3665639118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a01a2880 con 0x7f15a01040e0 2024-06-10T00:27:14.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159cff9700 1 -- 172.21.15.121:0/3665639118 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1598004bb0 con 0x7f15a01040e0 2024-06-10T00:27:14.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f159cff9700 1 -- 172.21.15.121:0/3665639118 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1598005520 con 0x7f15a01040e0 2024-06-10T00:27:14.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.577+0000 7f15a5bf3700 1 -- 172.21.15.121:0/3665639118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a01a2d40 con 0x7f15a01040e0 2024-06-10T00:27:14.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.578+0000 7f15a5bf3700 1 -- 172.21.15.121:0/3665639118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15a0069180 con 0x7f15a01040e0 2024-06-10T00:27:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.580+0000 7f159cff9700 1 -- 172.21.15.121:0/3665639118 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f15980056c0 con 0x7f15a01040e0 2024-06-10T00:27:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.580+0000 7f159cff9700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f15880a2c20 0x7f15880a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.580+0000 7f159cff9700 1 -- 172.21.15.121:0/3665639118 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f15980cf100 con 0x7f15a01040e0 2024-06-10T00:27:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.580+0000 7f159f7fe700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f15880a2c20 0x7f15880a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:14.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.581+0000 7f159f7fe700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f15880a2c20 0x7f15880a50e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f15a01035a0 tx=0x7f159000a680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:14.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.584+0000 7f159cff9700 1 -- 172.21.15.121:0/3665639118 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1598021020 con 0x7f15a01040e0 2024-06-10T00:27:14.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.795+0000 7f15a5bf3700 1 -- 172.21.15.121:0/3665639118 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f15a00628d0 con 0x7f15880a2c20 2024-06-10T00:27:14.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.796+0000 7f159cff9700 1 -- 172.21.15.121:0/3665639118 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f15a00628d0 con 0x7f15880a2c20 2024-06-10T00:27:14.795 INFO:teuthology.orchestra.run.smithi121.stdout:Backend: cephadm 2024-06-10T00:27:14.796 INFO:teuthology.orchestra.run.smithi121.stdout:Available: Yes 2024-06-10T00:27:14.796 INFO:teuthology.orchestra.run.smithi121.stdout:Paused: No 2024-06-10T00:27:14.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 -- 172.21.15.121:0/3665639118 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f15880a2c20 msgr2=0x7f15880a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:14.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f15880a2c20 0x7f15880a50e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f15a01035a0 tx=0x7f159000a680 comp rx=0 tx=0).stop 2024-06-10T00:27:14.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 -- 172.21.15.121:0/3665639118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a01040e0 msgr2=0x7f15a0074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:14.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a01040e0 0x7f15a0074dc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f159800ada0 tx=0x7f159800c330 comp rx=0 tx=0).stop 2024-06-10T00:27:14.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 -- 172.21.15.121:0/3665639118 shutdown_connections 2024-06-10T00:27:14.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f15880a2c20 0x7f15880a50e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:14.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15a01040e0 0x7f15a0074dc0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:14.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15a0102540 0x7f15a0076230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:14.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 --2- 172.21.15.121:0/3665639118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15a0103740 0x7f15a0074880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:14.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 -- 172.21.15.121:0/3665639118 >> 172.21.15.121:0/3665639118 conn(0x7f15a00fdae0 msgr2=0x7f15a0106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:14.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 -- 172.21.15.121:0/3665639118 shutdown_connections 2024-06-10T00:27:14.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:14.798+0000 7f15a5bf3700 1 -- 172.21.15.121:0/3665639118 wait complete. 2024-06-10T00:27:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:15 smithi121 ceph-mon[32378]: pgmap v406: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:15.489 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ps' 2024-06-10T00:27:15.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:15 smithi193 ceph-mon[36716]: pgmap v406: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:15.729 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:15.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:15 smithi135 ceph-mon[36424]: pgmap v406: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:16.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:16 smithi121 ceph-mon[32378]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:16.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:16 smithi193 ceph-mon[36716]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:16.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:16 smithi135 ceph-mon[36424]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:17.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.451+0000 7f189197b700 1 -- 172.21.15.121:0/40789605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f188c101f30 msgr2=0x7f188c1027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:17.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.451+0000 7f189197b700 1 --2- 172.21.15.121:0/40789605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f188c101f30 0x7f188c1027b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f188400b0a0 tx=0x7f188400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:27:17.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.451+0000 7f189197b700 1 -- 172.21.15.121:0/40789605 shutdown_connections 2024-06-10T00:27:17.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.451+0000 7f189197b700 1 --2- 172.21.15.121:0/40789605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f188c101f30 0x7f188c1027b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.451+0000 7f189197b700 1 --2- 172.21.15.121:0/40789605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f188c100800 0x7f188c100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.451+0000 7f189197b700 1 --2- 172.21.15.121:0/40789605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f188c101570 0x7f188c1019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.451+0000 7f189197b700 1 -- 172.21.15.121:0/40789605 >> 172.21.15.121:0/40789605 conn(0x7f188c0fc160 msgr2=0x7f188c0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.451+0000 7f189197b700 1 -- 172.21.15.121:0/40789605 shutdown_connections 2024-06-10T00:27:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.451+0000 7f189197b700 1 -- 172.21.15.121:0/40789605 wait complete. 2024-06-10T00:27:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.451+0000 7f189197b700 1 Processor -- start 2024-06-10T00:27:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f189197b700 1 -- start start 2024-06-10T00:27:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f189197b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f188c100800 0x7f188c19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f189197b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f188c101570 0x7f188c19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:17.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f189197b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f188c101f30 0x7f188c1a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:17.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f189197b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f188c110310 con 0x7f188c100800 2024-06-10T00:27:17.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f189197b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f188c110190 con 0x7f188c101f30 2024-06-10T00:27:17.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f189197b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f188c110490 con 0x7f188c101570 2024-06-10T00:27:17.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f188affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f188c100800 0x7f188c19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:17.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f188affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f188c100800 0x7f188c19a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58620/0 (socket says 172.21.15.121:58620) 2024-06-10T00:27:17.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f188affd700 1 -- 172.21.15.121:0/2350638199 learned_addr learned my addr 172.21.15.121:0/2350638199 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:17.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f188a7fc700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f188c101570 0x7f188c19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:17.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f188b7fe700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f188c101f30 0x7f188c1a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:17.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f188affd700 1 -- 172.21.15.121:0/2350638199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f188c101570 msgr2=0x7f188c19ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f188affd700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f188c101570 0x7f188c19ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f188affd700 1 -- 172.21.15.121:0/2350638199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f188c101f30 msgr2=0x7f188c1a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f188affd700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f188c101f30 0x7f188c1a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f188affd700 1 -- 172.21.15.121:0/2350638199 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f188c1a24c0 con 0x7f188c100800 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f188affd700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f188c100800 0x7f188c19a860 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f187c00c950 tx=0x7f187c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f1890979700 1 -- 172.21.15.121:0/2350638199 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187c0078d0 con 0x7f188c100800 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f1890979700 1 -- 172.21.15.121:0/2350638199 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f187c00f450 con 0x7f188c100800 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.452+0000 7f189197b700 1 -- 172.21.15.121:0/2350638199 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f188c1a27b0 con 0x7f188c100800 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.453+0000 7f1890979700 1 -- 172.21.15.121:0/2350638199 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f187c018660 con 0x7f188c100800 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.453+0000 7f189197b700 1 -- 172.21.15.121:0/2350638199 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f188c1a2cf0 con 0x7f188c100800 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.454+0000 7f189197b700 1 -- 172.21.15.121:0/2350638199 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f188c069180 con 0x7f188c100800 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.455+0000 7f1890979700 1 -- 172.21.15.121:0/2350638199 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f187c00f600 con 0x7f188c100800 2024-06-10T00:27:17.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.456+0000 7f1890979700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f18740a2c20 0x7f18740a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:17.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.456+0000 7f188a7fc700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f18740a2c20 0x7f18740a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:17.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.456+0000 7f1890979700 1 -- 172.21.15.121:0/2350638199 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f187c0cf2b0 con 0x7f188c100800 2024-06-10T00:27:17.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.456+0000 7f188a7fc700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f18740a2c20 0x7f18740a50e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1880004630 tx=0x7f188000b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:17.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.460+0000 7f1890979700 1 -- 172.21.15.121:0/2350638199 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f187c014030 con 0x7f188c100800 2024-06-10T00:27:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:17 smithi193 ceph-mon[36716]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:17.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.668+0000 7f189197b700 1 -- 172.21.15.121:0/2350638199 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f188c0628d0 con 0x7f18740a2c20 2024-06-10T00:27:17.668 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:17 smithi121 ceph-mon[32378]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:17.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.681+0000 7f1890979700 1 -- 172.21.15.121:0/2350638199 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f188c0628d0 con 0x7f18740a2c20 2024-06-10T00:27:17.681 INFO:teuthology.orchestra.run.smithi121.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-10T00:27:17.681 INFO:teuthology.orchestra.run.smithi121.stdout:mds.a smithi121 running (3m) 34s ago 3m 13.4M - 19.0.0-2946-g3d89accf b045ac545f82 ab8ef27468e7 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mds.b smithi135 running (3m) 84s ago 3m 26.0M - 19.0.0-2946-g3d89accf b045ac545f82 13bb6b83c0c3 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mds.c smithi193 running (3m) 84s ago 3m 18.2M - 19.0.0-2946-g3d89accf b045ac545f82 94260ed35f30 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mds.d smithi121 running (4m) 34s ago 4m 14.5M - 19.0.0-2946-g3d89accf b045ac545f82 cbb96db31993 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mds.e smithi135 running (3m) 84s ago 3m 16.8M - 19.0.0-2946-g3d89accf b045ac545f82 d4a9ad535a9d 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mds.f smithi193 running (3m) 84s ago 3m 18.4M - 19.0.0-2946-g3d89accf b045ac545f82 e76d9836ad77 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mds.g smithi121 running (3m) 34s ago 3m 16.6M - 19.0.0-2946-g3d89accf b045ac545f82 6b9283c82a03 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mds.h smithi135 running (3m) 84s ago 3m 20.3M - 19.0.0-2946-g3d89accf b045ac545f82 6707dedba949 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mds.i smithi193 running (3m) 84s ago 3m 19.7M - 19.0.0-2946-g3d89accf b045ac545f82 e78cfb13592a 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mds.j smithi121 running (4m) 34s ago 4m 16.9M - 19.0.0-2946-g3d89accf b045ac545f82 4bd66b898688 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mds.k smithi135 running (4m) 84s ago 4m 13.6M - 19.0.0-2946-g3d89accf b045ac545f82 e4647ca801fd 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mds.l smithi193 running (4m) 84s ago 4m 14.5M - 19.0.0-2946-g3d89accf b045ac545f82 3fe4e86aa09f 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.x smithi121 *:9283,8765 running (11m) 34s ago 11m 518M - 19.0.0-2946-g3d89accf b045ac545f82 a57b33200d7e 2024-06-10T00:27:17.682 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.y smithi135 *:8443,8765 running (9m) 84s ago 9m 454M - 19.0.0-2946-g3d89accf b045ac545f82 a4bbc28e66c7 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:mgr.z smithi193 *:8443,8765 running (9m) 84s ago 9m 454M - 19.0.0-2946-g3d89accf b045ac545f82 22997f2ae24b 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:mon.a smithi121 running (11m) 34s ago 11m 50.2M 2048M 19.0.0-2946-g3d89accf b045ac545f82 abafb9418971 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:mon.b smithi135 running (9m) 84s ago 9m 41.7M 2048M 19.0.0-2946-g3d89accf b045ac545f82 37fc9b55c2e6 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:mon.c smithi193 running (9m) 84s ago 9m 39.5M 2048M 19.0.0-2946-g3d89accf b045ac545f82 55317b68c716 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 smithi121 running (9m) 34s ago 9m 57.1M 4272M 19.0.0-2946-g3d89accf b045ac545f82 c6630c87df7f 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 smithi135 running (8m) 84s ago 8m 46.5M 4272M 19.0.0-2946-g3d89accf b045ac545f82 cd0a3fb1734d 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 smithi193 running (8m) 84s ago 8m 57.7M 4272M 19.0.0-2946-g3d89accf b045ac545f82 0448d3a1d335 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 smithi121 running (7m) 34s ago 7m 47.8M 4272M 19.0.0-2946-g3d89accf b045ac545f82 0e5f92768ccc 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 smithi135 running (7m) 84s ago 7m 43.9M 4272M 19.0.0-2946-g3d89accf b045ac545f82 dfb4c7ebd91f 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 smithi193 running (7m) 84s ago 7m 55.8M 4272M 19.0.0-2946-g3d89accf b045ac545f82 0ca640f8d67e 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6 smithi121 running (6m) 34s ago 6m 53.2M 4272M 19.0.0-2946-g3d89accf b045ac545f82 c6851f316f4a 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7 smithi135 running (6m) 84s ago 6m 44.1M 4272M 19.0.0-2946-g3d89accf b045ac545f82 a957e4720bb2 2024-06-10T00:27:17.683 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8 smithi193 running (5m) 84s ago 5m 44.1M 4272M 19.0.0-2946-g3d89accf b045ac545f82 c197cbbedb71 2024-06-10T00:27:17.684 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9 smithi121 running (5m) 34s ago 5m 45.3M 4272M 19.0.0-2946-g3d89accf b045ac545f82 ecc4e366ea0e 2024-06-10T00:27:17.684 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10 smithi135 running (5m) 84s ago 5m 58.7M 4272M 19.0.0-2946-g3d89accf b045ac545f82 9a00ffc84171 2024-06-10T00:27:17.684 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11 smithi193 running (4m) 84s ago 4m 42.5M 4272M 19.0.0-2946-g3d89accf b045ac545f82 ef2a891f6911 2024-06-10T00:27:17.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 -- 172.21.15.121:0/2350638199 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f18740a2c20 msgr2=0x7f18740a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:17.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f18740a2c20 0x7f18740a50e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1880004630 tx=0x7f188000b3c0 comp rx=0 tx=0).stop 2024-06-10T00:27:17.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 -- 172.21.15.121:0/2350638199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f188c100800 msgr2=0x7f188c19a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f188c100800 0x7f188c19a860 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f187c00c950 tx=0x7f187c00cc60 comp rx=0 tx=0).stop 2024-06-10T00:27:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 -- 172.21.15.121:0/2350638199 shutdown_connections 2024-06-10T00:27:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f18740a2c20 0x7f18740a50e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f188c101f30 0x7f188c1a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f188c100800 0x7f188c19a860 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 --2- 172.21.15.121:0/2350638199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f188c101570 0x7f188c19ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 -- 172.21.15.121:0/2350638199 >> 172.21.15.121:0/2350638199 conn(0x7f188c0fc160 msgr2=0x7f188c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 -- 172.21.15.121:0/2350638199 shutdown_connections 2024-06-10T00:27:17.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:17.684+0000 7f189197b700 1 -- 172.21.15.121:0/2350638199 wait complete. 2024-06-10T00:27:17.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:17 smithi135 ceph-mon[36424]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:18.348 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ls' 2024-06-10T00:27:18.590 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:19.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:19 smithi193 ceph-mon[36716]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:19.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:19 smithi193 ceph-mon[36716]: pgmap v408: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:19.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:19 smithi121 ceph-mon[32378]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:19.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:19 smithi121 ceph-mon[32378]: pgmap v408: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:19.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:19 smithi135 ceph-mon[36424]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:19.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:19 smithi135 ceph-mon[36424]: pgmap v408: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.304+0000 7f335ac93700 1 -- 172.21.15.121:0/3998418245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33541013d0 msgr2=0x7f33541017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.304+0000 7f335ac93700 1 --2- 172.21.15.121:0/3998418245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33541013d0 0x7f33541017f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f33480099c0 tx=0x7f3348009cd0 comp rx=0 tx=0).stop 2024-06-10T00:27:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.304+0000 7f335ac93700 1 -- 172.21.15.121:0/3998418245 shutdown_connections 2024-06-10T00:27:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.304+0000 7f335ac93700 1 --2- 172.21.15.121:0/3998418245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3354102f70 0x7f3354109820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.304+0000 7f335ac93700 1 --2- 172.21.15.121:0/3998418245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33541013d0 0x7f33541017f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.304+0000 7f335ac93700 1 --2- 172.21.15.121:0/3998418245 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33541025d0 0x7f3354102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.304+0000 7f335ac93700 1 -- 172.21.15.121:0/3998418245 >> 172.21.15.121:0/3998418245 conn(0x7f33540fc990 msgr2=0x7f33540fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.304+0000 7f335ac93700 1 -- 172.21.15.121:0/3998418245 shutdown_connections 2024-06-10T00:27:20.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.304+0000 7f335ac93700 1 -- 172.21.15.121:0/3998418245 wait complete. 2024-06-10T00:27:20.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.304+0000 7f335ac93700 1 Processor -- start 2024-06-10T00:27:20.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f335ac93700 1 -- start start 2024-06-10T00:27:20.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f335ac93700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33541013d0 0x7f33541976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:20.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f335ac93700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33541025d0 0x7f3354197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:20.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f335ac93700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3354102f70 0x7f335419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:20.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f335ac93700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f335410d030 con 0x7f33541025d0 2024-06-10T00:27:20.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f335ac93700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f335410ceb0 con 0x7f33541013d0 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f335ac93700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f335410d1b0 con 0x7f3354102f70 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f3353fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33541025d0 0x7f3354197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f3353fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33541025d0 0x7f3354197c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32784/0 (socket says 172.21.15.121:32784) 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f3353fff700 1 -- 172.21.15.121:0/2317975872 learned_addr learned my addr 172.21.15.121:0/2317975872 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f3359230700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3354102f70 0x7f335419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f3358a2f700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33541013d0 0x7f33541976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f3353fff700 1 -- 172.21.15.121:0/2317975872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3354102f70 msgr2=0x7f335419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f3353fff700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3354102f70 0x7f335419eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f3353fff700 1 -- 172.21.15.121:0/2317975872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33541013d0 msgr2=0x7f33541976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f3353fff700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33541013d0 0x7f33541976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f3353fff700 1 -- 172.21.15.121:0/2317975872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335406ace0 con 0x7f33541025d0 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.305+0000 7f3353fff700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33541025d0 0x7f3354197c20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f334000b530 tx=0x7f334000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.306+0000 7f3351ffb700 1 -- 172.21.15.121:0/2317975872 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3340015020 con 0x7f33541025d0 2024-06-10T00:27:20.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.306+0000 7f3351ffb700 1 -- 172.21.15.121:0/2317975872 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3340011d10 con 0x7f33541025d0 2024-06-10T00:27:20.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.306+0000 7f335ac93700 1 -- 172.21.15.121:0/2317975872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f335406afd0 con 0x7f33541025d0 2024-06-10T00:27:20.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.306+0000 7f3351ffb700 1 -- 172.21.15.121:0/2317975872 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334000f2e0 con 0x7f33541025d0 2024-06-10T00:27:20.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.306+0000 7f335ac93700 1 -- 172.21.15.121:0/2317975872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f335406b590 con 0x7f33541025d0 2024-06-10T00:27:20.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.307+0000 7f335ac93700 1 -- 172.21.15.121:0/2317975872 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3354069180 con 0x7f33541025d0 2024-06-10T00:27:20.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.308+0000 7f3351ffb700 1 -- 172.21.15.121:0/2317975872 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3340018020 con 0x7f33541025d0 2024-06-10T00:27:20.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.308+0000 7f3351ffb700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f33440a2c20 0x7f33440a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:20.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.308+0000 7f3358a2f700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f33440a2c20 0x7f33440a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:20.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.308+0000 7f3351ffb700 1 -- 172.21.15.121:0/2317975872 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f33400ceb10 con 0x7f33541025d0 2024-06-10T00:27:20.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.309+0000 7f3358a2f700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f33440a2c20 0x7f33440a50e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3348000c00 tx=0x7f3348005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:20.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.313+0000 7f3351ffb700 1 -- 172.21.15.121:0/2317975872 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f334001f020 con 0x7f33541025d0 2024-06-10T00:27:20.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.517+0000 7f335ac93700 1 -- 172.21.15.121:0/2317975872 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f335402dd10 con 0x7f33440a2c20 2024-06-10T00:27:20.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.523+0000 7f3351ffb700 1 -- 172.21.15.121:0/2317975872 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f335402dd10 con 0x7f33440a2c20 2024-06-10T00:27:20.523 INFO:teuthology.orchestra.run.smithi121.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-06-10T00:27:20.523 INFO:teuthology.orchestra.run.smithi121.stdout:mds.all 12/12 87s ago 4m smithi121=a;smithi121=d;smithi121=g;smithi121=j;smithi135=b;smithi135=e;smithi135=h;smithi135=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-06-10T00:27:20.523 INFO:teuthology.orchestra.run.smithi121.stdout:mgr 3/3 87s ago 9m smithi121=x;smithi135=y;smithi193=z;count:3 2024-06-10T00:27:20.523 INFO:teuthology.orchestra.run.smithi121.stdout:mon 3/3 87s ago 9m smithi121:172.21.15.121=a;smithi135:172.21.15.135=b;smithi193:172.21.15.193=c;count:3 2024-06-10T00:27:20.523 INFO:teuthology.orchestra.run.smithi121.stdout:osd 12 87s ago - 2024-06-10T00:27:20.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 -- 172.21.15.121:0/2317975872 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f33440a2c20 msgr2=0x7f33440a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:20.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f33440a2c20 0x7f33440a50e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3348000c00 tx=0x7f3348005fb0 comp rx=0 tx=0).stop 2024-06-10T00:27:20.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 -- 172.21.15.121:0/2317975872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33541025d0 msgr2=0x7f3354197c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:20.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33541025d0 0x7f3354197c20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f334000b530 tx=0x7f334000b840 comp rx=0 tx=0).stop 2024-06-10T00:27:20.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 -- 172.21.15.121:0/2317975872 shutdown_connections 2024-06-10T00:27:20.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f33440a2c20 0x7f33440a50e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:20.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3354102f70 0x7f335419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:20.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33541013d0 0x7f33541976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:20.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 --2- 172.21.15.121:0/2317975872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33541025d0 0x7f3354197c20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:20.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 -- 172.21.15.121:0/2317975872 >> 172.21.15.121:0/2317975872 conn(0x7f33540fc990 msgr2=0x7f33541058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:20.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 -- 172.21.15.121:0/2317975872 shutdown_connections 2024-06-10T00:27:20.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:20.525+0000 7f335ac93700 1 -- 172.21.15.121:0/2317975872 wait complete. 2024-06-10T00:27:21.217 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch host ls' 2024-06-10T00:27:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:21 smithi121 ceph-mon[32378]: pgmap v409: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:21.459 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:21.560 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:21 smithi135 ceph-mon[36424]: pgmap v409: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:21.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:21 smithi193 ceph-mon[36716]: pgmap v409: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:22.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:22 smithi193 ceph-mon[36716]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:22.646 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:22 smithi121 ceph-mon[32378]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:22.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:22 smithi135 ceph-mon[36424]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:23.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.203+0000 7f81fd908700 1 -- 172.21.15.121:0/4259415785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f81013f0 msgr2=0x7f81f8101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:23.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.203+0000 7f81fd908700 1 --2- 172.21.15.121:0/4259415785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f81013f0 0x7f81f8101810 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f81e00099c0 tx=0x7f81e0009cd0 comp rx=0 tx=0).stop 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.203+0000 7f81fd908700 1 -- 172.21.15.121:0/4259415785 shutdown_connections 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.203+0000 7f81fd908700 1 --2- 172.21.15.121:0/4259415785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8102f90 0x7f81f8109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.203+0000 7f81fd908700 1 --2- 172.21.15.121:0/4259415785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f81013f0 0x7f81f8101810 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.203+0000 7f81fd908700 1 --2- 172.21.15.121:0/4259415785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81f81025f0 0x7f81f8102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.203+0000 7f81fd908700 1 -- 172.21.15.121:0/4259415785 >> 172.21.15.121:0/4259415785 conn(0x7f81f80fc9b0 msgr2=0x7f81f80fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.203+0000 7f81fd908700 1 -- 172.21.15.121:0/4259415785 shutdown_connections 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.203+0000 7f81fd908700 1 -- 172.21.15.121:0/4259415785 wait complete. 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.204+0000 7f81fd908700 1 Processor -- start 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.204+0000 7f81fd908700 1 -- start start 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.204+0000 7f81fd908700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81f81013f0 0x7f81f8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.204+0000 7f81fd908700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f81025f0 0x7f81f8198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:23.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.204+0000 7f81fd908700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8102f90 0x7f81f819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.204+0000 7f81fd908700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f81f810cfb0 con 0x7f81f81025f0 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.204+0000 7f81fd908700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f81f810ce30 con 0x7f81f81013f0 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.204+0000 7f81fd908700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f81f810d130 con 0x7f81f8102f90 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.204+0000 7f81f67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f81025f0 0x7f81f8198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.204+0000 7f81f67fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f81025f0 0x7f81f8198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32814/0 (socket says 172.21.15.121:32814) 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.204+0000 7f81f67fc700 1 -- 172.21.15.121:0/592440996 learned_addr learned my addr 172.21.15.121:0/592440996 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81f6ffd700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81f81013f0 0x7f81f8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81f77fe700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8102f90 0x7f81f819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81f67fc700 1 -- 172.21.15.121:0/592440996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8102f90 msgr2=0x7f81f819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81f67fc700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8102f90 0x7f81f819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81f67fc700 1 -- 172.21.15.121:0/592440996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81f81013f0 msgr2=0x7f81f8198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81f67fc700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81f81013f0 0x7f81f8198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81f67fc700 1 -- 172.21.15.121:0/592440996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f81a0390 con 0x7f81f81025f0 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81f67fc700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f81025f0 0x7f81f8198c70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f81e800d980 tx=0x7f81e800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:23.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81fc906700 1 -- 172.21.15.121:0/592440996 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e8016070 con 0x7f81f81025f0 2024-06-10T00:27:23.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81fc906700 1 -- 172.21.15.121:0/592440996 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81e8009db0 con 0x7f81f81025f0 2024-06-10T00:27:23.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81fd908700 1 -- 172.21.15.121:0/592440996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81f81a0680 con 0x7f81f81025f0 2024-06-10T00:27:23.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81fc906700 1 -- 172.21.15.121:0/592440996 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e800f3c0 con 0x7f81f81025f0 2024-06-10T00:27:23.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.205+0000 7f81fd908700 1 -- 172.21.15.121:0/592440996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81f81a0b70 con 0x7f81f81025f0 2024-06-10T00:27:23.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.206+0000 7f81fd908700 1 -- 172.21.15.121:0/592440996 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81f8069180 con 0x7f81f81025f0 2024-06-10T00:27:23.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.208+0000 7f81fc906700 1 -- 172.21.15.121:0/592440996 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f81e8019070 con 0x7f81f81025f0 2024-06-10T00:27:23.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.209+0000 7f81fc906700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f81e40a2c20 0x7f81e40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:23.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.209+0000 7f81f6ffd700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f81e40a2c20 0x7f81e40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:23.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.209+0000 7f81fc906700 1 -- 172.21.15.121:0/592440996 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f81e80cee90 con 0x7f81f81025f0 2024-06-10T00:27:23.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.209+0000 7f81f6ffd700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f81e40a2c20 0x7f81e40a50e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f81e000b580 tx=0x7f81e0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:23.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.212+0000 7f81fc906700 1 -- 172.21.15.121:0/592440996 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f81e809ba40 con 0x7f81f81025f0 2024-06-10T00:27:23.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.415+0000 7f81fd908700 1 -- 172.21.15.121:0/592440996 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f81f806b270 con 0x7f81e40a2c20 2024-06-10T00:27:23.416 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.416+0000 7f81fc906700 1 -- 172.21.15.121:0/592440996 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f81f806b270 con 0x7f81e40a2c20 2024-06-10T00:27:23.416 INFO:teuthology.orchestra.run.smithi121.stdout:HOST ADDR LABELS STATUS 2024-06-10T00:27:23.416 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121 172.21.15.121 2024-06-10T00:27:23.416 INFO:teuthology.orchestra.run.smithi121.stdout:smithi135 172.21.15.135 2024-06-10T00:27:23.416 INFO:teuthology.orchestra.run.smithi121.stdout:smithi193 172.21.15.193 2024-06-10T00:27:23.416 INFO:teuthology.orchestra.run.smithi121.stdout:3 hosts in cluster 2024-06-10T00:27:23.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 -- 172.21.15.121:0/592440996 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f81e40a2c20 msgr2=0x7f81e40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:23.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f81e40a2c20 0x7f81e40a50e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f81e000b580 tx=0x7f81e0005fb0 comp rx=0 tx=0).stop 2024-06-10T00:27:23.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 -- 172.21.15.121:0/592440996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f81025f0 msgr2=0x7f81f8198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:23.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f81025f0 0x7f81f8198c70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f81e800d980 tx=0x7f81e800dc90 comp rx=0 tx=0).stop 2024-06-10T00:27:23.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 -- 172.21.15.121:0/592440996 shutdown_connections 2024-06-10T00:27:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f81e40a2c20 0x7f81e40a50e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8102f90 0x7f81f819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f81f81013f0 0x7f81f8198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 --2- 172.21.15.121:0/592440996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81f81025f0 0x7f81f8198c70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 -- 172.21.15.121:0/592440996 >> 172.21.15.121:0/592440996 conn(0x7f81f80fc9b0 msgr2=0x7f81f806ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 -- 172.21.15.121:0/592440996 shutdown_connections 2024-06-10T00:27:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:23.418+0000 7f81fd908700 1 -- 172.21.15.121:0/592440996 wait complete. 2024-06-10T00:27:23.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:23 smithi193 ceph-mon[36716]: pgmap v410: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:23.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:23 smithi121 ceph-mon[32378]: pgmap v410: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:23.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:23 smithi135 ceph-mon[36424]: pgmap v410: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:24.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch device ls' 2024-06-10T00:27:24.367 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:25.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:25 smithi121 ceph-mon[32378]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:25.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:25 smithi121 ceph-mon[32378]: pgmap v411: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:25.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:25 smithi193 ceph-mon[36716]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:25.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:25 smithi193 ceph-mon[36716]: pgmap v411: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:25.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:25 smithi135 ceph-mon[36424]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:25.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:25 smithi135 ceph-mon[36424]: pgmap v411: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.110+0000 7f25c225c700 1 -- 172.21.15.121:0/857500616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25bc104090 msgr2=0x7f25bc10a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.110+0000 7f25c225c700 1 --2- 172.21.15.121:0/857500616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25bc104090 0x7f25bc10a940 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f25b400b0d0 tx=0x7f25b400b3e0 comp rx=0 tx=0).stop 2024-06-10T00:27:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.110+0000 7f25c225c700 1 -- 172.21.15.121:0/857500616 shutdown_connections 2024-06-10T00:27:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.110+0000 7f25c225c700 1 --2- 172.21.15.121:0/857500616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25bc104090 0x7f25bc10a940 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.110+0000 7f25c225c700 1 --2- 172.21.15.121:0/857500616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25bc1024f0 0x7f25bc102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.110+0000 7f25c225c700 1 --2- 172.21.15.121:0/857500616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc1036f0 0x7f25bc103b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.110+0000 7f25c225c700 1 -- 172.21.15.121:0/857500616 >> 172.21.15.121:0/857500616 conn(0x7f25bc0fda90 msgr2=0x7f25bc0ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.110+0000 7f25c225c700 1 -- 172.21.15.121:0/857500616 shutdown_connections 2024-06-10T00:27:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.110+0000 7f25c225c700 1 -- 172.21.15.121:0/857500616 wait complete. 2024-06-10T00:27:26.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25c225c700 1 Processor -- start 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25c225c700 1 -- start start 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25c225c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc1024f0 0x7f25bc198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25c225c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25bc1036f0 0x7f25bc198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25c225c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25bc104090 0x7f25bc19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25c225c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc10e0c0 con 0x7f25bc1036f0 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25c225c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc10df40 con 0x7f25bc104090 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25c225c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc10e240 con 0x7f25bc1024f0 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25b2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25bc1036f0 0x7f25bc198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25b2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25bc1036f0 0x7f25bc198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32830/0 (socket says 172.21.15.121:32830) 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25b2ffd700 1 -- 172.21.15.121:0/729240926 learned_addr learned my addr 172.21.15.121:0/729240926 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25bb7fe700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc1024f0 0x7f25bc198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:26.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25bbfff700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25bc104090 0x7f25bc19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:26.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.111+0000 7f25b2ffd700 1 -- 172.21.15.121:0/729240926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc1024f0 msgr2=0x7f25bc198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:26.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.112+0000 7f25b2ffd700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc1024f0 0x7f25bc198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:26.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.112+0000 7f25b2ffd700 1 -- 172.21.15.121:0/729240926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25bc104090 msgr2=0x7f25bc19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:26.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.112+0000 7f25b2ffd700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25bc104090 0x7f25bc19fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:26.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.112+0000 7f25b2ffd700 1 -- 172.21.15.121:0/729240926 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25bc1a02e0 con 0x7f25bc1036f0 2024-06-10T00:27:26.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.112+0000 7f25bbfff700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25bc104090 0x7f25bc19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:26.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.112+0000 7f25b2ffd700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25bc1036f0 0x7f25bc198bc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f25a400d980 tx=0x7f25a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:26.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.112+0000 7f25b97fa700 1 -- 172.21.15.121:0/729240926 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a4016070 con 0x7f25bc1036f0 2024-06-10T00:27:26.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.112+0000 7f25b97fa700 1 -- 172.21.15.121:0/729240926 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25a400de70 con 0x7f25bc1036f0 2024-06-10T00:27:26.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.112+0000 7f25c225c700 1 -- 172.21.15.121:0/729240926 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25bc1a05d0 con 0x7f25bc1036f0 2024-06-10T00:27:26.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.112+0000 7f25b97fa700 1 -- 172.21.15.121:0/729240926 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a400f3c0 con 0x7f25bc1036f0 2024-06-10T00:27:26.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.112+0000 7f25c225c700 1 -- 172.21.15.121:0/729240926 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25bc1a0b60 con 0x7f25bc1036f0 2024-06-10T00:27:26.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.113+0000 7f25c225c700 1 -- 172.21.15.121:0/729240926 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25bc069180 con 0x7f25bc1036f0 2024-06-10T00:27:26.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.115+0000 7f25b97fa700 1 -- 172.21.15.121:0/729240926 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f25a4019070 con 0x7f25bc1036f0 2024-06-10T00:27:26.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.116+0000 7f25b97fa700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f25a80a2ca0 0x7f25a80a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:26.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.116+0000 7f25bb7fe700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f25a80a2ca0 0x7f25a80a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:26.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.116+0000 7f25b97fa700 1 -- 172.21.15.121:0/729240926 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f25a40cee50 con 0x7f25bc1036f0 2024-06-10T00:27:26.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.116+0000 7f25bb7fe700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f25a80a2ca0 0x7f25a80a5160 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f25bc103550 tx=0x7f25ac009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:26.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.118+0000 7f25b97fa700 1 -- 172.21.15.121:0/729240926 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f25a409ba00 con 0x7f25bc1036f0 2024-06-10T00:27:26.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.326+0000 7f25c225c700 1 -- 172.21.15.121:0/729240926 --> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f25bc0628d0 con 0x7f25a80a2ca0 2024-06-10T00:27:26.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.328+0000 7f25b97fa700 1 -- 172.21.15.121:0/729240926 <== mgr.14150 v2:172.21.15.121:6800/3506007074 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f25bc0628d0 con 0x7f25a80a2ca0 2024-06-10T00:27:26.328 INFO:teuthology.orchestra.run.smithi121.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-06-10T00:27:26.328 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZF400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-06-10T00:27:26.328 INFO:teuthology.orchestra.run.smithi121.stdout:smithi135 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620401BA400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-06-10T00:27:26.328 INFO:teuthology.orchestra.run.smithi121.stdout:smithi193 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500HB400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-06-10T00:27:26.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 -- 172.21.15.121:0/729240926 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f25a80a2ca0 msgr2=0x7f25a80a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:26.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f25a80a2ca0 0x7f25a80a5160 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f25bc103550 tx=0x7f25ac009260 comp rx=0 tx=0).stop 2024-06-10T00:27:26.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 -- 172.21.15.121:0/729240926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25bc1036f0 msgr2=0x7f25bc198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:26.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25bc1036f0 0x7f25bc198bc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f25a400d980 tx=0x7f25a400dc90 comp rx=0 tx=0).stop 2024-06-10T00:27:26.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 -- 172.21.15.121:0/729240926 shutdown_connections 2024-06-10T00:27:26.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f25a80a2ca0 0x7f25a80a5160 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:26.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f25bc104090 0x7f25bc19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:26.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25bc1024f0 0x7f25bc198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:26.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 --2- 172.21.15.121:0/729240926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25bc1036f0 0x7f25bc198bc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:26.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 -- 172.21.15.121:0/729240926 >> 172.21.15.121:0/729240926 conn(0x7f25bc0fda90 msgr2=0x7f25bc1069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:26.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 -- 172.21.15.121:0/729240926 shutdown_connections 2024-06-10T00:27:26.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:26.330+0000 7f25c225c700 1 -- 172.21.15.121:0/729240926 wait complete. 2024-06-10T00:27:27.050 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-10T00:27:27.058 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi121.front.sepia.ceph.com 2024-06-10T00:27:27.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- bash -c 'ceph fs dump' 2024-06-10T00:27:27.300 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:27.559 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:27 smithi121 ceph-mon[32378]: pgmap v412: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:27 smithi193 ceph-mon[36716]: pgmap v412: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:27.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:27 smithi135 ceph-mon[36424]: pgmap v412: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:28.440 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:28 smithi121 ceph-mon[32378]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:28.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:28 smithi193 ceph-mon[36716]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:28.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:28 smithi135 ceph-mon[36424]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:27:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.113+0000 7f491fbda700 1 -- 172.21.15.121:0/3213542601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4918102f80 msgr2=0x7f4918109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.113+0000 7f491fbda700 1 --2- 172.21.15.121:0/3213542601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4918102f80 0x7f4918109830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f491000b0d0 tx=0x7f491000b3e0 comp rx=0 tx=0).stop 2024-06-10T00:27:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.113+0000 7f491fbda700 1 -- 172.21.15.121:0/3213542601 shutdown_connections 2024-06-10T00:27:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.113+0000 7f491fbda700 1 --2- 172.21.15.121:0/3213542601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4918102f80 0x7f4918109830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.113+0000 7f491fbda700 1 --2- 172.21.15.121:0/3213542601 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49181013e0 0x7f4918101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:29.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.113+0000 7f491fbda700 1 --2- 172.21.15.121:0/3213542601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49181025e0 0x7f4918102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.113+0000 7f491fbda700 1 -- 172.21.15.121:0/3213542601 >> 172.21.15.121:0/3213542601 conn(0x7f49180fc980 msgr2=0x7f49180fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.113+0000 7f491fbda700 1 -- 172.21.15.121:0/3213542601 shutdown_connections 2024-06-10T00:27:29.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.113+0000 7f491fbda700 1 -- 172.21.15.121:0/3213542601 wait complete. 2024-06-10T00:27:29.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491fbda700 1 Processor -- start 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491fbda700 1 -- start start 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491fbda700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49181013e0 0x7f491806b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491fbda700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49181025e0 0x7f491819d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491fbda700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4918102f80 0x7f491819f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491fbda700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f491810cfc0 con 0x7f4918102f80 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491fbda700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f491810ce40 con 0x7f49181013e0 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491fbda700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f491810d140 con 0x7f49181025e0 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491d976700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49181013e0 0x7f491806b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491d976700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49181013e0 0x7f491806b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:39472/0 (socket says 172.21.15.121:39472) 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491d976700 1 -- 172.21.15.121:0/2848390006 learned_addr learned my addr 172.21.15.121:0/2848390006 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491d175700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49181025e0 0x7f491819d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:29.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.114+0000 7f491e177700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4918102f80 0x7f491819f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f491d976700 1 -- 172.21.15.121:0/2848390006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49181025e0 msgr2=0x7f491819d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f491d976700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49181025e0 0x7f491819d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f491d976700 1 -- 172.21.15.121:0/2848390006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4918102f80 msgr2=0x7f491819f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f491d976700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4918102f80 0x7f491819f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f491d976700 1 -- 172.21.15.121:0/2848390006 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49181a02b0 con 0x7f49181013e0 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f491e177700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4918102f80 0x7f491819f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f491d976700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49181013e0 0x7f491806b560 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f491400e950 tx=0x7f491400ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f490effd700 1 -- 172.21.15.121:0/2848390006 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4914004230 con 0x7f49181013e0 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f491fbda700 1 -- 172.21.15.121:0/2848390006 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49181a05a0 con 0x7f49181013e0 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f490effd700 1 -- 172.21.15.121:0/2848390006 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49140043d0 con 0x7f49181013e0 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f491fbda700 1 -- 172.21.15.121:0/2848390006 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49181a0ae0 con 0x7f49181013e0 2024-06-10T00:27:29.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.115+0000 7f490effd700 1 -- 172.21.15.121:0/2848390006 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4914003dd0 con 0x7f49181013e0 2024-06-10T00:27:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.116+0000 7f491fbda700 1 -- 172.21.15.121:0/2848390006 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4918069180 con 0x7f49181013e0 2024-06-10T00:27:29.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.118+0000 7f490effd700 1 -- 172.21.15.121:0/2848390006 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f491400c7e0 con 0x7f49181013e0 2024-06-10T00:27:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.118+0000 7f490effd700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49040a2ca0 0x7f49040a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.119+0000 7f491d175700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49040a2ca0 0x7f49040a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.119+0000 7f490effd700 1 -- 172.21.15.121:0/2848390006 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f49140cf6d0 con 0x7f49181013e0 2024-06-10T00:27:29.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.119+0000 7f491d175700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49040a2ca0 0x7f49040a5160 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f491806bae0 tx=0x7f490800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:29.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.122+0000 7f490effd700 1 -- 172.21.15.121:0/2848390006 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f491409c220 con 0x7f49181013e0 2024-06-10T00:27:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.373+0000 7f491fbda700 1 -- 172.21.15.121:0/2848390006 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f49180630a0 con 0x7f49181013e0 2024-06-10T00:27:29.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.373+0000 7f490effd700 1 -- 172.21.15.121:0/2848390006 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3060 (secure 0 0 0) 0x7f491409bb20 con 0x7f49181013e0 2024-06-10T00:27:29.375 INFO:teuthology.orchestra.run.smithi121.stdout:e25 2024-06-10T00:27:29.375 INFO:teuthology.orchestra.run.smithi121.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-06-10T00:27:29.375 INFO:teuthology.orchestra.run.smithi121.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-10T00:27:29.375 INFO:teuthology.orchestra.run.smithi121.stdout:legacy client fscid: 1 2024-06-10T00:27:29.375 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:27:29.375 INFO:teuthology.orchestra.run.smithi121.stdout:Filesystem 'cephfs' (1) 2024-06-10T00:27:29.375 INFO:teuthology.orchestra.run.smithi121.stdout:fs_name cephfs 2024-06-10T00:27:29.375 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 25 2024-06-10T00:27:29.375 INFO:teuthology.orchestra.run.smithi121.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-06-10T00:27:29.375 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-06-10T00:23:59.392036+0000 2024-06-10T00:27:29.375 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-06-10T00:25:08.157143+0000 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:tableserver 0 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:root 0 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:session_timeout 300 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:session_autoclose 300 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:max_file_size 1099511627776 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:max_xattr_size 65536 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:required_client_features {} 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure 0 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:last_failure_osd_epoch 0 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:max_mds 3 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:in 0,1,2 2024-06-10T00:27:29.376 INFO:teuthology.orchestra.run.smithi121.stdout:up {0=24479,1=24472,2=24455} 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:failed 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:damaged 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:stopped 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:data_pools [3,4] 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:metadata_pool 2 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:inline_data disabled 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:balancer 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:bal_rank_mask -1 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:standby_count_wanted 0 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:qdb_cluster leader: 24479 members: 24455,24479,24472 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b{0:24479} state up:active seq 6 addr [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c{0:24448} state up:standby-replay seq 1 addr [v2:172.21.15.193:6836/2863018656,v1:172.21.15.193:6837/2863018656] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.i{1:24472} state up:active seq 24 addr [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.377 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.f{1:24442} state up:standby-replay seq 1 addr [v2:172.21.15.193:6834/3023450024,v1:172.21.15.193:6835/3023450024] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.h{2:24455} state up:active seq 25 addr [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.e{2:24431} state up:standby-replay seq 1 addr [v2:172.21.15.135:6834/284701222,v1:172.21.15.135:6835/284701222] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout:Standby daemons: 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.j{-1:14514} state up:standby seq 1 addr [v2:172.21.15.121:6834/4144871914,v1:172.21.15.121:6835/4144871914] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d{-1:14526} state up:standby seq 1 addr [v2:172.21.15.121:6836/64109840,v1:172.21.15.121:6837/64109840] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.g{-1:14571} state up:standby seq 1 addr [v2:172.21.15.121:6838/4288408975,v1:172.21.15.121:6839/4288408975] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a{-1:14583} state up:standby seq 1 addr [v2:172.21.15.121:6840/691733071,v1:172.21.15.121:6841/691733071] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.l{-1:24412} state up:standby seq 1 addr [v2:172.21.15.193:6832/360772282,v1:172.21.15.193:6833/360772282] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.378 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.k{-1:24413} state up:standby seq 1 addr [v2:172.21.15.135:6832/1842796932,v1:172.21.15.135:6833/1842796932] compat {c=[1],r=[1],i=[1fff]}] 2024-06-10T00:27:29.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 -- 172.21.15.121:0/2848390006 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49040a2ca0 msgr2=0x7f49040a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:29.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49040a2ca0 0x7f49040a5160 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f491806bae0 tx=0x7f490800b410 comp rx=0 tx=0).stop 2024-06-10T00:27:29.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 -- 172.21.15.121:0/2848390006 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49181013e0 msgr2=0x7f491806b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:29.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49181013e0 0x7f491806b560 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f491400e950 tx=0x7f491400ec60 comp rx=0 tx=0).stop 2024-06-10T00:27:29.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 -- 172.21.15.121:0/2848390006 shutdown_connections 2024-06-10T00:27:29.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f49040a2ca0 0x7f49040a5160 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:29.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4918102f80 0x7f491819f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:29.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f49181013e0 0x7f491806b560 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:29.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 --2- 172.21.15.121:0/2848390006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49181025e0 0x7f491819d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:29.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 -- 172.21.15.121:0/2848390006 >> 172.21.15.121:0/2848390006 conn(0x7f49180fc980 msgr2=0x7f4918105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:29.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 -- 172.21.15.121:0/2848390006 shutdown_connections 2024-06-10T00:27:29.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:29.376+0000 7f491fbda700 1 -- 172.21.15.121:0/2848390006 wait complete. 2024-06-10T00:27:29.380 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 25 2024-06-10T00:27:29.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:29 smithi193 ceph-mon[36716]: pgmap v413: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:29.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:29 smithi121 ceph-mon[32378]: pgmap v413: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:29.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:29 smithi135 ceph-mon[36424]: pgmap v413: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:30.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- bash -c 'ceph osd dump' 2024-06-10T00:27:30.345 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:30 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2848390006' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-06-10T00:27:30.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:30 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2848390006' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-06-10T00:27:30.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:30 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2848390006' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-06-10T00:27:31.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:31 smithi193 ceph-mon[36716]: pgmap v414: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:31.644 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:31 smithi121 ceph-mon[32378]: pgmap v414: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:31.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:31 smithi135 ceph-mon[36424]: pgmap v414: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:32.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.007+0000 7f3e0869b700 1 -- 172.21.15.121:0/2891175788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00102fb0 msgr2=0x7f3e00109860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:32.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.007+0000 7f3e0869b700 1 --2- 172.21.15.121:0/2891175788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00102fb0 0x7f3e00109860 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3df800b0d0 tx=0x7f3df800b3e0 comp rx=0 tx=0).stop 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.007+0000 7f3e0869b700 1 -- 172.21.15.121:0/2891175788 shutdown_connections 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.007+0000 7f3e0869b700 1 --2- 172.21.15.121:0/2891175788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00102fb0 0x7f3e00109860 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.007+0000 7f3e0869b700 1 --2- 172.21.15.121:0/2891175788 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e00101410 0x7f3e00101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.007+0000 7f3e0869b700 1 --2- 172.21.15.121:0/2891175788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e00102610 0x7f3e00102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.007+0000 7f3e0869b700 1 -- 172.21.15.121:0/2891175788 >> 172.21.15.121:0/2891175788 conn(0x7f3e000fc9f0 msgr2=0x7f3e000fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.007+0000 7f3e0869b700 1 -- 172.21.15.121:0/2891175788 shutdown_connections 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.007+0000 7f3e0869b700 1 -- 172.21.15.121:0/2891175788 wait complete. 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.007+0000 7f3e0869b700 1 Processor -- start 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.007+0000 7f3e0869b700 1 -- start start 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e0869b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e00101410 0x7f3e00198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e0869b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e00102610 0x7f3e00198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:32.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e0869b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00102fb0 0x7f3e0019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e0869b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0010d090 con 0x7f3e00102fb0 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e0869b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0010cf10 con 0x7f3e00102610 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e0869b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0010d210 con 0x7f3e00101410 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e06c38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00102fb0 0x7f3e0019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e06c38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00102fb0 0x7f3e0019fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39242/0 (socket says 172.21.15.121:39242) 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e06c38700 1 -- 172.21.15.121:0/3323022892 learned_addr learned my addr 172.21.15.121:0/3323022892 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e06437700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e00101410 0x7f3e00198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e05c36700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e00102610 0x7f3e00198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e06c38700 1 -- 172.21.15.121:0/3323022892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e00101410 msgr2=0x7f3e00198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e06c38700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e00101410 0x7f3e00198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e06c38700 1 -- 172.21.15.121:0/3323022892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e00102610 msgr2=0x7f3e00198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e06c38700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e00102610 0x7f3e00198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:32.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e06c38700 1 -- 172.21.15.121:0/3323022892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e001a0370 con 0x7f3e00102fb0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e06c38700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00102fb0 0x7f3e0019fcd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3df8007d10 tx=0x7f3df8007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3df77fe700 1 -- 172.21.15.121:0/3323022892 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8021020 con 0x7f3e00102fb0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3df77fe700 1 -- 172.21.15.121:0/3323022892 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3df801d950 con 0x7f3e00102fb0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3e0869b700 1 -- 172.21.15.121:0/3323022892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e001a0600 con 0x7f3e00102fb0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.008+0000 7f3df77fe700 1 -- 172.21.15.121:0/3323022892 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8005450 con 0x7f3e00102fb0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.009+0000 7f3e0869b700 1 -- 172.21.15.121:0/3323022892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e001a0bc0 con 0x7f3e00102fb0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.009+0000 7f3e0869b700 1 -- 172.21.15.121:0/3323022892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e00069180 con 0x7f3e00102fb0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.011+0000 7f3df77fe700 1 -- 172.21.15.121:0/3323022892 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f3df8024020 con 0x7f3e00102fb0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.011+0000 7f3df77fe700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3dec0a2c20 0x7f3dec0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.011+0000 7f3e06437700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3dec0a2c20 0x7f3dec0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.011+0000 7f3df77fe700 1 -- 172.21.15.121:0/3323022892 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f3df800f020 con 0x7f3e00102fb0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.012+0000 7f3e06437700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3dec0a2c20 0x7f3dec0a50e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3e00102470 tx=0x7f3dfc009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:32.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.015+0000 7f3df77fe700 1 -- 172.21.15.121:0/3323022892 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3df809da00 con 0x7f3e00102fb0 2024-06-10T00:27:32.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.212+0000 7f3e0869b700 1 -- 172.21.15.121:0/3323022892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f3e001a0fe0 con 0x7f3e00102fb0 2024-06-10T00:27:32.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.213+0000 7f3df77fe700 1 -- 172.21.15.121:0/3323022892 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v100) ==== 56+0+5707 (secure 0 0 0) 0x7f3df809d300 con 0x7f3e00102fb0 2024-06-10T00:27:32.214 INFO:teuthology.orchestra.run.smithi121.stdout:epoch 100 2024-06-10T00:27:32.214 INFO:teuthology.orchestra.run.smithi121.stdout:fsid 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:27:32.214 INFO:teuthology.orchestra.run.smithi121.stdout:created 2024-06-10T00:15:42.548845+0000 2024-06-10T00:27:32.214 INFO:teuthology.orchestra.run.smithi121.stdout:modified 2024-06-10T00:25:44.074803+0000 2024-06-10T00:27:32.214 INFO:teuthology.orchestra.run.smithi121.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-06-10T00:27:32.214 INFO:teuthology.orchestra.run.smithi121.stdout:crush_version 27 2024-06-10T00:27:32.214 INFO:teuthology.orchestra.run.smithi121.stdout:full_ratio 0.95 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:backfillfull_ratio 0.9 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:nearfull_ratio 0.85 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:require_min_compat_client luminous 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:min_compat_client luminous 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:require_osd_release squid 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:stretch_mode_enabled false 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 89 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.69 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 96 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:max_osd 12 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:osd.0 up in weight 1 up_from 8 up_thru 99 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6802/1987481040,v1:172.21.15.121:6803/1987481040] [v2:172.21.15.121:6804/1987481040,v1:172.21.15.121:6805/1987481040] exists,up d80f9176-d911-4798-98cd-02587aa53bb5 2024-06-10T00:27:32.215 INFO:teuthology.orchestra.run.smithi121.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.135:6800/1153602323,v1:172.21.15.135:6801/1153602323] [v2:172.21.15.135:6802/1153602323,v1:172.21.15.135:6803/1153602323] exists,up 3887c2a6-d2b6-43e9-9632-c73154511128 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.2 up in weight 1 up_from 18 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6800/2430648346,v1:172.21.15.193:6801/2430648346] [v2:172.21.15.193:6802/2430648346,v1:172.21.15.193:6803/2430648346] exists,up d37465a0-81c9-499f-a0f3-fe5cdc1538ab 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.3 up in weight 1 up_from 26 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6810/1510255067,v1:172.21.15.121:6811/1510255067] [v2:172.21.15.121:6812/1510255067,v1:172.21.15.121:6813/1510255067] exists,up 1e4414c2-8c5d-4b4a-895e-b3d0157ab36a 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.4 up in weight 1 up_from 34 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.135:6808/759537459,v1:172.21.15.135:6809/759537459] [v2:172.21.15.135:6810/759537459,v1:172.21.15.135:6811/759537459] exists,up 5a6e3f25-d136-4698-8657-ca62ff8d4ffe 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.5 up in weight 1 up_from 39 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6808/3919206206,v1:172.21.15.193:6809/3919206206] [v2:172.21.15.193:6810/3919206206,v1:172.21.15.193:6811/3919206206] exists,up 62361748-4968-4095-9e32-756c6cfdfaf0 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.6 up in weight 1 up_from 47 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6818/2905970681,v1:172.21.15.121:6819/2905970681] [v2:172.21.15.121:6820/2905970681,v1:172.21.15.121:6821/2905970681] exists,up 3541db4e-839e-4ad2-bd35-2566a649d87f 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.7 up in weight 1 up_from 52 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.135:6816/3426631559,v1:172.21.15.135:6817/3426631559] [v2:172.21.15.135:6818/3426631559,v1:172.21.15.135:6819/3426631559] exists,up 3979b209-73a0-4891-85dc-b7aad76f0ff0 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.8 up in weight 1 up_from 60 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6816/3981402186,v1:172.21.15.193:6817/3981402186] [v2:172.21.15.193:6818/3981402186,v1:172.21.15.193:6819/3981402186] exists,up 6a099d63-2664-4c39-a992-2095f55cdfa7 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.9 up in weight 1 up_from 68 up_thru 99 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6826/3736625801,v1:172.21.15.121:6827/3736625801] [v2:172.21.15.121:6828/3736625801,v1:172.21.15.121:6829/3736625801] exists,up 109bd90d-d878-46b2-99c3-af0792345523 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.10 up in weight 1 up_from 73 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.135:6824/937452287,v1:172.21.15.135:6825/937452287] [v2:172.21.15.135:6826/937452287,v1:172.21.15.135:6827/937452287] exists,up 66e56ceb-7b01-4bec-9d2d-bff37873aa75 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:osd.11 up in weight 1 up_from 81 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6824/3765048609,v1:172.21.15.193:6825/3765048609] [v2:172.21.15.193:6826/3765048609,v1:172.21.15.193:6827/3765048609] exists,up 3b8b392a-5b98-4b28-8301-8ceecd572573 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 2.5 [8,11] 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 2.a [2,11] 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 3.0 [5,9] 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 3.2 [3,9] 2024-06-10T00:27:32.216 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 3.6 [3,8] 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 3.1b [4,6] 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 3.1d [4,8] 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 3.24 [4,9] 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 3.28 [5,7] 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 4.e [5,8] 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 4.16 [9,8] 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 4.17 [9,10] 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:pg_upmap_items 4.1f [9,8] 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:0/1710032094 expires 2024-06-11T00:16:42.011037+0000 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:6801/704666019 expires 2024-06-11T00:16:42.011037+0000 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:0/986214985 expires 2024-06-11T00:16:42.011037+0000 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:6801/3588432900 expires 2024-06-11T00:16:15.442852+0000 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:6800/3588432900 expires 2024-06-11T00:16:15.442852+0000 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:6800/704666019 expires 2024-06-11T00:16:42.011037+0000 2024-06-10T00:27:32.217 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:0/743919928 expires 2024-06-11T00:16:42.011037+0000 2024-06-10T00:27:32.218 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:0/2834833132 expires 2024-06-11T00:16:15.442852+0000 2024-06-10T00:27:32.218 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:0/1390094598 expires 2024-06-11T00:16:15.442852+0000 2024-06-10T00:27:32.218 INFO:teuthology.orchestra.run.smithi121.stdout:blocklist 172.21.15.121:0/1651500994 expires 2024-06-11T00:16:15.442852+0000 2024-06-10T00:27:32.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 -- 172.21.15.121:0/3323022892 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3dec0a2c20 msgr2=0x7f3dec0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:32.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3dec0a2c20 0x7f3dec0a50e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3e00102470 tx=0x7f3dfc009190 comp rx=0 tx=0).stop 2024-06-10T00:27:32.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 -- 172.21.15.121:0/3323022892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00102fb0 msgr2=0x7f3e0019fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:32.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00102fb0 0x7f3e0019fcd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3df8007d10 tx=0x7f3df8007d40 comp rx=0 tx=0).stop 2024-06-10T00:27:32.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 -- 172.21.15.121:0/3323022892 shutdown_connections 2024-06-10T00:27:32.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f3dec0a2c20 0x7f3dec0a50e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:32.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e00102fb0 0x7f3e0019fcd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:32.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e00101410 0x7f3e00198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:32.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 --2- 172.21.15.121:0/3323022892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3e00102610 0x7f3e00198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:32.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 -- 172.21.15.121:0/3323022892 >> 172.21.15.121:0/3323022892 conn(0x7f3e000fc9f0 msgr2=0x7f3e000fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:32.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 -- 172.21.15.121:0/3323022892 shutdown_connections 2024-06-10T00:27:32.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:32.215+0000 7f3e0869b700 1 -- 172.21.15.121:0/3323022892 wait complete. 2024-06-10T00:27:32.525 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:32 smithi121 ceph-mon[32378]: pgmap v415: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:32.525 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:32 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3323022892' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-10T00:27:32.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:32 smithi193 ceph-mon[36716]: pgmap v415: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:32.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:32 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3323022892' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-10T00:27:32.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:32 smithi135 ceph-mon[36424]: pgmap v415: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:32.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:32 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3323022892' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-06-10T00:27:32.873 INFO:teuthology.run_tasks:Running task fs.ready... 2024-06-10T00:27:32.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:27:33.125 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:34.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.806+0000 7f7075dd7700 1 -- 172.21.15.121:0/1342884676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7070107860 msgr2=0x7f707010f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:34.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.806+0000 7f7075dd7700 1 --2- 172.21.15.121:0/1342884676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7070107860 0x7f707010f3e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f70600099c0 tx=0x7f7060009cd0 comp rx=0 tx=0).stop 2024-06-10T00:27:34.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.806+0000 7f7075dd7700 1 -- 172.21.15.121:0/1342884676 shutdown_connections 2024-06-10T00:27:34.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.806+0000 7f7075dd7700 1 --2- 172.21.15.121:0/1342884676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7070107860 0x7f707010f3e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:34.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.806+0000 7f7075dd7700 1 --2- 172.21.15.121:0/1342884676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7070102770 0x7f7070102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:34.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.806+0000 7f7075dd7700 1 --2- 172.21.15.121:0/1342884676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7070103120 0x7f7070107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:34.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.806+0000 7f7075dd7700 1 -- 172.21.15.121:0/1342884676 >> 172.21.15.121:0/1342884676 conn(0x7f7070079600 msgr2=0x7f7070079a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:34.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.806+0000 7f7075dd7700 1 -- 172.21.15.121:0/1342884676 shutdown_connections 2024-06-10T00:27:34.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.806+0000 7f7075dd7700 1 -- 172.21.15.121:0/1342884676 wait complete. 2024-06-10T00:27:34.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.807+0000 7f7075dd7700 1 Processor -- start 2024-06-10T00:27:34.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.807+0000 7f7075dd7700 1 -- start start 2024-06-10T00:27:34.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.807+0000 7f7075dd7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7070102770 0x7f7070074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:34.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.807+0000 7f7075dd7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7070103120 0x7f7070074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:34.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.807+0000 7f7075dd7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7070107860 0x7f7070075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:34.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.807+0000 7f7075dd7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f70701127d0 con 0x7f7070103120 2024-06-10T00:27:34.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.807+0000 7f7075dd7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f7070112650 con 0x7f7070107860 2024-06-10T00:27:34.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.807+0000 7f7075dd7700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7070112950 con 0x7f7070102770 2024-06-10T00:27:34.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.807+0000 7f706effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7070103120 0x7f7070074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:34.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.807+0000 7f706effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7070103120 0x7f7070074cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39252/0 (socket says 172.21.15.121:39252) 2024-06-10T00:27:34.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.807+0000 7f706effd700 1 -- 172.21.15.121:0/4274599605 learned_addr learned my addr 172.21.15.121:0/4274599605 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f706f7fe700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7070102770 0x7f7070074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f706ffff700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7070107860 0x7f7070075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f706effd700 1 -- 172.21.15.121:0/4274599605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7070102770 msgr2=0x7f7070074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f706effd700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7070102770 0x7f7070074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f706effd700 1 -- 172.21.15.121:0/4274599605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7070107860 msgr2=0x7f7070075210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f706effd700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7070107860 0x7f7070075210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f706effd700 1 -- 172.21.15.121:0/4274599605 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70701a7090 con 0x7f7070103120 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f706effd700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7070103120 0x7f7070074cd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f705800c920 tx=0x7f705800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f706cff9700 1 -- 172.21.15.121:0/4274599605 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058007b00 con 0x7f7070103120 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f706cff9700 1 -- 172.21.15.121:0/4274599605 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7058004d10 con 0x7f7070103120 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f7075dd7700 1 -- 172.21.15.121:0/4274599605 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70701a7380 con 0x7f7070103120 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f706cff9700 1 -- 172.21.15.121:0/4274599605 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058005680 con 0x7f7070103120 2024-06-10T00:27:34.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.808+0000 7f7075dd7700 1 -- 172.21.15.121:0/4274599605 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70701a78c0 con 0x7f7070103120 2024-06-10T00:27:34.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.809+0000 7f7075dd7700 1 -- 172.21.15.121:0/4274599605 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7070069180 con 0x7f7070103120 2024-06-10T00:27:34.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.811+0000 7f706cff9700 1 -- 172.21.15.121:0/4274599605 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f7058004750 con 0x7f7070103120 2024-06-10T00:27:34.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.812+0000 7f706cff9700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f705c0a2c20 0x7f705c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:34.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.812+0000 7f706f7fe700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f705c0a2c20 0x7f705c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:34.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.812+0000 7f706cff9700 1 -- 172.21.15.121:0/4274599605 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f70580cf450 con 0x7f7070103120 2024-06-10T00:27:34.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.812+0000 7f706f7fe700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f705c0a2c20 0x7f705c0a50e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f70680060b0 tx=0x7f706800c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:34.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:34.815+0000 7f706cff9700 1 -- 172.21.15.121:0/4274599605 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7058014030 con 0x7f7070103120 2024-06-10T00:27:35.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.056+0000 7f7075dd7700 1 -- 172.21.15.121:0/4274599605 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7070075cc0 con 0x7f7070103120 2024-06-10T00:27:35.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.057+0000 7f706cff9700 1 -- 172.21.15.121:0/4274599605 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11885 (secure 0 0 0) 0x7f70580afe20 con 0x7f7070103120 2024-06-10T00:27:35.059 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:27:35.059 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:25:08.157143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:27:35.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.059+0000 7f7075dd7700 1 -- 172.21.15.121:0/4274599605 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f705c0a2c20 msgr2=0x7f705c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:35.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.059+0000 7f7075dd7700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f705c0a2c20 0x7f705c0a50e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f70680060b0 tx=0x7f706800c040 comp rx=0 tx=0).stop 2024-06-10T00:27:35.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.059+0000 7f7075dd7700 1 -- 172.21.15.121:0/4274599605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7070103120 msgr2=0x7f7070074cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:35.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.059+0000 7f7075dd7700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7070103120 0x7f7070074cd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f705800c920 tx=0x7f705800cc30 comp rx=0 tx=0).stop 2024-06-10T00:27:35.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.060+0000 7f7075dd7700 1 -- 172.21.15.121:0/4274599605 shutdown_connections 2024-06-10T00:27:35.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.060+0000 7f7075dd7700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f705c0a2c20 0x7f705c0a50e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:35.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.060+0000 7f7075dd7700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7070107860 0x7f7070075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:35.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.060+0000 7f7075dd7700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7070102770 0x7f7070074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:35.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.060+0000 7f7075dd7700 1 --2- 172.21.15.121:0/4274599605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7070103120 0x7f7070074cd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:35.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.060+0000 7f7075dd7700 1 -- 172.21.15.121:0/4274599605 >> 172.21.15.121:0/4274599605 conn(0x7f7070079600 msgr2=0x7f7070100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:35.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.060+0000 7f7075dd7700 1 -- 172.21.15.121:0/4274599605 shutdown_connections 2024-06-10T00:27:35.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:35.060+0000 7f7075dd7700 1 -- 172.21.15.121:0/4274599605 wait complete. 2024-06-10T00:27:35.062 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 25 2024-06-10T00:27:35.395 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:35 smithi121 ceph-mon[32378]: pgmap v416: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:35.395 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:35 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/4274599605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:27:35.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:35 smithi135 ceph-mon[36424]: pgmap v416: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:35 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/4274599605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:27:35.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:35 smithi193 ceph-mon[36716]: pgmap v416: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:35 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/4274599605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:27:35.740 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:27:35.981 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:37.419 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:37 smithi121 ceph-mon[32378]: pgmap v417: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:37.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:37 smithi135 ceph-mon[36424]: pgmap v417: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:37.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:37 smithi193 ceph-mon[36716]: pgmap v417: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:37.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.649+0000 7ff103dc6700 1 -- 172.21.15.121:0/2159232949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0fc1030f0 msgr2=0x7ff0fc107110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:37.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.649+0000 7ff103dc6700 1 --2- 172.21.15.121:0/2159232949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0fc1030f0 0x7ff0fc107110 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80099c0 tx=0x7ff0f8009cd0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.649+0000 7ff103dc6700 1 -- 172.21.15.121:0/2159232949 shutdown_connections 2024-06-10T00:27:37.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.649+0000 7ff103dc6700 1 --2- 172.21.15.121:0/2159232949 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0fc107840 0x7ff0fc10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.649+0000 7ff103dc6700 1 --2- 172.21.15.121:0/2159232949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0fc102740 0x7ff0fc102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.649+0000 7ff103dc6700 1 --2- 172.21.15.121:0/2159232949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0fc1030f0 0x7ff0fc107110 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.649+0000 7ff103dc6700 1 -- 172.21.15.121:0/2159232949 >> 172.21.15.121:0/2159232949 conn(0x7ff0fc0795f0 msgr2=0x7ff0fc079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:37.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.649+0000 7ff103dc6700 1 -- 172.21.15.121:0/2159232949 shutdown_connections 2024-06-10T00:27:37.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.649+0000 7ff103dc6700 1 -- 172.21.15.121:0/2159232949 wait complete. 2024-06-10T00:27:37.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.650+0000 7ff103dc6700 1 Processor -- start 2024-06-10T00:27:37.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.650+0000 7ff103dc6700 1 -- start start 2024-06-10T00:27:37.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff103dc6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0fc102740 0x7ff0fc19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff103dc6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0fc1030f0 0x7ff0fc19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff103dc6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0fc107840 0x7ff0fc1988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff103dc6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc112750 con 0x7ff0fc102740 2024-06-10T00:27:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff103dc6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc1125d0 con 0x7ff0fc1030f0 2024-06-10T00:27:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff103dc6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc1128d0 con 0x7ff0fc107840 2024-06-10T00:27:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff101b62700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0fc102740 0x7ff0fc19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff101b62700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0fc102740 0x7ff0fc19e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39280/0 (socket says 172.21.15.121:39280) 2024-06-10T00:27:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff101b62700 1 -- 172.21.15.121:0/3461885603 learned_addr learned my addr 172.21.15.121:0/3461885603 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff101361700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0fc1030f0 0x7ff0fc19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff102363700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0fc107840 0x7ff0fc1988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:37.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff101b62700 1 -- 172.21.15.121:0/3461885603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0fc107840 msgr2=0x7ff0fc1988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff101b62700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0fc107840 0x7ff0fc1988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff101b62700 1 -- 172.21.15.121:0/3461885603 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0fc1030f0 msgr2=0x7ff0fc19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff101b62700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0fc1030f0 0x7ff0fc19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff101b62700 1 -- 172.21.15.121:0/3461885603 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc199050 con 0x7ff0fc102740 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff102363700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0fc107840 0x7ff0fc1988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff101b62700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0fc102740 0x7ff0fc19e730 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000c9b0 tx=0x7ff0f000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff0eeffd700 1 -- 172.21.15.121:0/3461885603 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0007b00 con 0x7ff0fc102740 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff0eeffd700 1 -- 172.21.15.121:0/3461885603 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0f0007ca0 con 0x7ff0fc102740 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff103dc6700 1 -- 172.21.15.121:0/3461885603 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0fc199340 con 0x7ff0fc102740 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff0eeffd700 1 -- 172.21.15.121:0/3461885603 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0018730 con 0x7ff0fc102740 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.651+0000 7ff103dc6700 1 -- 172.21.15.121:0/3461885603 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0fc1a5600 con 0x7ff0fc102740 2024-06-10T00:27:37.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.652+0000 7ff103dc6700 1 -- 172.21.15.121:0/3461885603 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0fc069180 con 0x7ff0fc102740 2024-06-10T00:27:37.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.653+0000 7ff0eeffd700 1 -- 172.21.15.121:0/3461885603 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7ff0f00188d0 con 0x7ff0fc102740 2024-06-10T00:27:37.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.654+0000 7ff0eeffd700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff0e80a2c20 0x7ff0e80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:37.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.654+0000 7ff101361700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff0e80a2c20 0x7ff0e80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:37.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.654+0000 7ff0eeffd700 1 -- 172.21.15.121:0/3461885603 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7ff0f00cf360 con 0x7ff0fc102740 2024-06-10T00:27:37.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.655+0000 7ff101361700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff0e80a2c20 0x7ff0e80a50e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc108b40 tx=0x7ff0f8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:37.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.659+0000 7ff0eeffd700 1 -- 172.21.15.121:0/3461885603 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff0f0014030 con 0x7ff0fc102740 2024-06-10T00:27:37.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.899+0000 7ff103dc6700 1 -- 172.21.15.121:0/3461885603 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0fc1a5890 con 0x7ff0fc102740 2024-06-10T00:27:37.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.900+0000 7ff0eeffd700 1 -- 172.21.15.121:0/3461885603 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11885 (secure 0 0 0) 0x7ff0f009bfb0 con 0x7ff0fc102740 2024-06-10T00:27:37.901 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:27:37.901 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:25:08.157143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 -- 172.21.15.121:0/3461885603 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff0e80a2c20 msgr2=0x7ff0e80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff0e80a2c20 0x7ff0e80a50e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc108b40 tx=0x7ff0f8005fb0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 -- 172.21.15.121:0/3461885603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0fc102740 msgr2=0x7ff0fc19e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0fc102740 0x7ff0fc19e730 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000c9b0 tx=0x7ff0f000ccc0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 -- 172.21.15.121:0/3461885603 shutdown_connections 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7ff0e80a2c20 0x7ff0e80a50e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0fc107840 0x7ff0fc1988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0fc102740 0x7ff0fc19e730 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 --2- 172.21.15.121:0/3461885603 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff0fc1030f0 0x7ff0fc19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 -- 172.21.15.121:0/3461885603 >> 172.21.15.121:0/3461885603 conn(0x7ff0fc0795f0 msgr2=0x7ff0fc100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 -- 172.21.15.121:0/3461885603 shutdown_connections 2024-06-10T00:27:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:37.902+0000 7ff103dc6700 1 -- 172.21.15.121:0/3461885603 wait complete. 2024-06-10T00:27:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 25 2024-06-10T00:27:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:38 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3461885603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:27:38.579 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:38 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3461885603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:27:38.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:27:38.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:38 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3461885603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:27:38.847 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:39.299 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:39 smithi121 ceph-mon[32378]: pgmap v418: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:39.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:39 smithi135 ceph-mon[36424]: pgmap v418: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:39 smithi193 ceph-mon[36716]: pgmap v418: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:40.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.518+0000 7f781a05c700 1 -- 172.21.15.121:0/276000960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f78141024e0 msgr2=0x7f7814102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:40.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.518+0000 7f781a05c700 1 --2- 172.21.15.121:0/276000960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f78141024e0 0x7f7814102940 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009990 tx=0x7f77fc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:27:40.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.518+0000 7f781a05c700 1 -- 172.21.15.121:0/276000960 shutdown_connections 2024-06-10T00:27:40.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.518+0000 7f781a05c700 1 --2- 172.21.15.121:0/276000960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7814102e80 0x7f781410f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:40.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.518+0000 7f781a05c700 1 --2- 172.21.15.121:0/276000960 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78141084f0 0x7f78141088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:40.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.518+0000 7f781a05c700 1 --2- 172.21.15.121:0/276000960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f78141024e0 0x7f7814102940 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:40.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.518+0000 7f781a05c700 1 -- 172.21.15.121:0/276000960 >> 172.21.15.121:0/276000960 conn(0x7f7814079530 msgr2=0x7f7814079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.518+0000 7f781a05c700 1 -- 172.21.15.121:0/276000960 shutdown_connections 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.518+0000 7f781a05c700 1 -- 172.21.15.121:0/276000960 wait complete. 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.518+0000 7f781a05c700 1 Processor -- start 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.518+0000 7f781a05c700 1 -- start start 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f781a05c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78141024e0 0x7f781419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f781a05c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7814102e80 0x7f781419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f781a05c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78141084f0 0x7f7814198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f781a05c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7814112770 con 0x7f7814102e80 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f781a05c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f78141125f0 con 0x7f78141024e0 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f781a05c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f78141128f0 con 0x7f78141084f0 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f780affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7814102e80 0x7f781419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:40.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f78137fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78141024e0 0x7f781419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f780affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7814102e80 0x7f781419ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34702/0 (socket says 172.21.15.121:34702) 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f780affd700 1 -- 172.21.15.121:0/2831393706 learned_addr learned my addr 172.21.15.121:0/2831393706 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f7813fff700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78141084f0 0x7f7814198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f780affd700 1 -- 172.21.15.121:0/2831393706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78141084f0 msgr2=0x7f7814198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f780affd700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78141084f0 0x7f7814198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f780affd700 1 -- 172.21.15.121:0/2831393706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78141024e0 msgr2=0x7f781419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f780affd700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78141024e0 0x7f781419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f780affd700 1 -- 172.21.15.121:0/2831393706 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7814199210 con 0x7f7814102e80 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f78137fe700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78141024e0 0x7f781419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.519+0000 7f780affd700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7814102e80 0x7f781419ee30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00bae0 tx=0x7f77fc00bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.520+0000 7f78117fa700 1 -- 172.21.15.121:0/2831393706 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc026070 con 0x7f7814102e80 2024-06-10T00:27:40.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.520+0000 7f781a05c700 1 -- 172.21.15.121:0/2831393706 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78141994a0 con 0x7f7814102e80 2024-06-10T00:27:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.520+0000 7f78117fa700 1 -- 172.21.15.121:0/2831393706 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77fc003d70 con 0x7f7814102e80 2024-06-10T00:27:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.520+0000 7f78117fa700 1 -- 172.21.15.121:0/2831393706 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc021410 con 0x7f7814102e80 2024-06-10T00:27:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.520+0000 7f781a05c700 1 -- 172.21.15.121:0/2831393706 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78141a57c0 con 0x7f7814102e80 2024-06-10T00:27:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.520+0000 7f781a05c700 1 -- 172.21.15.121:0/2831393706 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7814069180 con 0x7f7814102e80 2024-06-10T00:27:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.522+0000 7f78117fa700 1 -- 172.21.15.121:0/2831393706 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f77fc029070 con 0x7f7814102e80 2024-06-10T00:27:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.522+0000 7f78117fa700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f78000a2ca0 0x7f78000a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.522+0000 7f78137fe700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f78000a2ca0 0x7f78000a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.522+0000 7f78117fa700 1 -- 172.21.15.121:0/2831393706 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f77fc0d0660 con 0x7f7814102e80 2024-06-10T00:27:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.523+0000 7f78137fe700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f78000a2ca0 0x7f78000a5160 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7814199bf0 tx=0x7f7804009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:40.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.526+0000 7f78117fa700 1 -- 172.21.15.121:0/2831393706 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f77fc030080 con 0x7f7814102e80 2024-06-10T00:27:40.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.722+0000 7f781a05c700 1 -- 172.21.15.121:0/2831393706 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f781402dca0 con 0x7f7814102e80 2024-06-10T00:27:40.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.724+0000 7f78117fa700 1 -- 172.21.15.121:0/2831393706 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+24919 (secure 0 0 0) 0x7f77fc013070 con 0x7f7814102e80 2024-06-10T00:27:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:27:40.724 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":100,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:25:44.074803+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:27:40.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.726+0000 7f781a05c700 1 -- 172.21.15.121:0/2831393706 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f78000a2ca0 msgr2=0x7f78000a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:40.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.726+0000 7f781a05c700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f78000a2ca0 0x7f78000a5160 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7814199bf0 tx=0x7f7804009260 comp rx=0 tx=0).stop 2024-06-10T00:27:40.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.726+0000 7f781a05c700 1 -- 172.21.15.121:0/2831393706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7814102e80 msgr2=0x7f781419ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:40.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.726+0000 7f781a05c700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7814102e80 0x7f781419ee30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00bae0 tx=0x7f77fc00bb10 comp rx=0 tx=0).stop 2024-06-10T00:27:40.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.727+0000 7f781a05c700 1 -- 172.21.15.121:0/2831393706 shutdown_connections 2024-06-10T00:27:40.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.727+0000 7f781a05c700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f78000a2ca0 0x7f78000a5160 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:40.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.727+0000 7f781a05c700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78141084f0 0x7f7814198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:40.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.727+0000 7f781a05c700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f78141024e0 0x7f781419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:40.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.727+0000 7f781a05c700 1 --2- 172.21.15.121:0/2831393706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7814102e80 0x7f781419ee30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:40.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.727+0000 7f781a05c700 1 -- 172.21.15.121:0/2831393706 >> 172.21.15.121:0/2831393706 conn(0x7f7814079530 msgr2=0x7f7814100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:40.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.727+0000 7f781a05c700 1 -- 172.21.15.121:0/2831393706 shutdown_connections 2024-06-10T00:27:40.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:40.727+0000 7f781a05c700 1 -- 172.21.15.121:0/2831393706 wait complete. 2024-06-10T00:27:41.424 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-06-10T00:23:59.392036+0000', 'modified': '2024-06-10T00:25:08.157143+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24479, 'mds_1': 24472, 'mds_2': 24455}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.135:6835/284701222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6834', 'nonce': 284701222}, {'type': 'v1', 'addr': '172.21.15.135:6835', 'nonce': 284701222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/3023450024', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 3023450024}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 3023450024}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24448': {'gid': 24448, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/2863018656', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 2863018656}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 2863018656}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.135:6837/1549198529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6836', 'nonce': 1549198529}, {'type': 'v1', 'addr': '172.21.15.135:6837', 'nonce': 1549198529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.193:6839/2677954983', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 2677954983}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 2677954983}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24479': {'gid': 24479, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.135:6839/1599631648', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6838', 'nonce': 1599631648}, {'type': 'v1', 'addr': '172.21.15.135:6839', 'nonce': 1599631648}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24479, 'qdb_cluster': [24455, 24479, 24472]} 2024-06-10T00:27:41.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-06-10T00:27:41.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-06-10T00:27:41.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:41 smithi135 ceph-mon[36424]: pgmap v419: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:41.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:41 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2831393706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:27:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:41 smithi193 ceph-mon[36716]: pgmap v419: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:41 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2831393706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:27:41.665 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:41.698 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:41 smithi121 ceph-mon[32378]: pgmap v419: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-06-10T00:27:41.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:41 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2831393706' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:27:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.344+0000 7fdaee4ef700 1 -- 172.21.15.121:0/2955403615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdae8108500 msgr2=0x7fdae81088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.344+0000 7fdaee4ef700 1 --2- 172.21.15.121:0/2955403615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdae8108500 0x7fdae81088e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdad8009990 tx=0x7fdad8009ca0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.344+0000 7fdaee4ef700 1 -- 172.21.15.121:0/2955403615 shutdown_connections 2024-06-10T00:27:43.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.344+0000 7fdaee4ef700 1 --2- 172.21.15.121:0/2955403615 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdae806ab30 0x7fdae810f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.344+0000 7fdaee4ef700 1 --2- 172.21.15.121:0/2955403615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdae8108500 0x7fdae81088e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.344+0000 7fdaee4ef700 1 --2- 172.21.15.121:0/2955403615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae806a170 0x7fdae806a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.344+0000 7fdaee4ef700 1 -- 172.21.15.121:0/2955403615 >> 172.21.15.121:0/2955403615 conn(0x7fdae8078340 msgr2=0x7fdae8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.344+0000 7fdaee4ef700 1 -- 172.21.15.121:0/2955403615 shutdown_connections 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.344+0000 7fdaee4ef700 1 -- 172.21.15.121:0/2955403615 wait complete. 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdaee4ef700 1 Processor -- start 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdaee4ef700 1 -- start start 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdaee4ef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdae806a170 0x7fdae819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdaee4ef700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdae806ab30 0x7fdae819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdaee4ef700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae8108500 0x7fdae8198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdaee4ef700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdae8112780 con 0x7fdae806a170 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdaee4ef700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fdae8112600 con 0x7fdae806ab30 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdaee4ef700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdae8112900 con 0x7fdae8108500 2024-06-10T00:27:43.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdae7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdae806a170 0x7fdae819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdae7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdae806a170 0x7fdae819e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34716/0 (socket says 172.21.15.121:34716) 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdae7fff700 1 -- 172.21.15.121:0/895033369 learned_addr learned my addr 172.21.15.121:0/895033369 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.345+0000 7fdae77fe700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdae806ab30 0x7fdae819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdaeca8c700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae8108500 0x7fdae8198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdae7fff700 1 -- 172.21.15.121:0/895033369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae8108500 msgr2=0x7fdae8198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdae7fff700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae8108500 0x7fdae8198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdae7fff700 1 -- 172.21.15.121:0/895033369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdae806ab30 msgr2=0x7fdae819ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdae7fff700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdae806ab30 0x7fdae819ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdae7fff700 1 -- 172.21.15.121:0/895033369 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae8199240 con 0x7fdae806a170 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdae7fff700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdae806a170 0x7fdae819e920 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdad8000c00 tx=0x7fdad800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdae57fa700 1 -- 172.21.15.121:0/895033369 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad8027070 con 0x7fdae806a170 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdae57fa700 1 -- 172.21.15.121:0/895033369 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdad8003d70 con 0x7fdae806a170 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdaee4ef700 1 -- 172.21.15.121:0/895033369 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae81994d0 con 0x7fdae806a170 2024-06-10T00:27:43.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdae57fa700 1 -- 172.21.15.121:0/895033369 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad800f460 con 0x7fdae806a170 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.346+0000 7fdaee4ef700 1 -- 172.21.15.121:0/895033369 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae81a57f0 con 0x7fdae806a170 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.348+0000 7fdae57fa700 1 -- 172.21.15.121:0/895033369 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fdad802a070 con 0x7fdae806a170 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.349+0000 7fdae57fa700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdad00a2c20 0x7fdad00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.349+0000 7fdae77fe700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdad00a2c20 0x7fdad00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.349+0000 7fdae57fa700 1 -- 172.21.15.121:0/895033369 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fdad80d0130 con 0x7fdae806a170 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdae77fe700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdad00a2c20 0x7fdad00a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdae8078a00 tx=0x7fdadc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdaf8d72b80 1 Processor -- start 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdaf8d72b80 1 -- start start 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f93f80 0x5600f3f94360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3f986a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f98be0 0x5600f3f99040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5600f3ee17c0 con 0x5600f3f98be0 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5600f3e896d0 con 0x5600f3f948a0 2024-06-10T00:27:43.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5600f3edd780 con 0x5600f3f93f80 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdaeca8c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f98be0 0x5600f3f99040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdaeca8c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f98be0 0x5600f3f99040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34730/0 (socket says 172.21.15.121:34730) 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.350+0000 7fdaeca8c700 1 -- 172.21.15.121:0/1063299665 learned_addr learned my addr 172.21.15.121:0/1063299665 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.351+0000 7fdae77fe700 1 --2- 172.21.15.121:0/1063299665 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3f986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.351+0000 7fdae7fff700 1 --2- 172.21.15.121:0/1063299665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f93f80 0x5600f3f94360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.351+0000 7fdaeca8c700 1 -- 172.21.15.121:0/1063299665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f93f80 msgr2=0x5600f3f94360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.351+0000 7fdaeca8c700 1 --2- 172.21.15.121:0/1063299665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f93f80 0x5600f3f94360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.351+0000 7fdaeca8c700 1 -- 172.21.15.121:0/1063299665 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 msgr2=0x5600f3f986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.351+0000 7fdaeca8c700 1 --2- 172.21.15.121:0/1063299665 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3f986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.351+0000 7fdaeca8c700 1 -- 172.21.15.121:0/1063299665 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600f3e7ff00 con 0x5600f3f98be0 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.351+0000 7fdae77fe700 1 --2- 172.21.15.121:0/1063299665 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3f986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:43.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.351+0000 7fdaeca8c700 1 --2- 172.21.15.121:0/1063299665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f98be0 0x5600f3f99040 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009440 tx=0x7fdae0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fcd4f59e10eda8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.352+0000 7fdace7fc700 1 -- 172.21.15.121:0/1063299665 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0013750 con 0x5600f3f98be0 2024-06-10T00:27:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.352+0000 7fdace7fc700 1 -- 172.21.15.121:0/1063299665 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdae0013dd0 con 0x5600f3f98be0 2024-06-10T00:27:43.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.352+0000 7fdace7fc700 1 -- 172.21.15.121:0/1063299665 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae00143e0 con 0x5600f3f98be0 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.353+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/1063299665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f98be0 msgr2=0x5600f3f99040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.353+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/1063299665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f98be0 0x5600f3f99040 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009440 tx=0x7fdae0009920 comp rx=0 tx=0).stop 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.353+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/1063299665 shutdown_connections 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.353+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/1063299665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f98be0 0x5600f3f99040 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.353+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/1063299665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f93f80 0x5600f3f94360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.353+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/1063299665 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3f986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.353+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/1063299665 >> 172.21.15.121:0/1063299665 conn(0x5600f3ebac70 msgr2=0x5600f3f8a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.353+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/1063299665 shutdown_connections 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.353+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/1063299665 wait complete. 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.353+0000 7fdaf8d72b80 1 -- start start 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.353+0000 7fdaf8d72b80 1 Processor -- start 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fa00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f948a0 0x5600f3f9d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f98be0 0x5600f3f9d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5600f3fa0d80 con 0x5600f3f93f80 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5600f3e81ba0 con 0x5600f3f98be0 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5600f3fa0f00 con 0x5600f3f948a0 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdaf8d72b80 10 client.0 _ll_register_callbacks cb 0x5600f3e7b8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdae7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fa00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdae7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fa00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34736/0 (socket says 172.21.15.121:34736) 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdae7fff700 1 -- 172.21.15.121:0/3544501974 learned_addr learned my addr 172.21.15.121:0/3544501974 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdae77fe700 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f948a0 0x5600f3f9d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdaeca8c700 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f98be0 0x5600f3f9d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdae7fff700 1 -- 172.21.15.121:0/3544501974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f948a0 msgr2=0x5600f3f9d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdae7fff700 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f948a0 0x5600f3f9d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdae7fff700 1 -- 172.21.15.121:0/3544501974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f98be0 msgr2=0x5600f3f9d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:43.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdae7fff700 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f98be0 0x5600f3f9d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdae7fff700 1 -- 172.21.15.121:0/3544501974 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600f3e30880 con 0x5600f3f93f80 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdae7fff700 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fa00e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdad80cd820 tx=0x7fdad80229e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a998d17a3961be server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdac77fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdac77fe700 1 -- 172.21.15.121:0/3544501974 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad809c610 con 0x5600f3f93f80 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdac77fe700 1 -- 172.21.15.121:0/3544501974 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fdad80a4d60 con 0x5600f3f93f80 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.354+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/3544501974 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdae0013dd0 con 0x5600f3f93f80 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdac77fe700 1 -- 172.21.15.121:0/3544501974 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdad80a09c0 con 0x5600f3f93f80 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdaf8d72b80 10 client.15153 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/3544501974 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5600f3fab690 con 0x5600f3f93f80 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdac77fe700 1 -- 172.21.15.121:0/3544501974 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdae0013dd0 con 0x5600f3f93f80 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdac77fe700 1 -- 172.21.15.121:0/3544501974 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fdad80cdce0 con 0x5600f3f93f80 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdaf8d72b80 10 client.15153 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdaf8d72b80 10 client.15153 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-10T00:27:43.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdaf8d72b80 20 client.15153 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5600f3fadc90 0x5600f3faf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdaf8d72b80 4 client.15153 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:27:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/3544501974 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600f3eeecc0 con 0x5600f3fadc90 2024-06-10T00:27:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.355+0000 7fdae77fe700 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5600f3fadc90 0x5600f3faf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.356+0000 7fdae77fe700 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5600f3fadc90 0x5600f3faf100 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16dfc99f20dcfe13 server_cookie=4d6364f5bb662571 in_seq=0 out_seq=0 2024-06-10T00:27:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.356+0000 7fdac77fe700 10 client.15153 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:27:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.356+0000 7fdac77fe700 1 -- 172.21.15.121:0/3544501974 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5600f3eeecc0 con 0x5600f3fadc90 2024-06-10T00:27:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.356+0000 7fdac77fe700 10 client.15153 handle_command_reply: tid=0 2024-06-10T00:27:43.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 1 client.15153 shutdown 2024-06-10T00:27:43.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:43 smithi121 ceph-mon[32378]: pgmap v420: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:43.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/3544501974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 msgr2=0x5600f3fa00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fa00e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdad80cd820 tx=0x7fdad80229e0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/3544501974 shutdown_connections 2024-06-10T00:27:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5600f3fadc90 0x5600f3faf100 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f98be0 0x5600f3f9d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fa00e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/3544501974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f948a0 0x5600f3f9d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/3544501974 >> 172.21.15.121:0/3544501974 conn(0x5600f3ebac70 msgr2=0x5600f3f8e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdac77fe700 0 client.15153 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:27:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/3544501974 shutdown_connections 2024-06-10T00:27:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/3544501974 wait complete. 2024-06-10T00:27:43.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.357+0000 7fdaf8d72b80 20 client.15153 trim_cache size 0 max 16384 2024-06-10T00:27:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.378+0000 7fdaf8d72b80 1 Processor -- start 2024-06-10T00:27:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.378+0000 7fdaf8d72b80 1 -- start start 2024-06-10T00:27:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3f9da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f98be0 0x5600f3f8a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5600f3fad440 con 0x5600f3f93f80 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5600f3fadb10 con 0x5600f3f948a0 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5600f3fae1e0 con 0x5600f3f98be0 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdae7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdae7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fad5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34744/0 (socket says 172.21.15.121:34744) 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdae7fff700 1 -- 172.21.15.121:0/2282302676 learned_addr learned my addr 172.21.15.121:0/2282302676 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdaeca8c700 1 --2- 172.21.15.121:0/2282302676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f98be0 0x5600f3f8a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdae77fe700 1 --2- 172.21.15.121:0/2282302676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3f9da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdae7fff700 1 -- 172.21.15.121:0/2282302676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f98be0 msgr2=0x5600f3f8a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdae7fff700 1 --2- 172.21.15.121:0/2282302676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f98be0 0x5600f3f8a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdae7fff700 1 -- 172.21.15.121:0/2282302676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 msgr2=0x5600f3f9da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdae7fff700 1 --2- 172.21.15.121:0/2282302676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3f9da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdae7fff700 1 -- 172.21.15.121:0/2282302676 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600f3fad270 con 0x5600f3f93f80 2024-06-10T00:27:43.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdaeca8c700 1 --2- 172.21.15.121:0/2282302676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f98be0 0x5600f3f8a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:43.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdae7fff700 1 --2- 172.21.15.121:0/2282302676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fad5c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdad809c310 tx=0x7fdad8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337f9f813183d669 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:43.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdacdffb700 1 -- 172.21.15.121:0/2282302676 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad80a51f0 con 0x5600f3f93f80 2024-06-10T00:27:43.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdacdffb700 1 -- 172.21.15.121:0/2282302676 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdad8098040 con 0x5600f3f93f80 2024-06-10T00:27:43.381 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.379+0000 7fdacdffb700 1 -- 172.21.15.121:0/2282302676 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad80a4fc0 con 0x5600f3f93f80 2024-06-10T00:27:43.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.380+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/2282302676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 msgr2=0x5600f3fad5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:43.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.380+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/2282302676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fad5c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdad809c310 tx=0x7fdad8009310 comp rx=0 tx=0).stop 2024-06-10T00:27:43.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.380+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/2282302676 shutdown_connections 2024-06-10T00:27:43.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.380+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/2282302676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f98be0 0x5600f3f8a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.380+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/2282302676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fad5c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.380+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/2282302676 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3f9da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.380+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/2282302676 >> 172.21.15.121:0/2282302676 conn(0x5600f3ebac70 msgr2=0x5600f3f8be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.380+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/2282302676 shutdown_connections 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.380+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/2282302676 wait complete. 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.380+0000 7fdaf8d72b80 1 -- start start 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdaf8d72b80 1 Processor -- start 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fb8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3fb1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdaf8d72b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f98be0 0x5600f3fb2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5600f3faef80 con 0x5600f3f93f80 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5600f3fae8b0 con 0x5600f3f948a0 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdaf8d72b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5600f3faf650 con 0x5600f3f98be0 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdaf8d72b80 10 client.0 _ll_register_callbacks cb 0x5600f3e7b8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:43.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdae7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fb8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdae7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fb8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34758/0 (socket says 172.21.15.121:34758) 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdae7fff700 1 -- 172.21.15.121:0/621725096 learned_addr learned my addr 172.21.15.121:0/621725096 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdae77fe700 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3fb1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdaeca8c700 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f98be0 0x5600f3fb2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdae7fff700 1 -- 172.21.15.121:0/621725096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f98be0 msgr2=0x5600f3fb2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdae7fff700 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f98be0 0x5600f3fb2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdae7fff700 1 -- 172.21.15.121:0/621725096 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 msgr2=0x5600f3fb1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdae7fff700 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3fb1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdae7fff700 1 -- 172.21.15.121:0/621725096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600f3fb2d60 con 0x5600f3f93f80 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdae7fff700 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fb8c80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdad809c7d0 tx=0x7fdad80229e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69dbfe72accdd96d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.381+0000 7fdac77fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdac77fe700 1 -- 172.21.15.121:0/621725096 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad807f020 con 0x5600f3f93f80 2024-06-10T00:27:43.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdac77fe700 1 -- 172.21.15.121:0/621725096 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fdad8085da0 con 0x5600f3f93f80 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/621725096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdad8098040 con 0x5600f3f93f80 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdac77fe700 1 -- 172.21.15.121:0/621725096 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdad809ba60 con 0x5600f3f93f80 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdaf8d72b80 10 client.15159 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/621725096 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5600f3fb2d60 con 0x5600f3f93f80 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdac77fe700 1 -- 172.21.15.121:0/621725096 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdad8098040 con 0x5600f3f93f80 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdac77fe700 1 -- 172.21.15.121:0/621725096 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fdad80cdce0 con 0x5600f3f93f80 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdaf8d72b80 10 client.15159 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdaf8d72b80 10 client.15159 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdaf8d72b80 20 client.15159 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5600f401d8d0 0x5600f401c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdaf8d72b80 4 client.15159 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/621725096 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "status"}) -- 0x5600f3ee9240 con 0x5600f401d8d0 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.382+0000 7fdae77fe700 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5600f401d8d0 0x5600f401c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:43.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdae77fe700 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5600f401d8d0 0x5600f401c5e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6146854a3499116a server_cookie=187cf82007187405 in_seq=0 out_seq=0 2024-06-10T00:27:43.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdac77fe700 10 client.15159 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:27:43.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdac77fe700 1 -- 172.21.15.121:0/621725096 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x5600f3ee9240 con 0x5600f401d8d0 2024-06-10T00:27:43.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdac77fe700 10 client.15159 handle_command_reply: tid=0 2024-06-10T00:27:43.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 1 client.15159 shutdown 2024-06-10T00:27:43.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/621725096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 msgr2=0x5600f3fb8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:43.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fb8c80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdad809c7d0 tx=0x7fdad80229e0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/621725096 shutdown_connections 2024-06-10T00:27:43.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5600f401d8d0 0x5600f401c5e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600f3f98be0 0x5600f3fb2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5600f3f93f80 0x5600f3fb8c80 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 1 --2- 172.21.15.121:0/621725096 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5600f3f948a0 0x5600f3fb1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/621725096 >> 172.21.15.121:0/621725096 conn(0x5600f3ebac70 msgr2=0x5600f3f8a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:43.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdac77fe700 0 client.15159 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:27:43.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/621725096 shutdown_connections 2024-06-10T00:27:43.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 1 -- 172.21.15.121:0/621725096 wait complete. 2024-06-10T00:27:43.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.383+0000 7fdaf8d72b80 20 client.15159 trim_cache size 0 max 16384 2024-06-10T00:27:43.390 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:27:43.390 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:27:43.390 INFO:teuthology.orchestra.run.smithi121.stdout: "whoami": 2, 2024-06-10T00:27:43.390 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 24455, 2024-06-10T00:27:43.390 INFO:teuthology.orchestra.run.smithi121.stdout: "want_state": "up:active", 2024-06-10T00:27:43.391 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "up:active", 2024-06-10T00:27:43.391 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs", 2024-06-10T00:27:43.391 INFO:teuthology.orchestra.run.smithi121.stdout: "rank_uptime": 160.128755044, 2024-06-10T00:27:43.391 INFO:teuthology.orchestra.run.smithi121.stdout: "mdsmap_epoch": 25, 2024-06-10T00:27:43.391 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch": 100, 2024-06-10T00:27:43.391 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch_barrier": 98, 2024-06-10T00:27:43.391 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 254.96219399099999 2024-06-10T00:27:43.391 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:27:43.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 -- 172.21.15.121:0/895033369 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdad00a2c20 msgr2=0x7fdad00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:43.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdad00a2c20 0x7fdad00a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdae8078a00 tx=0x7fdadc009260 comp rx=0 tx=0).stop 2024-06-10T00:27:43.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 -- 172.21.15.121:0/895033369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdae806a170 msgr2=0x7fdae819e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:43.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdae806a170 0x7fdae819e920 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fdad8000c00 tx=0x7fdad800bab0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 -- 172.21.15.121:0/895033369 shutdown_connections 2024-06-10T00:27:43.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fdad00a2c20 0x7fdad00a50e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdae8108500 0x7fdae8198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdae806a170 0x7fdae819e920 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 --2- 172.21.15.121:0/895033369 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdae806ab30 0x7fdae819ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:43.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 -- 172.21.15.121:0/895033369 >> 172.21.15.121:0/895033369 conn(0x7fdae8078340 msgr2=0x7fdae8100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:43.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 -- 172.21.15.121:0/895033369 shutdown_connections 2024-06-10T00:27:43.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:43.384+0000 7fdaee4ef700 1 -- 172.21.15.121:0/895033369 wait complete. 2024-06-10T00:27:43.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:43 smithi135 ceph-mon[36424]: pgmap v420: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:43 smithi193 ceph-mon[36716]: pgmap v420: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-06-10T00:27:44.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-06-10T00:27:44.366 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:44.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:44 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:27:44.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:44 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:27:44.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:44 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:27:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:45 smithi135 ceph-mon[36424]: pgmap v421: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:45 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:45 smithi121 ceph-mon[32378]: pgmap v421: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:45.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.536 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:45 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:45 smithi193 ceph-mon[36716]: pgmap v421: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-06-10T00:27:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:45 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.113+0000 7f8ff59c3700 1 -- 172.21.15.121:0/1997189588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff0103970 msgr2=0x7f8ff010e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.113+0000 7f8ff59c3700 1 --2- 172.21.15.121:0/1997189588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff0103970 0x7f8ff010e3f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800b0a0 tx=0x7f8fe800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.113+0000 7f8ff59c3700 1 -- 172.21.15.121:0/1997189588 shutdown_connections 2024-06-10T00:27:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.113+0000 7f8ff59c3700 1 --2- 172.21.15.121:0/1997189588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff0103970 0x7f8ff010e3f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.113+0000 7f8ff59c3700 1 --2- 172.21.15.121:0/1997189588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ff01017e0 0x7f8ff0101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.113+0000 7f8ff59c3700 1 --2- 172.21.15.121:0/1997189588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ff0102fb0 0x7f8ff0103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.113+0000 7f8ff59c3700 1 -- 172.21.15.121:0/1997189588 >> 172.21.15.121:0/1997189588 conn(0x7f8ff0077430 msgr2=0x7f8ff0077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.113+0000 7f8ff59c3700 1 -- 172.21.15.121:0/1997189588 shutdown_connections 2024-06-10T00:27:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.114+0000 7f8ff59c3700 1 -- 172.21.15.121:0/1997189588 wait complete. 2024-06-10T00:27:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.114+0000 7f8ff59c3700 1 Processor -- start 2024-06-10T00:27:46.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.114+0000 7f8ff59c3700 1 -- start start 2024-06-10T00:27:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8ff59c3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ff01017e0 0x7f8ff01979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8ff59c3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff0102fb0 0x7f8ff0197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8ff59c3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ff0103970 0x7f8ff0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8ff59c3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff0111840 con 0x7f8ff0102fb0 2024-06-10T00:27:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8ff59c3700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff01116c0 con 0x7f8ff0103970 2024-06-10T00:27:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8ff59c3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff01119c0 con 0x7f8ff01017e0 2024-06-10T00:27:46.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8fee7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff0102fb0 0x7f8ff0197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8feeffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ff01017e0 0x7f8ff01979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8fee7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff0102fb0 0x7f8ff0197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34772/0 (socket says 172.21.15.121:34772) 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8feeffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ff01017e0 0x7f8ff01979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:55836/0 (socket says 172.21.15.121:55836) 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2275121612 learned_addr learned my addr 172.21.15.121:0/2275121612 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8fef7fe700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ff0103970 0x7f8ff0198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2275121612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ff01017e0 msgr2=0x7f8ff01979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ff01017e0 0x7f8ff01979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2275121612 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ff0103970 msgr2=0x7f8ff0198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ff0103970 0x7f8ff0198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2275121612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff01a7220 con 0x7f8ff0102fb0 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.115+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff0102fb0 0x7f8ff0197ef0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c970 tx=0x7f8fe400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:46.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.116+0000 7f8ff49c1700 1 -- 172.21.15.121:0/2275121612 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40078d0 con 0x7f8ff0102fb0 2024-06-10T00:27:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.116+0000 7f8ff59c3700 1 -- 172.21.15.121:0/2275121612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff01a7510 con 0x7f8ff0102fb0 2024-06-10T00:27:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.116+0000 7f8ff49c1700 1 -- 172.21.15.121:0/2275121612 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fe4018480 con 0x7f8ff0102fb0 2024-06-10T00:27:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.116+0000 7f8ff49c1700 1 -- 172.21.15.121:0/2275121612 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe400f6b0 con 0x7f8ff0102fb0 2024-06-10T00:27:46.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.116+0000 7f8ff59c3700 1 -- 172.21.15.121:0/2275121612 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff01a7a00 con 0x7f8ff0102fb0 2024-06-10T00:27:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.118+0000 7f8ff49c1700 1 -- 172.21.15.121:0/2275121612 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8fe4018630 con 0x7f8ff0102fb0 2024-06-10T00:27:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.118+0000 7f8ff49c1700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fd80a2c20 0x7f8fd80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.118+0000 7f8feeffd700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fd80a2c20 0x7f8fd80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.118+0000 7f8ff49c1700 1 -- 172.21.15.121:0/2275121612 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f8fe40cf3a0 con 0x7f8ff0102fb0 2024-06-10T00:27:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.119+0000 7f8feeffd700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fd80a2c20 0x7f8fd80a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0007be0 tx=0x7f8fe00073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.119+0000 7f9000246b80 1 Processor -- start 2024-06-10T00:27:46.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.119+0000 7f9000246b80 1 -- start start 2024-06-10T00:27:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.119+0000 7f9000246b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f9000246b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cf9a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f9000246b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cf9b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f9000246b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556a7ceec7c0 con 0x556a7cfa05f0 2024-06-10T00:27:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f9000246b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x556a7ce946d0 con 0x556a7cf9aed0 2024-06-10T00:27:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f9000246b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556a7cee8780 con 0x556a7cfa0f10 2024-06-10T00:27:46.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8feeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8feeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34778/0 (socket says 172.21.15.121:34778) 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8feeffd700 1 -- 172.21.15.121:0/1755168496 learned_addr learned my addr 172.21.15.121:0/1755168496 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8fef7fe700 1 --2- 172.21.15.121:0/1755168496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cf9b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/1755168496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cf9a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8feeffd700 1 -- 172.21.15.121:0/1755168496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 msgr2=0x556a7cf9a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8feeffd700 1 --2- 172.21.15.121:0/1755168496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cf9a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8feeffd700 1 -- 172.21.15.121:0/1755168496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 msgr2=0x556a7cf9b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8feeffd700 1 --2- 172.21.15.121:0/1755168496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cf9b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8feeffd700 1 -- 172.21.15.121:0/1755168496 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a7ce8af00 con 0x556a7cfa05f0 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8feeffd700 1 --2- 172.21.15.121:0/1755168496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa09d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0010e80 tx=0x7f8fe0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=775bcdfafaf2be3b server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8fd5ffb700 1 -- 172.21.15.121:0/1755168496 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0024070 con 0x556a7cfa05f0 2024-06-10T00:27:46.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8fd5ffb700 1 -- 172.21.15.121:0/1755168496 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fe0015070 con 0x556a7cfa05f0 2024-06-10T00:27:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.120+0000 7f8fd5ffb700 1 -- 172.21.15.121:0/1755168496 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe00119d0 con 0x556a7cfa05f0 2024-06-10T00:27:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.121+0000 7f9000246b80 1 -- 172.21.15.121:0/1755168496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 msgr2=0x556a7cfa09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.121+0000 7f9000246b80 1 --2- 172.21.15.121:0/1755168496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa09d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0010e80 tx=0x7f8fe0013000 comp rx=0 tx=0).stop 2024-06-10T00:27:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.121+0000 7f9000246b80 1 -- 172.21.15.121:0/1755168496 shutdown_connections 2024-06-10T00:27:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.121+0000 7f9000246b80 1 --2- 172.21.15.121:0/1755168496 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cf9b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.121+0000 7f9000246b80 1 --2- 172.21.15.121:0/1755168496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa09d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.121+0000 7f9000246b80 1 --2- 172.21.15.121:0/1755168496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cf9a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.121+0000 7f9000246b80 1 -- 172.21.15.121:0/1755168496 >> 172.21.15.121:0/1755168496 conn(0x556a7cec5c70 msgr2=0x556a7cf95800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.121+0000 7f9000246b80 1 -- 172.21.15.121:0/1755168496 shutdown_connections 2024-06-10T00:27:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.121+0000 7f9000246b80 1 -- 172.21.15.121:0/1755168496 wait complete. 2024-06-10T00:27:46.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f9000246b80 1 -- start start 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f9000246b80 1 Processor -- start 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f9000246b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cfa6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f9000246b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f9000246b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cfa44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f9000246b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556a7cfa7980 con 0x556a7cfa05f0 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f9000246b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x556a7ce8cba0 con 0x556a7cf9aed0 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f9000246b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556a7cfa7b00 con 0x556a7cfa0f10 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f9000246b80 10 client.0 _ll_register_callbacks cb 0x556a7ce868d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f8fee7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f8fee7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34792/0 (socket says 172.21.15.121:34792) 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2943338226 learned_addr learned my addr 172.21.15.121:0/2943338226 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:46.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.122+0000 7f8feeffd700 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cfa6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fef7fe700 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cfa44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2943338226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 msgr2=0x556a7cfa44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cfa44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2943338226 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 msgr2=0x556a7cfa6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cfa6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2943338226 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a7ce3b880 con 0x556a7cfa05f0 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fef7fe700 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cfa44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8feeffd700 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cfa6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa3f80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4098350 tx=0x7f8fe40a2390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0156601d9664afb server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fceffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fceffd700 1 -- 172.21.15.121:0/2943338226 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40b4060 con 0x556a7cfa05f0 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fceffd700 1 -- 172.21.15.121:0/2943338226 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f8fe40189a0 con 0x556a7cfa05f0 2024-06-10T00:27:46.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f9000246b80 1 -- 172.21.15.121:0/2943338226 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fe0015070 con 0x556a7cfa05f0 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fceffd700 1 -- 172.21.15.121:0/2943338226 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8fe40a4560 con 0x556a7cfa05f0 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f9000246b80 10 client.15171 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f9000246b80 1 -- 172.21.15.121:0/2943338226 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556a7cfb22e0 con 0x556a7cfa05f0 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.123+0000 7f8fceffd700 1 -- 172.21.15.121:0/2943338226 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fe0015070 con 0x556a7cfa05f0 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.124+0000 7f8fceffd700 1 -- 172.21.15.121:0/2943338226 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8fe40860f0 con 0x556a7cfa05f0 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.124+0000 7f9000246b80 10 client.15171 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.124+0000 7f9000246b80 10 client.15171 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.124+0000 7f9000246b80 20 client.15171 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.124+0000 7f9000246b80 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556a7cfb4800 0x556a7cfb5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.124+0000 7f9000246b80 4 client.15171 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.124+0000 7f9000246b80 1 -- 172.21.15.121:0/2943338226 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a7cef9cc0 con 0x556a7cfb4800 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.124+0000 7f8feeffd700 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556a7cfb4800 0x556a7cfb5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.124+0000 7f8feeffd700 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556a7cfb4800 0x556a7cfb5c70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a51f0cf69e9e9951 server_cookie=74abe03e2d708d7d in_seq=0 out_seq=0 2024-06-10T00:27:46.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.124+0000 7f8fceffd700 10 client.15171 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.125+0000 7f8fceffd700 1 -- 172.21.15.121:0/2943338226 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x556a7cef9cc0 con 0x556a7cfb4800 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.125+0000 7f8fceffd700 10 client.15171 handle_command_reply: tid=0 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.125+0000 7f9000246b80 1 client.15171 shutdown 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.125+0000 7f9000246b80 1 -- 172.21.15.121:0/2943338226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 msgr2=0x556a7cfa3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.125+0000 7f9000246b80 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa3f80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4098350 tx=0x7f8fe40a2390 comp rx=0 tx=0).stop 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.125+0000 7f9000246b80 1 -- 172.21.15.121:0/2943338226 shutdown_connections 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.125+0000 7f9000246b80 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556a7cfb4800 0x556a7cfb5c70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.125+0000 7f9000246b80 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cfa44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.125+0000 7f9000246b80 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cfa6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.125+0000 7f9000246b80 1 --2- 172.21.15.121:0/2943338226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfa3f80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.125+0000 7f9000246b80 1 -- 172.21.15.121:0/2943338226 >> 172.21.15.121:0/2943338226 conn(0x556a7cec5c70 msgr2=0x556a7cf955a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.126+0000 7f8fceffd700 0 client.15171 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:27:46.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.126+0000 7f9000246b80 1 -- 172.21.15.121:0/2943338226 shutdown_connections 2024-06-10T00:27:46.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.126+0000 7f9000246b80 1 -- 172.21.15.121:0/2943338226 wait complete. 2024-06-10T00:27:46.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.126+0000 7f9000246b80 20 client.15171 trim_cache size 0 max 16384 2024-06-10T00:27:46.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f9000246b80 1 Processor -- start 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f9000246b80 1 -- start start 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f9000246b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cf9aed0 0x556a7cf9a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f9000246b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f9000246b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cfa0f10 0x556a7cfbf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f9000246b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556a7cfb3fb0 con 0x556a7cfa05f0 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f9000246b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x556a7cfb4680 con 0x556a7cfa0f10 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f9000246b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556a7cfb4d50 con 0x556a7cf9aed0 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f8fee7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f8fee7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34796/0 (socket says 172.21.15.121:34796) 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f8fee7fc700 1 -- 172.21.15.121:0/93239378 learned_addr learned my addr 172.21.15.121:0/93239378 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f8fee7fc700 1 -- 172.21.15.121:0/93239378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cf9aed0 msgr2=0x556a7cf9a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/93239378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cf9aed0 0x556a7cf9a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f8fee7fc700 1 -- 172.21.15.121:0/93239378 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cfa0f10 msgr2=0x556a7cfbf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:46.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f8fef7fe700 1 --2- 172.21.15.121:0/93239378 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cfa0f10 0x556a7cfbf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/93239378 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cfa0f10 0x556a7cfbf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f8fee7fc700 1 -- 172.21.15.121:0/93239378 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a7cfa68c0 con 0x556a7cfa05f0 2024-06-10T00:27:46.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.147+0000 7f8fef7fe700 1 --2- 172.21.15.121:0/93239378 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cfa0f10 0x556a7cfbf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:27:46.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/93239378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf0d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4018d20 tx=0x7f8fe409b3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e282f87e7f084a8c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:46.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f8fd57fa700 1 -- 172.21.15.121:0/93239378 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4099c80 con 0x556a7cfa05f0 2024-06-10T00:27:46.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f8fd57fa700 1 -- 172.21.15.121:0/93239378 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fe40189a0 con 0x556a7cfa05f0 2024-06-10T00:27:46.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f8fd57fa700 1 -- 172.21.15.121:0/93239378 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40a40d0 con 0x556a7cfa05f0 2024-06-10T00:27:46.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f9000246b80 1 -- 172.21.15.121:0/93239378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 msgr2=0x556a7cfbf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:46.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f9000246b80 1 --2- 172.21.15.121:0/93239378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf0d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4018d20 tx=0x7f8fe409b3d0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f9000246b80 1 -- 172.21.15.121:0/93239378 shutdown_connections 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f9000246b80 1 --2- 172.21.15.121:0/93239378 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cfa0f10 0x556a7cfbf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f9000246b80 1 --2- 172.21.15.121:0/93239378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cf9aed0 0x556a7cf9a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f9000246b80 1 --2- 172.21.15.121:0/93239378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf0d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f9000246b80 1 -- 172.21.15.121:0/93239378 >> 172.21.15.121:0/93239378 conn(0x556a7cec5c70 msgr2=0x556a7cfa2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f9000246b80 1 -- 172.21.15.121:0/93239378 shutdown_connections 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.148+0000 7f9000246b80 1 -- 172.21.15.121:0/93239378 wait complete. 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f9000246b80 1 -- start start 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f9000246b80 1 Processor -- start 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f9000246b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cfbf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f9000246b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f9000246b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cfba0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f9000246b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556a7cfb5af0 con 0x556a7cfa05f0 2024-06-10T00:27:46.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f9000246b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x556a7cfb5420 con 0x556a7cf9aed0 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f9000246b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556a7cfb61c0 con 0x556a7cfa0f10 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f9000246b80 10 client.0 _ll_register_callbacks cb 0x556a7ce868d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f8fee7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f8fee7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34798/0 (socket says 172.21.15.121:34798) 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2857626124 learned_addr learned my addr 172.21.15.121:0/2857626124 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.149+0000 7f8feeffd700 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cfbf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2857626124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 msgr2=0x556a7cfba0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fef7fe700 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cfba0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cfba0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2857626124 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 msgr2=0x556a7cfbf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cfbf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fee7fc700 1 -- 172.21.15.121:0/2857626124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a7cfba930 con 0x556a7cfa05f0 2024-06-10T00:27:46.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fef7fe700 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cfba0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8feeffd700 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cfbf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fee7fc700 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf880 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4099c30 tx=0x7f8fe4018bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a356c838a066b3 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fceffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fceffd700 1 -- 172.21.15.121:0/2857626124 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe407f020 con 0x556a7cfa05f0 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fceffd700 1 -- 172.21.15.121:0/2857626124 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f8fe40a42d0 con 0x556a7cfa05f0 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f9000246b80 1 -- 172.21.15.121:0/2857626124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fe40189a0 con 0x556a7cfa05f0 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fceffd700 1 -- 172.21.15.121:0/2857626124 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8fe4099d40 con 0x556a7cfa05f0 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f9000246b80 10 client.15177 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f9000246b80 1 -- 172.21.15.121:0/2857626124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556a7d020b60 con 0x556a7cfa05f0 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.150+0000 7f8fceffd700 1 -- 172.21.15.121:0/2857626124 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fe40189a0 con 0x556a7cfa05f0 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f8fceffd700 1 -- 172.21.15.121:0/2857626124 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8fe4097890 con 0x556a7cfa05f0 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f9000246b80 10 client.15177 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f9000246b80 10 client.15177 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-10T00:27:46.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f9000246b80 20 client.15177 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f9000246b80 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556a7d025220 0x556a7d023f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f9000246b80 4 client.15177 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f9000246b80 1 -- 172.21.15.121:0/2857626124 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "status"}) -- 0x556a7cef4240 con 0x556a7d025220 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f8feeffd700 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556a7d025220 0x556a7d023f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f8feeffd700 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556a7d025220 0x556a7d023f30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c956173440c5b00a server_cookie=a6fc168cc46b2296 in_seq=0 out_seq=0 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f8fceffd700 10 client.15177 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f8fceffd700 1 -- 172.21.15.121:0/2857626124 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x556a7cef4240 con 0x556a7d025220 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f8fceffd700 10 client.15177 handle_command_reply: tid=0 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.151+0000 7f9000246b80 1 client.15177 shutdown 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f9000246b80 1 -- 172.21.15.121:0/2857626124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 msgr2=0x556a7cfbf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f9000246b80 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf880 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4099c30 tx=0x7f8fe4018bc0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f9000246b80 1 -- 172.21.15.121:0/2857626124 shutdown_connections 2024-06-10T00:27:46.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f9000246b80 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556a7d025220 0x556a7d023f30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f9000246b80 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a7cfa0f10 0x556a7cfba0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f9000246b80 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556a7cf9aed0 0x556a7cfbf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f9000246b80 1 --2- 172.21.15.121:0/2857626124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a7cfa05f0 0x556a7cfbf880 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f9000246b80 1 -- 172.21.15.121:0/2857626124 >> 172.21.15.121:0/2857626124 conn(0x556a7cec5c70 msgr2=0x556a7cf8edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:46.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f8fceffd700 0 client.15177 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:27:46.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f9000246b80 1 -- 172.21.15.121:0/2857626124 shutdown_connections 2024-06-10T00:27:46.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f9000246b80 1 -- 172.21.15.121:0/2857626124 wait complete. 2024-06-10T00:27:46.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f9000246b80 20 client.15177 trim_cache size 0 max 16384 2024-06-10T00:27:46.157 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:27:46.157 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:27:46.157 INFO:teuthology.orchestra.run.smithi121.stdout: "whoami": 1, 2024-06-10T00:27:46.158 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 24472, 2024-06-10T00:27:46.158 INFO:teuthology.orchestra.run.smithi121.stdout: "want_state": "up:active", 2024-06-10T00:27:46.158 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "up:active", 2024-06-10T00:27:46.158 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs", 2024-06-10T00:27:46.158 INFO:teuthology.orchestra.run.smithi121.stdout: "rank_uptime": 163.90485249299999, 2024-06-10T00:27:46.158 INFO:teuthology.orchestra.run.smithi121.stdout: "mdsmap_epoch": 25, 2024-06-10T00:27:46.158 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch": 100, 2024-06-10T00:27:46.158 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch_barrier": 98, 2024-06-10T00:27:46.158 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 254.83482246200001 2024-06-10T00:27:46.158 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:27:46.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f8ff59c3700 1 -- 172.21.15.121:0/2275121612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fd80a2c20 msgr2=0x7f8fd80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:46.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f8ff59c3700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fd80a2c20 0x7f8fd80a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0007be0 tx=0x7f8fe00073d0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f8ff59c3700 1 -- 172.21.15.121:0/2275121612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff0102fb0 msgr2=0x7f8ff0197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:46.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f8ff59c3700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff0102fb0 0x7f8ff0197ef0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400c970 tx=0x7f8fe400cc80 comp rx=0 tx=0).stop 2024-06-10T00:27:46.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f8ff59c3700 1 -- 172.21.15.121:0/2275121612 shutdown_connections 2024-06-10T00:27:46.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f8ff59c3700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f8fd80a2c20 0x7f8fd80a50e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f8ff59c3700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ff0103970 0x7f8ff0198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f8ff59c3700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ff01017e0 0x7f8ff01979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f8ff59c3700 1 --2- 172.21.15.121:0/2275121612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ff0102fb0 0x7f8ff0197ef0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:46.159 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.152+0000 7f8ff59c3700 1 -- 172.21.15.121:0/2275121612 >> 172.21.15.121:0/2275121612 conn(0x7f8ff0077430 msgr2=0x7f8ff01062c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:46.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.153+0000 7f8ff59c3700 1 -- 172.21.15.121:0/2275121612 shutdown_connections 2024-06-10T00:27:46.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:46.153+0000 7f8ff59c3700 1 -- 172.21.15.121:0/2275121612 wait complete. 2024-06-10T00:27:46.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:27:46.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:27:46.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:46 smithi121 ceph-mon[32378]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:46.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:27:46.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:27:46.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:46 smithi135 ceph-mon[36424]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:46.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:27:46.708 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:27:46.708 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:46 smithi193 ceph-mon[36716]: from='mgr.14150 172.21.15.121:0/378961438' entity='mgr.x' 2024-06-10T00:27:46.836 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-06-10T00:27:47.073 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:47 smithi121 ceph-mon[32378]: pgmap v422: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:47.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:47 smithi135 ceph-mon[36424]: pgmap v422: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:47.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:47 smithi193 ceph-mon[36716]: pgmap v422: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:27:48.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.841+0000 7fc88b867700 1 -- 172.21.15.121:0/2276551045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc884101380 msgr2=0x7fc884101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:48.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.841+0000 7fc88b867700 1 --2- 172.21.15.121:0/2276551045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc884101380 0x7fc884101800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc874009990 tx=0x7fc874009ca0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.842+0000 7fc88b867700 1 -- 172.21.15.121:0/2276551045 shutdown_connections 2024-06-10T00:27:48.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.842+0000 7fc88b867700 1 --2- 172.21.15.121:0/2276551045 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc884101d40 0x7fc88410e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.842+0000 7fc88b867700 1 --2- 172.21.15.121:0/2276551045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8841073f0 0x7fc8841077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.842+0000 7fc88b867700 1 --2- 172.21.15.121:0/2276551045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc884101380 0x7fc884101800 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.842+0000 7fc88b867700 1 -- 172.21.15.121:0/2276551045 >> 172.21.15.121:0/2276551045 conn(0x7fc884078300 msgr2=0x7fc884078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:48.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.842+0000 7fc88b867700 1 -- 172.21.15.121:0/2276551045 shutdown_connections 2024-06-10T00:27:48.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.842+0000 7fc88b867700 1 -- 172.21.15.121:0/2276551045 wait complete. 2024-06-10T00:27:48.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.842+0000 7fc88b867700 1 Processor -- start 2024-06-10T00:27:48.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.842+0000 7fc88b867700 1 -- start start 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc88b867700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc884101380 0x7fc884199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc88b867700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc884101d40 0x7fc884199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc88b867700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8841073f0 0x7fc8841a4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc88b867700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc884111680 con 0x7fc884101380 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc88b867700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc884111500 con 0x7fc884101d40 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc88b867700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc884111800 con 0x7fc8841073f0 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc889603700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc884101380 0x7fc884199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc889603700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc884101380 0x7fc884199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34812/0 (socket says 172.21.15.121:34812) 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc889603700 1 -- 172.21.15.121:0/3353139669 learned_addr learned my addr 172.21.15.121:0/3353139669 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc888e02700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc884101d40 0x7fc884199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc889e04700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8841073f0 0x7fc8841a4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc889603700 1 -- 172.21.15.121:0/3353139669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8841073f0 msgr2=0x7fc8841a4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:48.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc889603700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8841073f0 0x7fc8841a4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc889603700 1 -- 172.21.15.121:0/3353139669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc884101d40 msgr2=0x7fc884199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:48.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc889603700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc884101d40 0x7fc884199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc889603700 1 -- 172.21.15.121:0/3353139669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8841a5060 con 0x7fc884101380 2024-06-10T00:27:48.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc889603700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc884101380 0x7fc884199aa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc880007f10 tx=0x7fc88000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:48.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc87a7fc700 1 -- 172.21.15.121:0/3353139669 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc880004bb0 con 0x7fc884101380 2024-06-10T00:27:48.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc87a7fc700 1 -- 172.21.15.121:0/3353139669 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc880004d50 con 0x7fc884101380 2024-06-10T00:27:48.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc88b867700 1 -- 172.21.15.121:0/3353139669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8841a5290 con 0x7fc884101380 2024-06-10T00:27:48.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc87a7fc700 1 -- 172.21.15.121:0/3353139669 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8800133f0 con 0x7fc884101380 2024-06-10T00:27:48.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.843+0000 7fc88b867700 1 -- 172.21.15.121:0/3353139669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8841a57a0 con 0x7fc884101380 2024-06-10T00:27:48.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.846+0000 7fc87a7fc700 1 -- 172.21.15.121:0/3353139669 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fc88001a020 con 0x7fc884101380 2024-06-10T00:27:48.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.846+0000 7fc87a7fc700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc8700a2ca0 0x7fc8700a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.846+0000 7fc888e02700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc8700a2ca0 0x7fc8700a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.846+0000 7fc87a7fc700 1 -- 172.21.15.121:0/3353139669 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fc8800cead0 con 0x7fc884101380 2024-06-10T00:27:48.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.847+0000 7fc888e02700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc8700a2ca0 0x7fc8700a5160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc87400b2e0 tx=0x7fc874005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:48.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.847+0000 7fc8960eab80 1 Processor -- start 2024-06-10T00:27:48.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.847+0000 7fc8960eab80 1 -- start start 2024-06-10T00:27:48.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.847+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b89420 0x55d996b89800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.847+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83620 0x55d996b83aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.847+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b83fe0 0x55d996b901c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.847+0000 7fc8960eab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d996a71f10 con 0x55d996b89420 2024-06-10T00:27:48.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.847+0000 7fc8960eab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d996ad0800 con 0x55d996b83fe0 2024-06-10T00:27:48.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.847+0000 7fc8960eab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d996a38410 con 0x55d996b83620 2024-06-10T00:27:48.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc889603700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b89420 0x55d996b89800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc889603700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b89420 0x55d996b89800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34828/0 (socket says 172.21.15.121:34828) 2024-06-10T00:27:48.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc889603700 1 -- 172.21.15.121:0/1998729373 learned_addr learned my addr 172.21.15.121:0/1998729373 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:48.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc889e04700 1 --2- 172.21.15.121:0/1998729373 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b83fe0 0x55d996b901c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc889603700 1 -- 172.21.15.121:0/1998729373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83620 msgr2=0x55d996b83aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:48.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc888e02700 1 --2- 172.21.15.121:0/1998729373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83620 0x55d996b83aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc889603700 1 --2- 172.21.15.121:0/1998729373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83620 0x55d996b83aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc889603700 1 -- 172.21.15.121:0/1998729373 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b83fe0 msgr2=0x55d996b901c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc889603700 1 --2- 172.21.15.121:0/1998729373 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b83fe0 0x55d996b901c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc889603700 1 -- 172.21.15.121:0/1998729373 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d996a1f880 con 0x55d996b89420 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc889e04700 1 --2- 172.21.15.121:0/1998729373 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b83fe0 0x55d996b901c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc889603700 1 --2- 172.21.15.121:0/1998729373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b89420 0x55d996b89800 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc88009ac80 tx=0x7fc880097350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8140cb018759d1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc86f7fe700 1 -- 172.21.15.121:0/1998729373 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88009b680 con 0x55d996b89420 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc86f7fe700 1 -- 172.21.15.121:0/1998729373 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8800b5070 con 0x55d996b89420 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.848+0000 7fc86f7fe700 1 -- 172.21.15.121:0/1998729373 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88000db20 con 0x55d996b89420 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.849+0000 7fc8960eab80 1 -- 172.21.15.121:0/1998729373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b89420 msgr2=0x55d996b89800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.849+0000 7fc8960eab80 1 --2- 172.21.15.121:0/1998729373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b89420 0x55d996b89800 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc88009ac80 tx=0x7fc880097350 comp rx=0 tx=0).stop 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.849+0000 7fc8960eab80 1 -- 172.21.15.121:0/1998729373 shutdown_connections 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.849+0000 7fc8960eab80 1 --2- 172.21.15.121:0/1998729373 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b83fe0 0x55d996b901c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.849+0000 7fc8960eab80 1 --2- 172.21.15.121:0/1998729373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b89420 0x55d996b89800 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.849+0000 7fc8960eab80 1 --2- 172.21.15.121:0/1998729373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83620 0x55d996b83aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.849+0000 7fc8960eab80 1 -- 172.21.15.121:0/1998729373 >> 172.21.15.121:0/1998729373 conn(0x55d996aac030 msgr2=0x55d996b7a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.849+0000 7fc8960eab80 1 -- 172.21.15.121:0/1998729373 shutdown_connections 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.849+0000 7fc8960eab80 1 -- 172.21.15.121:0/1998729373 wait complete. 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.849+0000 7fc8960eab80 1 -- start start 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc8960eab80 1 Processor -- start 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b8fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83fe0 0x55d996b8d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b8d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc8960eab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d996a70f70 con 0x55d996b83620 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc8960eab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d996acc7c0 con 0x55d996b89420 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc8960eab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d996b90700 con 0x55d996b83fe0 2024-06-10T00:27:48.855 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc8960eab80 10 client.0 _ll_register_callbacks cb 0x55d996a66560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc889603700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b8fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc889603700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b8fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34838/0 (socket says 172.21.15.121:34838) 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc889603700 1 -- 172.21.15.121:0/714488896 learned_addr learned my addr 172.21.15.121:0/714488896 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc889e04700 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b8d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc888e02700 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83fe0 0x55d996b8d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc889603700 1 -- 172.21.15.121:0/714488896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83fe0 msgr2=0x55d996b8d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc889603700 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83fe0 0x55d996b8d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc889603700 1 -- 172.21.15.121:0/714488896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 msgr2=0x55d996b8d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc889603700 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b8d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.850+0000 7fc889603700 1 -- 172.21.15.121:0/714488896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d996a96700 con 0x55d996b83620 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc889603700 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b8fe90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc880007ee0 tx=0x7fc88000d8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abcc189e199ab515 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc86cff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:48.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc86cff9700 1 -- 172.21.15.121:0/714488896 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88009b680 con 0x55d996b83620 2024-06-10T00:27:48.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc86cff9700 1 -- 172.21.15.121:0/714488896 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fc8800192c0 con 0x55d996b83620 2024-06-10T00:27:48.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc8960eab80 1 -- 172.21.15.121:0/714488896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc8800b5070 con 0x55d996b83620 2024-06-10T00:27:48.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc86cff9700 1 -- 172.21.15.121:0/714488896 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc8800ac050 con 0x55d996b83620 2024-06-10T00:27:48.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc8960eab80 10 client.15189 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:48.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc8960eab80 1 -- 172.21.15.121:0/714488896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d996b9ad30 con 0x55d996b83620 2024-06-10T00:27:48.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc86cff9700 1 -- 172.21.15.121:0/714488896 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8800b5070 con 0x55d996b83620 2024-06-10T00:27:48.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc86cff9700 1 -- 172.21.15.121:0/714488896 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc8800a3760 con 0x55d996b83620 2024-06-10T00:27:48.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc8960eab80 10 client.15189 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:48.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc8960eab80 10 client.15189 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-10T00:27:48.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc8960eab80 20 client.15189 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:48.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc8960eab80 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d996b9c080 0x55d996b9e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc8960eab80 4 client.15189 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:27:48.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.851+0000 7fc8960eab80 1 -- 172.21.15.121:0/714488896 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d996addd00 con 0x55d996b9c080 2024-06-10T00:27:48.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.852+0000 7fc888e02700 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d996b9c080 0x55d996b9e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.852+0000 7fc888e02700 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d996b9c080 0x55d996b9e180 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b6f8c82a3271bab server_cookie=12fb8d7e361ef731 in_seq=0 out_seq=0 2024-06-10T00:27:48.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.852+0000 7fc86cff9700 10 client.15189 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:27:48.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc86cff9700 1 -- 172.21.15.121:0/714488896 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d996addd00 con 0x55d996b9c080 2024-06-10T00:27:48.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc86cff9700 10 client.15189 handle_command_reply: tid=0 2024-06-10T00:27:48.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc8960eab80 1 client.15189 shutdown 2024-06-10T00:27:48.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc8960eab80 1 -- 172.21.15.121:0/714488896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 msgr2=0x55d996b8fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:48.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc8960eab80 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b8fe90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc880007ee0 tx=0x7fc88000d8a0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc8960eab80 1 -- 172.21.15.121:0/714488896 shutdown_connections 2024-06-10T00:27:48.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc8960eab80 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d996b9c080 0x55d996b9e180 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc8960eab80 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b8d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.861 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc8960eab80 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b8fe90 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc8960eab80 1 --2- 172.21.15.121:0/714488896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83fe0 0x55d996b8d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc8960eab80 1 -- 172.21.15.121:0/714488896 >> 172.21.15.121:0/714488896 conn(0x55d996aac030 msgr2=0x55d996b796b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:48.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc86cff9700 0 client.15189 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:27:48.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc8960eab80 1 -- 172.21.15.121:0/714488896 shutdown_connections 2024-06-10T00:27:48.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.853+0000 7fc8960eab80 1 -- 172.21.15.121:0/714488896 wait complete. 2024-06-10T00:27:48.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.854+0000 7fc8960eab80 20 client.15189 trim_cache size 0 max 16384 2024-06-10T00:27:48.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc8960eab80 1 Processor -- start 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc8960eab80 1 -- start start 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83620 0x55d996b9c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83fe0 0x55d996b995e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b8c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc8960eab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d996b9c4c0 con 0x55d996b83fe0 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc8960eab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d996b9cb90 con 0x55d996b89420 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc8960eab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d996b9d260 con 0x55d996b83620 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc888e02700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83fe0 0x55d996b995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc888e02700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83fe0 0x55d996b995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34842/0 (socket says 172.21.15.121:34842) 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc888e02700 1 -- 172.21.15.121:0/2669767285 learned_addr learned my addr 172.21.15.121:0/2669767285 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc889e04700 1 --2- 172.21.15.121:0/2669767285 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b8c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc889603700 1 --2- 172.21.15.121:0/2669767285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83620 0x55d996b9c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc888e02700 1 -- 172.21.15.121:0/2669767285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83620 msgr2=0x55d996b9c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc888e02700 1 --2- 172.21.15.121:0/2669767285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83620 0x55d996b9c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc888e02700 1 -- 172.21.15.121:0/2669767285 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 msgr2=0x55d996b8c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc888e02700 1 --2- 172.21.15.121:0/2669767285 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b8c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.875+0000 7fc888e02700 1 -- 172.21.15.121:0/2669767285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d996b71fa0 con 0x55d996b83fe0 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.876+0000 7fc888e02700 1 --2- 172.21.15.121:0/2669767285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83fe0 0x55d996b995e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc874009990 tx=0x7fc874021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee86494d2a575e0f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.876+0000 7fc86effd700 1 -- 172.21.15.121:0/2669767285 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874003680 con 0x55d996b83fe0 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.876+0000 7fc86effd700 1 -- 172.21.15.121:0/2669767285 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc874003d00 con 0x55d996b83fe0 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.876+0000 7fc86effd700 1 -- 172.21.15.121:0/2669767285 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874031ce0 con 0x55d996b83fe0 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.876+0000 7fc8960eab80 1 -- 172.21.15.121:0/2669767285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83fe0 msgr2=0x55d996b995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.876+0000 7fc8960eab80 1 --2- 172.21.15.121:0/2669767285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83fe0 0x55d996b995e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc874009990 tx=0x7fc874021000 comp rx=0 tx=0).stop 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 -- 172.21.15.121:0/2669767285 shutdown_connections 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 --2- 172.21.15.121:0/2669767285 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b8c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 --2- 172.21.15.121:0/2669767285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83620 0x55d996b9c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 --2- 172.21.15.121:0/2669767285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83fe0 0x55d996b995e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 -- 172.21.15.121:0/2669767285 >> 172.21.15.121:0/2669767285 conn(0x55d996aac030 msgr2=0x55d996b8b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 -- 172.21.15.121:0/2669767285 shutdown_connections 2024-06-10T00:27:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 -- 172.21.15.121:0/2669767285 wait complete. 2024-06-10T00:27:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 -- start start 2024-06-10T00:27:48.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 Processor -- start 2024-06-10T00:27:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b79820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83fe0 0x55d996b79d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b7a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d996b9e000 con 0x55d996b83620 2024-06-10T00:27:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.877+0000 7fc8960eab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d996b9d930 con 0x55d996b89420 2024-06-10T00:27:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc8960eab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d996b9e6d0 con 0x55d996b83fe0 2024-06-10T00:27:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc889603700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b79820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc8960eab80 10 client.0 _ll_register_callbacks cb 0x55d996a66560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:27:48.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc889603700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b79820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34856/0 (socket says 172.21.15.121:34856) 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc889603700 1 -- 172.21.15.121:0/502735097 learned_addr learned my addr 172.21.15.121:0/502735097 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc888e02700 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83fe0 0x55d996b79d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc889e04700 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b7a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc889603700 1 -- 172.21.15.121:0/502735097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83fe0 msgr2=0x55d996b79d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc889603700 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83fe0 0x55d996b79d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc889603700 1 -- 172.21.15.121:0/502735097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 msgr2=0x55d996b7a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc889603700 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b7a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc889603700 1 -- 172.21.15.121:0/502735097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d996ba5f60 con 0x55d996b83620 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc889603700 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b79820 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc880099230 tx=0x7fc88000d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d321d2e056b33f74 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc86cff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc86cff9700 1 -- 172.21.15.121:0/502735097 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88009a8e0 con 0x55d996b83620 2024-06-10T00:27:48.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc86cff9700 1 -- 172.21.15.121:0/502735097 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fc8800ce0e0 con 0x55d996b83620 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc8960eab80 1 -- 172.21.15.121:0/502735097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc874003d00 con 0x55d996b83620 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc86cff9700 1 -- 172.21.15.121:0/502735097 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc8800194c0 con 0x55d996b83620 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.878+0000 7fc8960eab80 10 client.15195 fetch_fsmap learned FSMap version 25 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.879+0000 7fc8960eab80 1 -- 172.21.15.121:0/502735097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d996c08a20 con 0x55d996b83620 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.879+0000 7fc86cff9700 1 -- 172.21.15.121:0/502735097 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc874003d00 con 0x55d996b83620 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.879+0000 7fc86cff9700 1 -- 172.21.15.121:0/502735097 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc88009b1e0 con 0x55d996b83620 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.879+0000 7fc8960eab80 10 client.15195 fetch_fsmap finished waiting for FSMap version 25 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.879+0000 7fc8960eab80 10 client.15195 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.879+0000 7fc8960eab80 20 client.15195 populate_metadata read hostname 'smithi121' 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.879+0000 7fc8960eab80 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d996c0d380 0x55d996c0c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.879+0000 7fc8960eab80 4 client.15195 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.879+0000 7fc8960eab80 1 -- 172.21.15.121:0/502735097 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "status"}) -- 0x55d996ad8280 con 0x55d996c0d380 2024-06-10T00:27:48.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.879+0000 7fc888e02700 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d996c0d380 0x55d996c0c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc888e02700 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d996c0d380 0x55d996c0c090 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6582af783792d10e server_cookie=6e6d2c09491950c3 in_seq=0 out_seq=0 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc86cff9700 10 client.15195 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc86cff9700 1 -- 172.21.15.121:0/502735097 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x55d996ad8280 con 0x55d996c0d380 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc86cff9700 10 client.15195 handle_command_reply: tid=0 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 1 client.15195 shutdown 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 1 -- 172.21.15.121:0/502735097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 msgr2=0x55d996b79820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b79820 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc880099230 tx=0x7fc88000d830 comp rx=0 tx=0).stop 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 1 -- 172.21.15.121:0/502735097 shutdown_connections 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d996c0d380 0x55d996c0c090 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d996b89420 0x55d996b7a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d996b83620 0x55d996b79820 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 1 --2- 172.21.15.121:0/502735097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d996b83fe0 0x55d996b79d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 1 -- 172.21.15.121:0/502735097 >> 172.21.15.121:0/502735097 conn(0x55d996aac030 msgr2=0x55d996b7ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc86cff9700 0 client.15195 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:27:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 1 -- 172.21.15.121:0/502735097 shutdown_connections 2024-06-10T00:27:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 1 -- 172.21.15.121:0/502735097 wait complete. 2024-06-10T00:27:48.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.880+0000 7fc8960eab80 20 client.15195 trim_cache size 0 max 16384 2024-06-10T00:27:48.886 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:27:48.886 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:27:48.886 INFO:teuthology.orchestra.run.smithi121.stdout: "whoami": 0, 2024-06-10T00:27:48.886 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 24479, 2024-06-10T00:27:48.886 INFO:teuthology.orchestra.run.smithi121.stdout: "want_state": "up:active", 2024-06-10T00:27:48.886 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "up:active", 2024-06-10T00:27:48.887 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs", 2024-06-10T00:27:48.887 INFO:teuthology.orchestra.run.smithi121.stdout: "rank_uptime": 229.33463317100001, 2024-06-10T00:27:48.887 INFO:teuthology.orchestra.run.smithi121.stdout: "mdsmap_epoch": 25, 2024-06-10T00:27:48.887 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch": 100, 2024-06-10T00:27:48.887 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch_barrier": 98, 2024-06-10T00:27:48.887 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 247.628625064 2024-06-10T00:27:48.887 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:27:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 -- 172.21.15.121:0/3353139669 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc8700a2ca0 msgr2=0x7fc8700a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc8700a2ca0 0x7fc8700a5160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc87400b2e0 tx=0x7fc874005960 comp rx=0 tx=0).stop 2024-06-10T00:27:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 -- 172.21.15.121:0/3353139669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc884101380 msgr2=0x7fc884199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc884101380 0x7fc884199aa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc880007f10 tx=0x7fc88000d3b0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 -- 172.21.15.121:0/3353139669 shutdown_connections 2024-06-10T00:27:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fc8700a2ca0 0x7fc8700a5160 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8841073f0 0x7fc8841a4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc884101380 0x7fc884199aa0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 --2- 172.21.15.121:0/3353139669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc884101d40 0x7fc884199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:48.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 -- 172.21.15.121:0/3353139669 >> 172.21.15.121:0/3353139669 conn(0x7fc884078300 msgr2=0x7fc8840fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 -- 172.21.15.121:0/3353139669 shutdown_connections 2024-06-10T00:27:48.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:48.881+0000 7fc88b867700 1 -- 172.21.15.121:0/3353139669 wait complete. 2024-06-10T00:27:49.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:49 smithi135 ceph-mon[36424]: pgmap v423: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:49.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:49 smithi193 ceph-mon[36716]: pgmap v423: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:49.644 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:49 smithi121 ceph-mon[32378]: pgmap v423: 129 pgs: 129 active+clean; 642 KiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-06-10T00:27:49.672 INFO:teuthology.run_tasks:Running task sequential... 2024-06-10T00:27:49.681 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-06-10T00:27:49.681 INFO:teuthology.task.sequential:In sequential, running task print... 2024-06-10T00:27:49.681 INFO:teuthology.task.print:Enabling mgr modules 2024-06-10T00:27:49.682 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-06-10T00:27:49.682 INFO:teuthology.task.exec:Executing custom commands... 2024-06-10T00:27:49.683 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi121.front.sepia.ceph.com 2024-06-10T00:27:49.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-06-10T00:27:49.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.785+0000 7fd804eb3640 1 -- 172.21.15.121:0/1492827271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800101ae0 msgr2=0x7fd800101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:49.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.785+0000 7fd804eb3640 1 --2- 172.21.15.121:0/1492827271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800101ae0 0x7fd800101f60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8009900 tx=0x7fd7e802a410 comp rx=0 tx=0).stop 2024-06-10T00:27:49.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.785+0000 7fd804eb3640 1 -- 172.21.15.121:0/1492827271 shutdown_connections 2024-06-10T00:27:49.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.785+0000 7fd804eb3640 1 --2- 172.21.15.121:0/1492827271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8001024a0 0x7fd800108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:49.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.785+0000 7fd804eb3640 1 --2- 172.21.15.121:0/1492827271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800101ae0 0x7fd800101f60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:49.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.785+0000 7fd804eb3640 1 --2- 172.21.15.121:0/1492827271 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd8001008f0 0x7fd800100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:49.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.785+0000 7fd804eb3640 1 -- 172.21.15.121:0/1492827271 >> 172.21.15.121:0/1492827271 conn(0x7fd8000fbe60 msgr2=0x7fd8000fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:49.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.785+0000 7fd804eb3640 1 -- 172.21.15.121:0/1492827271 shutdown_connections 2024-06-10T00:27:49.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.785+0000 7fd804eb3640 1 -- 172.21.15.121:0/1492827271 wait complete. 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.786+0000 7fd804eb3640 1 Processor -- start 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.786+0000 7fd804eb3640 1 -- start start 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.786+0000 7fd804eb3640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8001008f0 0x7fd800197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.786+0000 7fd804eb3640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd800101ae0 0x7fd8001981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.786+0000 7fd804eb3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8001024a0 0x7fd80019f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd804eb3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd80010d690 con 0x7fd8001024a0 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd804eb3640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd80010d510 con 0x7fd800101ae0 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd804eb3640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd80010d810 con 0x7fd8001008f0 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7fed9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8001024a0 0x7fd80019f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7fed9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8001024a0 0x7fd80019f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47270/0 (socket says 172.21.15.121:47270) 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7fed9d640 1 -- 172.21.15.121:0/1233249480 learned_addr learned my addr 172.21.15.121:0/1233249480 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7f5d9b640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd800101ae0 0x7fd8001981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7fe59c640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8001008f0 0x7fd800197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:49.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7fed9d640 1 -- 172.21.15.121:0/1233249480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8001008f0 msgr2=0x7fd800197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:49.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7fed9d640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8001008f0 0x7fd800197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:49.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7fed9d640 1 -- 172.21.15.121:0/1233249480 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd800101ae0 msgr2=0x7fd8001981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:49.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7fed9d640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd800101ae0 0x7fd8001981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:49.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7fed9d640 1 -- 172.21.15.121:0/1233249480 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80019f890 con 0x7fd8001024a0 2024-06-10T00:27:49.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7fed9d640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8001024a0 0x7fd80019f1f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800cad0 tx=0x7fd7f800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:49.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7f7fff640 1 -- 172.21.15.121:0/1233249480 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f800ae20 con 0x7fd8001024a0 2024-06-10T00:27:49.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7f7fff640 1 -- 172.21.15.121:0/1233249480 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7f8005020 con 0x7fd8001024a0 2024-06-10T00:27:49.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd804eb3640 1 -- 172.21.15.121:0/1233249480 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd80019fb80 con 0x7fd8001024a0 2024-06-10T00:27:49.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd7f7fff640 1 -- 172.21.15.121:0/1233249480 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f800f6f0 con 0x7fd8001024a0 2024-06-10T00:27:49.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.787+0000 7fd804eb3640 1 -- 172.21.15.121:0/1233249480 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8001a0080 con 0x7fd8001024a0 2024-06-10T00:27:49.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.788+0000 7fd804eb3640 1 -- 172.21.15.121:0/1233249480 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd800107bb0 con 0x7fd8001024a0 2024-06-10T00:27:49.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.789+0000 7fd7f7fff640 1 -- 172.21.15.121:0/1233249480 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd7f80051d0 con 0x7fd8001024a0 2024-06-10T00:27:49.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.790+0000 7fd7f7fff640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7ec0a2c30 0x7fd7ec0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:49.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.790+0000 7fd7fe59c640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7ec0a2c30 0x7fd7ec0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:49.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.790+0000 7fd7f7fff640 1 -- 172.21.15.121:0/1233249480 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fd7f80cf540 con 0x7fd8001024a0 2024-06-10T00:27:49.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.791+0000 7fd7fe59c640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7ec0a2c30 0x7fd7ec0a50f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0005fd0 tx=0x7fd7f0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:49.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:49.794+0000 7fd7f7fff640 1 -- 172.21.15.121:0/1233249480 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd7f8014030 con 0x7fd8001024a0 2024-06-10T00:27:50.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.023+0000 7fd804eb3640 1 -- 172.21.15.121:0/1233249480 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fd800062640 con 0x7fd8001024a0 2024-06-10T00:27:50.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.240+0000 7fd7f7fff640 1 -- 172.21.15.121:0/1233249480 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fd7f809c0f0 con 0x7fd8001024a0 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 -- 172.21.15.121:0/1233249480 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7ec0a2c30 msgr2=0x7fd7ec0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7ec0a2c30 0x7fd7ec0a50f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0005fd0 tx=0x7fd7f0005850 comp rx=0 tx=0).stop 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 -- 172.21.15.121:0/1233249480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8001024a0 msgr2=0x7fd80019f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8001024a0 0x7fd80019f1f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800cad0 tx=0x7fd7f800cf90 comp rx=0 tx=0).stop 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 -- 172.21.15.121:0/1233249480 shutdown_connections 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fd7ec0a2c30 0x7fd7ec0a50f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8001024a0 0x7fd80019f1f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd800101ae0 0x7fd8001981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 --2- 172.21.15.121:0/1233249480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8001008f0 0x7fd800197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 -- 172.21.15.121:0/1233249480 >> 172.21.15.121:0/1233249480 conn(0x7fd8000fbe60 msgr2=0x7fd800104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 -- 172.21.15.121:0/1233249480 shutdown_connections 2024-06-10T00:27:50.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:50.244+0000 7fd804eb3640 1 -- 172.21.15.121:0/1233249480 wait complete. 2024-06-10T00:27:50.261 INFO:teuthology.run_tasks:Running task kclient... 2024-06-10T00:27:50.276 INFO:tasks.kclient:Mounting kernel clients... 2024-06-10T00:27:50.276 INFO:tasks.kclient:config is {'client.0': {}} 2024-06-10T00:27:50.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:27:50.277 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-06-10T00:27:50.292 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-06-10T00:27:50.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:27:50.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-06-10T00:27:50.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:27:50.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2024-06-10T00:27:50.467 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-06-10T00:27:50.467 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-06-10T00:27:50.467 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-10T00:27:50.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs ls 2024-06-10T00:27:50.490 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:50 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1233249480' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-06-10T00:27:50.491 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:50 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: ignoring --setuser ceph since I am not root 2024-06-10T00:27:50.491 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:50 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: ignoring --setgroup ceph since I am not root 2024-06-10T00:27:50.491 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:50 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:50.428+0000 7fb015c11200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-10T00:27:50.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:50 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1233249480' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-06-10T00:27:50.508 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:50 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: ignoring --setuser ceph since I am not root 2024-06-10T00:27:50.508 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:50 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: ignoring --setgroup ceph since I am not root 2024-06-10T00:27:50.508 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:50 smithi135 ceph-mgr[37815]: -- 172.21.15.135:0/3900603585 <== mon.2 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563c187a85a0 con 0x563c1877cc00 2024-06-10T00:27:50.508 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:50 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:50.418+0000 7f50c51d8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-10T00:27:50.608 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:50 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1233249480' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-06-10T00:27:50.609 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:50 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: ignoring --setuser ceph since I am not root 2024-06-10T00:27:50.609 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:50 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: ignoring --setgroup ceph since I am not root 2024-06-10T00:27:50.609 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:50 smithi193 ceph-mgr[38541]: -- 172.21.15.193:0/3613270537 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d7e5bac5a0 con 0x55d7e5b80c00 2024-06-10T00:27:50.609 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:50 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:50.423+0000 7fccee99a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-10T00:27:50.609 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:50 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:50.608+0000 7fccee99a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:27:50.720 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:50.743 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:50 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:50.623+0000 7fb015c11200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:27:50.880 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:50 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:50.709+0000 7fccee99a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-10T00:27:50.993 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:50 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:50.743+0000 7fb015c11200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-10T00:27:51.007 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:50 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:50.603+0000 7f50c51d8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:27:51.007 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:50 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:50.704+0000 7f50c51d8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-10T00:27:51.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:51 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1233249480' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-06-10T00:27:51.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:51 smithi193 ceph-mon[36716]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-06-10T00:27:51.381 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:51 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:51.022+0000 7fccee99a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-10T00:27:51.381 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:51 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:51.107+0000 7fccee99a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-10T00:27:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:51 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1233249480' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-06-10T00:27:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:51 smithi121 ceph-mon[32378]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-06-10T00:27:51.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:51 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:51.067+0000 7fb015c11200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-10T00:27:51.449 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:51 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:51.154+0000 7fb015c11200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-10T00:27:51.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:51 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1233249480' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-06-10T00:27:51.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:51 smithi135 ceph-mon[36424]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-06-10T00:27:51.508 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:51 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:51.009+0000 7f50c51d8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-10T00:27:51.508 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:51 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:51.095+0000 7f50c51d8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-10T00:27:52.463 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:52 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:52.366+0000 7fb015c11200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-10T00:27:52.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.463+0000 7febac622700 1 -- 172.21.15.121:0/4034147678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feba4108530 msgr2=0x7feba4108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:52.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.463+0000 7febac622700 1 --2- 172.21.15.121:0/4034147678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feba4108530 0x7feba4108910 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7feba0009990 tx=0x7feba0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.463+0000 7febac622700 1 -- 172.21.15.121:0/4034147678 shutdown_connections 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.463+0000 7febac622700 1 --2- 172.21.15.121:0/4034147678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feba406b5c0 0x7feba410f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.463+0000 7febac622700 1 --2- 172.21.15.121:0/4034147678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feba4108530 0x7feba4108910 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.463+0000 7febac622700 1 --2- 172.21.15.121:0/4034147678 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feba406ac00 0x7feba406b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.463+0000 7febac622700 1 -- 172.21.15.121:0/4034147678 >> 172.21.15.121:0/4034147678 conn(0x7feba4077430 msgr2=0x7feba4077840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.463+0000 7febac622700 1 -- 172.21.15.121:0/4034147678 shutdown_connections 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.463+0000 7febac622700 1 -- 172.21.15.121:0/4034147678 wait complete. 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febac622700 1 Processor -- start 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febac622700 1 -- start start 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febac622700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feba406ac00 0x7feba41a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febac622700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feba406b5c0 0x7feba41a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febac622700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feba4108530 0x7feba410e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febac622700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feba4112790 con 0x7feba406ac00 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febac622700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7feba4112610 con 0x7feba4108530 2024-06-10T00:27:52.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febac622700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feba4112910 con 0x7feba406b5c0 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febaabbf700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feba4108530 0x7feba410e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7feba9bbd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feba406b5c0 0x7feba41a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febaabbf700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feba4108530 0x7feba410e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:57998/0 (socket says 172.21.15.121:57998) 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7feba9bbd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feba406b5c0 0x7feba41a5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:51166/0 (socket says 172.21.15.121:51166) 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febaabbf700 1 -- 172.21.15.121:0/1564007847 learned_addr learned my addr 172.21.15.121:0/1564007847 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.464+0000 7febaa3be700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feba406ac00 0x7feba41a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7febaabbf700 1 -- 172.21.15.121:0/1564007847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feba406b5c0 msgr2=0x7feba41a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7febaabbf700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feba406b5c0 0x7feba41a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7febaabbf700 1 -- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feba406ac00 msgr2=0x7feba41a4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7febaabbf700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feba406ac00 0x7feba41a4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7febaabbf700 1 -- 172.21.15.121:0/1564007847 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba410ecf0 con 0x7feba4108530 2024-06-10T00:27:52.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7feba9bbd700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feba406b5c0 0x7feba41a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:27:52.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7febaa3be700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feba406ac00 0x7feba41a4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:27:52.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7febaabbf700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feba4108530 0x7feba410e520 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feb9c007ee0 tx=0x7feb9c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:52.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7feb9b7fe700 1 -- 172.21.15.121:0/1564007847 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c00ddd0 con 0x7feba4108530 2024-06-10T00:27:52.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7febac622700 1 -- 172.21.15.121:0/1564007847 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba410efe0 con 0x7feba4108530 2024-06-10T00:27:52.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7feb9b7fe700 1 -- 172.21.15.121:0/1564007847 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb9c004020 con 0x7feba4108530 2024-06-10T00:27:52.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7feb9b7fe700 1 -- 172.21.15.121:0/1564007847 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c015660 con 0x7feba4108530 2024-06-10T00:27:52.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.465+0000 7febac622700 1 -- 172.21.15.121:0/1564007847 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba41abd40 con 0x7feba4108530 2024-06-10T00:27:52.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.466+0000 7febac622700 1 -- 172.21.15.121:0/1564007847 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feba4069180 con 0x7feba4108530 2024-06-10T00:27:52.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.468+0000 7feb9b7fe700 1 -- 172.21.15.121:0/1564007847 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7feb9c003a00 con 0x7feba4108530 2024-06-10T00:27:52.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.469+0000 7feb9b7fe700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7feb900a2cf0 0x7feb900a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:52.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.469+0000 7febaa3be700 1 -- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7feb900a2cf0 msgr2=0x7feb900a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3506007074 2024-06-10T00:27:52.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.469+0000 7feb9b7fe700 1 -- 172.21.15.121:0/1564007847 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7feb9c0cedc0 con 0x7feba4108530 2024-06-10T00:27:52.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.469+0000 7febaa3be700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7feb900a2cf0 0x7feb900a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:27:52.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.474+0000 7feb9b7fe700 1 -- 172.21.15.121:0/1564007847 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7feb9c010070 con 0x7feba4108530 2024-06-10T00:27:52.613 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:52 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:52.270+0000 7f50c51d8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-10T00:27:52.613 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:52 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:52.440+0000 7f50c51d8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-10T00:27:52.630 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:52 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:52.301+0000 7fccee99a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-10T00:27:52.630 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:52 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:52.473+0000 7fccee99a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-10T00:27:52.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.670+0000 7febaa3be700 1 -- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7feb900a2cf0 msgr2=0x7feb900a51b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3506007074 2024-06-10T00:27:52.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.670+0000 7febaa3be700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7feb900a2cf0 0x7feb900a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-10T00:27:52.722 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:52 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:52.547+0000 7fb015c11200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-10T00:27:52.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.723+0000 7febac622700 1 -- 172.21.15.121:0/1564007847 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7feba41ac060 con 0x7feba4108530 2024-06-10T00:27:52.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.724+0000 7feb9b7fe700 1 -- 172.21.15.121:0/1564007847 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+88 (secure 0 0 0) 0x7feb9c09b970 con 0x7feba4108530 2024-06-10T00:27:52.724 INFO:teuthology.orchestra.run.smithi121.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-06-10T00:27:52.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 -- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7feb900a2cf0 msgr2=0x7feb900a51b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:27:52.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7feb900a2cf0 0x7feb900a51b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 -- 172.21.15.121:0/1564007847 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feba4108530 msgr2=0x7feba410e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:52.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feba4108530 0x7feba410e520 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7feb9c007ee0 tx=0x7feb9c00d3b0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 -- 172.21.15.121:0/1564007847 shutdown_connections 2024-06-10T00:27:52.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7feb900a2cf0 0x7feb900a51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feba4108530 0x7feba410e520 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feba406ac00 0x7feba41a4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 --2- 172.21.15.121:0/1564007847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feba406b5c0 0x7feba41a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:52.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 -- 172.21.15.121:0/1564007847 >> 172.21.15.121:0/1564007847 conn(0x7feba4077430 msgr2=0x7feba40ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:52.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 -- 172.21.15.121:0/1564007847 shutdown_connections 2024-06-10T00:27:52.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:52.726+0000 7febac622700 1 -- 172.21.15.121:0/1564007847 wait complete. 2024-06-10T00:27:52.902 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:52 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1564007847' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-10T00:27:52.902 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:52 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:52.613+0000 7f50c51d8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-10T00:27:52.938 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:52 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1564007847' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-10T00:27:52.938 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:52 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:52.646+0000 7fccee99a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-10T00:27:53.025 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:52 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1564007847' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-06-10T00:27:53.025 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:52 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:52.723+0000 7fb015c11200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-10T00:27:53.198 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:52 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:52.902+0000 7f50c51d8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-10T00:27:53.198 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:52 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:52.993+0000 7f50c51d8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-10T00:27:53.238 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:52 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:52.938+0000 7fccee99a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-10T00:27:53.238 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:53.031+0000 7fccee99a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-10T00:27:53.327 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:53 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:53.025+0000 7fb015c11200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-10T00:27:53.327 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:53 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:53.120+0000 7fb015c11200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-10T00:27:53.458 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-06-10T00:27:53.458 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-06-10T00:27:53.458 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi121.front.sepia.ceph.com 2024-06-10T00:27:53.458 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-06-10T00:27:53.458 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-06-10T00:27:53.458 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-06-10T00:27:53.458 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-06-10T00:27:53.458 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-06-10T00:27:53.458 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-06-10T00:27:53.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:27:53.459 DEBUG:teuthology.orchestra.run.smithi121:> ip addr 2024-06-10T00:27:53.507 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:53 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:53.198+0000 7f50c51d8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-10T00:27:53.507 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:53 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:53.392+0000 7f50c51d8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:27:53.515 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-06-10T00:27:53.515 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-06-10T00:27:53.515 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2024-06-10T00:27:53.515 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-06-10T00:27:53.515 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:82 brd ff:ff:ff:ff:ff:ff 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f0 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:83 brd ff:ff:ff:ff:ff:ff 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f1 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a2 brd ff:ff:ff:ff:ff:ff 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a3 brd ff:ff:ff:ff:ff:ff 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout: inet 172.21.15.121/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-06-10T00:27:53.516 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft 41997sec preferred_lft 41997sec 2024-06-10T00:27:53.517 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2024-06-10T00:27:53.517 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-06-10T00:27:53.517 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-06-10T00:27:53.517 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-10T00:27:53.517 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-06-10T00:27:53.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add name ceph-brx type bridge 2024-06-10T00:27:53.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr flush dev ceph-brx 2024-06-10T00:27:53.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set ceph-brx up 2024-06-10T00:27:53.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-06-10T00:27:53.517 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-06-10T00:27:53.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-10T00:27:53.614 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:53 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:53.327+0000 7fb015c11200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-10T00:27:53.614 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:53 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:53.529+0000 7fb015c11200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:27:53.630 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:53.238+0000 7fccee99a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-10T00:27:53.630 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:53.435+0000 7fccee99a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-10T00:27:53.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:27:53.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:27:53.659 DEBUG:teuthology.orchestra.run.smithi121:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-06-10T00:27:53.684 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-06-10T00:27:53.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:27:53.686 DEBUG:teuthology.orchestra.run.smithi121:> route 2024-06-10T00:27:53.742 INFO:teuthology.orchestra.run.smithi121.stdout:Kernel IP routing table 2024-06-10T00:27:53.743 INFO:teuthology.orchestra.run.smithi121.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-06-10T00:27:53.743 INFO:teuthology.orchestra.run.smithi121.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-06-10T00:27:53.743 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-06-10T00:27:53.743 INFO:teuthology.orchestra.run.smithi121.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-06-10T00:27:53.743 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-10T00:27:53.743 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-06-10T00:27:53.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-06-10T00:27:53.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-06-10T00:27:53.743 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-06-10T00:27:53.743 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-06-10T00:27:53.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-10T00:27:53.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:27:53.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:27:53.872 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-06-10T00:27:53.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:27:53.887 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list-id 2024-06-10T00:27:53.943 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-10T00:27:53.943 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-06-10T00:27:53.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-10T00:27:53.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-06-10T00:27:53.943 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-06-10T00:27:54.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-10T00:27:54.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:27:54.066 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-06-10T00:27:54.066 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-10T00:27:54.066 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-06-10T00:27:54.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-06-10T00:27:54.066 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-06-10T00:27:54.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-06-10T00:27:54.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-06-10T00:27:54.067 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-06-10T00:27:54.067 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-06-10T00:27:54.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-10T00:27:54.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:27:54.183 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-06-10T00:27:54.183 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-06-10T00:27:54.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set brx.0 up 2024-06-10T00:27:54.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set dev brx.0 master ceph-brx 2024-06-10T00:27:54.183 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-06-10T00:27:54.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-10T00:27:54.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:27:54.257 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:53 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:53.991+0000 7f50c51d8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-10T00:27:54.262 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-10T00:27:54.262 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-06-10T00:27:54.277 INFO:teuthology.orchestra.run.smithi121.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-06-10T00:27:54.277 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-06-10T00:27:54.277 DEBUG:teuthology.orchestra.run.smithi121:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-06-10T00:27:54.332 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-06-10T00:27:54.333 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-06-10T00:27:54.333 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca 2024-06-10T00:27:54.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:27:54.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-06-10T00:27:54.380 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:54 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:54.041+0000 7fccee99a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-10T00:27:54.394 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:54 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:54.175+0000 7fb015c11200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-10T00:27:54.417 INFO:teuthology.orchestra.run.smithi121.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-06-10T00:27:54.417 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-06-10T00:27:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-06-10T00:27:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-06-10T00:27:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: resolved to: "172.21.15.121:3300,172.21.15.135:3300,172.21.15.193:3300" 2024-06-10T00:27:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: trying mount with new device syntax: 0@6c432244-26be-11ef-bc9b-c7b262605968.cephfs=/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca 2024-06-10T00:27:54.573 INFO:teuthology.orchestra.run.smithi121.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,name=0,key=0,mon_addr=172.21.15.121:3300/172.21.15.135:3300/172.21.15.193:3300" will pass to kernel 2024-06-10T00:27:54.574 INFO:teuthology.orchestra.run.smithi121.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-06-10T00:27:54.574 INFO:teuthology.orchestra.run.smithi121.stdout: You just mounted a file system that supports labels which does not 2024-06-10T00:27:54.574 INFO:teuthology.orchestra.run.smithi121.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-06-10T00:27:54.574 INFO:teuthology.orchestra.run.smithi121.stdout: applications will generate AVC messages and not be allowed access to 2024-06-10T00:27:54.575 INFO:teuthology.orchestra.run.smithi121.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-06-10T00:27:54.576 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-06-10T00:27:54.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:27:54.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-06-10T00:27:54.643 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-10T00:27:54.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:54 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:54.645+0000 7fb015c11200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-10T00:27:54.709 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:54 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:54.410+0000 7f50c51d8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-10T00:27:54.710 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:54 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:54.508+0000 7f50c51d8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-10T00:27:54.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-10T00:27:54.735 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:27:54.741 INFO:teuthology.orchestra.run.smithi121.stdout:client24871 2024-06-10T00:27:54.741 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:27:54.742 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-10T00:27:54.770 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:54 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:54.470+0000 7fccee99a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-10T00:27:54.770 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:54 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:54.566+0000 7fccee99a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-10T00:27:54.798 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-10T00:27:54.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-10T00:27:54.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:27:54.896 INFO:teuthology.orchestra.run.smithi121.stdout:client24871 2024-06-10T00:27:54.896 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-06-10T00:27:54.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-06-10T00:27:54.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:27:54.952 INFO:teuthology.orchestra.run.smithi121.stdout:client24871 2024-06-10T00:27:54.953 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/6c432244-26be-11ef-bc9b-c7b262605968.client24871/status') 2024-06-10T00:27:55.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-06-10T00:27:55.007 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:54 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:54.710+0000 7f50c51d8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-10T00:27:55.010 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:54 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:54.750+0000 7fb015c11200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-10T00:27:55.010 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:54 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:54.963+0000 7fb015c11200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-10T00:27:55.012 INFO:teuthology.orchestra.run.smithi121.stdout:instance: client.24871 (3)172.21.15.121:0/3990478629 2024-06-10T00:27:55.013 INFO:teuthology.orchestra.run.smithi121.stdout:blocklisted: false 2024-06-10T00:27:55.013 INFO:teuthology.orchestra.run.smithi121.stderr:0+1 records in 2024-06-10T00:27:55.013 INFO:teuthology.orchestra.run.smithi121.stderr:0+1 records out 2024-06-10T00:27:55.013 INFO:teuthology.orchestra.run.smithi121.stderr:72 bytes copied, 2.2622e-05 s, 3.2 MB/s 2024-06-10T00:27:55.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-06-10T00:27:55.020 INFO:teuthology.run_tasks:Running task exec... 2024-06-10T00:27:55.030 INFO:teuthology.task.exec:Executing custom commands... 2024-06-10T00:27:55.030 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi121.front.sepia.ceph.com 2024-06-10T00:27:55.030 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-06-10T00:27:55.131 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:54 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:54.770+0000 7fccee99a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-10T00:27:55.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.137+0000 7f349d13b640 1 -- 172.21.15.121:0/2225054657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3498101a10 msgr2=0x7f3498101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:55.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.137+0000 7f349d13b640 1 --2- 172.21.15.121:0/2225054657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3498101a10 0x7f3498101e30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3488009900 tx=0x7f348802a410 comp rx=0 tx=0).stop 2024-06-10T00:27:55.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.137+0000 7f349d13b640 1 -- 172.21.15.121:0/2225054657 shutdown_connections 2024-06-10T00:27:55.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.137+0000 7f349d13b640 1 --2- 172.21.15.121:0/2225054657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f34981035c0 0x7f3498109e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.137+0000 7f349d13b640 1 --2- 172.21.15.121:0/2225054657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3498102c00 0x7f3498103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.137+0000 7f349d13b640 1 --2- 172.21.15.121:0/2225054657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3498101a10 0x7f3498101e30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.137+0000 7f349d13b640 1 -- 172.21.15.121:0/2225054657 >> 172.21.15.121:0/2225054657 conn(0x7f34980fcf80 msgr2=0x7f34980ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.137+0000 7f349d13b640 1 -- 172.21.15.121:0/2225054657 shutdown_connections 2024-06-10T00:27:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.137+0000 7f349d13b640 1 -- 172.21.15.121:0/2225054657 wait complete. 2024-06-10T00:27:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349d13b640 1 Processor -- start 2024-06-10T00:27:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349d13b640 1 -- start start 2024-06-10T00:27:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349d13b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3498101a10 0x7f3498197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349d13b640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3498102c00 0x7f3498198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349d13b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34981035c0 0x7f349819f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349d13b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f349810d680 con 0x7f34981035c0 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349d13b640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f349810d500 con 0x7f3498102c00 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349d13b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f349810d800 con 0x7f3498101a10 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349759e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34981035c0 0x7f349819f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349659c640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3498102c00 0x7f3498198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349759e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34981035c0 0x7f349819f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47328/0 (socket says 172.21.15.121:47328) 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f3496d9d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3498101a10 0x7f3498197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349759e640 1 -- 172.21.15.121:0/4111054627 learned_addr learned my addr 172.21.15.121:0/4111054627 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349659c640 1 -- 172.21.15.121:0/4111054627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3498101a10 msgr2=0x7f3498197cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349659c640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3498101a10 0x7f3498197cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349659c640 1 -- 172.21.15.121:0/4111054627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34981035c0 msgr2=0x7f349819f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349659c640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34981035c0 0x7f349819f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.138+0000 7f349659c640 1 -- 172.21.15.121:0/4111054627 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349819f900 con 0x7f3498102c00 2024-06-10T00:27:55.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.139+0000 7f349659c640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3498102c00 0x7f3498198200 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f348000e9f0 tx=0x7f348000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:55.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.139+0000 7f348ffff640 1 -- 172.21.15.121:0/4111054627 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348000ccc0 con 0x7f3498102c00 2024-06-10T00:27:55.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.139+0000 7f348ffff640 1 -- 172.21.15.121:0/4111054627 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3480005020 con 0x7f3498102c00 2024-06-10T00:27:55.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.139+0000 7f349d13b640 1 -- 172.21.15.121:0/4111054627 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f349819fbf0 con 0x7f3498102c00 2024-06-10T00:27:55.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.139+0000 7f348ffff640 1 -- 172.21.15.121:0/4111054627 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34800049e0 con 0x7f3498102c00 2024-06-10T00:27:55.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.139+0000 7f349d13b640 1 -- 172.21.15.121:0/4111054627 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34981a00c0 con 0x7f3498102c00 2024-06-10T00:27:55.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.140+0000 7f349d13b640 1 -- 172.21.15.121:0/4111054627 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3498108d30 con 0x7f3498102c00 2024-06-10T00:27:55.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.141+0000 7f348ffff640 1 -- 172.21.15.121:0/4111054627 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f3480005560 con 0x7f3498102c00 2024-06-10T00:27:55.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.142+0000 7f348ffff640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f34840a2c80 0x7f34840a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.142+0000 7f3496d9d640 1 -- 172.21.15.121:0/4111054627 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f34840a2c80 msgr2=0x7f34840a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3506007074 2024-06-10T00:27:55.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.142+0000 7f348ffff640 1 -- 172.21.15.121:0/4111054627 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f34800cf3a0 con 0x7f3498102c00 2024-06-10T00:27:55.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.142+0000 7f3496d9d640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f34840a2c80 0x7f34840a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:27:55.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.145+0000 7f348ffff640 1 -- 172.21.15.121:0/4111054627 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f34800051d0 con 0x7f3498102c00 2024-06-10T00:27:55.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.343+0000 7f3496d9d640 1 -- 172.21.15.121:0/4111054627 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f34840a2c80 msgr2=0x7f34840a5140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3506007074 2024-06-10T00:27:55.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.343+0000 7f3496d9d640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f34840a2c80 0x7f34840a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-10T00:27:55.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.368+0000 7f349d13b640 1 -- 172.21.15.121:0/4111054627 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f349802d2e0 con 0x7f3498102c00 2024-06-10T00:27:55.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.424+0000 7f348ffff640 1 -- 172.21.15.121:0/4111054627 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v26) ==== 117+0+0 (secure 0 0 0) 0x7f348009bef0 con 0x7f3498102c00 2024-06-10T00:27:55.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 -- 172.21.15.121:0/4111054627 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f34840a2c80 msgr2=0x7f34840a5140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:27:55.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f34840a2c80 0x7f34840a5140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 -- 172.21.15.121:0/4111054627 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3498102c00 msgr2=0x7f3498198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:55.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3498102c00 0x7f3498198200 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f348000e9f0 tx=0x7f348000eeb0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 -- 172.21.15.121:0/4111054627 shutdown_connections 2024-06-10T00:27:55.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f34840a2c80 0x7f34840a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34981035c0 0x7f349819f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3498102c00 0x7f3498198200 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 --2- 172.21.15.121:0/4111054627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3498101a10 0x7f3498197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 -- 172.21.15.121:0/4111054627 >> 172.21.15.121:0/4111054627 conn(0x7f34980fcf80 msgr2=0x7f3498105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:55.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 -- 172.21.15.121:0/4111054627 shutdown_connections 2024-06-10T00:27:55.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.427+0000 7f349d13b640 1 -- 172.21.15.121:0/4111054627 wait complete. 2024-06-10T00:27:55.438 INFO:teuthology.run_tasks:Running task check-counter... 2024-06-10T00:27:55.448 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-06-10T00:27:55.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-10T00:27:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:55 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/4111054627' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-10T00:27:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:55 smithi135 ceph-mon[36424]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-10T00:27:55.508 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:55.233+0000 7f50c51d8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-10T00:27:55.509 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:55.326+0000 7f50c51d8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-10T00:27:55.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 -- 172.21.15.121:0/3896319408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f41008e0 msgr2=0x7f16f4100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 --2- 172.21.15.121:0/3896319408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f41008e0 0x7f16f4100d00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009900 tx=0x7f16e802a410 comp rx=0 tx=0).stop 2024-06-10T00:27:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 -- 172.21.15.121:0/3896319408 shutdown_connections 2024-06-10T00:27:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 --2- 172.21.15.121:0/3896319408 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16f4102490 0x7f16f4108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 --2- 172.21.15.121:0/3896319408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f4101ad0 0x7f16f4101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 --2- 172.21.15.121:0/3896319408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f41008e0 0x7f16f4100d00 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 -- 172.21.15.121:0/3896319408 >> 172.21.15.121:0/3896319408 conn(0x7f16f40fbe50 msgr2=0x7f16f40fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 -- 172.21.15.121:0/3896319408 shutdown_connections 2024-06-10T00:27:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 -- 172.21.15.121:0/3896319408 wait complete. 2024-06-10T00:27:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 Processor -- start 2024-06-10T00:27:55.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 -- start start 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16f41008e0 0x7f16f40696c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f4101ad0 0x7f16f419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f4102490 0x7f16f419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f16f410c550 con 0x7f16f4102490 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f16f410c3d0 con 0x7f16f41008e0 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.569+0000 7f16fb0a5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f16f410c6d0 con 0x7f16f4101ad0 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f9642640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f4102490 0x7f16f419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f8e41640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16f41008e0 0x7f16f40696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f9642640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f4102490 0x7f16f419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47352/0 (socket says 172.21.15.121:47352) 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f8e41640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16f41008e0 0x7f16f40696c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:58024/0 (socket says 172.21.15.121:58024) 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f9642640 1 -- 172.21.15.121:0/1217442787 learned_addr learned my addr 172.21.15.121:0/1217442787 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f3fff640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f4101ad0 0x7f16f419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:55.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f8e41640 1 -- 172.21.15.121:0/1217442787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f4101ad0 msgr2=0x7f16f419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f8e41640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f4101ad0 0x7f16f419ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f8e41640 1 -- 172.21.15.121:0/1217442787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f4102490 msgr2=0x7f16f419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f8e41640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f4102490 0x7f16f419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f8e41640 1 -- 172.21.15.121:0/1217442787 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f419f950 con 0x7f16f41008e0 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f3fff640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f4101ad0 0x7f16f419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f9642640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f4102490 0x7f16f419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f8e41640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16f41008e0 0x7f16f40696c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009230 tx=0x7f16e8004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f1ffb640 1 -- 172.21.15.121:0/1217442787 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e803f070 con 0x7f16f41008e0 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16fb0a5640 1 -- 172.21.15.121:0/1217442787 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f419fbe0 con 0x7f16f41008e0 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f1ffb640 1 -- 172.21.15.121:0/1217442787 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16e8003cd0 con 0x7f16f41008e0 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16f1ffb640 1 -- 172.21.15.121:0/1217442787 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e8035070 con 0x7f16f41008e0 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.570+0000 7f16fb0a5640 1 -- 172.21.15.121:0/1217442787 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f41a00f0 con 0x7f16f41008e0 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.571+0000 7f16fb0a5640 1 -- 172.21.15.121:0/1217442787 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f4107c70 con 0x7f16f41008e0 2024-06-10T00:27:55.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.573+0000 7f16f1ffb640 1 -- 172.21.15.121:0/1217442787 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f16e80037f0 con 0x7f16f41008e0 2024-06-10T00:27:55.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.573+0000 7f16f1ffb640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f16e40a2be0 0x7f16e40a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.573+0000 7f16f3fff640 1 -- 172.21.15.121:0/1217442787 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f16e40a2be0 msgr2=0x7f16e40a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3506007074 2024-06-10T00:27:55.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.573+0000 7f16f1ffb640 1 -- 172.21.15.121:0/1217442787 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f16e80efc70 con 0x7f16f41008e0 2024-06-10T00:27:55.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.573+0000 7f16f3fff640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f16e40a2be0 0x7f16e40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:27:55.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.577+0000 7f16f1ffb640 1 -- 172.21.15.121:0/1217442787 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f16e80bc170 con 0x7f16f41008e0 2024-06-10T00:27:55.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:55 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/4111054627' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-10T00:27:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:55 smithi193 ceph-mon[36716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-10T00:27:55.631 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:55.298+0000 7fccee99a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-10T00:27:55.631 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:55.392+0000 7fccee99a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-10T00:27:55.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:55 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/4111054627' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-10T00:27:55.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:55 smithi121 ceph-mon[32378]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-06-10T00:27:55.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:55 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:55.512+0000 7fb015c11200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-10T00:27:55.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:55 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:55.612+0000 7fb015c11200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-10T00:27:55.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.762+0000 7f16fb0a5640 1 -- 172.21.15.121:0/1217442787 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16f402d460 con 0x7f16f41008e0 2024-06-10T00:27:55.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.764+0000 7f16f1ffb640 1 -- 172.21.15.121:0/1217442787 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) ==== 74+0+24919 (secure 0 0 0) 0x7f16e80c0470 con 0x7f16f41008e0 2024-06-10T00:27:55.764 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:27:55.765 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":100,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:25:44.074803+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:27:55.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.766+0000 7f16fb0a5640 1 -- 172.21.15.121:0/1217442787 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f16e40a2be0 msgr2=0x7f16e40a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:27:55.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.766+0000 7f16fb0a5640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f16e40a2be0 0x7f16e40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.766+0000 7f16fb0a5640 1 -- 172.21.15.121:0/1217442787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16f41008e0 msgr2=0x7f16f40696c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:55.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.766+0000 7f16fb0a5640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16f41008e0 0x7f16f40696c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009230 tx=0x7f16e8004270 comp rx=0 tx=0).stop 2024-06-10T00:27:55.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.767+0000 7f16fb0a5640 1 -- 172.21.15.121:0/1217442787 shutdown_connections 2024-06-10T00:27:55.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.767+0000 7f16fb0a5640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f16e40a2be0 0x7f16e40a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.767+0000 7f16fb0a5640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f4102490 0x7f16f419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.767+0000 7f16fb0a5640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f4101ad0 0x7f16f419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.767+0000 7f16fb0a5640 1 --2- 172.21.15.121:0/1217442787 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16f41008e0 0x7f16f40696c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.767+0000 7f16fb0a5640 1 -- 172.21.15.121:0/1217442787 >> 172.21.15.121:0/1217442787 conn(0x7f16f40fbe50 msgr2=0x7f16f4104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:55.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.767+0000 7f16fb0a5640 1 -- 172.21.15.121:0/1217442787 shutdown_connections 2024-06-10T00:27:55.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.767+0000 7f16fb0a5640 1 -- 172.21.15.121:0/1217442787 wait complete. 2024-06-10T00:27:55.779 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-10T00:19:04.337527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-10T00:23:53.357210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-10T00:23:56.483170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-10T00:24:07.390280+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-10T00:27:55.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-10T00:27:55.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.884+0000 7f47dc1c4640 1 -- 172.21.15.121:0/2543673104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47d41008e0 msgr2=0x7f47d4100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:55.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.884+0000 7f47dc1c4640 1 --2- 172.21.15.121:0/2543673104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47d41008e0 0x7f47d4100d00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009900 tx=0x7f47d002a410 comp rx=0 tx=0).stop 2024-06-10T00:27:55.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.884+0000 7f47dc1c4640 1 -- 172.21.15.121:0/2543673104 shutdown_connections 2024-06-10T00:27:55.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.884+0000 7f47dc1c4640 1 --2- 172.21.15.121:0/2543673104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47d4102490 0x7f47d4108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.884+0000 7f47dc1c4640 1 --2- 172.21.15.121:0/2543673104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47d4101ad0 0x7f47d4101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.884+0000 7f47dc1c4640 1 --2- 172.21.15.121:0/2543673104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47d41008e0 0x7f47d4100d00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.884+0000 7f47dc1c4640 1 -- 172.21.15.121:0/2543673104 >> 172.21.15.121:0/2543673104 conn(0x7f47d40fbe50 msgr2=0x7f47d40fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.884+0000 7f47dc1c4640 1 -- 172.21.15.121:0/2543673104 shutdown_connections 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.884+0000 7f47dc1c4640 1 -- 172.21.15.121:0/2543673104 wait complete. 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47dc1c4640 1 Processor -- start 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47dc1c4640 1 -- start start 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47dc1c4640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47d41008e0 0x7f47d40696a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47dc1c4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47d4101ad0 0x7f47d419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47dc1c4640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47d4102490 0x7f47d419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47dc1c4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f47d410c560 con 0x7f47d4101ad0 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47dc1c4640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f47d410c3e0 con 0x7f47d4102490 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47dc1c4640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47d410c6e0 con 0x7f47d41008e0 2024-06-10T00:27:55.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47d975f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47d4101ad0 0x7f47d419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47d975f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47d4101ad0 0x7f47d419ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47372/0 (socket says 172.21.15.121:47372) 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47d975f640 1 -- 172.21.15.121:0/3711465391 learned_addr learned my addr 172.21.15.121:0/3711465391 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47d9f60640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47d41008e0 0x7f47d40696a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.885+0000 7f47da761640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47d4102490 0x7f47d419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.886+0000 7f47d975f640 1 -- 172.21.15.121:0/3711465391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47d41008e0 msgr2=0x7f47d40696a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.886+0000 7f47d975f640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47d41008e0 0x7f47d40696a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.886+0000 7f47d975f640 1 -- 172.21.15.121:0/3711465391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47d4102490 msgr2=0x7f47d419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.886+0000 7f47d975f640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47d4102490 0x7f47d419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.886+0000 7f47d975f640 1 -- 172.21.15.121:0/3711465391 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d419f9b0 con 0x7f47d4101ad0 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.886+0000 7f47d975f640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47d4101ad0 0x7f47d419ca80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f47c400d950 tx=0x7f47c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.886+0000 7f47caffd640 1 -- 172.21.15.121:0/3711465391 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c4017030 con 0x7f47d4101ad0 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.886+0000 7f47caffd640 1 -- 172.21.15.121:0/3711465391 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47c4004020 con 0x7f47d4101ad0 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.886+0000 7f47dc1c4640 1 -- 172.21.15.121:0/3711465391 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d419fc40 con 0x7f47d4101ad0 2024-06-10T00:27:55.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.886+0000 7f47caffd640 1 -- 172.21.15.121:0/3711465391 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c40047a0 con 0x7f47d4101ad0 2024-06-10T00:27:55.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.886+0000 7f47dc1c4640 1 -- 172.21.15.121:0/3711465391 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d41a0190 con 0x7f47d4101ad0 2024-06-10T00:27:55.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.887+0000 7f47dc1c4640 1 -- 172.21.15.121:0/3711465391 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d4107cd0 con 0x7f47d4101ad0 2024-06-10T00:27:55.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.889+0000 7f47caffd640 1 -- 172.21.15.121:0/3711465391 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f47c40041d0 con 0x7f47d4101ad0 2024-06-10T00:27:55.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.890+0000 7f47caffd640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f47c00a2c60 0x7f47c00a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:55.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.890+0000 7f47d9f60640 1 -- 172.21.15.121:0/3711465391 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f47c00a2c60 msgr2=0x7f47c00a5120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3506007074 2024-06-10T00:27:55.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.890+0000 7f47caffd640 1 -- 172.21.15.121:0/3711465391 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f47c40ced70 con 0x7f47d4101ad0 2024-06-10T00:27:55.890 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.890+0000 7f47d9f60640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f47c00a2c60 0x7f47c00a5120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:27:55.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:55.895+0000 7f47caffd640 1 -- 172.21.15.121:0/3711465391 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f47c409b920 con 0x7f47d4101ad0 2024-06-10T00:27:55.915 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:55.572+0000 7f50c51d8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-10T00:27:55.915 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:55.705+0000 7f50c51d8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-10T00:27:55.915 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:55.800+0000 7f50c51d8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-10T00:27:55.916 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:55.915+0000 7f50c51d8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-10T00:27:55.988 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:55.639+0000 7fccee99a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-10T00:27:55.988 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:55.775+0000 7fccee99a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-10T00:27:55.988 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:55.871+0000 7fccee99a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-10T00:27:55.988 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:55.987+0000 7fccee99a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-10T00:27:56.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.074+0000 7f47dc1c4640 1 -- 172.21.15.121:0/3711465391 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f47d40625a0 con 0x7f47d4101ad0 2024-06-10T00:27:56.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.075+0000 7f47caffd640 1 -- 172.21.15.121:0/3711465391 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v100) ==== 93+0+10 (secure 0 0 0) 0x7f47c409b220 con 0x7f47d4101ad0 2024-06-10T00:27:56.075 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-06-10T00:27:56.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 -- 172.21.15.121:0/3711465391 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f47c00a2c60 msgr2=0x7f47c00a5120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:27:56.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f47c00a2c60 0x7f47c00a5120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 -- 172.21.15.121:0/3711465391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47d4101ad0 msgr2=0x7f47d419ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47d4101ad0 0x7f47d419ca80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f47c400d950 tx=0x7f47c400de10 comp rx=0 tx=0).stop 2024-06-10T00:27:56.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 -- 172.21.15.121:0/3711465391 shutdown_connections 2024-06-10T00:27:56.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f47c00a2c60 0x7f47c00a5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f47d4102490 0x7f47d419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47d4101ad0 0x7f47d419ca80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 --2- 172.21.15.121:0/3711465391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47d41008e0 0x7f47d40696a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 -- 172.21.15.121:0/3711465391 >> 172.21.15.121:0/3711465391 conn(0x7f47d40fbe50 msgr2=0x7f47d4104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:56.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 -- 172.21.15.121:0/3711465391 shutdown_connections 2024-06-10T00:27:56.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.077+0000 7f47dc1c4640 1 -- 172.21.15.121:0/3711465391 wait complete. 2024-06-10T00:27:56.087 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:55 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:55.870+0000 7fb015c11200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-10T00:27:56.088 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:56 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:56.012+0000 7fb015c11200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-10T00:27:56.089 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-10T00:27:56.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.195+0000 7f1a111c1640 1 -- 172.21.15.121:0/2731133261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a0c1035a0 msgr2=0x7f1a0c109e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.195+0000 7f1a111c1640 1 --2- 172.21.15.121:0/2731133261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a0c1035a0 0x7f1a0c109e30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009ca0 tx=0x7f1a0402a8c0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.195+0000 7f1a111c1640 1 -- 172.21.15.121:0/2731133261 shutdown_connections 2024-06-10T00:27:56.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.195+0000 7f1a111c1640 1 --2- 172.21.15.121:0/2731133261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a0c1035a0 0x7f1a0c109e30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.195+0000 7f1a111c1640 1 --2- 172.21.15.121:0/2731133261 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a0c102be0 0x7f1a0c103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.195+0000 7f1a111c1640 1 --2- 172.21.15.121:0/2731133261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a0c1019f0 0x7f1a0c101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.195+0000 7f1a111c1640 1 -- 172.21.15.121:0/2731133261 >> 172.21.15.121:0/2731133261 conn(0x7f1a0c0fcf80 msgr2=0x7f1a0c0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:56.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.195+0000 7f1a111c1640 1 -- 172.21.15.121:0/2731133261 shutdown_connections 2024-06-10T00:27:56.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.195+0000 7f1a111c1640 1 -- 172.21.15.121:0/2731133261 wait complete. 2024-06-10T00:27:56.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.196+0000 7f1a111c1640 1 Processor -- start 2024-06-10T00:27:56.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.196+0000 7f1a111c1640 1 -- start start 2024-06-10T00:27:56.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.196+0000 7f1a111c1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a0c1019f0 0x7f1a0c197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.196+0000 7f1a111c1640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a0c102be0 0x7f1a0c1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.196+0000 7f1a111c1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a0c1035a0 0x7f1a0c19f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.196+0000 7f1a111c1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0c10d650 con 0x7f1a0c1019f0 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.196+0000 7f1a111c1640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0c10d4d0 con 0x7f1a0c102be0 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.196+0000 7f1a111c1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0c10d7d0 con 0x7f1a0c1035a0 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.196+0000 7f1a0ad9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a0c1019f0 0x7f1a0c197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.196+0000 7f1a0ad9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a0c1019f0 0x7f1a0c197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47400/0 (socket says 172.21.15.121:47400) 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f1a0ad9d640 1 -- 172.21.15.121:0/2414343143 learned_addr learned my addr 172.21.15.121:0/2414343143 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f1a0a59c640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a0c102be0 0x7f1a0c1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f1a0b59e640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a0c1035a0 0x7f1a0c19f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f1a0ad9d640 1 -- 172.21.15.121:0/2414343143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a0c1035a0 msgr2=0x7f1a0c19f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f1a0ad9d640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a0c1035a0 0x7f1a0c19f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f1a0ad9d640 1 -- 172.21.15.121:0/2414343143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a0c102be0 msgr2=0x7f1a0c1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f1a0ad9d640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a0c102be0 0x7f1a0c1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f1a0ad9d640 1 -- 172.21.15.121:0/2414343143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a0c19f8d0 con 0x7f1a0c1019f0 2024-06-10T00:27:56.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f1a0ad9d640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a0c1019f0 0x7f1a0c197c90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00d950 tx=0x7f19fc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:56.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f19f3fff640 1 -- 172.21.15.121:0/2414343143 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc017020 con 0x7f1a0c1019f0 2024-06-10T00:27:56.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f19f3fff640 1 -- 172.21.15.121:0/2414343143 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19fc009c30 con 0x7f1a0c1019f0 2024-06-10T00:27:56.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f1a111c1640 1 -- 172.21.15.121:0/2414343143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a0c19fbc0 con 0x7f1a0c1019f0 2024-06-10T00:27:56.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f19f3fff640 1 -- 172.21.15.121:0/2414343143 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19fc00f740 con 0x7f1a0c1019f0 2024-06-10T00:27:56.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.197+0000 7f1a111c1640 1 -- 172.21.15.121:0/2414343143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a0c1a0110 con 0x7f1a0c1019f0 2024-06-10T00:27:56.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.198+0000 7f1a111c1640 1 -- 172.21.15.121:0/2414343143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a0c108d10 con 0x7f1a0c1019f0 2024-06-10T00:27:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.200+0000 7f19f3fff640 1 -- 172.21.15.121:0/2414343143 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f19fc01a020 con 0x7f1a0c1019f0 2024-06-10T00:27:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.200+0000 7f19f3fff640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f19f40a2be0 0x7f19f40a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.200+0000 7f1a0a59c640 1 -- 172.21.15.121:0/2414343143 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f19f40a2be0 msgr2=0x7f19f40a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3506007074 2024-06-10T00:27:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.200+0000 7f19f3fff640 1 -- 172.21.15.121:0/2414343143 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f19fc0cf0f0 con 0x7f1a0c1019f0 2024-06-10T00:27:56.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.200+0000 7f1a0a59c640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f19f40a2be0 0x7f19f40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:27:56.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.204+0000 7f19f3fff640 1 -- 172.21.15.121:0/2414343143 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f19fc021070 con 0x7f1a0c1019f0 2024-06-10T00:27:56.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.384+0000 7f1a111c1640 1 -- 172.21.15.121:0/2414343143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1a0c02d470 con 0x7f1a0c1019f0 2024-06-10T00:27:56.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.385+0000 7f19f3fff640 1 -- 172.21.15.121:0/2414343143 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v100) ==== 104+0+11 (secure 0 0 0) 0x7f19fc009de0 con 0x7f1a0c1019f0 2024-06-10T00:27:56.385 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-06-10T00:27:56.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 -- 172.21.15.121:0/2414343143 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f19f40a2be0 msgr2=0x7f19f40a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:27:56.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f19f40a2be0 0x7f19f40a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 -- 172.21.15.121:0/2414343143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a0c1019f0 msgr2=0x7f1a0c197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a0c1019f0 0x7f1a0c197c90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00d950 tx=0x7f19fc00de10 comp rx=0 tx=0).stop 2024-06-10T00:27:56.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 -- 172.21.15.121:0/2414343143 shutdown_connections 2024-06-10T00:27:56.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f19f40a2be0 0x7f19f40a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a0c1035a0 0x7f1a0c19f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1a0c102be0 0x7f1a0c1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 --2- 172.21.15.121:0/2414343143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a0c1019f0 0x7f1a0c197c90 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 -- 172.21.15.121:0/2414343143 >> 172.21.15.121:0/2414343143 conn(0x7f1a0c0fcf80 msgr2=0x7f1a0c105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:56.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 -- 172.21.15.121:0/2414343143 shutdown_connections 2024-06-10T00:27:56.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.387+0000 7f1a111c1640 1 -- 172.21.15.121:0/2414343143 wait complete. 2024-06-10T00:27:56.398 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:56 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:56.112+0000 7fb015c11200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-10T00:27:56.398 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:56 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:56.238+0000 7fb015c11200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-10T00:27:56.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-10T00:27:56.507 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:56 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:56.219+0000 7f50c51d8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-10T00:27:56.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:56 smithi135 ceph-mon[36424]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-06-10T00:27:56.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:56 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:27:56.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:56 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1217442787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:27:56.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:56 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3711465391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-10T00:27:56.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:56 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2414343143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-10T00:27:56.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.508+0000 7f955dd7e640 1 -- 172.21.15.121:0/2990227561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558102490 msgr2=0x7f9558108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.508+0000 7f955dd7e640 1 --2- 172.21.15.121:0/2990227561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558102490 0x7f9558108d20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9550009ca0 tx=0x7f955002a8c0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.508+0000 7f955dd7e640 1 -- 172.21.15.121:0/2990227561 shutdown_connections 2024-06-10T00:27:56.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.508+0000 7f955dd7e640 1 --2- 172.21.15.121:0/2990227561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558102490 0x7f9558108d20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.508+0000 7f955dd7e640 1 --2- 172.21.15.121:0/2990227561 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9558101ad0 0x7f9558101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.508+0000 7f955dd7e640 1 --2- 172.21.15.121:0/2990227561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95581008e0 0x7f9558100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.508+0000 7f955dd7e640 1 -- 172.21.15.121:0/2990227561 >> 172.21.15.121:0/2990227561 conn(0x7f95580fbe50 msgr2=0x7f95580fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.508+0000 7f955dd7e640 1 -- 172.21.15.121:0/2990227561 shutdown_connections 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.508+0000 7f955dd7e640 1 -- 172.21.15.121:0/2990227561 wait complete. 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f955dd7e640 1 Processor -- start 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f955dd7e640 1 -- start start 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f955dd7e640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95581008e0 0x7f9558069700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f955dd7e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558101ad0 0x7f955819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f955dd7e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9558102490 0x7f955819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f955dd7e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f955810c560 con 0x7f9558101ad0 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f955dd7e640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f955810c3e0 con 0x7f95581008e0 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f955dd7e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f955810c6e0 con 0x7f9558102490 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f954ffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558101ad0 0x7f955819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:56.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f95577fe640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95581008e0 0x7f9558069700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:56.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f95577fe640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95581008e0 0x7f9558069700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:58076/0 (socket says 172.21.15.121:58076) 2024-06-10T00:27:56.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f95577fe640 1 -- 172.21.15.121:0/1237354781 learned_addr learned my addr 172.21.15.121:0/1237354781 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:56.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f954ffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558101ad0 0x7f955819c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47406/0 (socket says 172.21.15.121:47406) 2024-06-10T00:27:56.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f9557fff640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9558102490 0x7f955819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:56.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f95577fe640 1 -- 172.21.15.121:0/1237354781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9558102490 msgr2=0x7f955819f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f95577fe640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9558102490 0x7f955819f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f95577fe640 1 -- 172.21.15.121:0/1237354781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558101ad0 msgr2=0x7f955819c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f95577fe640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558101ad0 0x7f955819c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f95577fe640 1 -- 172.21.15.121:0/1237354781 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955819f8f0 con 0x7f95581008e0 2024-06-10T00:27:56.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f954ffff640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558101ad0 0x7f955819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.509+0000 7f9557fff640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9558102490 0x7f955819f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.510+0000 7f95577fe640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95581008e0 0x7f9558069700 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f954800e9c0 tx=0x7f954800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.510+0000 7f95557fa640 1 -- 172.21.15.121:0/1237354781 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954800cd80 con 0x7f95581008e0 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.510+0000 7f955dd7e640 1 -- 172.21.15.121:0/1237354781 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f955819fbe0 con 0x7f95581008e0 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.510+0000 7f95557fa640 1 -- 172.21.15.121:0/1237354781 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95480035f0 con 0x7f95581008e0 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.510+0000 7f95557fa640 1 -- 172.21.15.121:0/1237354781 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9548010690 con 0x7f95581008e0 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.510+0000 7f955dd7e640 1 -- 172.21.15.121:0/1237354781 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95581a00e0 con 0x7f95581008e0 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.511+0000 7f955dd7e640 1 -- 172.21.15.121:0/1237354781 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9558107c80 con 0x7f95581008e0 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.512+0000 7f95557fa640 1 -- 172.21.15.121:0/1237354781 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f95480037a0 con 0x7f95581008e0 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.513+0000 7f95557fa640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f95440a2be0 0x7f95440a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.513+0000 7f954ffff640 1 -- 172.21.15.121:0/1237354781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f95440a2be0 msgr2=0x7f95440a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3506007074 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.513+0000 7f95557fa640 1 -- 172.21.15.121:0/1237354781 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7f95480cf8a0 con 0x7f95581008e0 2024-06-10T00:27:56.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.513+0000 7f954ffff640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f95440a2be0 0x7f95440a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:27:56.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.516+0000 7f95557fa640 1 -- 172.21.15.121:0/1237354781 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f954809c450 con 0x7f95581008e0 2024-06-10T00:27:56.631 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:56.292+0000 7fccee99a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-10T00:27:56.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:56 smithi193 ceph-mon[36716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-06-10T00:27:56.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:56 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:27:56.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:56 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1217442787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:27:56.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:56 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3711465391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-10T00:27:56.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:56 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2414343143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-10T00:27:56.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:56 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:56.554+0000 7fb015c11200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-10T00:27:56.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:56 smithi121 ceph-mon[32378]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-06-10T00:27:56.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:56 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:27:56.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:56 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1217442787' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:27:56.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:56 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3711465391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-10T00:27:56.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:56 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2414343143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-10T00:27:56.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.700+0000 7f955dd7e640 1 -- 172.21.15.121:0/1237354781 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f955802d2e0 con 0x7f95581008e0 2024-06-10T00:27:56.701 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.701+0000 7f95557fa640 1 -- 172.21.15.121:0/1237354781 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v100) ==== 100+0+11 (secure 0 0 0) 0x7f954809bd50 con 0x7f95581008e0 2024-06-10T00:27:56.701 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 64 2024-06-10T00:27:56.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 -- 172.21.15.121:0/1237354781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f95440a2be0 msgr2=0x7f95440a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:27:56.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f95440a2be0 0x7f95440a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 -- 172.21.15.121:0/1237354781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95581008e0 msgr2=0x7f9558069700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95581008e0 0x7f9558069700 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f954800e9c0 tx=0x7f954800ee80 comp rx=0 tx=0).stop 2024-06-10T00:27:56.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 -- 172.21.15.121:0/1237354781 shutdown_connections 2024-06-10T00:27:56.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7f95440a2be0 0x7f95440a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9558102490 0x7f955819f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558101ad0 0x7f955819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 --2- 172.21.15.121:0/1237354781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95581008e0 0x7f9558069700 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 -- 172.21.15.121:0/1237354781 >> 172.21.15.121:0/1237354781 conn(0x7f95580fbe50 msgr2=0x7f9558104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:56.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 -- 172.21.15.121:0/1237354781 shutdown_connections 2024-06-10T00:27:56.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.703+0000 7f955dd7e640 1 -- 172.21.15.121:0/1237354781 wait complete. 2024-06-10T00:27:56.714 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-10T00:27:56.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.819+0000 7fb772da2640 1 -- 172.21.15.121:0/3183658012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb76c101ad0 msgr2=0x7fb76c101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.819+0000 7fb772da2640 1 --2- 172.21.15.121:0/3183658012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb76c101ad0 0x7fb76c101f50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb758009900 tx=0x7fb75802a410 comp rx=0 tx=0).stop 2024-06-10T00:27:56.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.819+0000 7fb772da2640 1 -- 172.21.15.121:0/3183658012 shutdown_connections 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.819+0000 7fb772da2640 1 --2- 172.21.15.121:0/3183658012 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb76c102490 0x7fb76c108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.819+0000 7fb772da2640 1 --2- 172.21.15.121:0/3183658012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb76c101ad0 0x7fb76c101f50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.819+0000 7fb772da2640 1 --2- 172.21.15.121:0/3183658012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c1008e0 0x7fb76c100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.819+0000 7fb772da2640 1 -- 172.21.15.121:0/3183658012 >> 172.21.15.121:0/3183658012 conn(0x7fb76c0fbe50 msgr2=0x7fb76c0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.819+0000 7fb772da2640 1 -- 172.21.15.121:0/3183658012 shutdown_connections 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb772da2640 1 -- 172.21.15.121:0/3183658012 wait complete. 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb772da2640 1 Processor -- start 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb772da2640 1 -- start start 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb772da2640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c1008e0 0x7fb76c0696c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb772da2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb76c101ad0 0x7fb76c19ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb772da2640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb76c102490 0x7fb76c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb772da2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c10c560 con 0x7fb76c101ad0 2024-06-10T00:27:56.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb772da2640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c10c3e0 con 0x7fb76c102490 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb772da2640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c10c6e0 con 0x7fb76c1008e0 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb76bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb76c101ad0 0x7fb76c19ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb76bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb76c101ad0 0x7fb76c19ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47420/0 (socket says 172.21.15.121:47420) 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb76bfff640 1 -- 172.21.15.121:0/1831426048 learned_addr learned my addr 172.21.15.121:0/1831426048 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb770b3e640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c1008e0 0x7fb76c0696c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.820+0000 7fb77133f640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb76c102490 0x7fb76c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.821+0000 7fb76bfff640 1 -- 172.21.15.121:0/1831426048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c1008e0 msgr2=0x7fb76c0696c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.821+0000 7fb76bfff640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c1008e0 0x7fb76c0696c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.821+0000 7fb76bfff640 1 -- 172.21.15.121:0/1831426048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb76c102490 msgr2=0x7fb76c19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.821+0000 7fb76bfff640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb76c102490 0x7fb76c19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.821+0000 7fb76bfff640 1 -- 172.21.15.121:0/1831426048 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb76c19f960 con 0x7fb76c101ad0 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.821+0000 7fb76bfff640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb76c101ad0 0x7fb76c19ca60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb758009230 tx=0x7fb7580042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:56.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.821+0000 7fb769ffb640 1 -- 172.21.15.121:0/1831426048 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75803f030 con 0x7fb76c101ad0 2024-06-10T00:27:56.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.821+0000 7fb769ffb640 1 -- 172.21.15.121:0/1831426048 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb758003ce0 con 0x7fb76c101ad0 2024-06-10T00:27:56.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.821+0000 7fb769ffb640 1 -- 172.21.15.121:0/1831426048 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb758035070 con 0x7fb76c101ad0 2024-06-10T00:27:56.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.821+0000 7fb772da2640 1 -- 172.21.15.121:0/1831426048 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb76c19fbf0 con 0x7fb76c101ad0 2024-06-10T00:27:56.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.821+0000 7fb772da2640 1 -- 172.21.15.121:0/1831426048 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb76c1a00e0 con 0x7fb76c101ad0 2024-06-10T00:27:56.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.822+0000 7fb772da2640 1 -- 172.21.15.121:0/1831426048 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb76c107c50 con 0x7fb76c101ad0 2024-06-10T00:27:56.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.824+0000 7fb769ffb640 1 -- 172.21.15.121:0/1831426048 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7fb758042030 con 0x7fb76c101ad0 2024-06-10T00:27:56.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.825+0000 7fb769ffb640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb75c0a2be0 0x7fb75c0a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:56.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.825+0000 7fb770b3e640 1 -- 172.21.15.121:0/1831426048 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb75c0a2be0 msgr2=0x7fb75c0a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3506007074 2024-06-10T00:27:56.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.825+0000 7fb769ffb640 1 -- 172.21.15.121:0/1831426048 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9179+0+0 (secure 0 0 0) 0x7fb7580ef8f0 con 0x7fb76c101ad0 2024-06-10T00:27:56.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.825+0000 7fb770b3e640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb75c0a2be0 0x7fb75c0a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-10T00:27:56.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:56.829+0000 7fb769ffb640 1 -- 172.21.15.121:0/1831426048 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb7580bbe50 con 0x7fb76c101ad0 2024-06-10T00:27:57.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.012+0000 7fb772da2640 1 -- 172.21.15.121:0/1831426048 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb76c0625a0 con 0x7fb76c101ad0 2024-06-10T00:27:57.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.012+0000 7fb769ffb640 1 -- 172.21.15.121:0/1831426048 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v100) ==== 103+0+11 (secure 0 0 0) 0x7fb7580c0150 con 0x7fb76c101ad0 2024-06-10T00:27:57.012 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-06-10T00:27:57.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 -- 172.21.15.121:0/1831426048 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb75c0a2be0 msgr2=0x7fb75c0a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:27:57.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb75c0a2be0 0x7fb75c0a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:57.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 -- 172.21.15.121:0/1831426048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb76c101ad0 msgr2=0x7fb76c19ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:57.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb76c101ad0 0x7fb76c19ca60 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb758009230 tx=0x7fb7580042f0 comp rx=0 tx=0).stop 2024-06-10T00:27:57.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 -- 172.21.15.121:0/1831426048 shutdown_connections 2024-06-10T00:27:57.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.121:6800/3506007074,v1:172.21.15.121:6801/3506007074] conn(0x7fb75c0a2be0 0x7fb75c0a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:57.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb76c102490 0x7fb76c19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:57.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb76c101ad0 0x7fb76c19ca60 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:57.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 --2- 172.21.15.121:0/1831426048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb76c1008e0 0x7fb76c0696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:57.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 -- 172.21.15.121:0/1831426048 >> 172.21.15.121:0/1831426048 conn(0x7fb76c0fbe50 msgr2=0x7fb76c104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:57.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 -- 172.21.15.121:0/1831426048 shutdown_connections 2024-06-10T00:27:57.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:57.014+0000 7fb772da2640 1 -- 172.21.15.121:0/1831426048 wait complete. 2024-06-10T00:27:57.025 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-06-10T00:27:57.026 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:27:57.258 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:56 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:56.821+0000 7f50c51d8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-10T00:27:57.258 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y[37811]: 2024-06-10T00:27:57.037+0000 7f50c51d8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-10T00:27:57.281 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:27:57.311 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.196+0000 7fb015c11200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-10T00:27:57.380 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:56.899+0000 7fccee99a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-10T00:27:57.381 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z[38537]: 2024-06-10T00:27:57.118+0000 7fccee99a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-10T00:27:57.571 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.449+0000 7fb015c11200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1237354781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1831426048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: Standby manager daemon y restarted 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: Standby manager daemon y started 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: Standby manager daemon z restarted 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: Standby manager daemon z started 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-10T00:27:57.572 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-mon[32378]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:27:57.715 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1237354781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-10T00:27:57.715 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1831426048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: Standby manager daemon y restarted 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: Standby manager daemon y started 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: Standby manager daemon z restarted 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: Standby manager daemon z started 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-10T00:27:57.716 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:57 smithi135 ceph-mon[36424]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1237354781' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1831426048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: Standby manager daemon y restarted 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: Standby manager daemon y started 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.135:0/713417077' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: Standby manager daemon z restarted 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: Standby manager daemon z started 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-10T00:27:57.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-06-10T00:27:57.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:57 smithi193 ceph-mon[36716]: from='mgr.? 172.21.15.193:0/521544645' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-10T00:27:57.950 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.743+0000 7faf77cd0700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:27:57.950 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.743+0000 7faf77cd0700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:27:57.950 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.743+0000 7faf77cd0700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:27:57.950 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.743+0000 7faf77cd0700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:27:57.950 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.743+0000 7faf77cd0700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:27:57.950 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.743+0000 7faf754cb700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:27:57.950 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.743+0000 7faf754cb700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:27:57.950 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.743+0000 7faf754cb700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:27:57.950 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.743+0000 7faf754cb700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:27:57.950 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:27:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:27:57.743+0000 7faf754cb700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:27:58.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-06-10T00:27:58.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: Active manager daemon x restarted 2024-06-10T00:27:58.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: Activating manager daemon x 2024-06-10T00:27:58.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: osdmap e101: 12 total, 12 up, 12 in 2024-06-10T00:27:58.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: mgrmap e21: x(active, starting, since 0.0143731s), standbys: z, y 2024-06-10T00:27:58.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:27:58.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:27:58.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:27:58.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-10T00:27:58.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:27:58.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-10T00:27:58.703 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-10T00:27:58.703 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-10T00:27:58.703 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: Manager daemon x is now available 2024-06-10T00:27:58.703 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:27:58.703 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-10T00:27:58.703 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-10T00:27:58.703 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: [10/Jun/2024:00:27:57] ENGINE Bus STARTING 2024-06-10T00:27:58.703 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: [10/Jun/2024:00:27:58] ENGINE Serving on https://172.21.15.121:7150 2024-06-10T00:27:58.703 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: [10/Jun/2024:00:27:58] ENGINE Serving on http://172.21.15.121:8765 2024-06-10T00:27:58.703 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:58 smithi121 ceph-mon[32378]: [10/Jun/2024:00:27:58] ENGINE Bus STARTED 2024-06-10T00:27:58.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-06-10T00:27:58.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: Active manager daemon x restarted 2024-06-10T00:27:58.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: Activating manager daemon x 2024-06-10T00:27:58.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: osdmap e101: 12 total, 12 up, 12 in 2024-06-10T00:27:58.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: mgrmap e21: x(active, starting, since 0.0143731s), standbys: z, y 2024-06-10T00:27:58.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:27:58.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:27:58.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:27:58.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-10T00:27:58.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-10T00:27:58.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:27:58.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:27:58.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-10T00:27:58.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-10T00:27:58.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-10T00:27:58.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: Manager daemon x is now available 2024-06-10T00:27:58.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:27:58.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-10T00:27:58.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-10T00:27:58.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: [10/Jun/2024:00:27:57] ENGINE Bus STARTING 2024-06-10T00:27:58.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: [10/Jun/2024:00:27:58] ENGINE Serving on https://172.21.15.121:7150 2024-06-10T00:27:58.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: [10/Jun/2024:00:27:58] ENGINE Serving on http://172.21.15.121:8765 2024-06-10T00:27:58.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:58 smithi193 ceph-mon[36716]: [10/Jun/2024:00:27:58] ENGINE Bus STARTED 2024-06-10T00:27:58.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-06-10T00:27:58.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: Active manager daemon x restarted 2024-06-10T00:27:58.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: Activating manager daemon x 2024-06-10T00:27:58.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: osdmap e101: 12 total, 12 up, 12 in 2024-06-10T00:27:58.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: mgrmap e21: x(active, starting, since 0.0143731s), standbys: z, y 2024-06-10T00:27:58.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-10T00:27:58.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-10T00:27:58.745 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-10T00:27:58.746 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-06-10T00:27:58.747 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-06-10T00:27:58.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-10T00:27:58.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-10T00:27:58.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-10T00:27:58.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: Manager daemon x is now available 2024-06-10T00:27:58.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:27:58.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-10T00:27:58.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-10T00:27:58.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: [10/Jun/2024:00:27:57] ENGINE Bus STARTING 2024-06-10T00:27:58.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: [10/Jun/2024:00:27:58] ENGINE Serving on https://172.21.15.121:7150 2024-06-10T00:27:58.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: [10/Jun/2024:00:27:58] ENGINE Serving on http://172.21.15.121:8765 2024-06-10T00:27:58.748 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:58 smithi135 ceph-mon[36424]: [10/Jun/2024:00:27:58] ENGINE Bus STARTED 2024-06-10T00:27:59.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.134+0000 7faa16188700 1 -- 172.21.15.121:0/2153817020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa10101390 msgr2=0x7faa10101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:59.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.134+0000 7faa16188700 1 --2- 172.21.15.121:0/2153817020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa10101390 0x7faa10101810 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009990 tx=0x7fa9f8009ca0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 -- 172.21.15.121:0/2153817020 shutdown_connections 2024-06-10T00:27:59.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 --2- 172.21.15.121:0/2153817020 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa10101d50 0x7faa1010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 --2- 172.21.15.121:0/2153817020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa10107400 0x7faa101077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 --2- 172.21.15.121:0/2153817020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa10101390 0x7faa10101810 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 -- 172.21.15.121:0/2153817020 >> 172.21.15.121:0/2153817020 conn(0x7faa10078340 msgr2=0x7faa10078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 -- 172.21.15.121:0/2153817020 shutdown_connections 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 -- 172.21.15.121:0/2153817020 wait complete. 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 Processor -- start 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 -- start start 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa10101390 0x7faa10103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa10101d50 0x7faa101035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa10107400 0x7faa10103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7faa10111670 con 0x7faa10107400 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7faa101114f0 con 0x7faa10101d50 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.135+0000 7faa16188700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faa101117f0 con 0x7faa10101390 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa10107400 0x7faa10103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa10107400 0x7faa10103af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41376/0 (socket says 172.21.15.121:41376) 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0ffff700 1 -- 172.21.15.121:0/1385561311 learned_addr learned my addr 172.21.15.121:0/1385561311 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:27:59.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0f7fe700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa10101390 0x7faa10103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa07fff700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa10101d50 0x7faa101035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0ffff700 1 -- 172.21.15.121:0/1385561311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa10101390 msgr2=0x7faa10103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0ffff700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa10101390 0x7faa10103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0ffff700 1 -- 172.21.15.121:0/1385561311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa10101d50 msgr2=0x7faa101035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0ffff700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa10101d50 0x7faa101035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0ffff700 1 -- 172.21.15.121:0/1385561311 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa101a7120 con 0x7faa10107400 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0ffff700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa10107400 0x7faa10103af0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7faa08007ee0 tx=0x7faa0800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0d7fa700 1 -- 172.21.15.121:0/1385561311 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0800dbd0 con 0x7faa10107400 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa16188700 1 -- 172.21.15.121:0/1385561311 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa101a7410 con 0x7faa10107400 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0d7fa700 1 -- 172.21.15.121:0/1385561311 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa0800dd70 con 0x7faa10107400 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa0d7fa700 1 -- 172.21.15.121:0/1385561311 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa080133f0 con 0x7faa10107400 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.136+0000 7faa16188700 1 -- 172.21.15.121:0/1385561311 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa101a7900 con 0x7faa10107400 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.137+0000 7faa16188700 1 -- 172.21.15.121:0/1385561311 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa10069180 con 0x7faa10107400 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.138+0000 7faa0d7fa700 1 -- 172.21.15.121:0/1385561311 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) ==== 137645+0+0 (secure 0 0 0) 0x7faa0801a020 con 0x7faa10107400 2024-06-10T00:27:59.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.139+0000 7faa0d7fa700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9fc0a2a20 0x7fa9fc0a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:27:59.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.139+0000 7faa0d7fa700 1 -- 172.21.15.121:0/1385561311 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7faa080ce910 con 0x7faa10107400 2024-06-10T00:27:59.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.139+0000 7faa0f7fe700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9fc0a2a20 0x7fa9fc0a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:27:59.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.140+0000 7faa0f7fe700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9fc0a2a20 0x7fa9fc0a4ee0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faa00005950 tx=0x7faa000058e0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:27:59.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.144+0000 7faa0d7fa700 1 -- 172.21.15.121:0/1385561311 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7faa08010070 con 0x7faa10107400 2024-06-10T00:27:59.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.395+0000 7faa16188700 1 -- 172.21.15.121:0/1385561311 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa101a7b60 con 0x7faa10107400 2024-06-10T00:27:59.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.396+0000 7faa0d7fa700 1 -- 172.21.15.121:0/1385561311 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7faa0809b6b0 con 0x7faa10107400 2024-06-10T00:27:59.396 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:27:59.396 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:27:55.370804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:27:59.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 -- 172.21.15.121:0/1385561311 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9fc0a2a20 msgr2=0x7fa9fc0a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:59.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9fc0a2a20 0x7fa9fc0a4ee0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faa00005950 tx=0x7faa000058e0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 -- 172.21.15.121:0/1385561311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa10107400 msgr2=0x7faa10103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:27:59.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa10107400 0x7faa10103af0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7faa08007ee0 tx=0x7faa0800d3b0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 -- 172.21.15.121:0/1385561311 shutdown_connections 2024-06-10T00:27:59.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9fc0a2a20 0x7fa9fc0a4ee0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faa10107400 0x7faa10103af0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faa10101390 0x7faa10103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 --2- 172.21.15.121:0/1385561311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7faa10101d50 0x7faa101035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:27:59.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 -- 172.21.15.121:0/1385561311 >> 172.21.15.121:0/1385561311 conn(0x7faa10078340 msgr2=0x7faa101005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:27:59.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 -- 172.21.15.121:0/1385561311 shutdown_connections 2024-06-10T00:27:59.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:27:59.398+0000 7faa16188700 1 -- 172.21.15.121:0/1385561311 wait complete. 2024-06-10T00:27:59.399 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-06-10T00:27:59.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:59 smithi121 ceph-mon[32378]: mgrmap e22: x(active, since 1.01778s), standbys: z, y 2024-06-10T00:27:59.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:59 smithi121 ceph-mon[32378]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:27:59.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:27:59 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1385561311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:27:59.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:59 smithi135 ceph-mon[36424]: mgrmap e22: x(active, since 1.01778s), standbys: z, y 2024-06-10T00:27:59.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:59 smithi135 ceph-mon[36424]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:27:59.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:27:59 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1385561311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:27:59.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:59 smithi193 ceph-mon[36716]: mgrmap e22: x(active, since 1.01778s), standbys: z, y 2024-06-10T00:27:59.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:59 smithi193 ceph-mon[36716]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:27:59.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:27:59 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1385561311' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:00.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:00 smithi121 ceph-mon[32378]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:28:00.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:00 smithi121 ceph-mon[32378]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-06-10T00:28:00.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:00 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:00 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:00 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:00 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:00 smithi135 ceph-mon[36424]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:28:00.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:00 smithi135 ceph-mon[36424]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-06-10T00:28:00.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:00 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:00 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:00 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:00 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.811 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-06-10T00:28:00.811 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-06-10T00:28:00.812 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-10T00:28:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:00 smithi193 ceph-mon[36716]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:28:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:00 smithi193 ceph-mon[36716]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-06-10T00:28:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:00 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:00 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:00 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:00 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:00.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.926+0000 7fd36c464640 1 -- 172.21.15.121:0/3310966645 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd364101a10 msgr2=0x7fd364101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:00.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.926+0000 7fd36c464640 1 --2- 172.21.15.121:0/3310966645 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd364101a10 0x7fd364101e30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd360009900 tx=0x7fd36002a410 comp rx=0 tx=0).stop 2024-06-10T00:28:00.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.926+0000 7fd36c464640 1 -- 172.21.15.121:0/3310966645 shutdown_connections 2024-06-10T00:28:00.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.926+0000 7fd36c464640 1 --2- 172.21.15.121:0/3310966645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3641035c0 0x7fd364109e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:00.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.926+0000 7fd36c464640 1 --2- 172.21.15.121:0/3310966645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364102c00 0x7fd364103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:00.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.926+0000 7fd36c464640 1 --2- 172.21.15.121:0/3310966645 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd364101a10 0x7fd364101e30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:00.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.926+0000 7fd36c464640 1 -- 172.21.15.121:0/3310966645 >> 172.21.15.121:0/3310966645 conn(0x7fd3640fcfa0 msgr2=0x7fd3640ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:00.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.926+0000 7fd36c464640 1 -- 172.21.15.121:0/3310966645 shutdown_connections 2024-06-10T00:28:00.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.926+0000 7fd36c464640 1 -- 172.21.15.121:0/3310966645 wait complete. 2024-06-10T00:28:00.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36c464640 1 Processor -- start 2024-06-10T00:28:00.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36c464640 1 -- start start 2024-06-10T00:28:00.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36c464640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101a10 0x7fd364199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:00.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36c464640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364102c00 0x7fd36419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:00.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36c464640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3641035c0 0x7fd3641a14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:00.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36c464640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd36410d660 con 0x7fd364101a10 2024-06-10T00:28:00.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36c464640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd36410d4e0 con 0x7fd3641035c0 2024-06-10T00:28:00.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36c464640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd36410d7e0 con 0x7fd364102c00 2024-06-10T00:28:00.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36a200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101a10 0x7fd364199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:00.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36a200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101a10 0x7fd364199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41402/0 (socket says 172.21.15.121:41402) 2024-06-10T00:28:00.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36a200640 1 -- 172.21.15.121:0/1180712094 learned_addr learned my addr 172.21.15.121:0/1180712094 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:00.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36aa01640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3641035c0 0x7fd3641a14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:00.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd3699ff640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364102c00 0x7fd36419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:00.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.927+0000 7fd36a200640 1 -- 172.21.15.121:0/1180712094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364102c00 msgr2=0x7fd36419a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:00.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.928+0000 7fd36a200640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364102c00 0x7fd36419a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:00.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.928+0000 7fd36a200640 1 -- 172.21.15.121:0/1180712094 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3641035c0 msgr2=0x7fd3641a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:00.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.928+0000 7fd36a200640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3641035c0 0x7fd3641a14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:00.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.928+0000 7fd36a200640 1 -- 172.21.15.121:0/1180712094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3641a1b10 con 0x7fd364101a10 2024-06-10T00:28:00.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.928+0000 7fd36aa01640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3641035c0 0x7fd3641a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:00.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.928+0000 7fd36a200640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101a10 0x7fd364199ef0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd360031610 tx=0x7fd360004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:00.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.928+0000 7fd35b7fe640 1 -- 172.21.15.121:0/1180712094 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36003f030 con 0x7fd364101a10 2024-06-10T00:28:00.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.928+0000 7fd36c464640 1 -- 172.21.15.121:0/1180712094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3641a1da0 con 0x7fd364101a10 2024-06-10T00:28:00.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.928+0000 7fd35b7fe640 1 -- 172.21.15.121:0/1180712094 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd360003ad0 con 0x7fd364101a10 2024-06-10T00:28:00.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.928+0000 7fd35b7fe640 1 -- 172.21.15.121:0/1180712094 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360035070 con 0x7fd364101a10 2024-06-10T00:28:00.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.928+0000 7fd36c464640 1 -- 172.21.15.121:0/1180712094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3641a2220 con 0x7fd364101a10 2024-06-10T00:28:00.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.929+0000 7fd36c464640 1 -- 172.21.15.121:0/1180712094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3641a24b0 con 0x7fd364101a10 2024-06-10T00:28:00.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.930+0000 7fd35b7fe640 1 -- 172.21.15.121:0/1180712094 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 23) ==== 137954+0+0 (secure 0 0 0) 0x7fd360042030 con 0x7fd364101a10 2024-06-10T00:28:00.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.931+0000 7fd35b7fe640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd3500a2d00 0x7fd3500a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:00.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.931+0000 7fd3699ff640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd3500a2d00 0x7fd3500a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:00.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.931+0000 7fd35b7fe640 1 -- 172.21.15.121:0/1180712094 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fd3600f0150 con 0x7fd364101a10 2024-06-10T00:28:00.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.931+0000 7fd3699ff640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd3500a2d00 0x7fd3500a51c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd3540095f0 tx=0x7fd354005f90 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:00.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:00.935+0000 7fd35b7fe640 1 -- 172.21.15.121:0/1180712094 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd3600bc9e0 con 0x7fd364101a10 2024-06-10T00:28:01.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.148+0000 7fd36c464640 1 -- 172.21.15.121:0/1180712094 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd3641a2740 con 0x7fd3500a2d00 2024-06-10T00:28:01.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.158+0000 7fd35b7fe640 1 -- 172.21.15.121:0/1180712094 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339875 (secure 0 0 0) 0x7fd3641a2740 con 0x7fd3500a2d00 2024-06-10T00:28:01.163 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:01.165 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-06-10T00:28:01.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.161+0000 7fd36c464640 1 -- 172.21.15.121:0/1180712094 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd3500a2d00 msgr2=0x7fd3500a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.161+0000 7fd36c464640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd3500a2d00 0x7fd3500a51c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd3540095f0 tx=0x7fd354005f90 comp rx=0 tx=0).stop 2024-06-10T00:28:01.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.161+0000 7fd36c464640 1 -- 172.21.15.121:0/1180712094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101a10 msgr2=0x7fd364199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.161+0000 7fd36c464640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101a10 0x7fd364199ef0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd360031610 tx=0x7fd360004270 comp rx=0 tx=0).stop 2024-06-10T00:28:01.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.161+0000 7fd36c464640 1 -- 172.21.15.121:0/1180712094 shutdown_connections 2024-06-10T00:28:01.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.162+0000 7fd36c464640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd3641035c0 0x7fd3641a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.162+0000 7fd36c464640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd3500a2d00 0x7fd3500a51c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.162+0000 7fd36c464640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364102c00 0x7fd36419a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.162+0000 7fd36c464640 1 --2- 172.21.15.121:0/1180712094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd364101a10 0x7fd364199ef0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.162+0000 7fd36c464640 1 -- 172.21.15.121:0/1180712094 >> 172.21.15.121:0/1180712094 conn(0x7fd3640fcfa0 msgr2=0x7fd364073bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.162+0000 7fd36c464640 1 -- 172.21.15.121:0/1180712094 shutdown_connections 2024-06-10T00:28:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.162+0000 7fd36c464640 1 -- 172.21.15.121:0/1180712094 wait complete. 2024-06-10T00:28:01.177 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-06-10T00:27:59.472459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190402,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1299,"num_read_kb":812,"num_write":482,"num_write_kb":4387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":874,"ondisk_log_size":874,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367044,"kb_used_data":13688,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574780,"statfs":{"total":1151940427776,"available":1151564574720,"internally_reserved":0,"allocated":14016512,"data_stored":10528477,"data_compressed":603376,"data_compressed_allocated":819200,"data_compressed_original":4489216,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.996009"},"pg_stats":[{"pgid":"3.3f","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592178+0000","last_change":"2024-06-10T00:23:57.564073+0000","last_active":"2024-06-10T00:27:57.592178+0000","last_peered":"2024-06-10T00:27:57.592178+0000","last_clean":"2024-06-10T00:27:57.592178+0000","last_became_active":"2024-06-10T00:23:57.563618+0000","last_became_peered":"2024-06-10T00:23:57.563618+0000","last_unstale":"2024-06-10T00:27:57.592178+0000","last_undegraded":"2024-06-10T00:27:57.592178+0000","last_fullsized":"2024-06-10T00:27:57.592178+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:26:25.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591355+0000","last_change":"2024-06-10T00:23:57.560148+0000","last_active":"2024-06-10T00:27:57.591355+0000","last_peered":"2024-06-10T00:27:57.591355+0000","last_clean":"2024-06-10T00:27:57.591355+0000","last_became_active":"2024-06-10T00:23:57.559794+0000","last_became_peered":"2024-06-10T00:23:57.559794+0000","last_unstale":"2024-06-10T00:27:57.591355+0000","last_undegraded":"2024-06-10T00:27:57.591355+0000","last_fullsized":"2024-06-10T00:27:57.591355+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:59:50.624542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593114+0000","last_change":"2024-06-10T00:23:57.563292+0000","last_active":"2024-06-10T00:27:57.593114+0000","last_peered":"2024-06-10T00:27:57.593114+0000","last_clean":"2024-06-10T00:27:57.593114+0000","last_became_active":"2024-06-10T00:23:57.563007+0000","last_became_peered":"2024-06-10T00:23:57.563007+0000","last_unstale":"2024-06-10T00:27:57.593114+0000","last_undegraded":"2024-06-10T00:27:57.593114+0000","last_fullsized":"2024-06-10T00:27:57.593114+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:27:44.897770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545262+0000","last_change":"2024-06-10T00:23:57.558789+0000","last_active":"2024-06-10T00:27:57.545262+0000","last_peered":"2024-06-10T00:27:57.545262+0000","last_clean":"2024-06-10T00:27:57.545262+0000","last_became_active":"2024-06-10T00:23:57.557943+0000","last_became_peered":"2024-06-10T00:23:57.557943+0000","last_unstale":"2024-06-10T00:27:57.545262+0000","last_undegraded":"2024-06-10T00:27:57.545262+0000","last_fullsized":"2024-06-10T00:27:57.545262+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:10:40.081996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569868+0000","last_change":"2024-06-10T00:23:57.550806+0000","last_active":"2024-06-10T00:27:57.569868+0000","last_peered":"2024-06-10T00:27:57.569868+0000","last_clean":"2024-06-10T00:27:57.569868+0000","last_became_active":"2024-06-10T00:23:57.550453+0000","last_became_peered":"2024-06-10T00:23:57.550453+0000","last_unstale":"2024-06-10T00:27:57.569868+0000","last_undegraded":"2024-06-10T00:27:57.569868+0000","last_fullsized":"2024-06-10T00:27:57.569868+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:11:58.765099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458657+0000","last_change":"2024-06-10T00:23:57.567307+0000","last_active":"2024-06-10T00:27:57.458657+0000","last_peered":"2024-06-10T00:27:57.458657+0000","last_clean":"2024-06-10T00:27:57.458657+0000","last_became_active":"2024-06-10T00:23:57.566973+0000","last_became_peered":"2024-06-10T00:23:57.566973+0000","last_unstale":"2024-06-10T00:27:57.458657+0000","last_undegraded":"2024-06-10T00:27:57.458657+0000","last_fullsized":"2024-06-10T00:27:57.458657+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:27:25.135046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591279+0000","last_change":"2024-06-10T00:23:57.547998+0000","last_active":"2024-06-10T00:27:57.591279+0000","last_peered":"2024-06-10T00:27:57.591279+0000","last_clean":"2024-06-10T00:27:57.591279+0000","last_became_active":"2024-06-10T00:23:57.547593+0000","last_became_peered":"2024-06-10T00:23:57.547593+0000","last_unstale":"2024-06-10T00:27:57.591279+0000","last_undegraded":"2024-06-10T00:27:57.591279+0000","last_fullsized":"2024-06-10T00:27:57.591279+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:34:04.219399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570182+0000","last_change":"2024-06-10T00:23:57.560252+0000","last_active":"2024-06-10T00:27:57.570182+0000","last_peered":"2024-06-10T00:27:57.570182+0000","last_clean":"2024-06-10T00:27:57.570182+0000","last_became_active":"2024-06-10T00:23:57.559924+0000","last_became_peered":"2024-06-10T00:23:57.559924+0000","last_unstale":"2024-06-10T00:27:57.570182+0000","last_undegraded":"2024-06-10T00:27:57.570182+0000","last_fullsized":"2024-06-10T00:27:57.570182+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:01:43.684262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590876+0000","last_change":"2024-06-10T00:23:57.564031+0000","last_active":"2024-06-10T00:27:57.590876+0000","last_peered":"2024-06-10T00:27:57.590876+0000","last_clean":"2024-06-10T00:27:57.590876+0000","last_became_active":"2024-06-10T00:23:57.563642+0000","last_became_peered":"2024-06-10T00:23:57.563642+0000","last_unstale":"2024-06-10T00:27:57.590876+0000","last_undegraded":"2024-06-10T00:27:57.590876+0000","last_fullsized":"2024-06-10T00:27:57.590876+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:16:23.003152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"100'16","reported_seq":57,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459536+0000","last_change":"2024-06-10T00:23:57.566877+0000","last_active":"2024-06-10T00:27:57.459536+0000","last_peered":"2024-06-10T00:27:57.459536+0000","last_clean":"2024-06-10T00:27:57.459536+0000","last_became_active":"2024-06-10T00:23:57.566529+0000","last_became_peered":"2024-06-10T00:23:57.566529+0000","last_unstale":"2024-06-10T00:27:57.459536+0000","last_undegraded":"2024-06-10T00:27:57.459536+0000","last_fullsized":"2024-06-10T00:27:57.459536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:30:43.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592151+0000","last_change":"2024-06-10T00:23:57.564979+0000","last_active":"2024-06-10T00:27:57.592151+0000","last_peered":"2024-06-10T00:27:57.592151+0000","last_clean":"2024-06-10T00:27:57.592151+0000","last_became_active":"2024-06-10T00:23:57.564578+0000","last_became_peered":"2024-06-10T00:23:57.564578+0000","last_unstale":"2024-06-10T00:27:57.592151+0000","last_undegraded":"2024-06-10T00:27:57.592151+0000","last_fullsized":"2024-06-10T00:27:57.592151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:40:53.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"100'7","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545061+0000","last_change":"2024-06-10T00:23:57.559126+0000","last_active":"2024-06-10T00:27:57.545061+0000","last_peered":"2024-06-10T00:27:57.545061+0000","last_clean":"2024-06-10T00:27:57.545061+0000","last_became_active":"2024-06-10T00:23:57.558401+0000","last_became_peered":"2024-06-10T00:23:57.558401+0000","last_unstale":"2024-06-10T00:27:57.545061+0000","last_undegraded":"2024-06-10T00:27:57.545061+0000","last_fullsized":"2024-06-10T00:27:57.545061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:57.894137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542053+0000","last_change":"2024-06-10T00:23:57.559272+0000","last_active":"2024-06-10T00:27:57.542053+0000","last_peered":"2024-06-10T00:27:57.542053+0000","last_clean":"2024-06-10T00:27:57.542053+0000","last_became_active":"2024-06-10T00:23:57.558941+0000","last_became_peered":"2024-06-10T00:23:57.558941+0000","last_unstale":"2024-06-10T00:27:57.542053+0000","last_undegraded":"2024-06-10T00:27:57.542053+0000","last_fullsized":"2024-06-10T00:27:57.542053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:36:48.088347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591361+0000","last_change":"2024-06-10T00:23:57.555875+0000","last_active":"2024-06-10T00:27:57.591361+0000","last_peered":"2024-06-10T00:27:57.591361+0000","last_clean":"2024-06-10T00:27:57.591361+0000","last_became_active":"2024-06-10T00:23:57.555492+0000","last_became_peered":"2024-06-10T00:23:57.555492+0000","last_unstale":"2024-06-10T00:27:57.591361+0000","last_undegraded":"2024-06-10T00:27:57.591361+0000","last_fullsized":"2024-06-10T00:27:57.591361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:49:10.827458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591360+0000","last_change":"2024-06-10T00:23:57.555316+0000","last_active":"2024-06-10T00:27:57.591360+0000","last_peered":"2024-06-10T00:27:57.591360+0000","last_clean":"2024-06-10T00:27:57.591360+0000","last_became_active":"2024-06-10T00:23:57.554952+0000","last_became_peered":"2024-06-10T00:23:57.554952+0000","last_unstale":"2024-06-10T00:27:57.591360+0000","last_undegraded":"2024-06-10T00:27:57.591360+0000","last_fullsized":"2024-06-10T00:27:57.591360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:03:57.557330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591053+0000","last_change":"2024-06-10T00:23:57.562442+0000","last_active":"2024-06-10T00:27:57.591053+0000","last_peered":"2024-06-10T00:27:57.591053+0000","last_clean":"2024-06-10T00:27:57.591053+0000","last_became_active":"2024-06-10T00:23:57.561965+0000","last_became_peered":"2024-06-10T00:23:57.561965+0000","last_unstale":"2024-06-10T00:27:57.591053+0000","last_undegraded":"2024-06-10T00:27:57.591053+0000","last_fullsized":"2024-06-10T00:27:57.591053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:40:16.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591981+0000","last_change":"2024-06-10T00:23:57.564422+0000","last_active":"2024-06-10T00:27:57.591981+0000","last_peered":"2024-06-10T00:27:57.591981+0000","last_clean":"2024-06-10T00:27:57.591981+0000","last_became_active":"2024-06-10T00:23:57.564069+0000","last_became_peered":"2024-06-10T00:23:57.564069+0000","last_unstale":"2024-06-10T00:27:57.591981+0000","last_undegraded":"2024-06-10T00:27:57.591981+0000","last_fullsized":"2024-06-10T00:27:57.591981+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:07.662465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458239+0000","last_change":"2024-06-10T00:23:57.561284+0000","last_active":"2024-06-10T00:27:57.458239+0000","last_peered":"2024-06-10T00:27:57.458239+0000","last_clean":"2024-06-10T00:27:57.458239+0000","last_became_active":"2024-06-10T00:23:57.560933+0000","last_became_peered":"2024-06-10T00:23:57.560933+0000","last_unstale":"2024-06-10T00:27:57.458239+0000","last_undegraded":"2024-06-10T00:27:57.458239+0000","last_fullsized":"2024-06-10T00:27:57.458239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:58:48.511774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590777+0000","last_change":"2024-06-10T00:23:57.557752+0000","last_active":"2024-06-10T00:27:57.590777+0000","last_peered":"2024-06-10T00:27:57.590777+0000","last_clean":"2024-06-10T00:27:57.590777+0000","last_became_active":"2024-06-10T00:23:57.557419+0000","last_became_peered":"2024-06-10T00:23:57.557419+0000","last_unstale":"2024-06-10T00:27:57.590777+0000","last_undegraded":"2024-06-10T00:27:57.590777+0000","last_fullsized":"2024-06-10T00:27:57.590777+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:48.234181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591122+0000","last_change":"2024-06-10T00:24:09.177183+0000","last_active":"2024-06-10T00:27:57.591122+0000","last_peered":"2024-06-10T00:27:57.591122+0000","last_clean":"2024-06-10T00:27:57.591122+0000","last_became_active":"2024-06-10T00:24:09.176680+0000","last_became_peered":"2024-06-10T00:24:09.176680+0000","last_unstale":"2024-06-10T00:27:57.591122+0000","last_undegraded":"2024-06-10T00:27:57.591122+0000","last_fullsized":"2024-06-10T00:27:57.591122+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:17:06.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"98'9","reported_seq":57,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458177+0000","last_change":"2024-06-10T00:23:54.426364+0000","last_active":"2024-06-10T00:27:57.458177+0000","last_peered":"2024-06-10T00:27:57.458177+0000","last_clean":"2024-06-10T00:27:57.458177+0000","last_became_active":"2024-06-10T00:23:54.426054+0000","last_became_peered":"2024-06-10T00:23:54.426054+0000","last_unstale":"2024-06-10T00:27:57.458177+0000","last_undegraded":"2024-06-10T00:27:57.458177+0000","last_fullsized":"2024-06-10T00:27:57.458177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:47.613402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570428+0000","last_change":"2024-06-10T00:23:57.557206+0000","last_active":"2024-06-10T00:27:57.570428+0000","last_peered":"2024-06-10T00:27:57.570428+0000","last_clean":"2024-06-10T00:27:57.570428+0000","last_became_active":"2024-06-10T00:23:57.556853+0000","last_became_peered":"2024-06-10T00:23:57.556853+0000","last_unstale":"2024-06-10T00:27:57.570428+0000","last_undegraded":"2024-06-10T00:27:57.570428+0000","last_fullsized":"2024-06-10T00:27:57.570428+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:09.686334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544782+0000","last_change":"2024-06-10T00:24:09.171047+0000","last_active":"2024-06-10T00:27:57.544782+0000","last_peered":"2024-06-10T00:27:57.544782+0000","last_clean":"2024-06-10T00:27:57.544782+0000","last_became_active":"2024-06-10T00:24:09.170501+0000","last_became_peered":"2024-06-10T00:24:09.170501+0000","last_unstale":"2024-06-10T00:27:57.544782+0000","last_undegraded":"2024-06-10T00:27:57.544782+0000","last_fullsized":"2024-06-10T00:27:57.544782+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:47:27.690439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"98'2","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571600+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:27:57.571600+0000","last_peered":"2024-06-10T00:27:57.571600+0000","last_clean":"2024-06-10T00:27:57.571600+0000","last_became_active":"2024-06-10T00:23:54.822447+0000","last_became_peered":"2024-06-10T00:23:54.822447+0000","last_unstale":"2024-06-10T00:27:57.571600+0000","last_undegraded":"2024-06-10T00:27:57.571600+0000","last_fullsized":"2024-06-10T00:27:57.571600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:45:14.956540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591811+0000","last_change":"2024-06-10T00:23:57.554958+0000","last_active":"2024-06-10T00:27:57.591811+0000","last_peered":"2024-06-10T00:27:57.591811+0000","last_clean":"2024-06-10T00:27:57.591811+0000","last_became_active":"2024-06-10T00:23:57.554477+0000","last_became_peered":"2024-06-10T00:23:57.554477+0000","last_unstale":"2024-06-10T00:27:57.591811+0000","last_undegraded":"2024-06-10T00:27:57.591811+0000","last_fullsized":"2024-06-10T00:27:57.591811+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:40:06.066967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544895+0000","last_change":"2024-06-10T00:24:09.169642+0000","last_active":"2024-06-10T00:27:57.544895+0000","last_peered":"2024-06-10T00:27:57.544895+0000","last_clean":"2024-06-10T00:27:57.544895+0000","last_became_active":"2024-06-10T00:24:09.168071+0000","last_became_peered":"2024-06-10T00:24:09.168071+0000","last_unstale":"2024-06-10T00:27:57.544895+0000","last_undegraded":"2024-06-10T00:27:57.544895+0000","last_fullsized":"2024-06-10T00:27:57.544895+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:44.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"98'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570379+0000","last_change":"2024-06-10T00:23:54.430543+0000","last_active":"2024-06-10T00:27:57.570379+0000","last_peered":"2024-06-10T00:27:57.570379+0000","last_clean":"2024-06-10T00:27:57.570379+0000","last_became_active":"2024-06-10T00:23:54.430138+0000","last_became_peered":"2024-06-10T00:23:54.430138+0000","last_unstale":"2024-06-10T00:27:57.570379+0000","last_undegraded":"2024-06-10T00:27:57.570379+0000","last_fullsized":"2024-06-10T00:27:57.570379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:39:53.212607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590796+0000","last_change":"2024-06-10T00:23:57.562251+0000","last_active":"2024-06-10T00:27:57.590796+0000","last_peered":"2024-06-10T00:27:57.590796+0000","last_clean":"2024-06-10T00:27:57.590796+0000","last_became_active":"2024-06-10T00:23:57.561903+0000","last_became_peered":"2024-06-10T00:23:57.561903+0000","last_unstale":"2024-06-10T00:27:57.590796+0000","last_undegraded":"2024-06-10T00:27:57.590796+0000","last_fullsized":"2024-06-10T00:27:57.590796+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:59.489475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544829+0000","last_change":"2024-06-10T00:24:09.165894+0000","last_active":"2024-06-10T00:27:57.544829+0000","last_peered":"2024-06-10T00:27:57.544829+0000","last_clean":"2024-06-10T00:27:57.544829+0000","last_became_active":"2024-06-10T00:24:09.165435+0000","last_became_peered":"2024-06-10T00:24:09.165435+0000","last_unstale":"2024-06-10T00:27:57.544829+0000","last_undegraded":"2024-06-10T00:27:57.544829+0000","last_fullsized":"2024-06-10T00:27:57.544829+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:13:36.058375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570296+0000","last_change":"2024-06-10T00:23:54.427868+0000","last_active":"2024-06-10T00:27:57.570296+0000","last_peered":"2024-06-10T00:27:57.570296+0000","last_clean":"2024-06-10T00:27:57.570296+0000","last_became_active":"2024-06-10T00:23:54.427366+0000","last_became_peered":"2024-06-10T00:23:54.427366+0000","last_unstale":"2024-06-10T00:27:57.570296+0000","last_undegraded":"2024-06-10T00:27:57.570296+0000","last_fullsized":"2024-06-10T00:27:57.570296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:15.499292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"100'15","reported_seq":62,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462390+0000","last_change":"2024-06-10T00:24:43.768556+0000","last_active":"2024-06-10T00:27:57.462390+0000","last_peered":"2024-06-10T00:27:57.462390+0000","last_clean":"2024-06-10T00:27:57.462390+0000","last_became_active":"2024-06-10T00:24:43.768090+0000","last_became_peered":"2024-06-10T00:24:43.768090+0000","last_unstale":"2024-06-10T00:27:57.462390+0000","last_undegraded":"2024-06-10T00:27:57.462390+0000","last_fullsized":"2024-06-10T00:27:57.462390+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:05:41.604184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591403+0000","last_change":"2024-06-10T00:24:09.170514+0000","last_active":"2024-06-10T00:27:57.591403+0000","last_peered":"2024-06-10T00:27:57.591403+0000","last_clean":"2024-06-10T00:27:57.591403+0000","last_became_active":"2024-06-10T00:24:09.168902+0000","last_became_peered":"2024-06-10T00:24:09.168902+0000","last_unstale":"2024-06-10T00:27:57.591403+0000","last_undegraded":"2024-06-10T00:27:57.591403+0000","last_fullsized":"2024-06-10T00:27:57.591403+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:33:10.534657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"101'9","reported_seq":234,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.552203+0000","last_change":"2024-06-10T00:23:54.426581+0000","last_active":"2024-06-10T00:27:57.552203+0000","last_peered":"2024-06-10T00:27:57.552203+0000","last_clean":"2024-06-10T00:27:57.552203+0000","last_became_active":"2024-06-10T00:23:54.426211+0000","last_became_peered":"2024-06-10T00:23:54.426211+0000","last_unstale":"2024-06-10T00:27:57.552203+0000","last_undegraded":"2024-06-10T00:27:57.552203+0000","last_fullsized":"2024-06-10T00:27:57.552203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:49:05.360742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":178,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569865+0000","last_change":"2024-06-10T00:23:57.552308+0000","last_active":"2024-06-10T00:27:57.569865+0000","last_peered":"2024-06-10T00:27:57.569865+0000","last_clean":"2024-06-10T00:27:57.569865+0000","last_became_active":"2024-06-10T00:23:57.551762+0000","last_became_peered":"2024-06-10T00:23:57.551762+0000","last_unstale":"2024-06-10T00:27:57.569865+0000","last_undegraded":"2024-06-10T00:27:57.569865+0000","last_fullsized":"2024-06-10T00:27:57.569865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:22:28.476019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591613+0000","last_change":"2024-06-10T00:24:09.171456+0000","last_active":"2024-06-10T00:27:57.591613+0000","last_peered":"2024-06-10T00:27:57.591613+0000","last_clean":"2024-06-10T00:27:57.591613+0000","last_became_active":"2024-06-10T00:24:09.170969+0000","last_became_peered":"2024-06-10T00:24:09.170969+0000","last_unstale":"2024-06-10T00:27:57.591613+0000","last_undegraded":"2024-06-10T00:27:57.591613+0000","last_fullsized":"2024-06-10T00:27:57.591613+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:22:46.023897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458602+0000","last_change":"2024-06-10T00:23:54.428384+0000","last_active":"2024-06-10T00:27:57.458602+0000","last_peered":"2024-06-10T00:27:57.458602+0000","last_clean":"2024-06-10T00:27:57.458602+0000","last_became_active":"2024-06-10T00:23:54.427967+0000","last_became_peered":"2024-06-10T00:23:54.427967+0000","last_unstale":"2024-06-10T00:27:57.458602+0000","last_undegraded":"2024-06-10T00:27:57.458602+0000","last_fullsized":"2024-06-10T00:27:57.458602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:54:30.408684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"100'8","reported_seq":55,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461853+0000","last_change":"2024-06-10T00:24:43.262593+0000","last_active":"2024-06-10T00:27:57.461853+0000","last_peered":"2024-06-10T00:27:57.461853+0000","last_clean":"2024-06-10T00:27:57.461853+0000","last_became_active":"2024-06-10T00:24:43.262212+0000","last_became_peered":"2024-06-10T00:24:43.262212+0000","last_unstale":"2024-06-10T00:27:57.461853+0000","last_undegraded":"2024-06-10T00:27:57.461853+0000","last_fullsized":"2024-06-10T00:27:57.461853+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:40:24.442712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591973+0000","last_change":"2024-06-10T00:24:43.752805+0000","last_active":"2024-06-10T00:27:57.591973+0000","last_peered":"2024-06-10T00:27:57.591973+0000","last_clean":"2024-06-10T00:27:57.591973+0000","last_became_active":"2024-06-10T00:24:43.752251+0000","last_became_peered":"2024-06-10T00:24:43.752251+0000","last_unstale":"2024-06-10T00:27:57.591973+0000","last_undegraded":"2024-06-10T00:27:57.591973+0000","last_fullsized":"2024-06-10T00:27:57.591973+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:27:57.617443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542566+0000","last_change":"2024-06-10T00:23:54.421493+0000","last_active":"2024-06-10T00:27:57.542566+0000","last_peered":"2024-06-10T00:27:57.542566+0000","last_clean":"2024-06-10T00:27:57.542566+0000","last_became_active":"2024-06-10T00:23:54.421151+0000","last_became_peered":"2024-06-10T00:23:54.421151+0000","last_unstale":"2024-06-10T00:27:57.542566+0000","last_undegraded":"2024-06-10T00:27:57.542566+0000","last_fullsized":"2024-06-10T00:27:57.542566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:03.995244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570099+0000","last_change":"2024-06-10T00:23:57.554020+0000","last_active":"2024-06-10T00:27:57.570099+0000","last_peered":"2024-06-10T00:27:57.570099+0000","last_clean":"2024-06-10T00:27:57.570099+0000","last_became_active":"2024-06-10T00:23:57.553690+0000","last_became_peered":"2024-06-10T00:23:57.553690+0000","last_unstale":"2024-06-10T00:27:57.570099+0000","last_undegraded":"2024-06-10T00:27:57.570099+0000","last_fullsized":"2024-06-10T00:27:57.570099+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:48.097074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593589+0000","last_change":"2024-06-10T00:24:09.166426+0000","last_active":"2024-06-10T00:27:57.593589+0000","last_peered":"2024-06-10T00:27:57.593589+0000","last_clean":"2024-06-10T00:27:57.593589+0000","last_became_active":"2024-06-10T00:24:09.166047+0000","last_became_peered":"2024-06-10T00:24:09.166047+0000","last_unstale":"2024-06-10T00:27:57.593589+0000","last_undegraded":"2024-06-10T00:27:57.593589+0000","last_fullsized":"2024-06-10T00:27:57.593589+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:56:11.608061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"98'1","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461240+0000","last_change":"2024-06-10T00:23:54.426409+0000","last_active":"2024-06-10T00:27:57.461240+0000","last_peered":"2024-06-10T00:27:57.461240+0000","last_clean":"2024-06-10T00:27:57.461240+0000","last_became_active":"2024-06-10T00:23:54.425385+0000","last_became_peered":"2024-06-10T00:23:54.425385+0000","last_unstale":"2024-06-10T00:27:57.461240+0000","last_undegraded":"2024-06-10T00:27:57.461240+0000","last_fullsized":"2024-06-10T00:27:57.461240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:27:00.379189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457389+0000","last_change":"2024-06-10T00:23:57.560132+0000","last_active":"2024-06-10T00:27:57.457389+0000","last_peered":"2024-06-10T00:27:57.457389+0000","last_clean":"2024-06-10T00:27:57.457389+0000","last_became_active":"2024-06-10T00:23:57.559803+0000","last_became_peered":"2024-06-10T00:23:57.559803+0000","last_unstale":"2024-06-10T00:27:57.457389+0000","last_undegraded":"2024-06-10T00:27:57.457389+0000","last_fullsized":"2024-06-10T00:27:57.457389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:13:16.004233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571137+0000","last_change":"2024-06-10T00:24:09.170962+0000","last_active":"2024-06-10T00:27:57.571137+0000","last_peered":"2024-06-10T00:27:57.571137+0000","last_clean":"2024-06-10T00:27:57.571137+0000","last_became_active":"2024-06-10T00:24:09.170090+0000","last_became_peered":"2024-06-10T00:24:09.170090+0000","last_unstale":"2024-06-10T00:27:57.571137+0000","last_undegraded":"2024-06-10T00:27:57.571137+0000","last_fullsized":"2024-06-10T00:27:57.571137+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:07:33.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'5","reported_seq":225,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459435+0000","last_change":"2024-06-10T00:23:54.421472+0000","last_active":"2024-06-10T00:27:57.459435+0000","last_peered":"2024-06-10T00:27:57.459435+0000","last_clean":"2024-06-10T00:27:57.459435+0000","last_became_active":"2024-06-10T00:23:54.421019+0000","last_became_peered":"2024-06-10T00:23:54.421019+0000","last_unstale":"2024-06-10T00:27:57.459435+0000","last_undegraded":"2024-06-10T00:27:57.459435+0000","last_fullsized":"2024-06-10T00:27:57.459435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:28:18.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":173,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590963+0000","last_change":"2024-06-10T00:23:57.563010+0000","last_active":"2024-06-10T00:27:57.590963+0000","last_peered":"2024-06-10T00:27:57.590963+0000","last_clean":"2024-06-10T00:27:57.590963+0000","last_became_active":"2024-06-10T00:23:57.561986+0000","last_became_peered":"2024-06-10T00:23:57.561986+0000","last_unstale":"2024-06-10T00:27:57.590963+0000","last_undegraded":"2024-06-10T00:27:57.590963+0000","last_fullsized":"2024-06-10T00:27:57.590963+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:31:17.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457415+0000","last_change":"2024-06-10T00:24:09.171746+0000","last_active":"2024-06-10T00:27:57.457415+0000","last_peered":"2024-06-10T00:27:57.457415+0000","last_clean":"2024-06-10T00:27:57.457415+0000","last_became_active":"2024-06-10T00:24:09.171030+0000","last_became_peered":"2024-06-10T00:24:09.171030+0000","last_unstale":"2024-06-10T00:27:57.457415+0000","last_undegraded":"2024-06-10T00:27:57.457415+0000","last_fullsized":"2024-06-10T00:27:57.457415+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:41.669471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591812+0000","last_change":"2024-06-10T00:23:54.428917+0000","last_active":"2024-06-10T00:27:57.591812+0000","last_peered":"2024-06-10T00:27:57.591812+0000","last_clean":"2024-06-10T00:27:57.591812+0000","last_became_active":"2024-06-10T00:23:54.428585+0000","last_became_peered":"2024-06-10T00:23:54.428585+0000","last_unstale":"2024-06-10T00:27:57.591812+0000","last_undegraded":"2024-06-10T00:27:57.591812+0000","last_fullsized":"2024-06-10T00:27:57.591812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:21:58.235848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591586+0000","last_change":"2024-06-10T00:23:57.551778+0000","last_active":"2024-06-10T00:27:57.591586+0000","last_peered":"2024-06-10T00:27:57.591586+0000","last_clean":"2024-06-10T00:27:57.591586+0000","last_became_active":"2024-06-10T00:23:57.551436+0000","last_became_peered":"2024-06-10T00:23:57.551436+0000","last_unstale":"2024-06-10T00:27:57.591586+0000","last_undegraded":"2024-06-10T00:27:57.591586+0000","last_fullsized":"2024-06-10T00:27:57.591586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:30.059783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545101+0000","last_change":"2024-06-10T00:24:09.167547+0000","last_active":"2024-06-10T00:27:57.545101+0000","last_peered":"2024-06-10T00:27:57.545101+0000","last_clean":"2024-06-10T00:27:57.545101+0000","last_became_active":"2024-06-10T00:24:09.167132+0000","last_became_peered":"2024-06-10T00:24:09.167132+0000","last_unstale":"2024-06-10T00:27:57.545101+0000","last_undegraded":"2024-06-10T00:27:57.545101+0000","last_fullsized":"2024-06-10T00:27:57.545101+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:11.520330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461450+0000","last_change":"2024-06-10T00:23:54.823754+0000","last_active":"2024-06-10T00:27:57.461450+0000","last_peered":"2024-06-10T00:27:57.461450+0000","last_clean":"2024-06-10T00:27:57.461450+0000","last_became_active":"2024-06-10T00:23:54.823362+0000","last_became_peered":"2024-06-10T00:23:54.823362+0000","last_unstale":"2024-06-10T00:27:57.461450+0000","last_undegraded":"2024-06-10T00:27:57.461450+0000","last_fullsized":"2024-06-10T00:27:57.461450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:48:50.915439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590772+0000","last_change":"2024-06-10T00:23:57.563313+0000","last_active":"2024-06-10T00:27:57.590772+0000","last_peered":"2024-06-10T00:27:57.590772+0000","last_clean":"2024-06-10T00:27:57.590772+0000","last_became_active":"2024-06-10T00:23:57.562307+0000","last_became_peered":"2024-06-10T00:23:57.562307+0000","last_unstale":"2024-06-10T00:27:57.590772+0000","last_undegraded":"2024-06-10T00:27:57.590772+0000","last_fullsized":"2024-06-10T00:27:57.590772+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:23.815267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457507+0000","last_change":"2024-06-10T00:24:09.174660+0000","last_active":"2024-06-10T00:27:57.457507+0000","last_peered":"2024-06-10T00:27:57.457507+0000","last_clean":"2024-06-10T00:27:57.457507+0000","last_became_active":"2024-06-10T00:24:09.174283+0000","last_became_peered":"2024-06-10T00:24:09.174283+0000","last_unstale":"2024-06-10T00:27:57.457507+0000","last_undegraded":"2024-06-10T00:27:57.457507+0000","last_fullsized":"2024-06-10T00:27:57.457507+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:15.236146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"101'41","reported_seq":446,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.749518+0000","last_change":"2024-06-10T00:23:54.426621+0000","last_active":"2024-06-10T00:27:57.749518+0000","last_peered":"2024-06-10T00:27:57.749518+0000","last_clean":"2024-06-10T00:27:57.749518+0000","last_became_active":"2024-06-10T00:23:54.426131+0000","last_became_peered":"2024-06-10T00:23:54.426131+0000","last_unstale":"2024-06-10T00:27:57.749518+0000","last_undegraded":"2024-06-10T00:27:57.749518+0000","last_fullsized":"2024-06-10T00:27:57.749518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:56:14.819454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69365,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":358,"num_read_kb":247,"num_write":42,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570532+0000","last_change":"2024-06-10T00:23:57.552606+0000","last_active":"2024-06-10T00:27:57.570532+0000","last_peered":"2024-06-10T00:27:57.570532+0000","last_clean":"2024-06-10T00:27:57.570532+0000","last_became_active":"2024-06-10T00:23:57.552246+0000","last_became_peered":"2024-06-10T00:23:57.552246+0000","last_unstale":"2024-06-10T00:27:57.570532+0000","last_undegraded":"2024-06-10T00:27:57.570532+0000","last_fullsized":"2024-06-10T00:27:57.570532+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:19:21.508721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462817+0000","last_change":"2024-06-10T00:24:09.174862+0000","last_active":"2024-06-10T00:27:57.462817+0000","last_peered":"2024-06-10T00:27:57.462817+0000","last_clean":"2024-06-10T00:27:57.462817+0000","last_became_active":"2024-06-10T00:24:09.173550+0000","last_became_peered":"2024-06-10T00:24:09.173550+0000","last_unstale":"2024-06-10T00:27:57.462817+0000","last_undegraded":"2024-06-10T00:27:57.462817+0000","last_fullsized":"2024-06-10T00:27:57.462817+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:49:25.549084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"98'6","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591325+0000","last_change":"2024-06-10T00:23:54.426074+0000","last_active":"2024-06-10T00:27:57.591325+0000","last_peered":"2024-06-10T00:27:57.591325+0000","last_clean":"2024-06-10T00:27:57.591325+0000","last_became_active":"2024-06-10T00:23:54.425707+0000","last_became_peered":"2024-06-10T00:23:54.425707+0000","last_unstale":"2024-06-10T00:27:57.591325+0000","last_undegraded":"2024-06-10T00:27:57.591325+0000","last_fullsized":"2024-06-10T00:27:57.591325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:11:10.563595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542128+0000","last_change":"2024-06-10T00:23:57.549784+0000","last_active":"2024-06-10T00:27:57.542128+0000","last_peered":"2024-06-10T00:27:57.542128+0000","last_clean":"2024-06-10T00:27:57.542128+0000","last_became_active":"2024-06-10T00:23:57.549457+0000","last_became_peered":"2024-06-10T00:23:57.549457+0000","last_unstale":"2024-06-10T00:27:57.542128+0000","last_undegraded":"2024-06-10T00:27:57.542128+0000","last_fullsized":"2024-06-10T00:27:57.542128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:59:27.352927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592095+0000","last_change":"2024-06-10T00:24:09.171028+0000","last_active":"2024-06-10T00:27:57.592095+0000","last_peered":"2024-06-10T00:27:57.592095+0000","last_clean":"2024-06-10T00:27:57.592095+0000","last_became_active":"2024-06-10T00:24:09.170543+0000","last_became_peered":"2024-06-10T00:24:09.170543+0000","last_unstale":"2024-06-10T00:27:57.592095+0000","last_undegraded":"2024-06-10T00:27:57.592095+0000","last_fullsized":"2024-06-10T00:27:57.592095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:57:40.816585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461583+0000","last_change":"2024-06-10T00:23:54.423332+0000","last_active":"2024-06-10T00:27:57.461583+0000","last_peered":"2024-06-10T00:27:57.461583+0000","last_clean":"2024-06-10T00:27:57.461583+0000","last_became_active":"2024-06-10T00:23:54.422792+0000","last_became_peered":"2024-06-10T00:23:54.422792+0000","last_unstale":"2024-06-10T00:27:57.461583+0000","last_undegraded":"2024-06-10T00:27:57.461583+0000","last_fullsized":"2024-06-10T00:27:57.461583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:50:52.655641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"100'15","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591219+0000","last_change":"2024-06-10T00:23:57.551411+0000","last_active":"2024-06-10T00:27:57.591219+0000","last_peered":"2024-06-10T00:27:57.591219+0000","last_clean":"2024-06-10T00:27:57.591219+0000","last_became_active":"2024-06-10T00:23:57.551053+0000","last_became_peered":"2024-06-10T00:23:57.551053+0000","last_unstale":"2024-06-10T00:27:57.591219+0000","last_undegraded":"2024-06-10T00:27:57.591219+0000","last_fullsized":"2024-06-10T00:27:57.591219+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:16:18.082777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459282+0000","last_change":"2024-06-10T00:24:43.648767+0000","last_active":"2024-06-10T00:27:57.459282+0000","last_peered":"2024-06-10T00:27:57.459282+0000","last_clean":"2024-06-10T00:27:57.459282+0000","last_became_active":"2024-06-10T00:24:43.648267+0000","last_became_peered":"2024-06-10T00:24:43.648267+0000","last_unstale":"2024-06-10T00:27:57.459282+0000","last_undegraded":"2024-06-10T00:27:57.459282+0000","last_fullsized":"2024-06-10T00:27:57.459282+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:41:09.005202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"101'5","reported_seq":226,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.752230+0000","last_change":"2024-06-10T00:23:54.425963+0000","last_active":"2024-06-10T00:27:57.752230+0000","last_peered":"2024-06-10T00:27:57.752230+0000","last_clean":"2024-06-10T00:27:57.752230+0000","last_became_active":"2024-06-10T00:23:54.425581+0000","last_became_peered":"2024-06-10T00:23:54.425581+0000","last_unstale":"2024-06-10T00:27:57.752230+0000","last_undegraded":"2024-06-10T00:27:57.752230+0000","last_fullsized":"2024-06-10T00:27:57.752230+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:54.760610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":176,"num_read_kb":3,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592340+0000","last_change":"2024-06-10T00:23:57.557464+0000","last_active":"2024-06-10T00:27:57.592340+0000","last_peered":"2024-06-10T00:27:57.592340+0000","last_clean":"2024-06-10T00:27:57.592340+0000","last_became_active":"2024-06-10T00:23:57.557130+0000","last_became_peered":"2024-06-10T00:23:57.557130+0000","last_unstale":"2024-06-10T00:27:57.592340+0000","last_undegraded":"2024-06-10T00:27:57.592340+0000","last_fullsized":"2024-06-10T00:27:57.592340+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:16:36.341210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":38,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543379+0000","last_change":"2024-06-10T00:24:43.753291+0000","last_active":"2024-06-10T00:27:57.543379+0000","last_peered":"2024-06-10T00:27:57.543379+0000","last_clean":"2024-06-10T00:27:57.543379+0000","last_became_active":"2024-06-10T00:24:43.752765+0000","last_became_peered":"2024-06-10T00:24:43.752765+0000","last_unstale":"2024-06-10T00:27:57.543379+0000","last_undegraded":"2024-06-10T00:27:57.543379+0000","last_fullsized":"2024-06-10T00:27:57.543379+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:55:06.349564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"98'15","reported_seq":63,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570040+0000","last_change":"2024-06-10T00:23:54.823489+0000","last_active":"2024-06-10T00:27:57.570040+0000","last_peered":"2024-06-10T00:27:57.570040+0000","last_clean":"2024-06-10T00:27:57.570040+0000","last_became_active":"2024-06-10T00:23:54.823073+0000","last_became_peered":"2024-06-10T00:23:54.823073+0000","last_unstale":"2024-06-10T00:27:57.570040+0000","last_undegraded":"2024-06-10T00:27:57.570040+0000","last_fullsized":"2024-06-10T00:27:57.570040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:07:52.563009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461389+0000","last_change":"2024-06-10T00:23:57.563388+0000","last_active":"2024-06-10T00:27:57.461389+0000","last_peered":"2024-06-10T00:27:57.461389+0000","last_clean":"2024-06-10T00:27:57.461389+0000","last_became_active":"2024-06-10T00:23:57.563004+0000","last_became_peered":"2024-06-10T00:23:57.563004+0000","last_unstale":"2024-06-10T00:27:57.461389+0000","last_undegraded":"2024-06-10T00:27:57.461389+0000","last_fullsized":"2024-06-10T00:27:57.461389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:51:18.805662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461226+0000","last_change":"2024-06-10T00:24:09.168447+0000","last_active":"2024-06-10T00:27:57.461226+0000","last_peered":"2024-06-10T00:27:57.461226+0000","last_clean":"2024-06-10T00:27:57.461226+0000","last_became_active":"2024-06-10T00:24:09.167439+0000","last_became_peered":"2024-06-10T00:24:09.167439+0000","last_unstale":"2024-06-10T00:27:57.461226+0000","last_undegraded":"2024-06-10T00:27:57.461226+0000","last_fullsized":"2024-06-10T00:27:57.461226+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:52.319395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"101'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570752+0000","last_change":"2024-06-10T00:23:54.427666+0000","last_active":"2024-06-10T00:27:57.570752+0000","last_peered":"2024-06-10T00:27:57.570752+0000","last_clean":"2024-06-10T00:27:57.570752+0000","last_became_active":"2024-06-10T00:23:54.427285+0000","last_became_peered":"2024-06-10T00:23:54.427285+0000","last_unstale":"2024-06-10T00:27:57.570752+0000","last_undegraded":"2024-06-10T00:27:57.570752+0000","last_fullsized":"2024-06-10T00:27:57.570752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:28:20.196168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"100'5","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.558787+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.558470+0000","last_became_peered":"2024-06-10T00:23:57.558470+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:02:14.758802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570142+0000","last_change":"2024-06-10T00:24:09.264952+0000","last_active":"2024-06-10T00:27:57.570142+0000","last_peered":"2024-06-10T00:27:57.570142+0000","last_clean":"2024-06-10T00:27:57.570142+0000","last_became_active":"2024-06-10T00:24:09.264379+0000","last_became_peered":"2024-06-10T00:24:09.264379+0000","last_unstale":"2024-06-10T00:27:57.570142+0000","last_undegraded":"2024-06-10T00:27:57.570142+0000","last_fullsized":"2024-06-10T00:27:57.570142+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:25:40.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458121+0000","last_change":"2024-06-10T00:23:54.823298+0000","last_active":"2024-06-10T00:27:57.458121+0000","last_peered":"2024-06-10T00:27:57.458121+0000","last_clean":"2024-06-10T00:27:57.458121+0000","last_became_active":"2024-06-10T00:23:54.822917+0000","last_became_peered":"2024-06-10T00:23:54.822917+0000","last_unstale":"2024-06-10T00:27:57.458121+0000","last_undegraded":"2024-06-10T00:27:57.458121+0000","last_fullsized":"2024-06-10T00:27:57.458121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:07:52.500641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459177+0000","last_change":"2024-06-10T00:23:57.559202+0000","last_active":"2024-06-10T00:27:57.459177+0000","last_peered":"2024-06-10T00:27:57.459177+0000","last_clean":"2024-06-10T00:27:57.459177+0000","last_became_active":"2024-06-10T00:23:57.558800+0000","last_became_peered":"2024-06-10T00:23:57.558800+0000","last_unstale":"2024-06-10T00:27:57.459177+0000","last_undegraded":"2024-06-10T00:27:57.459177+0000","last_fullsized":"2024-06-10T00:27:57.459177+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:49:51.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542486+0000","last_change":"2024-06-10T00:24:09.168603+0000","last_active":"2024-06-10T00:27:57.542486+0000","last_peered":"2024-06-10T00:27:57.542486+0000","last_clean":"2024-06-10T00:27:57.542486+0000","last_became_active":"2024-06-10T00:24:09.168233+0000","last_became_peered":"2024-06-10T00:24:09.168233+0000","last_unstale":"2024-06-10T00:27:57.542486+0000","last_undegraded":"2024-06-10T00:27:57.542486+0000","last_fullsized":"2024-06-10T00:27:57.542486+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:59.585023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'1","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592210+0000","last_change":"2024-06-10T00:23:54.425505+0000","last_active":"2024-06-10T00:27:57.592210+0000","last_peered":"2024-06-10T00:27:57.592210+0000","last_clean":"2024-06-10T00:27:57.592210+0000","last_became_active":"2024-06-10T00:23:54.425058+0000","last_became_peered":"2024-06-10T00:23:54.425058+0000","last_unstale":"2024-06-10T00:27:57.592210+0000","last_undegraded":"2024-06-10T00:27:57.592210+0000","last_fullsized":"2024-06-10T00:27:57.592210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:48:05.605947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458827+0000","last_change":"2024-06-10T00:23:57.566060+0000","last_active":"2024-06-10T00:27:57.458827+0000","last_peered":"2024-06-10T00:27:57.458827+0000","last_clean":"2024-06-10T00:27:57.458827+0000","last_became_active":"2024-06-10T00:23:57.565099+0000","last_became_peered":"2024-06-10T00:23:57.565099+0000","last_unstale":"2024-06-10T00:27:57.458827+0000","last_undegraded":"2024-06-10T00:27:57.458827+0000","last_fullsized":"2024-06-10T00:27:57.458827+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:24:54.204399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543284+0000","last_change":"2024-06-10T00:24:09.264741+0000","last_active":"2024-06-10T00:27:57.543284+0000","last_peered":"2024-06-10T00:27:57.543284+0000","last_clean":"2024-06-10T00:27:57.543284+0000","last_became_active":"2024-06-10T00:24:09.264267+0000","last_became_peered":"2024-06-10T00:24:09.264267+0000","last_unstale":"2024-06-10T00:27:57.543284+0000","last_undegraded":"2024-06-10T00:27:57.543284+0000","last_fullsized":"2024-06-10T00:27:57.543284+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:27.627391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457224+0000","last_change":"2024-06-10T00:25:44.346070+0000","last_active":"2024-06-10T00:27:57.457224+0000","last_peered":"2024-06-10T00:27:57.457224+0000","last_clean":"2024-06-10T00:27:57.457224+0000","last_became_active":"2024-06-10T00:25:44.345798+0000","last_became_peered":"2024-06-10T00:25:44.345798+0000","last_unstale":"2024-06-10T00:27:57.457224+0000","last_undegraded":"2024-06-10T00:27:57.457224+0000","last_fullsized":"2024-06-10T00:27:57.457224+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:04.506535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"101'5","reported_seq":225,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542310+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:27:57.542310+0000","last_peered":"2024-06-10T00:27:57.542310+0000","last_clean":"2024-06-10T00:27:57.542310+0000","last_became_active":"2024-06-10T00:23:54.822442+0000","last_became_peered":"2024-06-10T00:23:54.822442+0000","last_unstale":"2024-06-10T00:27:57.542310+0000","last_undegraded":"2024-06-10T00:27:57.542310+0000","last_fullsized":"2024-06-10T00:27:57.542310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:51:10.278257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":173,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"98'1","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591953+0000","last_change":"2024-06-10T00:24:09.168073+0000","last_active":"2024-06-10T00:27:57.591953+0000","last_peered":"2024-06-10T00:27:57.591953+0000","last_clean":"2024-06-10T00:27:57.591953+0000","last_became_active":"2024-06-10T00:24:09.167657+0000","last_became_peered":"2024-06-10T00:24:09.167657+0000","last_unstale":"2024-06-10T00:27:57.591953+0000","last_undegraded":"2024-06-10T00:27:57.591953+0000","last_fullsized":"2024-06-10T00:27:57.591953+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:20:23.481685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457629+0000","last_change":"2024-06-10T00:25:44.346629+0000","last_active":"2024-06-10T00:27:57.457629+0000","last_peered":"2024-06-10T00:27:57.457629+0000","last_clean":"2024-06-10T00:27:57.457629+0000","last_became_active":"2024-06-10T00:25:44.346388+0000","last_became_peered":"2024-06-10T00:25:44.346388+0000","last_unstale":"2024-06-10T00:27:57.457629+0000","last_undegraded":"2024-06-10T00:27:57.457629+0000","last_fullsized":"2024-06-10T00:27:57.457629+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:03:19.228632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544918+0000","last_change":"2024-06-10T00:23:54.423507+0000","last_active":"2024-06-10T00:27:57.544918+0000","last_peered":"2024-06-10T00:27:57.544918+0000","last_clean":"2024-06-10T00:27:57.544918+0000","last_became_active":"2024-06-10T00:23:54.423092+0000","last_became_peered":"2024-06-10T00:23:54.423092+0000","last_unstale":"2024-06-10T00:27:57.544918+0000","last_undegraded":"2024-06-10T00:27:57.544918+0000","last_fullsized":"2024-06-10T00:27:57.544918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:06:15.924827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":380,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592578+0000","last_change":"2024-06-10T00:22:28.025243+0000","last_active":"2024-06-10T00:27:57.592578+0000","last_peered":"2024-06-10T00:27:57.592578+0000","last_clean":"2024-06-10T00:27:57.592578+0000","last_became_active":"2024-06-10T00:22:28.024751+0000","last_became_peered":"2024-06-10T00:22:28.024751+0000","last_unstale":"2024-06-10T00:27:57.592578+0000","last_undegraded":"2024-06-10T00:27:57.592578+0000","last_fullsized":"2024-06-10T00:27:57.592578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_clean_scrub_stamp":"2024-06-10T00:19:05.039631+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:02:36.557931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461952+0000","last_change":"2024-06-10T00:24:09.265084+0000","last_active":"2024-06-10T00:27:57.461952+0000","last_peered":"2024-06-10T00:27:57.461952+0000","last_clean":"2024-06-10T00:27:57.461952+0000","last_became_active":"2024-06-10T00:24:09.264425+0000","last_became_peered":"2024-06-10T00:24:09.264425+0000","last_unstale":"2024-06-10T00:27:57.461952+0000","last_undegraded":"2024-06-10T00:27:57.461952+0000","last_fullsized":"2024-06-10T00:27:57.461952+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:37:31.577688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.541918+0000","last_change":"2024-06-10T00:23:57.558540+0000","last_active":"2024-06-10T00:27:57.541918+0000","last_peered":"2024-06-10T00:27:57.541918+0000","last_clean":"2024-06-10T00:27:57.541918+0000","last_became_active":"2024-06-10T00:23:57.558197+0000","last_became_peered":"2024-06-10T00:23:57.558197+0000","last_unstale":"2024-06-10T00:27:57.541918+0000","last_undegraded":"2024-06-10T00:27:57.541918+0000","last_fullsized":"2024-06-10T00:27:57.541918+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:04:25.566135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592707+0000","last_change":"2024-06-10T00:23:54.428126+0000","last_active":"2024-06-10T00:27:57.592707+0000","last_peered":"2024-06-10T00:27:57.592707+0000","last_clean":"2024-06-10T00:27:57.592707+0000","last_became_active":"2024-06-10T00:23:54.427630+0000","last_became_peered":"2024-06-10T00:23:54.427630+0000","last_unstale":"2024-06-10T00:27:57.592707+0000","last_undegraded":"2024-06-10T00:27:57.592707+0000","last_fullsized":"2024-06-10T00:27:57.592707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:05:23.679704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591957+0000","last_change":"2024-06-10T00:24:09.169984+0000","last_active":"2024-06-10T00:27:57.591957+0000","last_peered":"2024-06-10T00:27:57.591957+0000","last_clean":"2024-06-10T00:27:57.591957+0000","last_became_active":"2024-06-10T00:24:09.169556+0000","last_became_peered":"2024-06-10T00:24:09.169556+0000","last_unstale":"2024-06-10T00:27:57.591957+0000","last_undegraded":"2024-06-10T00:27:57.591957+0000","last_fullsized":"2024-06-10T00:27:57.591957+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:54:13.575207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"98'1","reported_seq":20,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591972+0000","last_change":"2024-06-10T00:24:43.767901+0000","last_active":"2024-06-10T00:27:57.591972+0000","last_peered":"2024-06-10T00:27:57.591972+0000","last_clean":"2024-06-10T00:27:57.591972+0000","last_became_active":"2024-06-10T00:24:43.767562+0000","last_became_peered":"2024-06-10T00:24:43.767562+0000","last_unstale":"2024-06-10T00:27:57.591972+0000","last_undegraded":"2024-06-10T00:27:57.591972+0000","last_fullsized":"2024-06-10T00:27:57.591972+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:46:36.099750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543049+0000","last_change":"2024-06-10T00:23:57.554906+0000","last_active":"2024-06-10T00:27:57.543049+0000","last_peered":"2024-06-10T00:27:57.543049+0000","last_clean":"2024-06-10T00:27:57.543049+0000","last_became_active":"2024-06-10T00:23:57.554608+0000","last_became_peered":"2024-06-10T00:23:57.554608+0000","last_unstale":"2024-06-10T00:27:57.543049+0000","last_undegraded":"2024-06-10T00:27:57.543049+0000","last_fullsized":"2024-06-10T00:27:57.543049+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:24.690594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570625+0000","last_change":"2024-06-10T00:24:09.264676+0000","last_active":"2024-06-10T00:27:57.570625+0000","last_peered":"2024-06-10T00:27:57.570625+0000","last_clean":"2024-06-10T00:27:57.570625+0000","last_became_active":"2024-06-10T00:24:09.264259+0000","last_became_peered":"2024-06-10T00:24:09.264259+0000","last_unstale":"2024-06-10T00:27:57.570625+0000","last_undegraded":"2024-06-10T00:27:57.570625+0000","last_fullsized":"2024-06-10T00:27:57.570625+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:37:53.787039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"101'48","reported_seq":118,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.689379+0000","last_change":"2024-06-10T00:23:54.424201+0000","last_active":"2024-06-10T00:27:57.689379+0000","last_peered":"2024-06-10T00:27:57.689379+0000","last_clean":"2024-06-10T00:27:57.689379+0000","last_became_active":"2024-06-10T00:23:54.423853+0000","last_became_peered":"2024-06-10T00:23:54.423853+0000","last_unstale":"2024-06-10T00:27:57.689379+0000","last_undegraded":"2024-06-10T00:27:57.689379+0000","last_fullsized":"2024-06-10T00:27:57.689379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:00.884041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":86,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571556+0000","last_change":"2024-06-10T00:23:57.554276+0000","last_active":"2024-06-10T00:27:57.571556+0000","last_peered":"2024-06-10T00:27:57.571556+0000","last_clean":"2024-06-10T00:27:57.571556+0000","last_became_active":"2024-06-10T00:23:57.553964+0000","last_became_peered":"2024-06-10T00:23:57.553964+0000","last_unstale":"2024-06-10T00:27:57.571556+0000","last_undegraded":"2024-06-10T00:27:57.571556+0000","last_fullsized":"2024-06-10T00:27:57.571556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:39:51.974770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591089+0000","last_change":"2024-06-10T00:24:09.170249+0000","last_active":"2024-06-10T00:27:57.591089+0000","last_peered":"2024-06-10T00:27:57.591089+0000","last_clean":"2024-06-10T00:27:57.591089+0000","last_became_active":"2024-06-10T00:24:09.169830+0000","last_became_peered":"2024-06-10T00:24:09.169830+0000","last_unstale":"2024-06-10T00:27:57.591089+0000","last_undegraded":"2024-06-10T00:27:57.591089+0000","last_fullsized":"2024-06-10T00:27:57.591089+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.320609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"98'4","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545304+0000","last_change":"2024-06-10T00:23:54.473943+0000","last_active":"2024-06-10T00:27:57.545304+0000","last_peered":"2024-06-10T00:27:57.545304+0000","last_clean":"2024-06-10T00:27:57.545304+0000","last_became_active":"2024-06-10T00:23:54.473652+0000","last_became_peered":"2024-06-10T00:23:54.473652+0000","last_unstale":"2024-06-10T00:27:57.545304+0000","last_undegraded":"2024-06-10T00:27:57.545304+0000","last_fullsized":"2024-06-10T00:27:57.545304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:52:06.773684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"100'7","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593713+0000","last_change":"2024-06-10T00:25:44.735050+0000","last_active":"2024-06-10T00:27:57.593713+0000","last_peered":"2024-06-10T00:27:57.593713+0000","last_clean":"2024-06-10T00:27:57.593713+0000","last_became_active":"2024-06-10T00:25:44.734756+0000","last_became_peered":"2024-06-10T00:25:44.734756+0000","last_unstale":"2024-06-10T00:27:57.593713+0000","last_undegraded":"2024-06-10T00:27:57.593713+0000","last_fullsized":"2024-06-10T00:27:57.593713+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:16:03.924238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545083+0000","last_change":"2024-06-10T00:24:43.768837+0000","last_active":"2024-06-10T00:27:57.545083+0000","last_peered":"2024-06-10T00:27:57.545083+0000","last_clean":"2024-06-10T00:27:57.545083+0000","last_became_active":"2024-06-10T00:24:43.768295+0000","last_became_peered":"2024-06-10T00:24:43.768295+0000","last_unstale":"2024-06-10T00:27:57.545083+0000","last_undegraded":"2024-06-10T00:27:57.545083+0000","last_fullsized":"2024-06-10T00:27:57.545083+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:59.600864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591709+0000","last_change":"2024-06-10T00:23:54.823448+0000","last_active":"2024-06-10T00:27:57.591709+0000","last_peered":"2024-06-10T00:27:57.591709+0000","last_clean":"2024-06-10T00:27:57.591709+0000","last_became_active":"2024-06-10T00:23:54.823107+0000","last_became_peered":"2024-06-10T00:23:54.823107+0000","last_unstale":"2024-06-10T00:27:57.591709+0000","last_undegraded":"2024-06-10T00:27:57.591709+0000","last_fullsized":"2024-06-10T00:27:57.591709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:50.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"100'2","reported_seq":43,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569862+0000","last_change":"2024-06-10T00:23:57.556733+0000","last_active":"2024-06-10T00:27:57.569862+0000","last_peered":"2024-06-10T00:27:57.569862+0000","last_clean":"2024-06-10T00:27:57.569862+0000","last_became_active":"2024-06-10T00:23:57.556386+0000","last_became_peered":"2024-06-10T00:23:57.556386+0000","last_unstale":"2024-06-10T00:27:57.569862+0000","last_undegraded":"2024-06-10T00:27:57.569862+0000","last_fullsized":"2024-06-10T00:27:57.569862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:39:00.572311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458015+0000","last_change":"2024-06-10T00:24:09.174600+0000","last_active":"2024-06-10T00:27:57.458015+0000","last_peered":"2024-06-10T00:27:57.458015+0000","last_clean":"2024-06-10T00:27:57.458015+0000","last_became_active":"2024-06-10T00:24:09.174236+0000","last_became_peered":"2024-06-10T00:24:09.174236+0000","last_unstale":"2024-06-10T00:27:57.458015+0000","last_undegraded":"2024-06-10T00:27:57.458015+0000","last_fullsized":"2024-06-10T00:27:57.458015+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:58:18.447768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571190+0000","last_change":"2024-06-10T00:23:54.425111+0000","last_active":"2024-06-10T00:27:57.571190+0000","last_peered":"2024-06-10T00:27:57.571190+0000","last_clean":"2024-06-10T00:27:57.571190+0000","last_became_active":"2024-06-10T00:23:54.424724+0000","last_became_peered":"2024-06-10T00:23:54.424724+0000","last_unstale":"2024-06-10T00:27:57.571190+0000","last_undegraded":"2024-06-10T00:27:57.571190+0000","last_fullsized":"2024-06-10T00:27:57.571190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:54.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"100'9","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.556355+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.555479+0000","last_became_peered":"2024-06-10T00:23:57.555479+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:08.817381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"98'1","reported_seq":30,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544968+0000","last_change":"2024-06-10T00:24:09.169589+0000","last_active":"2024-06-10T00:27:57.544968+0000","last_peered":"2024-06-10T00:27:57.544968+0000","last_clean":"2024-06-10T00:27:57.544968+0000","last_became_active":"2024-06-10T00:24:09.168436+0000","last_became_peered":"2024-06-10T00:24:09.168436+0000","last_unstale":"2024-06-10T00:27:57.544968+0000","last_undegraded":"2024-06-10T00:27:57.544968+0000","last_fullsized":"2024-06-10T00:27:57.544968+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:17:32.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"98'1","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591500+0000","last_change":"2024-06-10T00:24:43.751399+0000","last_active":"2024-06-10T00:27:57.591500+0000","last_peered":"2024-06-10T00:27:57.591500+0000","last_clean":"2024-06-10T00:27:57.591500+0000","last_became_active":"2024-06-10T00:24:43.750987+0000","last_became_peered":"2024-06-10T00:24:43.750987+0000","last_unstale":"2024-06-10T00:27:57.591500+0000","last_undegraded":"2024-06-10T00:27:57.591500+0000","last_fullsized":"2024-06-10T00:27:57.591500+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:14:47.902017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570338+0000","last_change":"2024-06-10T00:23:57.560213+0000","last_active":"2024-06-10T00:27:57.570338+0000","last_peered":"2024-06-10T00:27:57.570338+0000","last_clean":"2024-06-10T00:27:57.570338+0000","last_became_active":"2024-06-10T00:23:57.559925+0000","last_became_peered":"2024-06-10T00:23:57.559925+0000","last_unstale":"2024-06-10T00:27:57.570338+0000","last_undegraded":"2024-06-10T00:27:57.570338+0000","last_fullsized":"2024-06-10T00:27:57.570338+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:00:37.403087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545095+0000","last_change":"2024-06-10T00:24:09.264758+0000","last_active":"2024-06-10T00:27:57.545095+0000","last_peered":"2024-06-10T00:27:57.545095+0000","last_clean":"2024-06-10T00:27:57.545095+0000","last_became_active":"2024-06-10T00:24:09.263928+0000","last_became_peered":"2024-06-10T00:24:09.263928+0000","last_unstale":"2024-06-10T00:27:57.545095+0000","last_undegraded":"2024-06-10T00:27:57.545095+0000","last_fullsized":"2024-06-10T00:27:57.545095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:23:41.083091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570103+0000","last_change":"2024-06-10T00:23:54.428553+0000","last_active":"2024-06-10T00:27:57.570103+0000","last_peered":"2024-06-10T00:27:57.570103+0000","last_clean":"2024-06-10T00:27:57.570103+0000","last_became_active":"2024-06-10T00:23:54.428108+0000","last_became_peered":"2024-06-10T00:23:54.428108+0000","last_unstale":"2024-06-10T00:27:57.570103+0000","last_undegraded":"2024-06-10T00:27:57.570103+0000","last_fullsized":"2024-06-10T00:27:57.570103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:57:26.207915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591119+0000","last_change":"2024-06-10T00:23:57.560686+0000","last_active":"2024-06-10T00:27:57.591119+0000","last_peered":"2024-06-10T00:27:57.591119+0000","last_clean":"2024-06-10T00:27:57.591119+0000","last_became_active":"2024-06-10T00:23:57.560326+0000","last_became_peered":"2024-06-10T00:23:57.560326+0000","last_unstale":"2024-06-10T00:27:57.591119+0000","last_undegraded":"2024-06-10T00:27:57.591119+0000","last_fullsized":"2024-06-10T00:27:57.591119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:08:39.506567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591281+0000","last_change":"2024-06-10T00:24:09.166947+0000","last_active":"2024-06-10T00:27:57.591281+0000","last_peered":"2024-06-10T00:27:57.591281+0000","last_clean":"2024-06-10T00:27:57.591281+0000","last_became_active":"2024-06-10T00:24:09.166503+0000","last_became_peered":"2024-06-10T00:24:09.166503+0000","last_unstale":"2024-06-10T00:27:57.591281+0000","last_undegraded":"2024-06-10T00:27:57.591281+0000","last_fullsized":"2024-06-10T00:27:57.591281+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:16:45.094642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591369+0000","last_change":"2024-06-10T00:23:54.823522+0000","last_active":"2024-06-10T00:27:57.591369+0000","last_peered":"2024-06-10T00:27:57.591369+0000","last_clean":"2024-06-10T00:27:57.591369+0000","last_became_active":"2024-06-10T00:23:54.823142+0000","last_became_peered":"2024-06-10T00:23:54.823142+0000","last_unstale":"2024-06-10T00:27:57.591369+0000","last_undegraded":"2024-06-10T00:27:57.591369+0000","last_fullsized":"2024-06-10T00:27:57.591369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:03.836850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592288+0000","last_change":"2024-06-10T00:23:57.563710+0000","last_active":"2024-06-10T00:27:57.592288+0000","last_peered":"2024-06-10T00:27:57.592288+0000","last_clean":"2024-06-10T00:27:57.592288+0000","last_became_active":"2024-06-10T00:23:57.563331+0000","last_became_peered":"2024-06-10T00:23:57.563331+0000","last_unstale":"2024-06-10T00:27:57.592288+0000","last_undegraded":"2024-06-10T00:27:57.592288+0000","last_fullsized":"2024-06-10T00:27:57.592288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:08:48.061885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592820+0000","last_change":"2024-06-10T00:24:09.171203+0000","last_active":"2024-06-10T00:27:57.592820+0000","last_peered":"2024-06-10T00:27:57.592820+0000","last_clean":"2024-06-10T00:27:57.592820+0000","last_became_active":"2024-06-10T00:24:09.170824+0000","last_became_peered":"2024-06-10T00:24:09.170824+0000","last_unstale":"2024-06-10T00:27:57.592820+0000","last_undegraded":"2024-06-10T00:27:57.592820+0000","last_fullsized":"2024-06-10T00:27:57.592820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:47:28.045602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"101'39","reported_seq":94,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.686846+0000","last_change":"2024-06-10T00:23:54.421307+0000","last_active":"2024-06-10T00:27:57.686846+0000","last_peered":"2024-06-10T00:27:57.686846+0000","last_clean":"2024-06-10T00:27:57.686846+0000","last_became_active":"2024-06-10T00:23:54.420928+0000","last_became_peered":"2024-06-10T00:23:54.420928+0000","last_unstale":"2024-06-10T00:27:57.686846+0000","last_undegraded":"2024-06-10T00:27:57.686846+0000","last_fullsized":"2024-06-10T00:27:57.686846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:59:41.433605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591623+0000","last_change":"2024-06-10T00:23:57.556494+0000","last_active":"2024-06-10T00:27:57.591623+0000","last_peered":"2024-06-10T00:27:57.591623+0000","last_clean":"2024-06-10T00:27:57.591623+0000","last_became_active":"2024-06-10T00:23:57.556088+0000","last_became_peered":"2024-06-10T00:23:57.556088+0000","last_unstale":"2024-06-10T00:27:57.591623+0000","last_undegraded":"2024-06-10T00:27:57.591623+0000","last_fullsized":"2024-06-10T00:27:57.591623+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:06:23.873281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461068+0000","last_change":"2024-06-10T00:24:09.166734+0000","last_active":"2024-06-10T00:27:57.461068+0000","last_peered":"2024-06-10T00:27:57.461068+0000","last_clean":"2024-06-10T00:27:57.461068+0000","last_became_active":"2024-06-10T00:24:09.166270+0000","last_became_peered":"2024-06-10T00:24:09.166270+0000","last_unstale":"2024-06-10T00:27:57.461068+0000","last_undegraded":"2024-06-10T00:27:57.461068+0000","last_fullsized":"2024-06-10T00:27:57.461068+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:38.039633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543256+0000","last_change":"2024-06-10T00:23:54.426226+0000","last_active":"2024-06-10T00:27:57.543256+0000","last_peered":"2024-06-10T00:27:57.543256+0000","last_clean":"2024-06-10T00:27:57.543256+0000","last_became_active":"2024-06-10T00:23:54.425879+0000","last_became_peered":"2024-06-10T00:23:54.425879+0000","last_unstale":"2024-06-10T00:27:57.543256+0000","last_undegraded":"2024-06-10T00:27:57.543256+0000","last_fullsized":"2024-06-10T00:27:57.543256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:15:32.936556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592043+0000","last_change":"2024-06-10T00:23:57.553814+0000","last_active":"2024-06-10T00:27:57.592043+0000","last_peered":"2024-06-10T00:27:57.592043+0000","last_clean":"2024-06-10T00:27:57.592043+0000","last_became_active":"2024-06-10T00:23:57.553478+0000","last_became_peered":"2024-06-10T00:23:57.553478+0000","last_unstale":"2024-06-10T00:27:57.592043+0000","last_undegraded":"2024-06-10T00:27:57.592043+0000","last_fullsized":"2024-06-10T00:27:57.592043+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:54:39.264526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"100'9","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591390+0000","last_change":"2024-06-10T00:23:57.556676+0000","last_active":"2024-06-10T00:27:57.591390+0000","last_peered":"2024-06-10T00:27:57.591390+0000","last_clean":"2024-06-10T00:27:57.591390+0000","last_became_active":"2024-06-10T00:23:57.555946+0000","last_became_peered":"2024-06-10T00:23:57.555946+0000","last_unstale":"2024-06-10T00:27:57.591390+0000","last_undegraded":"2024-06-10T00:27:57.591390+0000","last_fullsized":"2024-06-10T00:27:57.591390+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:10:10.868799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"100'12","reported_seq":53,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590743+0000","last_change":"2024-06-10T00:23:57.565423+0000","last_active":"2024-06-10T00:27:57.590743+0000","last_peered":"2024-06-10T00:27:57.590743+0000","last_clean":"2024-06-10T00:27:57.590743+0000","last_became_active":"2024-06-10T00:23:57.564830+0000","last_became_peered":"2024-06-10T00:23:57.564830+0000","last_unstale":"2024-06-10T00:27:57.590743+0000","last_undegraded":"2024-06-10T00:27:57.590743+0000","last_fullsized":"2024-06-10T00:27:57.590743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:11:06.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590982+0000","last_change":"2024-06-10T00:23:57.558971+0000","last_active":"2024-06-10T00:27:57.590982+0000","last_peered":"2024-06-10T00:27:57.590982+0000","last_clean":"2024-06-10T00:27:57.590982+0000","last_became_active":"2024-06-10T00:23:57.558620+0000","last_became_peered":"2024-06-10T00:23:57.558620+0000","last_unstale":"2024-06-10T00:27:57.590982+0000","last_undegraded":"2024-06-10T00:27:57.590982+0000","last_fullsized":"2024-06-10T00:27:57.590982+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:45:43.308008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591491+0000","last_change":"2024-06-10T00:23:57.552341+0000","last_active":"2024-06-10T00:27:57.591491+0000","last_peered":"2024-06-10T00:27:57.591491+0000","last_clean":"2024-06-10T00:27:57.591491+0000","last_became_active":"2024-06-10T00:23:57.551464+0000","last_became_peered":"2024-06-10T00:23:57.551464+0000","last_unstale":"2024-06-10T00:27:57.591491+0000","last_undegraded":"2024-06-10T00:27:57.591491+0000","last_fullsized":"2024-06-10T00:27:57.591491+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:44:41.063516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"100'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543209+0000","last_change":"2024-06-10T00:24:43.768961+0000","last_active":"2024-06-10T00:27:57.543209+0000","last_peered":"2024-06-10T00:27:57.543209+0000","last_clean":"2024-06-10T00:27:57.543209+0000","last_became_active":"2024-06-10T00:24:43.768542+0000","last_became_peered":"2024-06-10T00:24:43.768542+0000","last_unstale":"2024-06-10T00:27:57.543209+0000","last_undegraded":"2024-06-10T00:27:57.543209+0000","last_fullsized":"2024-06-10T00:27:57.543209+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:23:58.412054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"100'6","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591613+0000","last_change":"2024-06-10T00:23:57.554924+0000","last_active":"2024-06-10T00:27:57.591613+0000","last_peered":"2024-06-10T00:27:57.591613+0000","last_clean":"2024-06-10T00:27:57.591613+0000","last_became_active":"2024-06-10T00:23:57.554515+0000","last_became_peered":"2024-06-10T00:23:57.554515+0000","last_unstale":"2024-06-10T00:27:57.591613+0000","last_undegraded":"2024-06-10T00:27:57.591613+0000","last_fullsized":"2024-06-10T00:27:57.591613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:58:15.624775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"100'15","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591485+0000","last_change":"2024-06-10T00:23:57.561940+0000","last_active":"2024-06-10T00:27:57.591485+0000","last_peered":"2024-06-10T00:27:57.591485+0000","last_clean":"2024-06-10T00:27:57.591485+0000","last_became_active":"2024-06-10T00:23:57.561512+0000","last_became_peered":"2024-06-10T00:23:57.561512+0000","last_unstale":"2024-06-10T00:27:57.591485+0000","last_undegraded":"2024-06-10T00:27:57.591485+0000","last_fullsized":"2024-06-10T00:27:57.591485+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:57:57.147311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"100'2","reported_seq":43,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570333+0000","last_change":"2024-06-10T00:23:57.557229+0000","last_active":"2024-06-10T00:27:57.570333+0000","last_peered":"2024-06-10T00:27:57.570333+0000","last_clean":"2024-06-10T00:27:57.570333+0000","last_became_active":"2024-06-10T00:23:57.556905+0000","last_became_peered":"2024-06-10T00:23:57.556905+0000","last_unstale":"2024-06-10T00:27:57.570333+0000","last_undegraded":"2024-06-10T00:27:57.570333+0000","last_fullsized":"2024-06-10T00:27:57.570333+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:30.513517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"100'6","reported_seq":24,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592589+0000","last_change":"2024-06-10T00:24:43.768509+0000","last_active":"2024-06-10T00:27:57.592589+0000","last_peered":"2024-06-10T00:27:57.592589+0000","last_clean":"2024-06-10T00:27:57.592589+0000","last_became_active":"2024-06-10T00:24:43.768038+0000","last_became_peered":"2024-06-10T00:24:43.768038+0000","last_unstale":"2024-06-10T00:27:57.592589+0000","last_undegraded":"2024-06-10T00:27:57.592589+0000","last_fullsized":"2024-06-10T00:27:57.592589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:38.344375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571244+0000","last_change":"2024-06-10T00:23:57.557830+0000","last_active":"2024-06-10T00:27:57.571244+0000","last_peered":"2024-06-10T00:27:57.571244+0000","last_clean":"2024-06-10T00:27:57.571244+0000","last_became_active":"2024-06-10T00:23:57.556986+0000","last_became_peered":"2024-06-10T00:23:57.556986+0000","last_unstale":"2024-06-10T00:27:57.571244+0000","last_undegraded":"2024-06-10T00:27:57.571244+0000","last_fullsized":"2024-06-10T00:27:57.571244+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:53.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590760+0000","last_change":"2024-06-10T00:23:57.565602+0000","last_active":"2024-06-10T00:27:57.590760+0000","last_peered":"2024-06-10T00:27:57.590760+0000","last_clean":"2024-06-10T00:27:57.590760+0000","last_became_active":"2024-06-10T00:23:57.565264+0000","last_became_peered":"2024-06-10T00:23:57.565264+0000","last_unstale":"2024-06-10T00:27:57.590760+0000","last_undegraded":"2024-06-10T00:27:57.590760+0000","last_fullsized":"2024-06-10T00:27:57.590760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:09:12.773087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542211+0000","last_change":"2024-06-10T00:23:57.548516+0000","last_active":"2024-06-10T00:27:57.542211+0000","last_peered":"2024-06-10T00:27:57.542211+0000","last_clean":"2024-06-10T00:27:57.542211+0000","last_became_active":"2024-06-10T00:23:57.548202+0000","last_became_peered":"2024-06-10T00:23:57.548202+0000","last_unstale":"2024-06-10T00:27:57.542211+0000","last_undegraded":"2024-06-10T00:27:57.542211+0000","last_fullsized":"2024-06-10T00:27:57.542211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:13.901478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591566+0000","last_change":"2024-06-10T00:23:57.565900+0000","last_active":"2024-06-10T00:27:57.591566+0000","last_peered":"2024-06-10T00:27:57.591566+0000","last_clean":"2024-06-10T00:27:57.591566+0000","last_became_active":"2024-06-10T00:23:57.565573+0000","last_became_peered":"2024-06-10T00:23:57.565573+0000","last_unstale":"2024-06-10T00:27:57.591566+0000","last_undegraded":"2024-06-10T00:27:57.591566+0000","last_fullsized":"2024-06-10T00:27:57.591566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:08.255905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107100,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599800,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1163,"num_read_kb":693,"num_write":299,"num_write_kb":1447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":1799400,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351040,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1118208,"data_stored":582820,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612677,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32432,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712720,"statfs":{"total":95995035648,"available":95961825280,"internally_reserved":0,"allocated":1286144,"data_stored":1371811,"data_compressed":77135,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776203,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1159168,"data_stored":980346,"data_compressed":54425,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037839,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28144,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717008,"statfs":{"total":95995035648,"available":95966216192,"internally_reserved":0,"allocated":1150976,"data_stored":976148,"data_compressed":54425,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299476,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":1179311,"data_compressed":70979,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463001,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1110016,"data_stored":578210,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1183744,"data_stored":647485,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":579598,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411438,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36504,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708648,"statfs":{"total":95995035648,"available":95957655552,"internally_reserved":0,"allocated":1196032,"data_stored":638999,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1167360,"data_stored":1103220,"data_compressed":60581,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888163,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":583640,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724638,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32356,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712796,"statfs":{"total":95995035648,"available":95961903104,"internally_reserved":0,"allocated":1208320,"data_stored":1306889,"data_compressed":77135,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134305,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":203323,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-10T00:28:01.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-10T00:28:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.321+0000 7f1812aae640 1 -- 172.21.15.121:0/1457688193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c1008e0 msgr2=0x7f180c100d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.321+0000 7f1812aae640 1 --2- 172.21.15.121:0/1457688193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c1008e0 0x7f180c100d00 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1804009900 tx=0x7f180402a410 comp rx=0 tx=0).stop 2024-06-10T00:28:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.321+0000 7f1812aae640 1 -- 172.21.15.121:0/1457688193 shutdown_connections 2024-06-10T00:28:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.321+0000 7f1812aae640 1 --2- 172.21.15.121:0/1457688193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f180c102490 0x7f180c108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.321+0000 7f1812aae640 1 --2- 172.21.15.121:0/1457688193 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f180c101ad0 0x7f180c101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.321+0000 7f1812aae640 1 --2- 172.21.15.121:0/1457688193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c1008e0 0x7f180c100d00 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.321+0000 7f1812aae640 1 -- 172.21.15.121:0/1457688193 >> 172.21.15.121:0/1457688193 conn(0x7f180c0fbe50 msgr2=0x7f180c0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.321+0000 7f1812aae640 1 -- 172.21.15.121:0/1457688193 shutdown_connections 2024-06-10T00:28:01.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.321+0000 7f1812aae640 1 -- 172.21.15.121:0/1457688193 wait complete. 2024-06-10T00:28:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f1812aae640 1 Processor -- start 2024-06-10T00:28:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f1812aae640 1 -- start start 2024-06-10T00:28:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f1812aae640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f180c1008e0 0x7f180c196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f1812aae640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f180c101ad0 0x7f180c197190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f1812aae640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c102490 0x7f180c19e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f1812aae640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f180c10c530 con 0x7f180c102490 2024-06-10T00:28:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f1812aae640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f180c10c3b0 con 0x7f180c1008e0 2024-06-10T00:28:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f1812aae640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f180c10c6b0 con 0x7f180c101ad0 2024-06-10T00:28:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f181104b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c102490 0x7f180c19e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f181104b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c102490 0x7f180c19e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41432/0 (socket says 172.21.15.121:41432) 2024-06-10T00:28:01.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f181104b640 1 -- 172.21.15.121:0/2620896711 learned_addr learned my addr 172.21.15.121:0/2620896711 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f180bfff640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f180c101ad0 0x7f180c197190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f181084a640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f180c1008e0 0x7f180c196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f181104b640 1 -- 172.21.15.121:0/2620896711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f180c101ad0 msgr2=0x7f180c197190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f181104b640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f180c101ad0 0x7f180c197190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f181104b640 1 -- 172.21.15.121:0/2620896711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f180c1008e0 msgr2=0x7f180c196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f181104b640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f180c1008e0 0x7f180c196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f181104b640 1 -- 172.21.15.121:0/2620896711 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180c19e860 con 0x7f180c102490 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.322+0000 7f181084a640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f180c1008e0 0x7f180c196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.323+0000 7f181104b640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c102490 0x7f180c19e1c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f17f40098a0 tx=0x7f17f4009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.323+0000 7f1809ffb640 1 -- 172.21.15.121:0/2620896711 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f400d030 con 0x7f180c102490 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.323+0000 7f1809ffb640 1 -- 172.21.15.121:0/2620896711 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17f400d1d0 con 0x7f180c102490 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.323+0000 7f1812aae640 1 -- 172.21.15.121:0/2620896711 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180c068b90 con 0x7f180c102490 2024-06-10T00:28:01.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.323+0000 7f1809ffb640 1 -- 172.21.15.121:0/2620896711 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f4003850 con 0x7f180c102490 2024-06-10T00:28:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.323+0000 7f1812aae640 1 -- 172.21.15.121:0/2620896711 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f180c069090 con 0x7f180c102490 2024-06-10T00:28:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.324+0000 7f1812aae640 1 -- 172.21.15.121:0/2620896711 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f180c0687c0 con 0x7f180c102490 2024-06-10T00:28:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.325+0000 7f1809ffb640 1 -- 172.21.15.121:0/2620896711 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 23) ==== 137954+0+0 (secure 0 0 0) 0x7f17f400c460 con 0x7f180c102490 2024-06-10T00:28:01.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.325+0000 7f1809ffb640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f17f80a2d00 0x7f17f80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:01.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.326+0000 7f181084a640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f17f80a2d00 0x7f17f80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:01.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.326+0000 7f1809ffb640 1 -- 172.21.15.121:0/2620896711 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f17f40d0ef0 con 0x7f180c102490 2024-06-10T00:28:01.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.326+0000 7f181084a640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f17f80a2d00 0x7f17f80a51c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18040314e0 tx=0x7f1804032cf0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:01.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.330+0000 7f1809ffb640 1 -- 172.21.15.121:0/2620896711 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f17f409d780 con 0x7f180c102490 2024-06-10T00:28:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.512+0000 7f1812aae640 1 -- 172.21.15.121:0/2620896711 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f180c069350 con 0x7f17f80a2d00 2024-06-10T00:28:01.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.521+0000 7f1809ffb640 1 -- 172.21.15.121:0/2620896711 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339875 (secure 0 0 0) 0x7f180c069350 con 0x7f17f80a2d00 2024-06-10T00:28:01.525 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:01.527 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-06-10T00:28:01.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 -- 172.21.15.121:0/2620896711 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f17f80a2d00 msgr2=0x7f17f80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f17f80a2d00 0x7f17f80a51c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18040314e0 tx=0x7f1804032cf0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 -- 172.21.15.121:0/2620896711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c102490 msgr2=0x7f180c19e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c102490 0x7f180c19e1c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f17f40098a0 tx=0x7f17f4009d60 comp rx=0 tx=0).stop 2024-06-10T00:28:01.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 -- 172.21.15.121:0/2620896711 shutdown_connections 2024-06-10T00:28:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c102490 0x7f180c19e1c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f17f80a2d00 0x7f17f80a51c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f180c101ad0 0x7f180c197190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 --2- 172.21.15.121:0/2620896711 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f180c1008e0 0x7f180c196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 -- 172.21.15.121:0/2620896711 >> 172.21.15.121:0/2620896711 conn(0x7f180c0fbe50 msgr2=0x7f180c104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 -- 172.21.15.121:0/2620896711 shutdown_connections 2024-06-10T00:28:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.525+0000 7f1812aae640 1 -- 172.21.15.121:0/2620896711 wait complete. 2024-06-10T00:28:01.536 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-06-10T00:28:01.472808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190402,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1299,"num_read_kb":812,"num_write":482,"num_write_kb":4387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":874,"ondisk_log_size":874,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367044,"kb_used_data":13688,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574780,"statfs":{"total":1151940427776,"available":1151564574720,"internally_reserved":0,"allocated":14016512,"data_stored":10528477,"data_compressed":603376,"data_compressed_allocated":819200,"data_compressed_original":4489216,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.996358"},"pg_stats":[{"pgid":"3.3f","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592178+0000","last_change":"2024-06-10T00:23:57.564073+0000","last_active":"2024-06-10T00:27:57.592178+0000","last_peered":"2024-06-10T00:27:57.592178+0000","last_clean":"2024-06-10T00:27:57.592178+0000","last_became_active":"2024-06-10T00:23:57.563618+0000","last_became_peered":"2024-06-10T00:23:57.563618+0000","last_unstale":"2024-06-10T00:27:57.592178+0000","last_undegraded":"2024-06-10T00:27:57.592178+0000","last_fullsized":"2024-06-10T00:27:57.592178+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:26:25.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591355+0000","last_change":"2024-06-10T00:23:57.560148+0000","last_active":"2024-06-10T00:27:57.591355+0000","last_peered":"2024-06-10T00:27:57.591355+0000","last_clean":"2024-06-10T00:27:57.591355+0000","last_became_active":"2024-06-10T00:23:57.559794+0000","last_became_peered":"2024-06-10T00:23:57.559794+0000","last_unstale":"2024-06-10T00:27:57.591355+0000","last_undegraded":"2024-06-10T00:27:57.591355+0000","last_fullsized":"2024-06-10T00:27:57.591355+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:59:50.624542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593114+0000","last_change":"2024-06-10T00:23:57.563292+0000","last_active":"2024-06-10T00:27:57.593114+0000","last_peered":"2024-06-10T00:27:57.593114+0000","last_clean":"2024-06-10T00:27:57.593114+0000","last_became_active":"2024-06-10T00:23:57.563007+0000","last_became_peered":"2024-06-10T00:23:57.563007+0000","last_unstale":"2024-06-10T00:27:57.593114+0000","last_undegraded":"2024-06-10T00:27:57.593114+0000","last_fullsized":"2024-06-10T00:27:57.593114+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:27:44.897770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545262+0000","last_change":"2024-06-10T00:23:57.558789+0000","last_active":"2024-06-10T00:27:57.545262+0000","last_peered":"2024-06-10T00:27:57.545262+0000","last_clean":"2024-06-10T00:27:57.545262+0000","last_became_active":"2024-06-10T00:23:57.557943+0000","last_became_peered":"2024-06-10T00:23:57.557943+0000","last_unstale":"2024-06-10T00:27:57.545262+0000","last_undegraded":"2024-06-10T00:27:57.545262+0000","last_fullsized":"2024-06-10T00:27:57.545262+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:10:40.081996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569868+0000","last_change":"2024-06-10T00:23:57.550806+0000","last_active":"2024-06-10T00:27:57.569868+0000","last_peered":"2024-06-10T00:27:57.569868+0000","last_clean":"2024-06-10T00:27:57.569868+0000","last_became_active":"2024-06-10T00:23:57.550453+0000","last_became_peered":"2024-06-10T00:23:57.550453+0000","last_unstale":"2024-06-10T00:27:57.569868+0000","last_undegraded":"2024-06-10T00:27:57.569868+0000","last_fullsized":"2024-06-10T00:27:57.569868+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:11:58.765099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458657+0000","last_change":"2024-06-10T00:23:57.567307+0000","last_active":"2024-06-10T00:27:57.458657+0000","last_peered":"2024-06-10T00:27:57.458657+0000","last_clean":"2024-06-10T00:27:57.458657+0000","last_became_active":"2024-06-10T00:23:57.566973+0000","last_became_peered":"2024-06-10T00:23:57.566973+0000","last_unstale":"2024-06-10T00:27:57.458657+0000","last_undegraded":"2024-06-10T00:27:57.458657+0000","last_fullsized":"2024-06-10T00:27:57.458657+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:27:25.135046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591279+0000","last_change":"2024-06-10T00:23:57.547998+0000","last_active":"2024-06-10T00:27:57.591279+0000","last_peered":"2024-06-10T00:27:57.591279+0000","last_clean":"2024-06-10T00:27:57.591279+0000","last_became_active":"2024-06-10T00:23:57.547593+0000","last_became_peered":"2024-06-10T00:23:57.547593+0000","last_unstale":"2024-06-10T00:27:57.591279+0000","last_undegraded":"2024-06-10T00:27:57.591279+0000","last_fullsized":"2024-06-10T00:27:57.591279+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:34:04.219399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570182+0000","last_change":"2024-06-10T00:23:57.560252+0000","last_active":"2024-06-10T00:27:57.570182+0000","last_peered":"2024-06-10T00:27:57.570182+0000","last_clean":"2024-06-10T00:27:57.570182+0000","last_became_active":"2024-06-10T00:23:57.559924+0000","last_became_peered":"2024-06-10T00:23:57.559924+0000","last_unstale":"2024-06-10T00:27:57.570182+0000","last_undegraded":"2024-06-10T00:27:57.570182+0000","last_fullsized":"2024-06-10T00:27:57.570182+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:01:43.684262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590876+0000","last_change":"2024-06-10T00:23:57.564031+0000","last_active":"2024-06-10T00:27:57.590876+0000","last_peered":"2024-06-10T00:27:57.590876+0000","last_clean":"2024-06-10T00:27:57.590876+0000","last_became_active":"2024-06-10T00:23:57.563642+0000","last_became_peered":"2024-06-10T00:23:57.563642+0000","last_unstale":"2024-06-10T00:27:57.590876+0000","last_undegraded":"2024-06-10T00:27:57.590876+0000","last_fullsized":"2024-06-10T00:27:57.590876+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:16:23.003152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"100'16","reported_seq":57,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459536+0000","last_change":"2024-06-10T00:23:57.566877+0000","last_active":"2024-06-10T00:27:57.459536+0000","last_peered":"2024-06-10T00:27:57.459536+0000","last_clean":"2024-06-10T00:27:57.459536+0000","last_became_active":"2024-06-10T00:23:57.566529+0000","last_became_peered":"2024-06-10T00:23:57.566529+0000","last_unstale":"2024-06-10T00:27:57.459536+0000","last_undegraded":"2024-06-10T00:27:57.459536+0000","last_fullsized":"2024-06-10T00:27:57.459536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:30:43.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592151+0000","last_change":"2024-06-10T00:23:57.564979+0000","last_active":"2024-06-10T00:27:57.592151+0000","last_peered":"2024-06-10T00:27:57.592151+0000","last_clean":"2024-06-10T00:27:57.592151+0000","last_became_active":"2024-06-10T00:23:57.564578+0000","last_became_peered":"2024-06-10T00:23:57.564578+0000","last_unstale":"2024-06-10T00:27:57.592151+0000","last_undegraded":"2024-06-10T00:27:57.592151+0000","last_fullsized":"2024-06-10T00:27:57.592151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:40:53.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"100'7","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545061+0000","last_change":"2024-06-10T00:23:57.559126+0000","last_active":"2024-06-10T00:27:57.545061+0000","last_peered":"2024-06-10T00:27:57.545061+0000","last_clean":"2024-06-10T00:27:57.545061+0000","last_became_active":"2024-06-10T00:23:57.558401+0000","last_became_peered":"2024-06-10T00:23:57.558401+0000","last_unstale":"2024-06-10T00:27:57.545061+0000","last_undegraded":"2024-06-10T00:27:57.545061+0000","last_fullsized":"2024-06-10T00:27:57.545061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:57.894137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542053+0000","last_change":"2024-06-10T00:23:57.559272+0000","last_active":"2024-06-10T00:27:57.542053+0000","last_peered":"2024-06-10T00:27:57.542053+0000","last_clean":"2024-06-10T00:27:57.542053+0000","last_became_active":"2024-06-10T00:23:57.558941+0000","last_became_peered":"2024-06-10T00:23:57.558941+0000","last_unstale":"2024-06-10T00:27:57.542053+0000","last_undegraded":"2024-06-10T00:27:57.542053+0000","last_fullsized":"2024-06-10T00:27:57.542053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:36:48.088347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591361+0000","last_change":"2024-06-10T00:23:57.555875+0000","last_active":"2024-06-10T00:27:57.591361+0000","last_peered":"2024-06-10T00:27:57.591361+0000","last_clean":"2024-06-10T00:27:57.591361+0000","last_became_active":"2024-06-10T00:23:57.555492+0000","last_became_peered":"2024-06-10T00:23:57.555492+0000","last_unstale":"2024-06-10T00:27:57.591361+0000","last_undegraded":"2024-06-10T00:27:57.591361+0000","last_fullsized":"2024-06-10T00:27:57.591361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:49:10.827458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591360+0000","last_change":"2024-06-10T00:23:57.555316+0000","last_active":"2024-06-10T00:27:57.591360+0000","last_peered":"2024-06-10T00:27:57.591360+0000","last_clean":"2024-06-10T00:27:57.591360+0000","last_became_active":"2024-06-10T00:23:57.554952+0000","last_became_peered":"2024-06-10T00:23:57.554952+0000","last_unstale":"2024-06-10T00:27:57.591360+0000","last_undegraded":"2024-06-10T00:27:57.591360+0000","last_fullsized":"2024-06-10T00:27:57.591360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:03:57.557330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591053+0000","last_change":"2024-06-10T00:23:57.562442+0000","last_active":"2024-06-10T00:27:57.591053+0000","last_peered":"2024-06-10T00:27:57.591053+0000","last_clean":"2024-06-10T00:27:57.591053+0000","last_became_active":"2024-06-10T00:23:57.561965+0000","last_became_peered":"2024-06-10T00:23:57.561965+0000","last_unstale":"2024-06-10T00:27:57.591053+0000","last_undegraded":"2024-06-10T00:27:57.591053+0000","last_fullsized":"2024-06-10T00:27:57.591053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:40:16.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591981+0000","last_change":"2024-06-10T00:23:57.564422+0000","last_active":"2024-06-10T00:27:57.591981+0000","last_peered":"2024-06-10T00:27:57.591981+0000","last_clean":"2024-06-10T00:27:57.591981+0000","last_became_active":"2024-06-10T00:23:57.564069+0000","last_became_peered":"2024-06-10T00:23:57.564069+0000","last_unstale":"2024-06-10T00:27:57.591981+0000","last_undegraded":"2024-06-10T00:27:57.591981+0000","last_fullsized":"2024-06-10T00:27:57.591981+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:07.662465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458239+0000","last_change":"2024-06-10T00:23:57.561284+0000","last_active":"2024-06-10T00:27:57.458239+0000","last_peered":"2024-06-10T00:27:57.458239+0000","last_clean":"2024-06-10T00:27:57.458239+0000","last_became_active":"2024-06-10T00:23:57.560933+0000","last_became_peered":"2024-06-10T00:23:57.560933+0000","last_unstale":"2024-06-10T00:27:57.458239+0000","last_undegraded":"2024-06-10T00:27:57.458239+0000","last_fullsized":"2024-06-10T00:27:57.458239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:58:48.511774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590777+0000","last_change":"2024-06-10T00:23:57.557752+0000","last_active":"2024-06-10T00:27:57.590777+0000","last_peered":"2024-06-10T00:27:57.590777+0000","last_clean":"2024-06-10T00:27:57.590777+0000","last_became_active":"2024-06-10T00:23:57.557419+0000","last_became_peered":"2024-06-10T00:23:57.557419+0000","last_unstale":"2024-06-10T00:27:57.590777+0000","last_undegraded":"2024-06-10T00:27:57.590777+0000","last_fullsized":"2024-06-10T00:27:57.590777+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:48.234181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591122+0000","last_change":"2024-06-10T00:24:09.177183+0000","last_active":"2024-06-10T00:27:57.591122+0000","last_peered":"2024-06-10T00:27:57.591122+0000","last_clean":"2024-06-10T00:27:57.591122+0000","last_became_active":"2024-06-10T00:24:09.176680+0000","last_became_peered":"2024-06-10T00:24:09.176680+0000","last_unstale":"2024-06-10T00:27:57.591122+0000","last_undegraded":"2024-06-10T00:27:57.591122+0000","last_fullsized":"2024-06-10T00:27:57.591122+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:17:06.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"98'9","reported_seq":57,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458177+0000","last_change":"2024-06-10T00:23:54.426364+0000","last_active":"2024-06-10T00:27:57.458177+0000","last_peered":"2024-06-10T00:27:57.458177+0000","last_clean":"2024-06-10T00:27:57.458177+0000","last_became_active":"2024-06-10T00:23:54.426054+0000","last_became_peered":"2024-06-10T00:23:54.426054+0000","last_unstale":"2024-06-10T00:27:57.458177+0000","last_undegraded":"2024-06-10T00:27:57.458177+0000","last_fullsized":"2024-06-10T00:27:57.458177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:47.613402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570428+0000","last_change":"2024-06-10T00:23:57.557206+0000","last_active":"2024-06-10T00:27:57.570428+0000","last_peered":"2024-06-10T00:27:57.570428+0000","last_clean":"2024-06-10T00:27:57.570428+0000","last_became_active":"2024-06-10T00:23:57.556853+0000","last_became_peered":"2024-06-10T00:23:57.556853+0000","last_unstale":"2024-06-10T00:27:57.570428+0000","last_undegraded":"2024-06-10T00:27:57.570428+0000","last_fullsized":"2024-06-10T00:27:57.570428+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:09.686334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544782+0000","last_change":"2024-06-10T00:24:09.171047+0000","last_active":"2024-06-10T00:27:57.544782+0000","last_peered":"2024-06-10T00:27:57.544782+0000","last_clean":"2024-06-10T00:27:57.544782+0000","last_became_active":"2024-06-10T00:24:09.170501+0000","last_became_peered":"2024-06-10T00:24:09.170501+0000","last_unstale":"2024-06-10T00:27:57.544782+0000","last_undegraded":"2024-06-10T00:27:57.544782+0000","last_fullsized":"2024-06-10T00:27:57.544782+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:47:27.690439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"98'2","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571600+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:27:57.571600+0000","last_peered":"2024-06-10T00:27:57.571600+0000","last_clean":"2024-06-10T00:27:57.571600+0000","last_became_active":"2024-06-10T00:23:54.822447+0000","last_became_peered":"2024-06-10T00:23:54.822447+0000","last_unstale":"2024-06-10T00:27:57.571600+0000","last_undegraded":"2024-06-10T00:27:57.571600+0000","last_fullsized":"2024-06-10T00:27:57.571600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:45:14.956540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591811+0000","last_change":"2024-06-10T00:23:57.554958+0000","last_active":"2024-06-10T00:27:57.591811+0000","last_peered":"2024-06-10T00:27:57.591811+0000","last_clean":"2024-06-10T00:27:57.591811+0000","last_became_active":"2024-06-10T00:23:57.554477+0000","last_became_peered":"2024-06-10T00:23:57.554477+0000","last_unstale":"2024-06-10T00:27:57.591811+0000","last_undegraded":"2024-06-10T00:27:57.591811+0000","last_fullsized":"2024-06-10T00:27:57.591811+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:40:06.066967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544895+0000","last_change":"2024-06-10T00:24:09.169642+0000","last_active":"2024-06-10T00:27:57.544895+0000","last_peered":"2024-06-10T00:27:57.544895+0000","last_clean":"2024-06-10T00:27:57.544895+0000","last_became_active":"2024-06-10T00:24:09.168071+0000","last_became_peered":"2024-06-10T00:24:09.168071+0000","last_unstale":"2024-06-10T00:27:57.544895+0000","last_undegraded":"2024-06-10T00:27:57.544895+0000","last_fullsized":"2024-06-10T00:27:57.544895+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:44.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"98'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570379+0000","last_change":"2024-06-10T00:23:54.430543+0000","last_active":"2024-06-10T00:27:57.570379+0000","last_peered":"2024-06-10T00:27:57.570379+0000","last_clean":"2024-06-10T00:27:57.570379+0000","last_became_active":"2024-06-10T00:23:54.430138+0000","last_became_peered":"2024-06-10T00:23:54.430138+0000","last_unstale":"2024-06-10T00:27:57.570379+0000","last_undegraded":"2024-06-10T00:27:57.570379+0000","last_fullsized":"2024-06-10T00:27:57.570379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:39:53.212607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590796+0000","last_change":"2024-06-10T00:23:57.562251+0000","last_active":"2024-06-10T00:27:57.590796+0000","last_peered":"2024-06-10T00:27:57.590796+0000","last_clean":"2024-06-10T00:27:57.590796+0000","last_became_active":"2024-06-10T00:23:57.561903+0000","last_became_peered":"2024-06-10T00:23:57.561903+0000","last_unstale":"2024-06-10T00:27:57.590796+0000","last_undegraded":"2024-06-10T00:27:57.590796+0000","last_fullsized":"2024-06-10T00:27:57.590796+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:59.489475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544829+0000","last_change":"2024-06-10T00:24:09.165894+0000","last_active":"2024-06-10T00:27:57.544829+0000","last_peered":"2024-06-10T00:27:57.544829+0000","last_clean":"2024-06-10T00:27:57.544829+0000","last_became_active":"2024-06-10T00:24:09.165435+0000","last_became_peered":"2024-06-10T00:24:09.165435+0000","last_unstale":"2024-06-10T00:27:57.544829+0000","last_undegraded":"2024-06-10T00:27:57.544829+0000","last_fullsized":"2024-06-10T00:27:57.544829+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:13:36.058375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570296+0000","last_change":"2024-06-10T00:23:54.427868+0000","last_active":"2024-06-10T00:27:57.570296+0000","last_peered":"2024-06-10T00:27:57.570296+0000","last_clean":"2024-06-10T00:27:57.570296+0000","last_became_active":"2024-06-10T00:23:54.427366+0000","last_became_peered":"2024-06-10T00:23:54.427366+0000","last_unstale":"2024-06-10T00:27:57.570296+0000","last_undegraded":"2024-06-10T00:27:57.570296+0000","last_fullsized":"2024-06-10T00:27:57.570296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:15.499292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"100'15","reported_seq":62,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462390+0000","last_change":"2024-06-10T00:24:43.768556+0000","last_active":"2024-06-10T00:27:57.462390+0000","last_peered":"2024-06-10T00:27:57.462390+0000","last_clean":"2024-06-10T00:27:57.462390+0000","last_became_active":"2024-06-10T00:24:43.768090+0000","last_became_peered":"2024-06-10T00:24:43.768090+0000","last_unstale":"2024-06-10T00:27:57.462390+0000","last_undegraded":"2024-06-10T00:27:57.462390+0000","last_fullsized":"2024-06-10T00:27:57.462390+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:05:41.604184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591403+0000","last_change":"2024-06-10T00:24:09.170514+0000","last_active":"2024-06-10T00:27:57.591403+0000","last_peered":"2024-06-10T00:27:57.591403+0000","last_clean":"2024-06-10T00:27:57.591403+0000","last_became_active":"2024-06-10T00:24:09.168902+0000","last_became_peered":"2024-06-10T00:24:09.168902+0000","last_unstale":"2024-06-10T00:27:57.591403+0000","last_undegraded":"2024-06-10T00:27:57.591403+0000","last_fullsized":"2024-06-10T00:27:57.591403+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:33:10.534657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"101'9","reported_seq":234,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.552203+0000","last_change":"2024-06-10T00:23:54.426581+0000","last_active":"2024-06-10T00:27:57.552203+0000","last_peered":"2024-06-10T00:27:57.552203+0000","last_clean":"2024-06-10T00:27:57.552203+0000","last_became_active":"2024-06-10T00:23:54.426211+0000","last_became_peered":"2024-06-10T00:23:54.426211+0000","last_unstale":"2024-06-10T00:27:57.552203+0000","last_undegraded":"2024-06-10T00:27:57.552203+0000","last_fullsized":"2024-06-10T00:27:57.552203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:49:05.360742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":178,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569865+0000","last_change":"2024-06-10T00:23:57.552308+0000","last_active":"2024-06-10T00:27:57.569865+0000","last_peered":"2024-06-10T00:27:57.569865+0000","last_clean":"2024-06-10T00:27:57.569865+0000","last_became_active":"2024-06-10T00:23:57.551762+0000","last_became_peered":"2024-06-10T00:23:57.551762+0000","last_unstale":"2024-06-10T00:27:57.569865+0000","last_undegraded":"2024-06-10T00:27:57.569865+0000","last_fullsized":"2024-06-10T00:27:57.569865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:22:28.476019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591613+0000","last_change":"2024-06-10T00:24:09.171456+0000","last_active":"2024-06-10T00:27:57.591613+0000","last_peered":"2024-06-10T00:27:57.591613+0000","last_clean":"2024-06-10T00:27:57.591613+0000","last_became_active":"2024-06-10T00:24:09.170969+0000","last_became_peered":"2024-06-10T00:24:09.170969+0000","last_unstale":"2024-06-10T00:27:57.591613+0000","last_undegraded":"2024-06-10T00:27:57.591613+0000","last_fullsized":"2024-06-10T00:27:57.591613+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:22:46.023897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458602+0000","last_change":"2024-06-10T00:23:54.428384+0000","last_active":"2024-06-10T00:27:57.458602+0000","last_peered":"2024-06-10T00:27:57.458602+0000","last_clean":"2024-06-10T00:27:57.458602+0000","last_became_active":"2024-06-10T00:23:54.427967+0000","last_became_peered":"2024-06-10T00:23:54.427967+0000","last_unstale":"2024-06-10T00:27:57.458602+0000","last_undegraded":"2024-06-10T00:27:57.458602+0000","last_fullsized":"2024-06-10T00:27:57.458602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:54:30.408684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"100'8","reported_seq":55,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461853+0000","last_change":"2024-06-10T00:24:43.262593+0000","last_active":"2024-06-10T00:27:57.461853+0000","last_peered":"2024-06-10T00:27:57.461853+0000","last_clean":"2024-06-10T00:27:57.461853+0000","last_became_active":"2024-06-10T00:24:43.262212+0000","last_became_peered":"2024-06-10T00:24:43.262212+0000","last_unstale":"2024-06-10T00:27:57.461853+0000","last_undegraded":"2024-06-10T00:27:57.461853+0000","last_fullsized":"2024-06-10T00:27:57.461853+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:40:24.442712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591973+0000","last_change":"2024-06-10T00:24:43.752805+0000","last_active":"2024-06-10T00:27:57.591973+0000","last_peered":"2024-06-10T00:27:57.591973+0000","last_clean":"2024-06-10T00:27:57.591973+0000","last_became_active":"2024-06-10T00:24:43.752251+0000","last_became_peered":"2024-06-10T00:24:43.752251+0000","last_unstale":"2024-06-10T00:27:57.591973+0000","last_undegraded":"2024-06-10T00:27:57.591973+0000","last_fullsized":"2024-06-10T00:27:57.591973+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:27:57.617443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542566+0000","last_change":"2024-06-10T00:23:54.421493+0000","last_active":"2024-06-10T00:27:57.542566+0000","last_peered":"2024-06-10T00:27:57.542566+0000","last_clean":"2024-06-10T00:27:57.542566+0000","last_became_active":"2024-06-10T00:23:54.421151+0000","last_became_peered":"2024-06-10T00:23:54.421151+0000","last_unstale":"2024-06-10T00:27:57.542566+0000","last_undegraded":"2024-06-10T00:27:57.542566+0000","last_fullsized":"2024-06-10T00:27:57.542566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:03.995244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570099+0000","last_change":"2024-06-10T00:23:57.554020+0000","last_active":"2024-06-10T00:27:57.570099+0000","last_peered":"2024-06-10T00:27:57.570099+0000","last_clean":"2024-06-10T00:27:57.570099+0000","last_became_active":"2024-06-10T00:23:57.553690+0000","last_became_peered":"2024-06-10T00:23:57.553690+0000","last_unstale":"2024-06-10T00:27:57.570099+0000","last_undegraded":"2024-06-10T00:27:57.570099+0000","last_fullsized":"2024-06-10T00:27:57.570099+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:48.097074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593589+0000","last_change":"2024-06-10T00:24:09.166426+0000","last_active":"2024-06-10T00:27:57.593589+0000","last_peered":"2024-06-10T00:27:57.593589+0000","last_clean":"2024-06-10T00:27:57.593589+0000","last_became_active":"2024-06-10T00:24:09.166047+0000","last_became_peered":"2024-06-10T00:24:09.166047+0000","last_unstale":"2024-06-10T00:27:57.593589+0000","last_undegraded":"2024-06-10T00:27:57.593589+0000","last_fullsized":"2024-06-10T00:27:57.593589+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:56:11.608061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"98'1","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461240+0000","last_change":"2024-06-10T00:23:54.426409+0000","last_active":"2024-06-10T00:27:57.461240+0000","last_peered":"2024-06-10T00:27:57.461240+0000","last_clean":"2024-06-10T00:27:57.461240+0000","last_became_active":"2024-06-10T00:23:54.425385+0000","last_became_peered":"2024-06-10T00:23:54.425385+0000","last_unstale":"2024-06-10T00:27:57.461240+0000","last_undegraded":"2024-06-10T00:27:57.461240+0000","last_fullsized":"2024-06-10T00:27:57.461240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:27:00.379189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457389+0000","last_change":"2024-06-10T00:23:57.560132+0000","last_active":"2024-06-10T00:27:57.457389+0000","last_peered":"2024-06-10T00:27:57.457389+0000","last_clean":"2024-06-10T00:27:57.457389+0000","last_became_active":"2024-06-10T00:23:57.559803+0000","last_became_peered":"2024-06-10T00:23:57.559803+0000","last_unstale":"2024-06-10T00:27:57.457389+0000","last_undegraded":"2024-06-10T00:27:57.457389+0000","last_fullsized":"2024-06-10T00:27:57.457389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:13:16.004233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571137+0000","last_change":"2024-06-10T00:24:09.170962+0000","last_active":"2024-06-10T00:27:57.571137+0000","last_peered":"2024-06-10T00:27:57.571137+0000","last_clean":"2024-06-10T00:27:57.571137+0000","last_became_active":"2024-06-10T00:24:09.170090+0000","last_became_peered":"2024-06-10T00:24:09.170090+0000","last_unstale":"2024-06-10T00:27:57.571137+0000","last_undegraded":"2024-06-10T00:27:57.571137+0000","last_fullsized":"2024-06-10T00:27:57.571137+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:07:33.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'5","reported_seq":225,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459435+0000","last_change":"2024-06-10T00:23:54.421472+0000","last_active":"2024-06-10T00:27:57.459435+0000","last_peered":"2024-06-10T00:27:57.459435+0000","last_clean":"2024-06-10T00:27:57.459435+0000","last_became_active":"2024-06-10T00:23:54.421019+0000","last_became_peered":"2024-06-10T00:23:54.421019+0000","last_unstale":"2024-06-10T00:27:57.459435+0000","last_undegraded":"2024-06-10T00:27:57.459435+0000","last_fullsized":"2024-06-10T00:27:57.459435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:28:18.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":173,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590963+0000","last_change":"2024-06-10T00:23:57.563010+0000","last_active":"2024-06-10T00:27:57.590963+0000","last_peered":"2024-06-10T00:27:57.590963+0000","last_clean":"2024-06-10T00:27:57.590963+0000","last_became_active":"2024-06-10T00:23:57.561986+0000","last_became_peered":"2024-06-10T00:23:57.561986+0000","last_unstale":"2024-06-10T00:27:57.590963+0000","last_undegraded":"2024-06-10T00:27:57.590963+0000","last_fullsized":"2024-06-10T00:27:57.590963+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:31:17.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457415+0000","last_change":"2024-06-10T00:24:09.171746+0000","last_active":"2024-06-10T00:27:57.457415+0000","last_peered":"2024-06-10T00:27:57.457415+0000","last_clean":"2024-06-10T00:27:57.457415+0000","last_became_active":"2024-06-10T00:24:09.171030+0000","last_became_peered":"2024-06-10T00:24:09.171030+0000","last_unstale":"2024-06-10T00:27:57.457415+0000","last_undegraded":"2024-06-10T00:27:57.457415+0000","last_fullsized":"2024-06-10T00:27:57.457415+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:41.669471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591812+0000","last_change":"2024-06-10T00:23:54.428917+0000","last_active":"2024-06-10T00:27:57.591812+0000","last_peered":"2024-06-10T00:27:57.591812+0000","last_clean":"2024-06-10T00:27:57.591812+0000","last_became_active":"2024-06-10T00:23:54.428585+0000","last_became_peered":"2024-06-10T00:23:54.428585+0000","last_unstale":"2024-06-10T00:27:57.591812+0000","last_undegraded":"2024-06-10T00:27:57.591812+0000","last_fullsized":"2024-06-10T00:27:57.591812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:21:58.235848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591586+0000","last_change":"2024-06-10T00:23:57.551778+0000","last_active":"2024-06-10T00:27:57.591586+0000","last_peered":"2024-06-10T00:27:57.591586+0000","last_clean":"2024-06-10T00:27:57.591586+0000","last_became_active":"2024-06-10T00:23:57.551436+0000","last_became_peered":"2024-06-10T00:23:57.551436+0000","last_unstale":"2024-06-10T00:27:57.591586+0000","last_undegraded":"2024-06-10T00:27:57.591586+0000","last_fullsized":"2024-06-10T00:27:57.591586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:30.059783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545101+0000","last_change":"2024-06-10T00:24:09.167547+0000","last_active":"2024-06-10T00:27:57.545101+0000","last_peered":"2024-06-10T00:27:57.545101+0000","last_clean":"2024-06-10T00:27:57.545101+0000","last_became_active":"2024-06-10T00:24:09.167132+0000","last_became_peered":"2024-06-10T00:24:09.167132+0000","last_unstale":"2024-06-10T00:27:57.545101+0000","last_undegraded":"2024-06-10T00:27:57.545101+0000","last_fullsized":"2024-06-10T00:27:57.545101+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:11.520330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461450+0000","last_change":"2024-06-10T00:23:54.823754+0000","last_active":"2024-06-10T00:27:57.461450+0000","last_peered":"2024-06-10T00:27:57.461450+0000","last_clean":"2024-06-10T00:27:57.461450+0000","last_became_active":"2024-06-10T00:23:54.823362+0000","last_became_peered":"2024-06-10T00:23:54.823362+0000","last_unstale":"2024-06-10T00:27:57.461450+0000","last_undegraded":"2024-06-10T00:27:57.461450+0000","last_fullsized":"2024-06-10T00:27:57.461450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:48:50.915439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590772+0000","last_change":"2024-06-10T00:23:57.563313+0000","last_active":"2024-06-10T00:27:57.590772+0000","last_peered":"2024-06-10T00:27:57.590772+0000","last_clean":"2024-06-10T00:27:57.590772+0000","last_became_active":"2024-06-10T00:23:57.562307+0000","last_became_peered":"2024-06-10T00:23:57.562307+0000","last_unstale":"2024-06-10T00:27:57.590772+0000","last_undegraded":"2024-06-10T00:27:57.590772+0000","last_fullsized":"2024-06-10T00:27:57.590772+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:23.815267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457507+0000","last_change":"2024-06-10T00:24:09.174660+0000","last_active":"2024-06-10T00:27:57.457507+0000","last_peered":"2024-06-10T00:27:57.457507+0000","last_clean":"2024-06-10T00:27:57.457507+0000","last_became_active":"2024-06-10T00:24:09.174283+0000","last_became_peered":"2024-06-10T00:24:09.174283+0000","last_unstale":"2024-06-10T00:27:57.457507+0000","last_undegraded":"2024-06-10T00:27:57.457507+0000","last_fullsized":"2024-06-10T00:27:57.457507+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:15.236146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"101'41","reported_seq":446,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.749518+0000","last_change":"2024-06-10T00:23:54.426621+0000","last_active":"2024-06-10T00:27:57.749518+0000","last_peered":"2024-06-10T00:27:57.749518+0000","last_clean":"2024-06-10T00:27:57.749518+0000","last_became_active":"2024-06-10T00:23:54.426131+0000","last_became_peered":"2024-06-10T00:23:54.426131+0000","last_unstale":"2024-06-10T00:27:57.749518+0000","last_undegraded":"2024-06-10T00:27:57.749518+0000","last_fullsized":"2024-06-10T00:27:57.749518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:56:14.819454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69365,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":358,"num_read_kb":247,"num_write":42,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570532+0000","last_change":"2024-06-10T00:23:57.552606+0000","last_active":"2024-06-10T00:27:57.570532+0000","last_peered":"2024-06-10T00:27:57.570532+0000","last_clean":"2024-06-10T00:27:57.570532+0000","last_became_active":"2024-06-10T00:23:57.552246+0000","last_became_peered":"2024-06-10T00:23:57.552246+0000","last_unstale":"2024-06-10T00:27:57.570532+0000","last_undegraded":"2024-06-10T00:27:57.570532+0000","last_fullsized":"2024-06-10T00:27:57.570532+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:19:21.508721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462817+0000","last_change":"2024-06-10T00:24:09.174862+0000","last_active":"2024-06-10T00:27:57.462817+0000","last_peered":"2024-06-10T00:27:57.462817+0000","last_clean":"2024-06-10T00:27:57.462817+0000","last_became_active":"2024-06-10T00:24:09.173550+0000","last_became_peered":"2024-06-10T00:24:09.173550+0000","last_unstale":"2024-06-10T00:27:57.462817+0000","last_undegraded":"2024-06-10T00:27:57.462817+0000","last_fullsized":"2024-06-10T00:27:57.462817+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:49:25.549084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"98'6","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591325+0000","last_change":"2024-06-10T00:23:54.426074+0000","last_active":"2024-06-10T00:27:57.591325+0000","last_peered":"2024-06-10T00:27:57.591325+0000","last_clean":"2024-06-10T00:27:57.591325+0000","last_became_active":"2024-06-10T00:23:54.425707+0000","last_became_peered":"2024-06-10T00:23:54.425707+0000","last_unstale":"2024-06-10T00:27:57.591325+0000","last_undegraded":"2024-06-10T00:27:57.591325+0000","last_fullsized":"2024-06-10T00:27:57.591325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:11:10.563595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542128+0000","last_change":"2024-06-10T00:23:57.549784+0000","last_active":"2024-06-10T00:27:57.542128+0000","last_peered":"2024-06-10T00:27:57.542128+0000","last_clean":"2024-06-10T00:27:57.542128+0000","last_became_active":"2024-06-10T00:23:57.549457+0000","last_became_peered":"2024-06-10T00:23:57.549457+0000","last_unstale":"2024-06-10T00:27:57.542128+0000","last_undegraded":"2024-06-10T00:27:57.542128+0000","last_fullsized":"2024-06-10T00:27:57.542128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:59:27.352927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592095+0000","last_change":"2024-06-10T00:24:09.171028+0000","last_active":"2024-06-10T00:27:57.592095+0000","last_peered":"2024-06-10T00:27:57.592095+0000","last_clean":"2024-06-10T00:27:57.592095+0000","last_became_active":"2024-06-10T00:24:09.170543+0000","last_became_peered":"2024-06-10T00:24:09.170543+0000","last_unstale":"2024-06-10T00:27:57.592095+0000","last_undegraded":"2024-06-10T00:27:57.592095+0000","last_fullsized":"2024-06-10T00:27:57.592095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:57:40.816585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461583+0000","last_change":"2024-06-10T00:23:54.423332+0000","last_active":"2024-06-10T00:27:57.461583+0000","last_peered":"2024-06-10T00:27:57.461583+0000","last_clean":"2024-06-10T00:27:57.461583+0000","last_became_active":"2024-06-10T00:23:54.422792+0000","last_became_peered":"2024-06-10T00:23:54.422792+0000","last_unstale":"2024-06-10T00:27:57.461583+0000","last_undegraded":"2024-06-10T00:27:57.461583+0000","last_fullsized":"2024-06-10T00:27:57.461583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:50:52.655641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"100'15","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591219+0000","last_change":"2024-06-10T00:23:57.551411+0000","last_active":"2024-06-10T00:27:57.591219+0000","last_peered":"2024-06-10T00:27:57.591219+0000","last_clean":"2024-06-10T00:27:57.591219+0000","last_became_active":"2024-06-10T00:23:57.551053+0000","last_became_peered":"2024-06-10T00:23:57.551053+0000","last_unstale":"2024-06-10T00:27:57.591219+0000","last_undegraded":"2024-06-10T00:27:57.591219+0000","last_fullsized":"2024-06-10T00:27:57.591219+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:16:18.082777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459282+0000","last_change":"2024-06-10T00:24:43.648767+0000","last_active":"2024-06-10T00:27:57.459282+0000","last_peered":"2024-06-10T00:27:57.459282+0000","last_clean":"2024-06-10T00:27:57.459282+0000","last_became_active":"2024-06-10T00:24:43.648267+0000","last_became_peered":"2024-06-10T00:24:43.648267+0000","last_unstale":"2024-06-10T00:27:57.459282+0000","last_undegraded":"2024-06-10T00:27:57.459282+0000","last_fullsized":"2024-06-10T00:27:57.459282+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:41:09.005202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"101'5","reported_seq":226,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.752230+0000","last_change":"2024-06-10T00:23:54.425963+0000","last_active":"2024-06-10T00:27:57.752230+0000","last_peered":"2024-06-10T00:27:57.752230+0000","last_clean":"2024-06-10T00:27:57.752230+0000","last_became_active":"2024-06-10T00:23:54.425581+0000","last_became_peered":"2024-06-10T00:23:54.425581+0000","last_unstale":"2024-06-10T00:27:57.752230+0000","last_undegraded":"2024-06-10T00:27:57.752230+0000","last_fullsized":"2024-06-10T00:27:57.752230+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:54.760610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":176,"num_read_kb":3,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592340+0000","last_change":"2024-06-10T00:23:57.557464+0000","last_active":"2024-06-10T00:27:57.592340+0000","last_peered":"2024-06-10T00:27:57.592340+0000","last_clean":"2024-06-10T00:27:57.592340+0000","last_became_active":"2024-06-10T00:23:57.557130+0000","last_became_peered":"2024-06-10T00:23:57.557130+0000","last_unstale":"2024-06-10T00:27:57.592340+0000","last_undegraded":"2024-06-10T00:27:57.592340+0000","last_fullsized":"2024-06-10T00:27:57.592340+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:16:36.341210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":38,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543379+0000","last_change":"2024-06-10T00:24:43.753291+0000","last_active":"2024-06-10T00:27:57.543379+0000","last_peered":"2024-06-10T00:27:57.543379+0000","last_clean":"2024-06-10T00:27:57.543379+0000","last_became_active":"2024-06-10T00:24:43.752765+0000","last_became_peered":"2024-06-10T00:24:43.752765+0000","last_unstale":"2024-06-10T00:27:57.543379+0000","last_undegraded":"2024-06-10T00:27:57.543379+0000","last_fullsized":"2024-06-10T00:27:57.543379+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:55:06.349564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"98'15","reported_seq":63,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570040+0000","last_change":"2024-06-10T00:23:54.823489+0000","last_active":"2024-06-10T00:27:57.570040+0000","last_peered":"2024-06-10T00:27:57.570040+0000","last_clean":"2024-06-10T00:27:57.570040+0000","last_became_active":"2024-06-10T00:23:54.823073+0000","last_became_peered":"2024-06-10T00:23:54.823073+0000","last_unstale":"2024-06-10T00:27:57.570040+0000","last_undegraded":"2024-06-10T00:27:57.570040+0000","last_fullsized":"2024-06-10T00:27:57.570040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:07:52.563009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461389+0000","last_change":"2024-06-10T00:23:57.563388+0000","last_active":"2024-06-10T00:27:57.461389+0000","last_peered":"2024-06-10T00:27:57.461389+0000","last_clean":"2024-06-10T00:27:57.461389+0000","last_became_active":"2024-06-10T00:23:57.563004+0000","last_became_peered":"2024-06-10T00:23:57.563004+0000","last_unstale":"2024-06-10T00:27:57.461389+0000","last_undegraded":"2024-06-10T00:27:57.461389+0000","last_fullsized":"2024-06-10T00:27:57.461389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:51:18.805662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461226+0000","last_change":"2024-06-10T00:24:09.168447+0000","last_active":"2024-06-10T00:27:57.461226+0000","last_peered":"2024-06-10T00:27:57.461226+0000","last_clean":"2024-06-10T00:27:57.461226+0000","last_became_active":"2024-06-10T00:24:09.167439+0000","last_became_peered":"2024-06-10T00:24:09.167439+0000","last_unstale":"2024-06-10T00:27:57.461226+0000","last_undegraded":"2024-06-10T00:27:57.461226+0000","last_fullsized":"2024-06-10T00:27:57.461226+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:52.319395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"101'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570752+0000","last_change":"2024-06-10T00:23:54.427666+0000","last_active":"2024-06-10T00:27:57.570752+0000","last_peered":"2024-06-10T00:27:57.570752+0000","last_clean":"2024-06-10T00:27:57.570752+0000","last_became_active":"2024-06-10T00:23:54.427285+0000","last_became_peered":"2024-06-10T00:23:54.427285+0000","last_unstale":"2024-06-10T00:27:57.570752+0000","last_undegraded":"2024-06-10T00:27:57.570752+0000","last_fullsized":"2024-06-10T00:27:57.570752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:28:20.196168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"100'5","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.558787+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.558470+0000","last_became_peered":"2024-06-10T00:23:57.558470+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:02:14.758802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570142+0000","last_change":"2024-06-10T00:24:09.264952+0000","last_active":"2024-06-10T00:27:57.570142+0000","last_peered":"2024-06-10T00:27:57.570142+0000","last_clean":"2024-06-10T00:27:57.570142+0000","last_became_active":"2024-06-10T00:24:09.264379+0000","last_became_peered":"2024-06-10T00:24:09.264379+0000","last_unstale":"2024-06-10T00:27:57.570142+0000","last_undegraded":"2024-06-10T00:27:57.570142+0000","last_fullsized":"2024-06-10T00:27:57.570142+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:25:40.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458121+0000","last_change":"2024-06-10T00:23:54.823298+0000","last_active":"2024-06-10T00:27:57.458121+0000","last_peered":"2024-06-10T00:27:57.458121+0000","last_clean":"2024-06-10T00:27:57.458121+0000","last_became_active":"2024-06-10T00:23:54.822917+0000","last_became_peered":"2024-06-10T00:23:54.822917+0000","last_unstale":"2024-06-10T00:27:57.458121+0000","last_undegraded":"2024-06-10T00:27:57.458121+0000","last_fullsized":"2024-06-10T00:27:57.458121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:07:52.500641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459177+0000","last_change":"2024-06-10T00:23:57.559202+0000","last_active":"2024-06-10T00:27:57.459177+0000","last_peered":"2024-06-10T00:27:57.459177+0000","last_clean":"2024-06-10T00:27:57.459177+0000","last_became_active":"2024-06-10T00:23:57.558800+0000","last_became_peered":"2024-06-10T00:23:57.558800+0000","last_unstale":"2024-06-10T00:27:57.459177+0000","last_undegraded":"2024-06-10T00:27:57.459177+0000","last_fullsized":"2024-06-10T00:27:57.459177+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:49:51.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542486+0000","last_change":"2024-06-10T00:24:09.168603+0000","last_active":"2024-06-10T00:27:57.542486+0000","last_peered":"2024-06-10T00:27:57.542486+0000","last_clean":"2024-06-10T00:27:57.542486+0000","last_became_active":"2024-06-10T00:24:09.168233+0000","last_became_peered":"2024-06-10T00:24:09.168233+0000","last_unstale":"2024-06-10T00:27:57.542486+0000","last_undegraded":"2024-06-10T00:27:57.542486+0000","last_fullsized":"2024-06-10T00:27:57.542486+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:59.585023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'1","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592210+0000","last_change":"2024-06-10T00:23:54.425505+0000","last_active":"2024-06-10T00:27:57.592210+0000","last_peered":"2024-06-10T00:27:57.592210+0000","last_clean":"2024-06-10T00:27:57.592210+0000","last_became_active":"2024-06-10T00:23:54.425058+0000","last_became_peered":"2024-06-10T00:23:54.425058+0000","last_unstale":"2024-06-10T00:27:57.592210+0000","last_undegraded":"2024-06-10T00:27:57.592210+0000","last_fullsized":"2024-06-10T00:27:57.592210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:48:05.605947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458827+0000","last_change":"2024-06-10T00:23:57.566060+0000","last_active":"2024-06-10T00:27:57.458827+0000","last_peered":"2024-06-10T00:27:57.458827+0000","last_clean":"2024-06-10T00:27:57.458827+0000","last_became_active":"2024-06-10T00:23:57.565099+0000","last_became_peered":"2024-06-10T00:23:57.565099+0000","last_unstale":"2024-06-10T00:27:57.458827+0000","last_undegraded":"2024-06-10T00:27:57.458827+0000","last_fullsized":"2024-06-10T00:27:57.458827+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:24:54.204399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543284+0000","last_change":"2024-06-10T00:24:09.264741+0000","last_active":"2024-06-10T00:27:57.543284+0000","last_peered":"2024-06-10T00:27:57.543284+0000","last_clean":"2024-06-10T00:27:57.543284+0000","last_became_active":"2024-06-10T00:24:09.264267+0000","last_became_peered":"2024-06-10T00:24:09.264267+0000","last_unstale":"2024-06-10T00:27:57.543284+0000","last_undegraded":"2024-06-10T00:27:57.543284+0000","last_fullsized":"2024-06-10T00:27:57.543284+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:27.627391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457224+0000","last_change":"2024-06-10T00:25:44.346070+0000","last_active":"2024-06-10T00:27:57.457224+0000","last_peered":"2024-06-10T00:27:57.457224+0000","last_clean":"2024-06-10T00:27:57.457224+0000","last_became_active":"2024-06-10T00:25:44.345798+0000","last_became_peered":"2024-06-10T00:25:44.345798+0000","last_unstale":"2024-06-10T00:27:57.457224+0000","last_undegraded":"2024-06-10T00:27:57.457224+0000","last_fullsized":"2024-06-10T00:27:57.457224+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:04.506535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"101'5","reported_seq":225,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542310+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:27:57.542310+0000","last_peered":"2024-06-10T00:27:57.542310+0000","last_clean":"2024-06-10T00:27:57.542310+0000","last_became_active":"2024-06-10T00:23:54.822442+0000","last_became_peered":"2024-06-10T00:23:54.822442+0000","last_unstale":"2024-06-10T00:27:57.542310+0000","last_undegraded":"2024-06-10T00:27:57.542310+0000","last_fullsized":"2024-06-10T00:27:57.542310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:51:10.278257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":173,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"98'1","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591953+0000","last_change":"2024-06-10T00:24:09.168073+0000","last_active":"2024-06-10T00:27:57.591953+0000","last_peered":"2024-06-10T00:27:57.591953+0000","last_clean":"2024-06-10T00:27:57.591953+0000","last_became_active":"2024-06-10T00:24:09.167657+0000","last_became_peered":"2024-06-10T00:24:09.167657+0000","last_unstale":"2024-06-10T00:27:57.591953+0000","last_undegraded":"2024-06-10T00:27:57.591953+0000","last_fullsized":"2024-06-10T00:27:57.591953+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:20:23.481685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457629+0000","last_change":"2024-06-10T00:25:44.346629+0000","last_active":"2024-06-10T00:27:57.457629+0000","last_peered":"2024-06-10T00:27:57.457629+0000","last_clean":"2024-06-10T00:27:57.457629+0000","last_became_active":"2024-06-10T00:25:44.346388+0000","last_became_peered":"2024-06-10T00:25:44.346388+0000","last_unstale":"2024-06-10T00:27:57.457629+0000","last_undegraded":"2024-06-10T00:27:57.457629+0000","last_fullsized":"2024-06-10T00:27:57.457629+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:03:19.228632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544918+0000","last_change":"2024-06-10T00:23:54.423507+0000","last_active":"2024-06-10T00:27:57.544918+0000","last_peered":"2024-06-10T00:27:57.544918+0000","last_clean":"2024-06-10T00:27:57.544918+0000","last_became_active":"2024-06-10T00:23:54.423092+0000","last_became_peered":"2024-06-10T00:23:54.423092+0000","last_unstale":"2024-06-10T00:27:57.544918+0000","last_undegraded":"2024-06-10T00:27:57.544918+0000","last_fullsized":"2024-06-10T00:27:57.544918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:06:15.924827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":380,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592578+0000","last_change":"2024-06-10T00:22:28.025243+0000","last_active":"2024-06-10T00:27:57.592578+0000","last_peered":"2024-06-10T00:27:57.592578+0000","last_clean":"2024-06-10T00:27:57.592578+0000","last_became_active":"2024-06-10T00:22:28.024751+0000","last_became_peered":"2024-06-10T00:22:28.024751+0000","last_unstale":"2024-06-10T00:27:57.592578+0000","last_undegraded":"2024-06-10T00:27:57.592578+0000","last_fullsized":"2024-06-10T00:27:57.592578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_clean_scrub_stamp":"2024-06-10T00:19:05.039631+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:02:36.557931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461952+0000","last_change":"2024-06-10T00:24:09.265084+0000","last_active":"2024-06-10T00:27:57.461952+0000","last_peered":"2024-06-10T00:27:57.461952+0000","last_clean":"2024-06-10T00:27:57.461952+0000","last_became_active":"2024-06-10T00:24:09.264425+0000","last_became_peered":"2024-06-10T00:24:09.264425+0000","last_unstale":"2024-06-10T00:27:57.461952+0000","last_undegraded":"2024-06-10T00:27:57.461952+0000","last_fullsized":"2024-06-10T00:27:57.461952+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:37:31.577688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.541918+0000","last_change":"2024-06-10T00:23:57.558540+0000","last_active":"2024-06-10T00:27:57.541918+0000","last_peered":"2024-06-10T00:27:57.541918+0000","last_clean":"2024-06-10T00:27:57.541918+0000","last_became_active":"2024-06-10T00:23:57.558197+0000","last_became_peered":"2024-06-10T00:23:57.558197+0000","last_unstale":"2024-06-10T00:27:57.541918+0000","last_undegraded":"2024-06-10T00:27:57.541918+0000","last_fullsized":"2024-06-10T00:27:57.541918+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:04:25.566135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592707+0000","last_change":"2024-06-10T00:23:54.428126+0000","last_active":"2024-06-10T00:27:57.592707+0000","last_peered":"2024-06-10T00:27:57.592707+0000","last_clean":"2024-06-10T00:27:57.592707+0000","last_became_active":"2024-06-10T00:23:54.427630+0000","last_became_peered":"2024-06-10T00:23:54.427630+0000","last_unstale":"2024-06-10T00:27:57.592707+0000","last_undegraded":"2024-06-10T00:27:57.592707+0000","last_fullsized":"2024-06-10T00:27:57.592707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:05:23.679704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591957+0000","last_change":"2024-06-10T00:24:09.169984+0000","last_active":"2024-06-10T00:27:57.591957+0000","last_peered":"2024-06-10T00:27:57.591957+0000","last_clean":"2024-06-10T00:27:57.591957+0000","last_became_active":"2024-06-10T00:24:09.169556+0000","last_became_peered":"2024-06-10T00:24:09.169556+0000","last_unstale":"2024-06-10T00:27:57.591957+0000","last_undegraded":"2024-06-10T00:27:57.591957+0000","last_fullsized":"2024-06-10T00:27:57.591957+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:54:13.575207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"98'1","reported_seq":20,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591972+0000","last_change":"2024-06-10T00:24:43.767901+0000","last_active":"2024-06-10T00:27:57.591972+0000","last_peered":"2024-06-10T00:27:57.591972+0000","last_clean":"2024-06-10T00:27:57.591972+0000","last_became_active":"2024-06-10T00:24:43.767562+0000","last_became_peered":"2024-06-10T00:24:43.767562+0000","last_unstale":"2024-06-10T00:27:57.591972+0000","last_undegraded":"2024-06-10T00:27:57.591972+0000","last_fullsized":"2024-06-10T00:27:57.591972+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:46:36.099750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543049+0000","last_change":"2024-06-10T00:23:57.554906+0000","last_active":"2024-06-10T00:27:57.543049+0000","last_peered":"2024-06-10T00:27:57.543049+0000","last_clean":"2024-06-10T00:27:57.543049+0000","last_became_active":"2024-06-10T00:23:57.554608+0000","last_became_peered":"2024-06-10T00:23:57.554608+0000","last_unstale":"2024-06-10T00:27:57.543049+0000","last_undegraded":"2024-06-10T00:27:57.543049+0000","last_fullsized":"2024-06-10T00:27:57.543049+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:24.690594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570625+0000","last_change":"2024-06-10T00:24:09.264676+0000","last_active":"2024-06-10T00:27:57.570625+0000","last_peered":"2024-06-10T00:27:57.570625+0000","last_clean":"2024-06-10T00:27:57.570625+0000","last_became_active":"2024-06-10T00:24:09.264259+0000","last_became_peered":"2024-06-10T00:24:09.264259+0000","last_unstale":"2024-06-10T00:27:57.570625+0000","last_undegraded":"2024-06-10T00:27:57.570625+0000","last_fullsized":"2024-06-10T00:27:57.570625+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:37:53.787039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"101'48","reported_seq":118,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.689379+0000","last_change":"2024-06-10T00:23:54.424201+0000","last_active":"2024-06-10T00:27:57.689379+0000","last_peered":"2024-06-10T00:27:57.689379+0000","last_clean":"2024-06-10T00:27:57.689379+0000","last_became_active":"2024-06-10T00:23:54.423853+0000","last_became_peered":"2024-06-10T00:23:54.423853+0000","last_unstale":"2024-06-10T00:27:57.689379+0000","last_undegraded":"2024-06-10T00:27:57.689379+0000","last_fullsized":"2024-06-10T00:27:57.689379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:00.884041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":86,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571556+0000","last_change":"2024-06-10T00:23:57.554276+0000","last_active":"2024-06-10T00:27:57.571556+0000","last_peered":"2024-06-10T00:27:57.571556+0000","last_clean":"2024-06-10T00:27:57.571556+0000","last_became_active":"2024-06-10T00:23:57.553964+0000","last_became_peered":"2024-06-10T00:23:57.553964+0000","last_unstale":"2024-06-10T00:27:57.571556+0000","last_undegraded":"2024-06-10T00:27:57.571556+0000","last_fullsized":"2024-06-10T00:27:57.571556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:39:51.974770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591089+0000","last_change":"2024-06-10T00:24:09.170249+0000","last_active":"2024-06-10T00:27:57.591089+0000","last_peered":"2024-06-10T00:27:57.591089+0000","last_clean":"2024-06-10T00:27:57.591089+0000","last_became_active":"2024-06-10T00:24:09.169830+0000","last_became_peered":"2024-06-10T00:24:09.169830+0000","last_unstale":"2024-06-10T00:27:57.591089+0000","last_undegraded":"2024-06-10T00:27:57.591089+0000","last_fullsized":"2024-06-10T00:27:57.591089+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.320609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"98'4","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545304+0000","last_change":"2024-06-10T00:23:54.473943+0000","last_active":"2024-06-10T00:27:57.545304+0000","last_peered":"2024-06-10T00:27:57.545304+0000","last_clean":"2024-06-10T00:27:57.545304+0000","last_became_active":"2024-06-10T00:23:54.473652+0000","last_became_peered":"2024-06-10T00:23:54.473652+0000","last_unstale":"2024-06-10T00:27:57.545304+0000","last_undegraded":"2024-06-10T00:27:57.545304+0000","last_fullsized":"2024-06-10T00:27:57.545304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:52:06.773684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"100'7","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593713+0000","last_change":"2024-06-10T00:25:44.735050+0000","last_active":"2024-06-10T00:27:57.593713+0000","last_peered":"2024-06-10T00:27:57.593713+0000","last_clean":"2024-06-10T00:27:57.593713+0000","last_became_active":"2024-06-10T00:25:44.734756+0000","last_became_peered":"2024-06-10T00:25:44.734756+0000","last_unstale":"2024-06-10T00:27:57.593713+0000","last_undegraded":"2024-06-10T00:27:57.593713+0000","last_fullsized":"2024-06-10T00:27:57.593713+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:16:03.924238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545083+0000","last_change":"2024-06-10T00:24:43.768837+0000","last_active":"2024-06-10T00:27:57.545083+0000","last_peered":"2024-06-10T00:27:57.545083+0000","last_clean":"2024-06-10T00:27:57.545083+0000","last_became_active":"2024-06-10T00:24:43.768295+0000","last_became_peered":"2024-06-10T00:24:43.768295+0000","last_unstale":"2024-06-10T00:27:57.545083+0000","last_undegraded":"2024-06-10T00:27:57.545083+0000","last_fullsized":"2024-06-10T00:27:57.545083+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:59.600864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591709+0000","last_change":"2024-06-10T00:23:54.823448+0000","last_active":"2024-06-10T00:27:57.591709+0000","last_peered":"2024-06-10T00:27:57.591709+0000","last_clean":"2024-06-10T00:27:57.591709+0000","last_became_active":"2024-06-10T00:23:54.823107+0000","last_became_peered":"2024-06-10T00:23:54.823107+0000","last_unstale":"2024-06-10T00:27:57.591709+0000","last_undegraded":"2024-06-10T00:27:57.591709+0000","last_fullsized":"2024-06-10T00:27:57.591709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:50.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"100'2","reported_seq":43,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569862+0000","last_change":"2024-06-10T00:23:57.556733+0000","last_active":"2024-06-10T00:27:57.569862+0000","last_peered":"2024-06-10T00:27:57.569862+0000","last_clean":"2024-06-10T00:27:57.569862+0000","last_became_active":"2024-06-10T00:23:57.556386+0000","last_became_peered":"2024-06-10T00:23:57.556386+0000","last_unstale":"2024-06-10T00:27:57.569862+0000","last_undegraded":"2024-06-10T00:27:57.569862+0000","last_fullsized":"2024-06-10T00:27:57.569862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:39:00.572311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458015+0000","last_change":"2024-06-10T00:24:09.174600+0000","last_active":"2024-06-10T00:27:57.458015+0000","last_peered":"2024-06-10T00:27:57.458015+0000","last_clean":"2024-06-10T00:27:57.458015+0000","last_became_active":"2024-06-10T00:24:09.174236+0000","last_became_peered":"2024-06-10T00:24:09.174236+0000","last_unstale":"2024-06-10T00:27:57.458015+0000","last_undegraded":"2024-06-10T00:27:57.458015+0000","last_fullsized":"2024-06-10T00:27:57.458015+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:58:18.447768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571190+0000","last_change":"2024-06-10T00:23:54.425111+0000","last_active":"2024-06-10T00:27:57.571190+0000","last_peered":"2024-06-10T00:27:57.571190+0000","last_clean":"2024-06-10T00:27:57.571190+0000","last_became_active":"2024-06-10T00:23:54.424724+0000","last_became_peered":"2024-06-10T00:23:54.424724+0000","last_unstale":"2024-06-10T00:27:57.571190+0000","last_undegraded":"2024-06-10T00:27:57.571190+0000","last_fullsized":"2024-06-10T00:27:57.571190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:54.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"100'9","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.556355+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.555479+0000","last_became_peered":"2024-06-10T00:23:57.555479+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:08.817381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"98'1","reported_seq":30,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544968+0000","last_change":"2024-06-10T00:24:09.169589+0000","last_active":"2024-06-10T00:27:57.544968+0000","last_peered":"2024-06-10T00:27:57.544968+0000","last_clean":"2024-06-10T00:27:57.544968+0000","last_became_active":"2024-06-10T00:24:09.168436+0000","last_became_peered":"2024-06-10T00:24:09.168436+0000","last_unstale":"2024-06-10T00:27:57.544968+0000","last_undegraded":"2024-06-10T00:27:57.544968+0000","last_fullsized":"2024-06-10T00:27:57.544968+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:17:32.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"98'1","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591500+0000","last_change":"2024-06-10T00:24:43.751399+0000","last_active":"2024-06-10T00:27:57.591500+0000","last_peered":"2024-06-10T00:27:57.591500+0000","last_clean":"2024-06-10T00:27:57.591500+0000","last_became_active":"2024-06-10T00:24:43.750987+0000","last_became_peered":"2024-06-10T00:24:43.750987+0000","last_unstale":"2024-06-10T00:27:57.591500+0000","last_undegraded":"2024-06-10T00:27:57.591500+0000","last_fullsized":"2024-06-10T00:27:57.591500+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:14:47.902017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570338+0000","last_change":"2024-06-10T00:23:57.560213+0000","last_active":"2024-06-10T00:27:57.570338+0000","last_peered":"2024-06-10T00:27:57.570338+0000","last_clean":"2024-06-10T00:27:57.570338+0000","last_became_active":"2024-06-10T00:23:57.559925+0000","last_became_peered":"2024-06-10T00:23:57.559925+0000","last_unstale":"2024-06-10T00:27:57.570338+0000","last_undegraded":"2024-06-10T00:27:57.570338+0000","last_fullsized":"2024-06-10T00:27:57.570338+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:00:37.403087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545095+0000","last_change":"2024-06-10T00:24:09.264758+0000","last_active":"2024-06-10T00:27:57.545095+0000","last_peered":"2024-06-10T00:27:57.545095+0000","last_clean":"2024-06-10T00:27:57.545095+0000","last_became_active":"2024-06-10T00:24:09.263928+0000","last_became_peered":"2024-06-10T00:24:09.263928+0000","last_unstale":"2024-06-10T00:27:57.545095+0000","last_undegraded":"2024-06-10T00:27:57.545095+0000","last_fullsized":"2024-06-10T00:27:57.545095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:23:41.083091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570103+0000","last_change":"2024-06-10T00:23:54.428553+0000","last_active":"2024-06-10T00:27:57.570103+0000","last_peered":"2024-06-10T00:27:57.570103+0000","last_clean":"2024-06-10T00:27:57.570103+0000","last_became_active":"2024-06-10T00:23:54.428108+0000","last_became_peered":"2024-06-10T00:23:54.428108+0000","last_unstale":"2024-06-10T00:27:57.570103+0000","last_undegraded":"2024-06-10T00:27:57.570103+0000","last_fullsized":"2024-06-10T00:27:57.570103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:57:26.207915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591119+0000","last_change":"2024-06-10T00:23:57.560686+0000","last_active":"2024-06-10T00:27:57.591119+0000","last_peered":"2024-06-10T00:27:57.591119+0000","last_clean":"2024-06-10T00:27:57.591119+0000","last_became_active":"2024-06-10T00:23:57.560326+0000","last_became_peered":"2024-06-10T00:23:57.560326+0000","last_unstale":"2024-06-10T00:27:57.591119+0000","last_undegraded":"2024-06-10T00:27:57.591119+0000","last_fullsized":"2024-06-10T00:27:57.591119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:08:39.506567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591281+0000","last_change":"2024-06-10T00:24:09.166947+0000","last_active":"2024-06-10T00:27:57.591281+0000","last_peered":"2024-06-10T00:27:57.591281+0000","last_clean":"2024-06-10T00:27:57.591281+0000","last_became_active":"2024-06-10T00:24:09.166503+0000","last_became_peered":"2024-06-10T00:24:09.166503+0000","last_unstale":"2024-06-10T00:27:57.591281+0000","last_undegraded":"2024-06-10T00:27:57.591281+0000","last_fullsized":"2024-06-10T00:27:57.591281+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:16:45.094642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591369+0000","last_change":"2024-06-10T00:23:54.823522+0000","last_active":"2024-06-10T00:27:57.591369+0000","last_peered":"2024-06-10T00:27:57.591369+0000","last_clean":"2024-06-10T00:27:57.591369+0000","last_became_active":"2024-06-10T00:23:54.823142+0000","last_became_peered":"2024-06-10T00:23:54.823142+0000","last_unstale":"2024-06-10T00:27:57.591369+0000","last_undegraded":"2024-06-10T00:27:57.591369+0000","last_fullsized":"2024-06-10T00:27:57.591369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:03.836850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592288+0000","last_change":"2024-06-10T00:23:57.563710+0000","last_active":"2024-06-10T00:27:57.592288+0000","last_peered":"2024-06-10T00:27:57.592288+0000","last_clean":"2024-06-10T00:27:57.592288+0000","last_became_active":"2024-06-10T00:23:57.563331+0000","last_became_peered":"2024-06-10T00:23:57.563331+0000","last_unstale":"2024-06-10T00:27:57.592288+0000","last_undegraded":"2024-06-10T00:27:57.592288+0000","last_fullsized":"2024-06-10T00:27:57.592288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:08:48.061885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592820+0000","last_change":"2024-06-10T00:24:09.171203+0000","last_active":"2024-06-10T00:27:57.592820+0000","last_peered":"2024-06-10T00:27:57.592820+0000","last_clean":"2024-06-10T00:27:57.592820+0000","last_became_active":"2024-06-10T00:24:09.170824+0000","last_became_peered":"2024-06-10T00:24:09.170824+0000","last_unstale":"2024-06-10T00:27:57.592820+0000","last_undegraded":"2024-06-10T00:27:57.592820+0000","last_fullsized":"2024-06-10T00:27:57.592820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:47:28.045602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"101'39","reported_seq":94,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.686846+0000","last_change":"2024-06-10T00:23:54.421307+0000","last_active":"2024-06-10T00:27:57.686846+0000","last_peered":"2024-06-10T00:27:57.686846+0000","last_clean":"2024-06-10T00:27:57.686846+0000","last_became_active":"2024-06-10T00:23:54.420928+0000","last_became_peered":"2024-06-10T00:23:54.420928+0000","last_unstale":"2024-06-10T00:27:57.686846+0000","last_undegraded":"2024-06-10T00:27:57.686846+0000","last_fullsized":"2024-06-10T00:27:57.686846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:59:41.433605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591623+0000","last_change":"2024-06-10T00:23:57.556494+0000","last_active":"2024-06-10T00:27:57.591623+0000","last_peered":"2024-06-10T00:27:57.591623+0000","last_clean":"2024-06-10T00:27:57.591623+0000","last_became_active":"2024-06-10T00:23:57.556088+0000","last_became_peered":"2024-06-10T00:23:57.556088+0000","last_unstale":"2024-06-10T00:27:57.591623+0000","last_undegraded":"2024-06-10T00:27:57.591623+0000","last_fullsized":"2024-06-10T00:27:57.591623+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:06:23.873281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461068+0000","last_change":"2024-06-10T00:24:09.166734+0000","last_active":"2024-06-10T00:27:57.461068+0000","last_peered":"2024-06-10T00:27:57.461068+0000","last_clean":"2024-06-10T00:27:57.461068+0000","last_became_active":"2024-06-10T00:24:09.166270+0000","last_became_peered":"2024-06-10T00:24:09.166270+0000","last_unstale":"2024-06-10T00:27:57.461068+0000","last_undegraded":"2024-06-10T00:27:57.461068+0000","last_fullsized":"2024-06-10T00:27:57.461068+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:38.039633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543256+0000","last_change":"2024-06-10T00:23:54.426226+0000","last_active":"2024-06-10T00:27:57.543256+0000","last_peered":"2024-06-10T00:27:57.543256+0000","last_clean":"2024-06-10T00:27:57.543256+0000","last_became_active":"2024-06-10T00:23:54.425879+0000","last_became_peered":"2024-06-10T00:23:54.425879+0000","last_unstale":"2024-06-10T00:27:57.543256+0000","last_undegraded":"2024-06-10T00:27:57.543256+0000","last_fullsized":"2024-06-10T00:27:57.543256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:15:32.936556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592043+0000","last_change":"2024-06-10T00:23:57.553814+0000","last_active":"2024-06-10T00:27:57.592043+0000","last_peered":"2024-06-10T00:27:57.592043+0000","last_clean":"2024-06-10T00:27:57.592043+0000","last_became_active":"2024-06-10T00:23:57.553478+0000","last_became_peered":"2024-06-10T00:23:57.553478+0000","last_unstale":"2024-06-10T00:27:57.592043+0000","last_undegraded":"2024-06-10T00:27:57.592043+0000","last_fullsized":"2024-06-10T00:27:57.592043+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:54:39.264526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"100'9","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591390+0000","last_change":"2024-06-10T00:23:57.556676+0000","last_active":"2024-06-10T00:27:57.591390+0000","last_peered":"2024-06-10T00:27:57.591390+0000","last_clean":"2024-06-10T00:27:57.591390+0000","last_became_active":"2024-06-10T00:23:57.555946+0000","last_became_peered":"2024-06-10T00:23:57.555946+0000","last_unstale":"2024-06-10T00:27:57.591390+0000","last_undegraded":"2024-06-10T00:27:57.591390+0000","last_fullsized":"2024-06-10T00:27:57.591390+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:10:10.868799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"100'12","reported_seq":53,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590743+0000","last_change":"2024-06-10T00:23:57.565423+0000","last_active":"2024-06-10T00:27:57.590743+0000","last_peered":"2024-06-10T00:27:57.590743+0000","last_clean":"2024-06-10T00:27:57.590743+0000","last_became_active":"2024-06-10T00:23:57.564830+0000","last_became_peered":"2024-06-10T00:23:57.564830+0000","last_unstale":"2024-06-10T00:27:57.590743+0000","last_undegraded":"2024-06-10T00:27:57.590743+0000","last_fullsized":"2024-06-10T00:27:57.590743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:11:06.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590982+0000","last_change":"2024-06-10T00:23:57.558971+0000","last_active":"2024-06-10T00:27:57.590982+0000","last_peered":"2024-06-10T00:27:57.590982+0000","last_clean":"2024-06-10T00:27:57.590982+0000","last_became_active":"2024-06-10T00:23:57.558620+0000","last_became_peered":"2024-06-10T00:23:57.558620+0000","last_unstale":"2024-06-10T00:27:57.590982+0000","last_undegraded":"2024-06-10T00:27:57.590982+0000","last_fullsized":"2024-06-10T00:27:57.590982+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:45:43.308008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591491+0000","last_change":"2024-06-10T00:23:57.552341+0000","last_active":"2024-06-10T00:27:57.591491+0000","last_peered":"2024-06-10T00:27:57.591491+0000","last_clean":"2024-06-10T00:27:57.591491+0000","last_became_active":"2024-06-10T00:23:57.551464+0000","last_became_peered":"2024-06-10T00:23:57.551464+0000","last_unstale":"2024-06-10T00:27:57.591491+0000","last_undegraded":"2024-06-10T00:27:57.591491+0000","last_fullsized":"2024-06-10T00:27:57.591491+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:44:41.063516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"100'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543209+0000","last_change":"2024-06-10T00:24:43.768961+0000","last_active":"2024-06-10T00:27:57.543209+0000","last_peered":"2024-06-10T00:27:57.543209+0000","last_clean":"2024-06-10T00:27:57.543209+0000","last_became_active":"2024-06-10T00:24:43.768542+0000","last_became_peered":"2024-06-10T00:24:43.768542+0000","last_unstale":"2024-06-10T00:27:57.543209+0000","last_undegraded":"2024-06-10T00:27:57.543209+0000","last_fullsized":"2024-06-10T00:27:57.543209+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:23:58.412054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"100'6","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591613+0000","last_change":"2024-06-10T00:23:57.554924+0000","last_active":"2024-06-10T00:27:57.591613+0000","last_peered":"2024-06-10T00:27:57.591613+0000","last_clean":"2024-06-10T00:27:57.591613+0000","last_became_active":"2024-06-10T00:23:57.554515+0000","last_became_peered":"2024-06-10T00:23:57.554515+0000","last_unstale":"2024-06-10T00:27:57.591613+0000","last_undegraded":"2024-06-10T00:27:57.591613+0000","last_fullsized":"2024-06-10T00:27:57.591613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:58:15.624775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"100'15","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591485+0000","last_change":"2024-06-10T00:23:57.561940+0000","last_active":"2024-06-10T00:27:57.591485+0000","last_peered":"2024-06-10T00:27:57.591485+0000","last_clean":"2024-06-10T00:27:57.591485+0000","last_became_active":"2024-06-10T00:23:57.561512+0000","last_became_peered":"2024-06-10T00:23:57.561512+0000","last_unstale":"2024-06-10T00:27:57.591485+0000","last_undegraded":"2024-06-10T00:27:57.591485+0000","last_fullsized":"2024-06-10T00:27:57.591485+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:57:57.147311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"100'2","reported_seq":43,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570333+0000","last_change":"2024-06-10T00:23:57.557229+0000","last_active":"2024-06-10T00:27:57.570333+0000","last_peered":"2024-06-10T00:27:57.570333+0000","last_clean":"2024-06-10T00:27:57.570333+0000","last_became_active":"2024-06-10T00:23:57.556905+0000","last_became_peered":"2024-06-10T00:23:57.556905+0000","last_unstale":"2024-06-10T00:27:57.570333+0000","last_undegraded":"2024-06-10T00:27:57.570333+0000","last_fullsized":"2024-06-10T00:27:57.570333+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:30.513517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"100'6","reported_seq":24,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592589+0000","last_change":"2024-06-10T00:24:43.768509+0000","last_active":"2024-06-10T00:27:57.592589+0000","last_peered":"2024-06-10T00:27:57.592589+0000","last_clean":"2024-06-10T00:27:57.592589+0000","last_became_active":"2024-06-10T00:24:43.768038+0000","last_became_peered":"2024-06-10T00:24:43.768038+0000","last_unstale":"2024-06-10T00:27:57.592589+0000","last_undegraded":"2024-06-10T00:27:57.592589+0000","last_fullsized":"2024-06-10T00:27:57.592589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:38.344375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571244+0000","last_change":"2024-06-10T00:23:57.557830+0000","last_active":"2024-06-10T00:27:57.571244+0000","last_peered":"2024-06-10T00:27:57.571244+0000","last_clean":"2024-06-10T00:27:57.571244+0000","last_became_active":"2024-06-10T00:23:57.556986+0000","last_became_peered":"2024-06-10T00:23:57.556986+0000","last_unstale":"2024-06-10T00:27:57.571244+0000","last_undegraded":"2024-06-10T00:27:57.571244+0000","last_fullsized":"2024-06-10T00:27:57.571244+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:53.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590760+0000","last_change":"2024-06-10T00:23:57.565602+0000","last_active":"2024-06-10T00:27:57.590760+0000","last_peered":"2024-06-10T00:27:57.590760+0000","last_clean":"2024-06-10T00:27:57.590760+0000","last_became_active":"2024-06-10T00:23:57.565264+0000","last_became_peered":"2024-06-10T00:23:57.565264+0000","last_unstale":"2024-06-10T00:27:57.590760+0000","last_undegraded":"2024-06-10T00:27:57.590760+0000","last_fullsized":"2024-06-10T00:27:57.590760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:09:12.773087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542211+0000","last_change":"2024-06-10T00:23:57.548516+0000","last_active":"2024-06-10T00:27:57.542211+0000","last_peered":"2024-06-10T00:27:57.542211+0000","last_clean":"2024-06-10T00:27:57.542211+0000","last_became_active":"2024-06-10T00:23:57.548202+0000","last_became_peered":"2024-06-10T00:23:57.548202+0000","last_unstale":"2024-06-10T00:27:57.542211+0000","last_undegraded":"2024-06-10T00:27:57.542211+0000","last_fullsized":"2024-06-10T00:27:57.542211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:13.901478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591566+0000","last_change":"2024-06-10T00:23:57.565900+0000","last_active":"2024-06-10T00:27:57.591566+0000","last_peered":"2024-06-10T00:27:57.591566+0000","last_clean":"2024-06-10T00:27:57.591566+0000","last_became_active":"2024-06-10T00:23:57.565573+0000","last_became_peered":"2024-06-10T00:23:57.565573+0000","last_unstale":"2024-06-10T00:27:57.591566+0000","last_undegraded":"2024-06-10T00:27:57.591566+0000","last_fullsized":"2024-06-10T00:27:57.591566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:08.255905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107100,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599800,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1163,"num_read_kb":693,"num_write":299,"num_write_kb":1447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":1799400,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351040,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1118208,"data_stored":582820,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612677,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32432,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712720,"statfs":{"total":95995035648,"available":95961825280,"internally_reserved":0,"allocated":1286144,"data_stored":1371811,"data_compressed":77135,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776203,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1159168,"data_stored":980346,"data_compressed":54425,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037839,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28144,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717008,"statfs":{"total":95995035648,"available":95966216192,"internally_reserved":0,"allocated":1150976,"data_stored":976148,"data_compressed":54425,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299476,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":1179311,"data_compressed":70979,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463001,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1110016,"data_stored":578210,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1183744,"data_stored":647485,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":579598,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411438,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36504,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708648,"statfs":{"total":95995035648,"available":95957655552,"internally_reserved":0,"allocated":1196032,"data_stored":638999,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1167360,"data_stored":1103220,"data_compressed":60581,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888163,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":583640,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724638,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32356,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712796,"statfs":{"total":95995035648,"available":95961903104,"internally_reserved":0,"allocated":1208320,"data_stored":1306889,"data_compressed":77135,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134305,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":203323,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-10T00:28:01.541 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-06-10T00:28:01.542 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-10T00:28:01.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.682+0000 7f8c741ee640 1 -- 172.21.15.121:0/2683704413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c6c101ae0 msgr2=0x7f8c6c101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.682+0000 7f8c741ee640 1 --2- 172.21.15.121:0/2683704413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c6c101ae0 0x7f8c6c101f60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009900 tx=0x7f8c5c02a410 comp rx=0 tx=0).stop 2024-06-10T00:28:01.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.682+0000 7f8c741ee640 1 -- 172.21.15.121:0/2683704413 shutdown_connections 2024-06-10T00:28:01.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.682+0000 7f8c741ee640 1 --2- 172.21.15.121:0/2683704413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c6c1024a0 0x7f8c6c108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.682+0000 7f8c741ee640 1 --2- 172.21.15.121:0/2683704413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c6c101ae0 0x7f8c6c101f60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.682+0000 7f8c741ee640 1 --2- 172.21.15.121:0/2683704413 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c6c1008f0 0x7f8c6c100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.682+0000 7f8c741ee640 1 -- 172.21.15.121:0/2683704413 >> 172.21.15.121:0/2683704413 conn(0x7f8c6c0fbe80 msgr2=0x7f8c6c0fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:01.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.682+0000 7f8c741ee640 1 -- 172.21.15.121:0/2683704413 shutdown_connections 2024-06-10T00:28:01.683 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.683+0000 7f8c741ee640 1 -- 172.21.15.121:0/2683704413 wait complete. 2024-06-10T00:28:01.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.683+0000 7f8c741ee640 1 Processor -- start 2024-06-10T00:28:01.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.683+0000 7f8c741ee640 1 -- start start 2024-06-10T00:28:01.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.683+0000 7f8c741ee640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c6c1008f0 0x7f8c6c197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:01.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.683+0000 7f8c741ee640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c6c101ae0 0x7f8c6c198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:01.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.683+0000 7f8c741ee640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c6c1024a0 0x7f8c6c19f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:01.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.683+0000 7f8c741ee640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c10c550 con 0x7f8c6c1008f0 2024-06-10T00:28:01.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c741ee640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c10c3d0 con 0x7f8c6c101ae0 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c741ee640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c10c6d0 con 0x7f8c6c1024a0 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c71f8a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c6c1008f0 0x7f8c6c197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c71f8a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c6c1008f0 0x7f8c6c197d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41464/0 (socket says 172.21.15.121:41464) 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c71f8a640 1 -- 172.21.15.121:0/2469363265 learned_addr learned my addr 172.21.15.121:0/2469363265 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c71789640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c6c101ae0 0x7f8c6c198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c7278b640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c6c1024a0 0x7f8c6c19f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c71f8a640 1 -- 172.21.15.121:0/2469363265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c6c1024a0 msgr2=0x7f8c6c19f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c71f8a640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c6c1024a0 0x7f8c6c19f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c71f8a640 1 -- 172.21.15.121:0/2469363265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c6c101ae0 msgr2=0x7f8c6c198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c71f8a640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c6c101ae0 0x7f8c6c198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c71f8a640 1 -- 172.21.15.121:0/2469363265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c6c19f920 con 0x7f8c6c1008f0 2024-06-10T00:28:01.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.684+0000 7f8c71f8a640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c6c1008f0 0x7f8c6c197d10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8c6800d770 tx=0x7f8c6800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:01.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.688+0000 7f8c62ffd640 1 -- 172.21.15.121:0/2469363265 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68017070 con 0x7f8c6c1008f0 2024-06-10T00:28:01.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.688+0000 7f8c62ffd640 1 -- 172.21.15.121:0/2469363265 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c68009c30 con 0x7f8c6c1008f0 2024-06-10T00:28:01.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.688+0000 7f8c741ee640 1 -- 172.21.15.121:0/2469363265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c6c19fc10 con 0x7f8c6c1008f0 2024-06-10T00:28:01.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.688+0000 7f8c62ffd640 1 -- 172.21.15.121:0/2469363265 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68003dd0 con 0x7f8c6c1008f0 2024-06-10T00:28:01.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.688+0000 7f8c741ee640 1 -- 172.21.15.121:0/2469363265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c6c1a01b0 con 0x7f8c6c1008f0 2024-06-10T00:28:01.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.689+0000 7f8c741ee640 1 -- 172.21.15.121:0/2469363265 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c6c1a0380 con 0x7f8c6c1008f0 2024-06-10T00:28:01.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.690+0000 7f8c62ffd640 1 -- 172.21.15.121:0/2469363265 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 23) ==== 137954+0+0 (secure 0 0 0) 0x7f8c68003fb0 con 0x7f8c6c1008f0 2024-06-10T00:28:01.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.691+0000 7f8c62ffd640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c580a2d80 0x7f8c580a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:01.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.691+0000 7f8c71789640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c580a2d80 0x7f8c580a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:01.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.691+0000 7f8c62ffd640 1 -- 172.21.15.121:0/2469363265 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8c680cf130 con 0x7f8c6c1008f0 2024-06-10T00:28:01.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.691+0000 7f8c71789640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c580a2d80 0x7f8c580a5240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0314e0 tx=0x7f8c5c000bc0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:01.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.695+0000 7f8c62ffd640 1 -- 172.21.15.121:0/2469363265 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8c68009de0 con 0x7f8c6c1008f0 2024-06-10T00:28:01.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.876+0000 7f8c741ee640 1 -- 172.21.15.121:0/2469363265 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8c6c0fe010 con 0x7f8c580a2d80 2024-06-10T00:28:01.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.886+0000 7f8c62ffd640 1 -- 172.21.15.121:0/2469363265 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339875 (secure 0 0 0) 0x7f8c6c0fe010 con 0x7f8c580a2d80 2024-06-10T00:28:01.890 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:01.892 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-06-10T00:28:01.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.889+0000 7f8c741ee640 1 -- 172.21.15.121:0/2469363265 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c580a2d80 msgr2=0x7f8c580a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.889+0000 7f8c741ee640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c580a2d80 0x7f8c580a5240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c0314e0 tx=0x7f8c5c000bc0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.890+0000 7f8c741ee640 1 -- 172.21.15.121:0/2469363265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c6c1008f0 msgr2=0x7f8c6c197d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:01.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.890+0000 7f8c741ee640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c6c1008f0 0x7f8c6c197d10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8c6800d770 tx=0x7f8c6800dc30 comp rx=0 tx=0).stop 2024-06-10T00:28:01.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.890+0000 7f8c741ee640 1 -- 172.21.15.121:0/2469363265 shutdown_connections 2024-06-10T00:28:01.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.890+0000 7f8c741ee640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c580a2d80 0x7f8c580a5240 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.890+0000 7f8c741ee640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c6c1024a0 0x7f8c6c19f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.890+0000 7f8c741ee640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c6c101ae0 0x7f8c6c198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.890+0000 7f8c741ee640 1 --2- 172.21.15.121:0/2469363265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c6c1008f0 0x7f8c6c197d10 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:01.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.890+0000 7f8c741ee640 1 -- 172.21.15.121:0/2469363265 >> 172.21.15.121:0/2469363265 conn(0x7f8c6c0fbe80 msgr2=0x7f8c6c0fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:01.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.890+0000 7f8c741ee640 1 -- 172.21.15.121:0/2469363265 shutdown_connections 2024-06-10T00:28:01.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:01.890+0000 7f8c741ee640 1 -- 172.21.15.121:0/2469363265 wait complete. 2024-06-10T00:28:01.900 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:01.900 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:01.900 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:01.900 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:01.900 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:01.900 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi135 to 176.2M 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: Unable to set osd_memory_target on smithi135 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: Unable to set osd_memory_target on smithi193 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:01.901 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:01 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:01.902 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-06-10T00:28:01.472808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190402,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1299,"num_read_kb":812,"num_write":482,"num_write_kb":4387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":874,"ondisk_log_size":874,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367044,"kb_used_data":13688,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574780,"statfs":{"total":1151940427776,"available":1151564574720,"internally_reserved":0,"allocated":14016512,"data_stored":10528477,"data_compressed":603376,"data_compressed_allocated":819200,"data_compressed_original":4489216,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.996358"},"pg_stats":[{"pgid":"3.3f","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592178+0000","last_change":"2024-06-10T00:23:57.564073+0000","last_active":"2024-06-10T00:27:57.592178+0000","last_peered":"2024-06-10T00:27:57.592178+0000","last_clean":"2024-06-10T00:27:57.592178+0000","last_became_active":"2024-06-10T00:23:57.563618+0000","last_became_peered":"2024-06-10T00:23:57.563618+0000","last_unstale":"2024-06-10T00:27:57.592178+0000","last_undegraded":"2024-06-10T00:27:57.592178+0000","last_fullsized":"2024-06-10T00:27:57.592178+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:26:25.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591355+0000","last_change":"2024-06-10T00:23:57.560148+0000","last_active":"2024-06-10T00:27:57.591355+0000","last_peered":"2024-06-10T00:27:57.591355+0000","last_clean":"2024-06-10T00:27:57.591355+0000","last_became_active":"2024-06-10T00:23:57.559794+0000","last_became_peered":"2024-06-10T00:23:57.559794+0000","last_unstale":"2024-06-10T00:27:57.591355+0000","last_undegraded":"2024-06-10T00:27:57.591355+0000","last_fullsized":"2024-06-10T00:27:57.591355+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:59:50.624542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593114+0000","last_change":"2024-06-10T00:23:57.563292+0000","last_active":"2024-06-10T00:27:57.593114+0000","last_peered":"2024-06-10T00:27:57.593114+0000","last_clean":"2024-06-10T00:27:57.593114+0000","last_became_active":"2024-06-10T00:23:57.563007+0000","last_became_peered":"2024-06-10T00:23:57.563007+0000","last_unstale":"2024-06-10T00:27:57.593114+0000","last_undegraded":"2024-06-10T00:27:57.593114+0000","last_fullsized":"2024-06-10T00:27:57.593114+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:27:44.897770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545262+0000","last_change":"2024-06-10T00:23:57.558789+0000","last_active":"2024-06-10T00:27:57.545262+0000","last_peered":"2024-06-10T00:27:57.545262+0000","last_clean":"2024-06-10T00:27:57.545262+0000","last_became_active":"2024-06-10T00:23:57.557943+0000","last_became_peered":"2024-06-10T00:23:57.557943+0000","last_unstale":"2024-06-10T00:27:57.545262+0000","last_undegraded":"2024-06-10T00:27:57.545262+0000","last_fullsized":"2024-06-10T00:27:57.545262+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:10:40.081996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569868+0000","last_change":"2024-06-10T00:23:57.550806+0000","last_active":"2024-06-10T00:27:57.569868+0000","last_peered":"2024-06-10T00:27:57.569868+0000","last_clean":"2024-06-10T00:27:57.569868+0000","last_became_active":"2024-06-10T00:23:57.550453+0000","last_became_peered":"2024-06-10T00:23:57.550453+0000","last_unstale":"2024-06-10T00:27:57.569868+0000","last_undegraded":"2024-06-10T00:27:57.569868+0000","last_fullsized":"2024-06-10T00:27:57.569868+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:11:58.765099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458657+0000","last_change":"2024-06-10T00:23:57.567307+0000","last_active":"2024-06-10T00:27:57.458657+0000","last_peered":"2024-06-10T00:27:57.458657+0000","last_clean":"2024-06-10T00:27:57.458657+0000","last_became_active":"2024-06-10T00:23:57.566973+0000","last_became_peered":"2024-06-10T00:23:57.566973+0000","last_unstale":"2024-06-10T00:27:57.458657+0000","last_undegraded":"2024-06-10T00:27:57.458657+0000","last_fullsized":"2024-06-10T00:27:57.458657+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:27:25.135046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591279+0000","last_change":"2024-06-10T00:23:57.547998+0000","last_active":"2024-06-10T00:27:57.591279+0000","last_peered":"2024-06-10T00:27:57.591279+0000","last_clean":"2024-06-10T00:27:57.591279+0000","last_became_active":"2024-06-10T00:23:57.547593+0000","last_became_peered":"2024-06-10T00:23:57.547593+0000","last_unstale":"2024-06-10T00:27:57.591279+0000","last_undegraded":"2024-06-10T00:27:57.591279+0000","last_fullsized":"2024-06-10T00:27:57.591279+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:34:04.219399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570182+0000","last_change":"2024-06-10T00:23:57.560252+0000","last_active":"2024-06-10T00:27:57.570182+0000","last_peered":"2024-06-10T00:27:57.570182+0000","last_clean":"2024-06-10T00:27:57.570182+0000","last_became_active":"2024-06-10T00:23:57.559924+0000","last_became_peered":"2024-06-10T00:23:57.559924+0000","last_unstale":"2024-06-10T00:27:57.570182+0000","last_undegraded":"2024-06-10T00:27:57.570182+0000","last_fullsized":"2024-06-10T00:27:57.570182+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:01:43.684262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590876+0000","last_change":"2024-06-10T00:23:57.564031+0000","last_active":"2024-06-10T00:27:57.590876+0000","last_peered":"2024-06-10T00:27:57.590876+0000","last_clean":"2024-06-10T00:27:57.590876+0000","last_became_active":"2024-06-10T00:23:57.563642+0000","last_became_peered":"2024-06-10T00:23:57.563642+0000","last_unstale":"2024-06-10T00:27:57.590876+0000","last_undegraded":"2024-06-10T00:27:57.590876+0000","last_fullsized":"2024-06-10T00:27:57.590876+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:16:23.003152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"100'16","reported_seq":57,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459536+0000","last_change":"2024-06-10T00:23:57.566877+0000","last_active":"2024-06-10T00:27:57.459536+0000","last_peered":"2024-06-10T00:27:57.459536+0000","last_clean":"2024-06-10T00:27:57.459536+0000","last_became_active":"2024-06-10T00:23:57.566529+0000","last_became_peered":"2024-06-10T00:23:57.566529+0000","last_unstale":"2024-06-10T00:27:57.459536+0000","last_undegraded":"2024-06-10T00:27:57.459536+0000","last_fullsized":"2024-06-10T00:27:57.459536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:30:43.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592151+0000","last_change":"2024-06-10T00:23:57.564979+0000","last_active":"2024-06-10T00:27:57.592151+0000","last_peered":"2024-06-10T00:27:57.592151+0000","last_clean":"2024-06-10T00:27:57.592151+0000","last_became_active":"2024-06-10T00:23:57.564578+0000","last_became_peered":"2024-06-10T00:23:57.564578+0000","last_unstale":"2024-06-10T00:27:57.592151+0000","last_undegraded":"2024-06-10T00:27:57.592151+0000","last_fullsized":"2024-06-10T00:27:57.592151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:40:53.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"100'7","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545061+0000","last_change":"2024-06-10T00:23:57.559126+0000","last_active":"2024-06-10T00:27:57.545061+0000","last_peered":"2024-06-10T00:27:57.545061+0000","last_clean":"2024-06-10T00:27:57.545061+0000","last_became_active":"2024-06-10T00:23:57.558401+0000","last_became_peered":"2024-06-10T00:23:57.558401+0000","last_unstale":"2024-06-10T00:27:57.545061+0000","last_undegraded":"2024-06-10T00:27:57.545061+0000","last_fullsized":"2024-06-10T00:27:57.545061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:57.894137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542053+0000","last_change":"2024-06-10T00:23:57.559272+0000","last_active":"2024-06-10T00:27:57.542053+0000","last_peered":"2024-06-10T00:27:57.542053+0000","last_clean":"2024-06-10T00:27:57.542053+0000","last_became_active":"2024-06-10T00:23:57.558941+0000","last_became_peered":"2024-06-10T00:23:57.558941+0000","last_unstale":"2024-06-10T00:27:57.542053+0000","last_undegraded":"2024-06-10T00:27:57.542053+0000","last_fullsized":"2024-06-10T00:27:57.542053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:36:48.088347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591361+0000","last_change":"2024-06-10T00:23:57.555875+0000","last_active":"2024-06-10T00:27:57.591361+0000","last_peered":"2024-06-10T00:27:57.591361+0000","last_clean":"2024-06-10T00:27:57.591361+0000","last_became_active":"2024-06-10T00:23:57.555492+0000","last_became_peered":"2024-06-10T00:23:57.555492+0000","last_unstale":"2024-06-10T00:27:57.591361+0000","last_undegraded":"2024-06-10T00:27:57.591361+0000","last_fullsized":"2024-06-10T00:27:57.591361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:49:10.827458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591360+0000","last_change":"2024-06-10T00:23:57.555316+0000","last_active":"2024-06-10T00:27:57.591360+0000","last_peered":"2024-06-10T00:27:57.591360+0000","last_clean":"2024-06-10T00:27:57.591360+0000","last_became_active":"2024-06-10T00:23:57.554952+0000","last_became_peered":"2024-06-10T00:23:57.554952+0000","last_unstale":"2024-06-10T00:27:57.591360+0000","last_undegraded":"2024-06-10T00:27:57.591360+0000","last_fullsized":"2024-06-10T00:27:57.591360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:03:57.557330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591053+0000","last_change":"2024-06-10T00:23:57.562442+0000","last_active":"2024-06-10T00:27:57.591053+0000","last_peered":"2024-06-10T00:27:57.591053+0000","last_clean":"2024-06-10T00:27:57.591053+0000","last_became_active":"2024-06-10T00:23:57.561965+0000","last_became_peered":"2024-06-10T00:23:57.561965+0000","last_unstale":"2024-06-10T00:27:57.591053+0000","last_undegraded":"2024-06-10T00:27:57.591053+0000","last_fullsized":"2024-06-10T00:27:57.591053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:40:16.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591981+0000","last_change":"2024-06-10T00:23:57.564422+0000","last_active":"2024-06-10T00:27:57.591981+0000","last_peered":"2024-06-10T00:27:57.591981+0000","last_clean":"2024-06-10T00:27:57.591981+0000","last_became_active":"2024-06-10T00:23:57.564069+0000","last_became_peered":"2024-06-10T00:23:57.564069+0000","last_unstale":"2024-06-10T00:27:57.591981+0000","last_undegraded":"2024-06-10T00:27:57.591981+0000","last_fullsized":"2024-06-10T00:27:57.591981+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:07.662465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458239+0000","last_change":"2024-06-10T00:23:57.561284+0000","last_active":"2024-06-10T00:27:57.458239+0000","last_peered":"2024-06-10T00:27:57.458239+0000","last_clean":"2024-06-10T00:27:57.458239+0000","last_became_active":"2024-06-10T00:23:57.560933+0000","last_became_peered":"2024-06-10T00:23:57.560933+0000","last_unstale":"2024-06-10T00:27:57.458239+0000","last_undegraded":"2024-06-10T00:27:57.458239+0000","last_fullsized":"2024-06-10T00:27:57.458239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:58:48.511774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590777+0000","last_change":"2024-06-10T00:23:57.557752+0000","last_active":"2024-06-10T00:27:57.590777+0000","last_peered":"2024-06-10T00:27:57.590777+0000","last_clean":"2024-06-10T00:27:57.590777+0000","last_became_active":"2024-06-10T00:23:57.557419+0000","last_became_peered":"2024-06-10T00:23:57.557419+0000","last_unstale":"2024-06-10T00:27:57.590777+0000","last_undegraded":"2024-06-10T00:27:57.590777+0000","last_fullsized":"2024-06-10T00:27:57.590777+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:48.234181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591122+0000","last_change":"2024-06-10T00:24:09.177183+0000","last_active":"2024-06-10T00:27:57.591122+0000","last_peered":"2024-06-10T00:27:57.591122+0000","last_clean":"2024-06-10T00:27:57.591122+0000","last_became_active":"2024-06-10T00:24:09.176680+0000","last_became_peered":"2024-06-10T00:24:09.176680+0000","last_unstale":"2024-06-10T00:27:57.591122+0000","last_undegraded":"2024-06-10T00:27:57.591122+0000","last_fullsized":"2024-06-10T00:27:57.591122+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:17:06.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"98'9","reported_seq":57,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458177+0000","last_change":"2024-06-10T00:23:54.426364+0000","last_active":"2024-06-10T00:27:57.458177+0000","last_peered":"2024-06-10T00:27:57.458177+0000","last_clean":"2024-06-10T00:27:57.458177+0000","last_became_active":"2024-06-10T00:23:54.426054+0000","last_became_peered":"2024-06-10T00:23:54.426054+0000","last_unstale":"2024-06-10T00:27:57.458177+0000","last_undegraded":"2024-06-10T00:27:57.458177+0000","last_fullsized":"2024-06-10T00:27:57.458177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:47.613402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570428+0000","last_change":"2024-06-10T00:23:57.557206+0000","last_active":"2024-06-10T00:27:57.570428+0000","last_peered":"2024-06-10T00:27:57.570428+0000","last_clean":"2024-06-10T00:27:57.570428+0000","last_became_active":"2024-06-10T00:23:57.556853+0000","last_became_peered":"2024-06-10T00:23:57.556853+0000","last_unstale":"2024-06-10T00:27:57.570428+0000","last_undegraded":"2024-06-10T00:27:57.570428+0000","last_fullsized":"2024-06-10T00:27:57.570428+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:09.686334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544782+0000","last_change":"2024-06-10T00:24:09.171047+0000","last_active":"2024-06-10T00:27:57.544782+0000","last_peered":"2024-06-10T00:27:57.544782+0000","last_clean":"2024-06-10T00:27:57.544782+0000","last_became_active":"2024-06-10T00:24:09.170501+0000","last_became_peered":"2024-06-10T00:24:09.170501+0000","last_unstale":"2024-06-10T00:27:57.544782+0000","last_undegraded":"2024-06-10T00:27:57.544782+0000","last_fullsized":"2024-06-10T00:27:57.544782+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:47:27.690439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"98'2","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571600+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:27:57.571600+0000","last_peered":"2024-06-10T00:27:57.571600+0000","last_clean":"2024-06-10T00:27:57.571600+0000","last_became_active":"2024-06-10T00:23:54.822447+0000","last_became_peered":"2024-06-10T00:23:54.822447+0000","last_unstale":"2024-06-10T00:27:57.571600+0000","last_undegraded":"2024-06-10T00:27:57.571600+0000","last_fullsized":"2024-06-10T00:27:57.571600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:45:14.956540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591811+0000","last_change":"2024-06-10T00:23:57.554958+0000","last_active":"2024-06-10T00:27:57.591811+0000","last_peered":"2024-06-10T00:27:57.591811+0000","last_clean":"2024-06-10T00:27:57.591811+0000","last_became_active":"2024-06-10T00:23:57.554477+0000","last_became_peered":"2024-06-10T00:23:57.554477+0000","last_unstale":"2024-06-10T00:27:57.591811+0000","last_undegraded":"2024-06-10T00:27:57.591811+0000","last_fullsized":"2024-06-10T00:27:57.591811+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:40:06.066967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544895+0000","last_change":"2024-06-10T00:24:09.169642+0000","last_active":"2024-06-10T00:27:57.544895+0000","last_peered":"2024-06-10T00:27:57.544895+0000","last_clean":"2024-06-10T00:27:57.544895+0000","last_became_active":"2024-06-10T00:24:09.168071+0000","last_became_peered":"2024-06-10T00:24:09.168071+0000","last_unstale":"2024-06-10T00:27:57.544895+0000","last_undegraded":"2024-06-10T00:27:57.544895+0000","last_fullsized":"2024-06-10T00:27:57.544895+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:44.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"98'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570379+0000","last_change":"2024-06-10T00:23:54.430543+0000","last_active":"2024-06-10T00:27:57.570379+0000","last_peered":"2024-06-10T00:27:57.570379+0000","last_clean":"2024-06-10T00:27:57.570379+0000","last_became_active":"2024-06-10T00:23:54.430138+0000","last_became_peered":"2024-06-10T00:23:54.430138+0000","last_unstale":"2024-06-10T00:27:57.570379+0000","last_undegraded":"2024-06-10T00:27:57.570379+0000","last_fullsized":"2024-06-10T00:27:57.570379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:39:53.212607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590796+0000","last_change":"2024-06-10T00:23:57.562251+0000","last_active":"2024-06-10T00:27:57.590796+0000","last_peered":"2024-06-10T00:27:57.590796+0000","last_clean":"2024-06-10T00:27:57.590796+0000","last_became_active":"2024-06-10T00:23:57.561903+0000","last_became_peered":"2024-06-10T00:23:57.561903+0000","last_unstale":"2024-06-10T00:27:57.590796+0000","last_undegraded":"2024-06-10T00:27:57.590796+0000","last_fullsized":"2024-06-10T00:27:57.590796+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:59.489475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544829+0000","last_change":"2024-06-10T00:24:09.165894+0000","last_active":"2024-06-10T00:27:57.544829+0000","last_peered":"2024-06-10T00:27:57.544829+0000","last_clean":"2024-06-10T00:27:57.544829+0000","last_became_active":"2024-06-10T00:24:09.165435+0000","last_became_peered":"2024-06-10T00:24:09.165435+0000","last_unstale":"2024-06-10T00:27:57.544829+0000","last_undegraded":"2024-06-10T00:27:57.544829+0000","last_fullsized":"2024-06-10T00:27:57.544829+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:13:36.058375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570296+0000","last_change":"2024-06-10T00:23:54.427868+0000","last_active":"2024-06-10T00:27:57.570296+0000","last_peered":"2024-06-10T00:27:57.570296+0000","last_clean":"2024-06-10T00:27:57.570296+0000","last_became_active":"2024-06-10T00:23:54.427366+0000","last_became_peered":"2024-06-10T00:23:54.427366+0000","last_unstale":"2024-06-10T00:27:57.570296+0000","last_undegraded":"2024-06-10T00:27:57.570296+0000","last_fullsized":"2024-06-10T00:27:57.570296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:15.499292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"100'15","reported_seq":62,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462390+0000","last_change":"2024-06-10T00:24:43.768556+0000","last_active":"2024-06-10T00:27:57.462390+0000","last_peered":"2024-06-10T00:27:57.462390+0000","last_clean":"2024-06-10T00:27:57.462390+0000","last_became_active":"2024-06-10T00:24:43.768090+0000","last_became_peered":"2024-06-10T00:24:43.768090+0000","last_unstale":"2024-06-10T00:27:57.462390+0000","last_undegraded":"2024-06-10T00:27:57.462390+0000","last_fullsized":"2024-06-10T00:27:57.462390+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:05:41.604184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591403+0000","last_change":"2024-06-10T00:24:09.170514+0000","last_active":"2024-06-10T00:27:57.591403+0000","last_peered":"2024-06-10T00:27:57.591403+0000","last_clean":"2024-06-10T00:27:57.591403+0000","last_became_active":"2024-06-10T00:24:09.168902+0000","last_became_peered":"2024-06-10T00:24:09.168902+0000","last_unstale":"2024-06-10T00:27:57.591403+0000","last_undegraded":"2024-06-10T00:27:57.591403+0000","last_fullsized":"2024-06-10T00:27:57.591403+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:33:10.534657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"101'9","reported_seq":234,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.552203+0000","last_change":"2024-06-10T00:23:54.426581+0000","last_active":"2024-06-10T00:27:57.552203+0000","last_peered":"2024-06-10T00:27:57.552203+0000","last_clean":"2024-06-10T00:27:57.552203+0000","last_became_active":"2024-06-10T00:23:54.426211+0000","last_became_peered":"2024-06-10T00:23:54.426211+0000","last_unstale":"2024-06-10T00:27:57.552203+0000","last_undegraded":"2024-06-10T00:27:57.552203+0000","last_fullsized":"2024-06-10T00:27:57.552203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:49:05.360742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":178,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569865+0000","last_change":"2024-06-10T00:23:57.552308+0000","last_active":"2024-06-10T00:27:57.569865+0000","last_peered":"2024-06-10T00:27:57.569865+0000","last_clean":"2024-06-10T00:27:57.569865+0000","last_became_active":"2024-06-10T00:23:57.551762+0000","last_became_peered":"2024-06-10T00:23:57.551762+0000","last_unstale":"2024-06-10T00:27:57.569865+0000","last_undegraded":"2024-06-10T00:27:57.569865+0000","last_fullsized":"2024-06-10T00:27:57.569865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:22:28.476019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591613+0000","last_change":"2024-06-10T00:24:09.171456+0000","last_active":"2024-06-10T00:27:57.591613+0000","last_peered":"2024-06-10T00:27:57.591613+0000","last_clean":"2024-06-10T00:27:57.591613+0000","last_became_active":"2024-06-10T00:24:09.170969+0000","last_became_peered":"2024-06-10T00:24:09.170969+0000","last_unstale":"2024-06-10T00:27:57.591613+0000","last_undegraded":"2024-06-10T00:27:57.591613+0000","last_fullsized":"2024-06-10T00:27:57.591613+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:22:46.023897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458602+0000","last_change":"2024-06-10T00:23:54.428384+0000","last_active":"2024-06-10T00:27:57.458602+0000","last_peered":"2024-06-10T00:27:57.458602+0000","last_clean":"2024-06-10T00:27:57.458602+0000","last_became_active":"2024-06-10T00:23:54.427967+0000","last_became_peered":"2024-06-10T00:23:54.427967+0000","last_unstale":"2024-06-10T00:27:57.458602+0000","last_undegraded":"2024-06-10T00:27:57.458602+0000","last_fullsized":"2024-06-10T00:27:57.458602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:54:30.408684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"100'8","reported_seq":55,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461853+0000","last_change":"2024-06-10T00:24:43.262593+0000","last_active":"2024-06-10T00:27:57.461853+0000","last_peered":"2024-06-10T00:27:57.461853+0000","last_clean":"2024-06-10T00:27:57.461853+0000","last_became_active":"2024-06-10T00:24:43.262212+0000","last_became_peered":"2024-06-10T00:24:43.262212+0000","last_unstale":"2024-06-10T00:27:57.461853+0000","last_undegraded":"2024-06-10T00:27:57.461853+0000","last_fullsized":"2024-06-10T00:27:57.461853+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:40:24.442712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591973+0000","last_change":"2024-06-10T00:24:43.752805+0000","last_active":"2024-06-10T00:27:57.591973+0000","last_peered":"2024-06-10T00:27:57.591973+0000","last_clean":"2024-06-10T00:27:57.591973+0000","last_became_active":"2024-06-10T00:24:43.752251+0000","last_became_peered":"2024-06-10T00:24:43.752251+0000","last_unstale":"2024-06-10T00:27:57.591973+0000","last_undegraded":"2024-06-10T00:27:57.591973+0000","last_fullsized":"2024-06-10T00:27:57.591973+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:27:57.617443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542566+0000","last_change":"2024-06-10T00:23:54.421493+0000","last_active":"2024-06-10T00:27:57.542566+0000","last_peered":"2024-06-10T00:27:57.542566+0000","last_clean":"2024-06-10T00:27:57.542566+0000","last_became_active":"2024-06-10T00:23:54.421151+0000","last_became_peered":"2024-06-10T00:23:54.421151+0000","last_unstale":"2024-06-10T00:27:57.542566+0000","last_undegraded":"2024-06-10T00:27:57.542566+0000","last_fullsized":"2024-06-10T00:27:57.542566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:03.995244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570099+0000","last_change":"2024-06-10T00:23:57.554020+0000","last_active":"2024-06-10T00:27:57.570099+0000","last_peered":"2024-06-10T00:27:57.570099+0000","last_clean":"2024-06-10T00:27:57.570099+0000","last_became_active":"2024-06-10T00:23:57.553690+0000","last_became_peered":"2024-06-10T00:23:57.553690+0000","last_unstale":"2024-06-10T00:27:57.570099+0000","last_undegraded":"2024-06-10T00:27:57.570099+0000","last_fullsized":"2024-06-10T00:27:57.570099+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:48.097074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593589+0000","last_change":"2024-06-10T00:24:09.166426+0000","last_active":"2024-06-10T00:27:57.593589+0000","last_peered":"2024-06-10T00:27:57.593589+0000","last_clean":"2024-06-10T00:27:57.593589+0000","last_became_active":"2024-06-10T00:24:09.166047+0000","last_became_peered":"2024-06-10T00:24:09.166047+0000","last_unstale":"2024-06-10T00:27:57.593589+0000","last_undegraded":"2024-06-10T00:27:57.593589+0000","last_fullsized":"2024-06-10T00:27:57.593589+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:56:11.608061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"98'1","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461240+0000","last_change":"2024-06-10T00:23:54.426409+0000","last_active":"2024-06-10T00:27:57.461240+0000","last_peered":"2024-06-10T00:27:57.461240+0000","last_clean":"2024-06-10T00:27:57.461240+0000","last_became_active":"2024-06-10T00:23:54.425385+0000","last_became_peered":"2024-06-10T00:23:54.425385+0000","last_unstale":"2024-06-10T00:27:57.461240+0000","last_undegraded":"2024-06-10T00:27:57.461240+0000","last_fullsized":"2024-06-10T00:27:57.461240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:27:00.379189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457389+0000","last_change":"2024-06-10T00:23:57.560132+0000","last_active":"2024-06-10T00:27:57.457389+0000","last_peered":"2024-06-10T00:27:57.457389+0000","last_clean":"2024-06-10T00:27:57.457389+0000","last_became_active":"2024-06-10T00:23:57.559803+0000","last_became_peered":"2024-06-10T00:23:57.559803+0000","last_unstale":"2024-06-10T00:27:57.457389+0000","last_undegraded":"2024-06-10T00:27:57.457389+0000","last_fullsized":"2024-06-10T00:27:57.457389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:13:16.004233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571137+0000","last_change":"2024-06-10T00:24:09.170962+0000","last_active":"2024-06-10T00:27:57.571137+0000","last_peered":"2024-06-10T00:27:57.571137+0000","last_clean":"2024-06-10T00:27:57.571137+0000","last_became_active":"2024-06-10T00:24:09.170090+0000","last_became_peered":"2024-06-10T00:24:09.170090+0000","last_unstale":"2024-06-10T00:27:57.571137+0000","last_undegraded":"2024-06-10T00:27:57.571137+0000","last_fullsized":"2024-06-10T00:27:57.571137+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:07:33.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'5","reported_seq":225,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459435+0000","last_change":"2024-06-10T00:23:54.421472+0000","last_active":"2024-06-10T00:27:57.459435+0000","last_peered":"2024-06-10T00:27:57.459435+0000","last_clean":"2024-06-10T00:27:57.459435+0000","last_became_active":"2024-06-10T00:23:54.421019+0000","last_became_peered":"2024-06-10T00:23:54.421019+0000","last_unstale":"2024-06-10T00:27:57.459435+0000","last_undegraded":"2024-06-10T00:27:57.459435+0000","last_fullsized":"2024-06-10T00:27:57.459435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:28:18.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":173,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590963+0000","last_change":"2024-06-10T00:23:57.563010+0000","last_active":"2024-06-10T00:27:57.590963+0000","last_peered":"2024-06-10T00:27:57.590963+0000","last_clean":"2024-06-10T00:27:57.590963+0000","last_became_active":"2024-06-10T00:23:57.561986+0000","last_became_peered":"2024-06-10T00:23:57.561986+0000","last_unstale":"2024-06-10T00:27:57.590963+0000","last_undegraded":"2024-06-10T00:27:57.590963+0000","last_fullsized":"2024-06-10T00:27:57.590963+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:31:17.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457415+0000","last_change":"2024-06-10T00:24:09.171746+0000","last_active":"2024-06-10T00:27:57.457415+0000","last_peered":"2024-06-10T00:27:57.457415+0000","last_clean":"2024-06-10T00:27:57.457415+0000","last_became_active":"2024-06-10T00:24:09.171030+0000","last_became_peered":"2024-06-10T00:24:09.171030+0000","last_unstale":"2024-06-10T00:27:57.457415+0000","last_undegraded":"2024-06-10T00:27:57.457415+0000","last_fullsized":"2024-06-10T00:27:57.457415+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:41.669471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591812+0000","last_change":"2024-06-10T00:23:54.428917+0000","last_active":"2024-06-10T00:27:57.591812+0000","last_peered":"2024-06-10T00:27:57.591812+0000","last_clean":"2024-06-10T00:27:57.591812+0000","last_became_active":"2024-06-10T00:23:54.428585+0000","last_became_peered":"2024-06-10T00:23:54.428585+0000","last_unstale":"2024-06-10T00:27:57.591812+0000","last_undegraded":"2024-06-10T00:27:57.591812+0000","last_fullsized":"2024-06-10T00:27:57.591812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:21:58.235848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591586+0000","last_change":"2024-06-10T00:23:57.551778+0000","last_active":"2024-06-10T00:27:57.591586+0000","last_peered":"2024-06-10T00:27:57.591586+0000","last_clean":"2024-06-10T00:27:57.591586+0000","last_became_active":"2024-06-10T00:23:57.551436+0000","last_became_peered":"2024-06-10T00:23:57.551436+0000","last_unstale":"2024-06-10T00:27:57.591586+0000","last_undegraded":"2024-06-10T00:27:57.591586+0000","last_fullsized":"2024-06-10T00:27:57.591586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:30.059783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545101+0000","last_change":"2024-06-10T00:24:09.167547+0000","last_active":"2024-06-10T00:27:57.545101+0000","last_peered":"2024-06-10T00:27:57.545101+0000","last_clean":"2024-06-10T00:27:57.545101+0000","last_became_active":"2024-06-10T00:24:09.167132+0000","last_became_peered":"2024-06-10T00:24:09.167132+0000","last_unstale":"2024-06-10T00:27:57.545101+0000","last_undegraded":"2024-06-10T00:27:57.545101+0000","last_fullsized":"2024-06-10T00:27:57.545101+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:11.520330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461450+0000","last_change":"2024-06-10T00:23:54.823754+0000","last_active":"2024-06-10T00:27:57.461450+0000","last_peered":"2024-06-10T00:27:57.461450+0000","last_clean":"2024-06-10T00:27:57.461450+0000","last_became_active":"2024-06-10T00:23:54.823362+0000","last_became_peered":"2024-06-10T00:23:54.823362+0000","last_unstale":"2024-06-10T00:27:57.461450+0000","last_undegraded":"2024-06-10T00:27:57.461450+0000","last_fullsized":"2024-06-10T00:27:57.461450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:48:50.915439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590772+0000","last_change":"2024-06-10T00:23:57.563313+0000","last_active":"2024-06-10T00:27:57.590772+0000","last_peered":"2024-06-10T00:27:57.590772+0000","last_clean":"2024-06-10T00:27:57.590772+0000","last_became_active":"2024-06-10T00:23:57.562307+0000","last_became_peered":"2024-06-10T00:23:57.562307+0000","last_unstale":"2024-06-10T00:27:57.590772+0000","last_undegraded":"2024-06-10T00:27:57.590772+0000","last_fullsized":"2024-06-10T00:27:57.590772+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:23.815267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457507+0000","last_change":"2024-06-10T00:24:09.174660+0000","last_active":"2024-06-10T00:27:57.457507+0000","last_peered":"2024-06-10T00:27:57.457507+0000","last_clean":"2024-06-10T00:27:57.457507+0000","last_became_active":"2024-06-10T00:24:09.174283+0000","last_became_peered":"2024-06-10T00:24:09.174283+0000","last_unstale":"2024-06-10T00:27:57.457507+0000","last_undegraded":"2024-06-10T00:27:57.457507+0000","last_fullsized":"2024-06-10T00:27:57.457507+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:15.236146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"101'41","reported_seq":446,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.749518+0000","last_change":"2024-06-10T00:23:54.426621+0000","last_active":"2024-06-10T00:27:57.749518+0000","last_peered":"2024-06-10T00:27:57.749518+0000","last_clean":"2024-06-10T00:27:57.749518+0000","last_became_active":"2024-06-10T00:23:54.426131+0000","last_became_peered":"2024-06-10T00:23:54.426131+0000","last_unstale":"2024-06-10T00:27:57.749518+0000","last_undegraded":"2024-06-10T00:27:57.749518+0000","last_fullsized":"2024-06-10T00:27:57.749518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:56:14.819454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69365,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":358,"num_read_kb":247,"num_write":42,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570532+0000","last_change":"2024-06-10T00:23:57.552606+0000","last_active":"2024-06-10T00:27:57.570532+0000","last_peered":"2024-06-10T00:27:57.570532+0000","last_clean":"2024-06-10T00:27:57.570532+0000","last_became_active":"2024-06-10T00:23:57.552246+0000","last_became_peered":"2024-06-10T00:23:57.552246+0000","last_unstale":"2024-06-10T00:27:57.570532+0000","last_undegraded":"2024-06-10T00:27:57.570532+0000","last_fullsized":"2024-06-10T00:27:57.570532+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:19:21.508721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462817+0000","last_change":"2024-06-10T00:24:09.174862+0000","last_active":"2024-06-10T00:27:57.462817+0000","last_peered":"2024-06-10T00:27:57.462817+0000","last_clean":"2024-06-10T00:27:57.462817+0000","last_became_active":"2024-06-10T00:24:09.173550+0000","last_became_peered":"2024-06-10T00:24:09.173550+0000","last_unstale":"2024-06-10T00:27:57.462817+0000","last_undegraded":"2024-06-10T00:27:57.462817+0000","last_fullsized":"2024-06-10T00:27:57.462817+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:49:25.549084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"98'6","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591325+0000","last_change":"2024-06-10T00:23:54.426074+0000","last_active":"2024-06-10T00:27:57.591325+0000","last_peered":"2024-06-10T00:27:57.591325+0000","last_clean":"2024-06-10T00:27:57.591325+0000","last_became_active":"2024-06-10T00:23:54.425707+0000","last_became_peered":"2024-06-10T00:23:54.425707+0000","last_unstale":"2024-06-10T00:27:57.591325+0000","last_undegraded":"2024-06-10T00:27:57.591325+0000","last_fullsized":"2024-06-10T00:27:57.591325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:11:10.563595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542128+0000","last_change":"2024-06-10T00:23:57.549784+0000","last_active":"2024-06-10T00:27:57.542128+0000","last_peered":"2024-06-10T00:27:57.542128+0000","last_clean":"2024-06-10T00:27:57.542128+0000","last_became_active":"2024-06-10T00:23:57.549457+0000","last_became_peered":"2024-06-10T00:23:57.549457+0000","last_unstale":"2024-06-10T00:27:57.542128+0000","last_undegraded":"2024-06-10T00:27:57.542128+0000","last_fullsized":"2024-06-10T00:27:57.542128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:59:27.352927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592095+0000","last_change":"2024-06-10T00:24:09.171028+0000","last_active":"2024-06-10T00:27:57.592095+0000","last_peered":"2024-06-10T00:27:57.592095+0000","last_clean":"2024-06-10T00:27:57.592095+0000","last_became_active":"2024-06-10T00:24:09.170543+0000","last_became_peered":"2024-06-10T00:24:09.170543+0000","last_unstale":"2024-06-10T00:27:57.592095+0000","last_undegraded":"2024-06-10T00:27:57.592095+0000","last_fullsized":"2024-06-10T00:27:57.592095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:57:40.816585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461583+0000","last_change":"2024-06-10T00:23:54.423332+0000","last_active":"2024-06-10T00:27:57.461583+0000","last_peered":"2024-06-10T00:27:57.461583+0000","last_clean":"2024-06-10T00:27:57.461583+0000","last_became_active":"2024-06-10T00:23:54.422792+0000","last_became_peered":"2024-06-10T00:23:54.422792+0000","last_unstale":"2024-06-10T00:27:57.461583+0000","last_undegraded":"2024-06-10T00:27:57.461583+0000","last_fullsized":"2024-06-10T00:27:57.461583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:50:52.655641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"100'15","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591219+0000","last_change":"2024-06-10T00:23:57.551411+0000","last_active":"2024-06-10T00:27:57.591219+0000","last_peered":"2024-06-10T00:27:57.591219+0000","last_clean":"2024-06-10T00:27:57.591219+0000","last_became_active":"2024-06-10T00:23:57.551053+0000","last_became_peered":"2024-06-10T00:23:57.551053+0000","last_unstale":"2024-06-10T00:27:57.591219+0000","last_undegraded":"2024-06-10T00:27:57.591219+0000","last_fullsized":"2024-06-10T00:27:57.591219+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:16:18.082777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459282+0000","last_change":"2024-06-10T00:24:43.648767+0000","last_active":"2024-06-10T00:27:57.459282+0000","last_peered":"2024-06-10T00:27:57.459282+0000","last_clean":"2024-06-10T00:27:57.459282+0000","last_became_active":"2024-06-10T00:24:43.648267+0000","last_became_peered":"2024-06-10T00:24:43.648267+0000","last_unstale":"2024-06-10T00:27:57.459282+0000","last_undegraded":"2024-06-10T00:27:57.459282+0000","last_fullsized":"2024-06-10T00:27:57.459282+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:41:09.005202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"101'5","reported_seq":226,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.752230+0000","last_change":"2024-06-10T00:23:54.425963+0000","last_active":"2024-06-10T00:27:57.752230+0000","last_peered":"2024-06-10T00:27:57.752230+0000","last_clean":"2024-06-10T00:27:57.752230+0000","last_became_active":"2024-06-10T00:23:54.425581+0000","last_became_peered":"2024-06-10T00:23:54.425581+0000","last_unstale":"2024-06-10T00:27:57.752230+0000","last_undegraded":"2024-06-10T00:27:57.752230+0000","last_fullsized":"2024-06-10T00:27:57.752230+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:54.760610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":176,"num_read_kb":3,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592340+0000","last_change":"2024-06-10T00:23:57.557464+0000","last_active":"2024-06-10T00:27:57.592340+0000","last_peered":"2024-06-10T00:27:57.592340+0000","last_clean":"2024-06-10T00:27:57.592340+0000","last_became_active":"2024-06-10T00:23:57.557130+0000","last_became_peered":"2024-06-10T00:23:57.557130+0000","last_unstale":"2024-06-10T00:27:57.592340+0000","last_undegraded":"2024-06-10T00:27:57.592340+0000","last_fullsized":"2024-06-10T00:27:57.592340+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:16:36.341210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":38,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543379+0000","last_change":"2024-06-10T00:24:43.753291+0000","last_active":"2024-06-10T00:27:57.543379+0000","last_peered":"2024-06-10T00:27:57.543379+0000","last_clean":"2024-06-10T00:27:57.543379+0000","last_became_active":"2024-06-10T00:24:43.752765+0000","last_became_peered":"2024-06-10T00:24:43.752765+0000","last_unstale":"2024-06-10T00:27:57.543379+0000","last_undegraded":"2024-06-10T00:27:57.543379+0000","last_fullsized":"2024-06-10T00:27:57.543379+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:55:06.349564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"98'15","reported_seq":63,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570040+0000","last_change":"2024-06-10T00:23:54.823489+0000","last_active":"2024-06-10T00:27:57.570040+0000","last_peered":"2024-06-10T00:27:57.570040+0000","last_clean":"2024-06-10T00:27:57.570040+0000","last_became_active":"2024-06-10T00:23:54.823073+0000","last_became_peered":"2024-06-10T00:23:54.823073+0000","last_unstale":"2024-06-10T00:27:57.570040+0000","last_undegraded":"2024-06-10T00:27:57.570040+0000","last_fullsized":"2024-06-10T00:27:57.570040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:07:52.563009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461389+0000","last_change":"2024-06-10T00:23:57.563388+0000","last_active":"2024-06-10T00:27:57.461389+0000","last_peered":"2024-06-10T00:27:57.461389+0000","last_clean":"2024-06-10T00:27:57.461389+0000","last_became_active":"2024-06-10T00:23:57.563004+0000","last_became_peered":"2024-06-10T00:23:57.563004+0000","last_unstale":"2024-06-10T00:27:57.461389+0000","last_undegraded":"2024-06-10T00:27:57.461389+0000","last_fullsized":"2024-06-10T00:27:57.461389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:51:18.805662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461226+0000","last_change":"2024-06-10T00:24:09.168447+0000","last_active":"2024-06-10T00:27:57.461226+0000","last_peered":"2024-06-10T00:27:57.461226+0000","last_clean":"2024-06-10T00:27:57.461226+0000","last_became_active":"2024-06-10T00:24:09.167439+0000","last_became_peered":"2024-06-10T00:24:09.167439+0000","last_unstale":"2024-06-10T00:27:57.461226+0000","last_undegraded":"2024-06-10T00:27:57.461226+0000","last_fullsized":"2024-06-10T00:27:57.461226+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:52.319395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"101'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570752+0000","last_change":"2024-06-10T00:23:54.427666+0000","last_active":"2024-06-10T00:27:57.570752+0000","last_peered":"2024-06-10T00:27:57.570752+0000","last_clean":"2024-06-10T00:27:57.570752+0000","last_became_active":"2024-06-10T00:23:54.427285+0000","last_became_peered":"2024-06-10T00:23:54.427285+0000","last_unstale":"2024-06-10T00:27:57.570752+0000","last_undegraded":"2024-06-10T00:27:57.570752+0000","last_fullsized":"2024-06-10T00:27:57.570752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:28:20.196168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"100'5","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.558787+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.558470+0000","last_became_peered":"2024-06-10T00:23:57.558470+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:02:14.758802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570142+0000","last_change":"2024-06-10T00:24:09.264952+0000","last_active":"2024-06-10T00:27:57.570142+0000","last_peered":"2024-06-10T00:27:57.570142+0000","last_clean":"2024-06-10T00:27:57.570142+0000","last_became_active":"2024-06-10T00:24:09.264379+0000","last_became_peered":"2024-06-10T00:24:09.264379+0000","last_unstale":"2024-06-10T00:27:57.570142+0000","last_undegraded":"2024-06-10T00:27:57.570142+0000","last_fullsized":"2024-06-10T00:27:57.570142+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:25:40.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458121+0000","last_change":"2024-06-10T00:23:54.823298+0000","last_active":"2024-06-10T00:27:57.458121+0000","last_peered":"2024-06-10T00:27:57.458121+0000","last_clean":"2024-06-10T00:27:57.458121+0000","last_became_active":"2024-06-10T00:23:54.822917+0000","last_became_peered":"2024-06-10T00:23:54.822917+0000","last_unstale":"2024-06-10T00:27:57.458121+0000","last_undegraded":"2024-06-10T00:27:57.458121+0000","last_fullsized":"2024-06-10T00:27:57.458121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:07:52.500641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459177+0000","last_change":"2024-06-10T00:23:57.559202+0000","last_active":"2024-06-10T00:27:57.459177+0000","last_peered":"2024-06-10T00:27:57.459177+0000","last_clean":"2024-06-10T00:27:57.459177+0000","last_became_active":"2024-06-10T00:23:57.558800+0000","last_became_peered":"2024-06-10T00:23:57.558800+0000","last_unstale":"2024-06-10T00:27:57.459177+0000","last_undegraded":"2024-06-10T00:27:57.459177+0000","last_fullsized":"2024-06-10T00:27:57.459177+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:49:51.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542486+0000","last_change":"2024-06-10T00:24:09.168603+0000","last_active":"2024-06-10T00:27:57.542486+0000","last_peered":"2024-06-10T00:27:57.542486+0000","last_clean":"2024-06-10T00:27:57.542486+0000","last_became_active":"2024-06-10T00:24:09.168233+0000","last_became_peered":"2024-06-10T00:24:09.168233+0000","last_unstale":"2024-06-10T00:27:57.542486+0000","last_undegraded":"2024-06-10T00:27:57.542486+0000","last_fullsized":"2024-06-10T00:27:57.542486+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:59.585023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'1","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592210+0000","last_change":"2024-06-10T00:23:54.425505+0000","last_active":"2024-06-10T00:27:57.592210+0000","last_peered":"2024-06-10T00:27:57.592210+0000","last_clean":"2024-06-10T00:27:57.592210+0000","last_became_active":"2024-06-10T00:23:54.425058+0000","last_became_peered":"2024-06-10T00:23:54.425058+0000","last_unstale":"2024-06-10T00:27:57.592210+0000","last_undegraded":"2024-06-10T00:27:57.592210+0000","last_fullsized":"2024-06-10T00:27:57.592210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:48:05.605947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458827+0000","last_change":"2024-06-10T00:23:57.566060+0000","last_active":"2024-06-10T00:27:57.458827+0000","last_peered":"2024-06-10T00:27:57.458827+0000","last_clean":"2024-06-10T00:27:57.458827+0000","last_became_active":"2024-06-10T00:23:57.565099+0000","last_became_peered":"2024-06-10T00:23:57.565099+0000","last_unstale":"2024-06-10T00:27:57.458827+0000","last_undegraded":"2024-06-10T00:27:57.458827+0000","last_fullsized":"2024-06-10T00:27:57.458827+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:24:54.204399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543284+0000","last_change":"2024-06-10T00:24:09.264741+0000","last_active":"2024-06-10T00:27:57.543284+0000","last_peered":"2024-06-10T00:27:57.543284+0000","last_clean":"2024-06-10T00:27:57.543284+0000","last_became_active":"2024-06-10T00:24:09.264267+0000","last_became_peered":"2024-06-10T00:24:09.264267+0000","last_unstale":"2024-06-10T00:27:57.543284+0000","last_undegraded":"2024-06-10T00:27:57.543284+0000","last_fullsized":"2024-06-10T00:27:57.543284+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:27.627391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457224+0000","last_change":"2024-06-10T00:25:44.346070+0000","last_active":"2024-06-10T00:27:57.457224+0000","last_peered":"2024-06-10T00:27:57.457224+0000","last_clean":"2024-06-10T00:27:57.457224+0000","last_became_active":"2024-06-10T00:25:44.345798+0000","last_became_peered":"2024-06-10T00:25:44.345798+0000","last_unstale":"2024-06-10T00:27:57.457224+0000","last_undegraded":"2024-06-10T00:27:57.457224+0000","last_fullsized":"2024-06-10T00:27:57.457224+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:04.506535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"101'5","reported_seq":225,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542310+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:27:57.542310+0000","last_peered":"2024-06-10T00:27:57.542310+0000","last_clean":"2024-06-10T00:27:57.542310+0000","last_became_active":"2024-06-10T00:23:54.822442+0000","last_became_peered":"2024-06-10T00:23:54.822442+0000","last_unstale":"2024-06-10T00:27:57.542310+0000","last_undegraded":"2024-06-10T00:27:57.542310+0000","last_fullsized":"2024-06-10T00:27:57.542310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:51:10.278257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":173,"num_read_kb":173,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"98'1","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591953+0000","last_change":"2024-06-10T00:24:09.168073+0000","last_active":"2024-06-10T00:27:57.591953+0000","last_peered":"2024-06-10T00:27:57.591953+0000","last_clean":"2024-06-10T00:27:57.591953+0000","last_became_active":"2024-06-10T00:24:09.167657+0000","last_became_peered":"2024-06-10T00:24:09.167657+0000","last_unstale":"2024-06-10T00:27:57.591953+0000","last_undegraded":"2024-06-10T00:27:57.591953+0000","last_fullsized":"2024-06-10T00:27:57.591953+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:20:23.481685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457629+0000","last_change":"2024-06-10T00:25:44.346629+0000","last_active":"2024-06-10T00:27:57.457629+0000","last_peered":"2024-06-10T00:27:57.457629+0000","last_clean":"2024-06-10T00:27:57.457629+0000","last_became_active":"2024-06-10T00:25:44.346388+0000","last_became_peered":"2024-06-10T00:25:44.346388+0000","last_unstale":"2024-06-10T00:27:57.457629+0000","last_undegraded":"2024-06-10T00:27:57.457629+0000","last_fullsized":"2024-06-10T00:27:57.457629+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:03:19.228632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544918+0000","last_change":"2024-06-10T00:23:54.423507+0000","last_active":"2024-06-10T00:27:57.544918+0000","last_peered":"2024-06-10T00:27:57.544918+0000","last_clean":"2024-06-10T00:27:57.544918+0000","last_became_active":"2024-06-10T00:23:54.423092+0000","last_became_peered":"2024-06-10T00:23:54.423092+0000","last_unstale":"2024-06-10T00:27:57.544918+0000","last_undegraded":"2024-06-10T00:27:57.544918+0000","last_fullsized":"2024-06-10T00:27:57.544918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:06:15.924827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":380,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592578+0000","last_change":"2024-06-10T00:22:28.025243+0000","last_active":"2024-06-10T00:27:57.592578+0000","last_peered":"2024-06-10T00:27:57.592578+0000","last_clean":"2024-06-10T00:27:57.592578+0000","last_became_active":"2024-06-10T00:22:28.024751+0000","last_became_peered":"2024-06-10T00:22:28.024751+0000","last_unstale":"2024-06-10T00:27:57.592578+0000","last_undegraded":"2024-06-10T00:27:57.592578+0000","last_fullsized":"2024-06-10T00:27:57.592578+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_clean_scrub_stamp":"2024-06-10T00:19:05.039631+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:02:36.557931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461952+0000","last_change":"2024-06-10T00:24:09.265084+0000","last_active":"2024-06-10T00:27:57.461952+0000","last_peered":"2024-06-10T00:27:57.461952+0000","last_clean":"2024-06-10T00:27:57.461952+0000","last_became_active":"2024-06-10T00:24:09.264425+0000","last_became_peered":"2024-06-10T00:24:09.264425+0000","last_unstale":"2024-06-10T00:27:57.461952+0000","last_undegraded":"2024-06-10T00:27:57.461952+0000","last_fullsized":"2024-06-10T00:27:57.461952+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:37:31.577688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.541918+0000","last_change":"2024-06-10T00:23:57.558540+0000","last_active":"2024-06-10T00:27:57.541918+0000","last_peered":"2024-06-10T00:27:57.541918+0000","last_clean":"2024-06-10T00:27:57.541918+0000","last_became_active":"2024-06-10T00:23:57.558197+0000","last_became_peered":"2024-06-10T00:23:57.558197+0000","last_unstale":"2024-06-10T00:27:57.541918+0000","last_undegraded":"2024-06-10T00:27:57.541918+0000","last_fullsized":"2024-06-10T00:27:57.541918+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:04:25.566135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592707+0000","last_change":"2024-06-10T00:23:54.428126+0000","last_active":"2024-06-10T00:27:57.592707+0000","last_peered":"2024-06-10T00:27:57.592707+0000","last_clean":"2024-06-10T00:27:57.592707+0000","last_became_active":"2024-06-10T00:23:54.427630+0000","last_became_peered":"2024-06-10T00:23:54.427630+0000","last_unstale":"2024-06-10T00:27:57.592707+0000","last_undegraded":"2024-06-10T00:27:57.592707+0000","last_fullsized":"2024-06-10T00:27:57.592707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:05:23.679704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591957+0000","last_change":"2024-06-10T00:24:09.169984+0000","last_active":"2024-06-10T00:27:57.591957+0000","last_peered":"2024-06-10T00:27:57.591957+0000","last_clean":"2024-06-10T00:27:57.591957+0000","last_became_active":"2024-06-10T00:24:09.169556+0000","last_became_peered":"2024-06-10T00:24:09.169556+0000","last_unstale":"2024-06-10T00:27:57.591957+0000","last_undegraded":"2024-06-10T00:27:57.591957+0000","last_fullsized":"2024-06-10T00:27:57.591957+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:54:13.575207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"98'1","reported_seq":20,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591972+0000","last_change":"2024-06-10T00:24:43.767901+0000","last_active":"2024-06-10T00:27:57.591972+0000","last_peered":"2024-06-10T00:27:57.591972+0000","last_clean":"2024-06-10T00:27:57.591972+0000","last_became_active":"2024-06-10T00:24:43.767562+0000","last_became_peered":"2024-06-10T00:24:43.767562+0000","last_unstale":"2024-06-10T00:27:57.591972+0000","last_undegraded":"2024-06-10T00:27:57.591972+0000","last_fullsized":"2024-06-10T00:27:57.591972+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:46:36.099750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543049+0000","last_change":"2024-06-10T00:23:57.554906+0000","last_active":"2024-06-10T00:27:57.543049+0000","last_peered":"2024-06-10T00:27:57.543049+0000","last_clean":"2024-06-10T00:27:57.543049+0000","last_became_active":"2024-06-10T00:23:57.554608+0000","last_became_peered":"2024-06-10T00:23:57.554608+0000","last_unstale":"2024-06-10T00:27:57.543049+0000","last_undegraded":"2024-06-10T00:27:57.543049+0000","last_fullsized":"2024-06-10T00:27:57.543049+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:24.690594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570625+0000","last_change":"2024-06-10T00:24:09.264676+0000","last_active":"2024-06-10T00:27:57.570625+0000","last_peered":"2024-06-10T00:27:57.570625+0000","last_clean":"2024-06-10T00:27:57.570625+0000","last_became_active":"2024-06-10T00:24:09.264259+0000","last_became_peered":"2024-06-10T00:24:09.264259+0000","last_unstale":"2024-06-10T00:27:57.570625+0000","last_undegraded":"2024-06-10T00:27:57.570625+0000","last_fullsized":"2024-06-10T00:27:57.570625+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:37:53.787039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"101'48","reported_seq":118,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.689379+0000","last_change":"2024-06-10T00:23:54.424201+0000","last_active":"2024-06-10T00:27:57.689379+0000","last_peered":"2024-06-10T00:27:57.689379+0000","last_clean":"2024-06-10T00:27:57.689379+0000","last_became_active":"2024-06-10T00:23:54.423853+0000","last_became_peered":"2024-06-10T00:23:54.423853+0000","last_unstale":"2024-06-10T00:27:57.689379+0000","last_undegraded":"2024-06-10T00:27:57.689379+0000","last_fullsized":"2024-06-10T00:27:57.689379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:00.884041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":86,"num_write_kb":838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571556+0000","last_change":"2024-06-10T00:23:57.554276+0000","last_active":"2024-06-10T00:27:57.571556+0000","last_peered":"2024-06-10T00:27:57.571556+0000","last_clean":"2024-06-10T00:27:57.571556+0000","last_became_active":"2024-06-10T00:23:57.553964+0000","last_became_peered":"2024-06-10T00:23:57.553964+0000","last_unstale":"2024-06-10T00:27:57.571556+0000","last_undegraded":"2024-06-10T00:27:57.571556+0000","last_fullsized":"2024-06-10T00:27:57.571556+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:39:51.974770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591089+0000","last_change":"2024-06-10T00:24:09.170249+0000","last_active":"2024-06-10T00:27:57.591089+0000","last_peered":"2024-06-10T00:27:57.591089+0000","last_clean":"2024-06-10T00:27:57.591089+0000","last_became_active":"2024-06-10T00:24:09.169830+0000","last_became_peered":"2024-06-10T00:24:09.169830+0000","last_unstale":"2024-06-10T00:27:57.591089+0000","last_undegraded":"2024-06-10T00:27:57.591089+0000","last_fullsized":"2024-06-10T00:27:57.591089+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.320609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"98'4","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545304+0000","last_change":"2024-06-10T00:23:54.473943+0000","last_active":"2024-06-10T00:27:57.545304+0000","last_peered":"2024-06-10T00:27:57.545304+0000","last_clean":"2024-06-10T00:27:57.545304+0000","last_became_active":"2024-06-10T00:23:54.473652+0000","last_became_peered":"2024-06-10T00:23:54.473652+0000","last_unstale":"2024-06-10T00:27:57.545304+0000","last_undegraded":"2024-06-10T00:27:57.545304+0000","last_fullsized":"2024-06-10T00:27:57.545304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:52:06.773684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"100'7","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593713+0000","last_change":"2024-06-10T00:25:44.735050+0000","last_active":"2024-06-10T00:27:57.593713+0000","last_peered":"2024-06-10T00:27:57.593713+0000","last_clean":"2024-06-10T00:27:57.593713+0000","last_became_active":"2024-06-10T00:25:44.734756+0000","last_became_peered":"2024-06-10T00:25:44.734756+0000","last_unstale":"2024-06-10T00:27:57.593713+0000","last_undegraded":"2024-06-10T00:27:57.593713+0000","last_fullsized":"2024-06-10T00:27:57.593713+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:16:03.924238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545083+0000","last_change":"2024-06-10T00:24:43.768837+0000","last_active":"2024-06-10T00:27:57.545083+0000","last_peered":"2024-06-10T00:27:57.545083+0000","last_clean":"2024-06-10T00:27:57.545083+0000","last_became_active":"2024-06-10T00:24:43.768295+0000","last_became_peered":"2024-06-10T00:24:43.768295+0000","last_unstale":"2024-06-10T00:27:57.545083+0000","last_undegraded":"2024-06-10T00:27:57.545083+0000","last_fullsized":"2024-06-10T00:27:57.545083+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:59.600864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591709+0000","last_change":"2024-06-10T00:23:54.823448+0000","last_active":"2024-06-10T00:27:57.591709+0000","last_peered":"2024-06-10T00:27:57.591709+0000","last_clean":"2024-06-10T00:27:57.591709+0000","last_became_active":"2024-06-10T00:23:54.823107+0000","last_became_peered":"2024-06-10T00:23:54.823107+0000","last_unstale":"2024-06-10T00:27:57.591709+0000","last_undegraded":"2024-06-10T00:27:57.591709+0000","last_fullsized":"2024-06-10T00:27:57.591709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:50.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"100'2","reported_seq":43,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569862+0000","last_change":"2024-06-10T00:23:57.556733+0000","last_active":"2024-06-10T00:27:57.569862+0000","last_peered":"2024-06-10T00:27:57.569862+0000","last_clean":"2024-06-10T00:27:57.569862+0000","last_became_active":"2024-06-10T00:23:57.556386+0000","last_became_peered":"2024-06-10T00:23:57.556386+0000","last_unstale":"2024-06-10T00:27:57.569862+0000","last_undegraded":"2024-06-10T00:27:57.569862+0000","last_fullsized":"2024-06-10T00:27:57.569862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:39:00.572311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458015+0000","last_change":"2024-06-10T00:24:09.174600+0000","last_active":"2024-06-10T00:27:57.458015+0000","last_peered":"2024-06-10T00:27:57.458015+0000","last_clean":"2024-06-10T00:27:57.458015+0000","last_became_active":"2024-06-10T00:24:09.174236+0000","last_became_peered":"2024-06-10T00:24:09.174236+0000","last_unstale":"2024-06-10T00:27:57.458015+0000","last_undegraded":"2024-06-10T00:27:57.458015+0000","last_fullsized":"2024-06-10T00:27:57.458015+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:58:18.447768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571190+0000","last_change":"2024-06-10T00:23:54.425111+0000","last_active":"2024-06-10T00:27:57.571190+0000","last_peered":"2024-06-10T00:27:57.571190+0000","last_clean":"2024-06-10T00:27:57.571190+0000","last_became_active":"2024-06-10T00:23:54.424724+0000","last_became_peered":"2024-06-10T00:23:54.424724+0000","last_unstale":"2024-06-10T00:27:57.571190+0000","last_undegraded":"2024-06-10T00:27:57.571190+0000","last_fullsized":"2024-06-10T00:27:57.571190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:54.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"100'9","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.556355+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.555479+0000","last_became_peered":"2024-06-10T00:23:57.555479+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:08.817381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"98'1","reported_seq":30,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544968+0000","last_change":"2024-06-10T00:24:09.169589+0000","last_active":"2024-06-10T00:27:57.544968+0000","last_peered":"2024-06-10T00:27:57.544968+0000","last_clean":"2024-06-10T00:27:57.544968+0000","last_became_active":"2024-06-10T00:24:09.168436+0000","last_became_peered":"2024-06-10T00:24:09.168436+0000","last_unstale":"2024-06-10T00:27:57.544968+0000","last_undegraded":"2024-06-10T00:27:57.544968+0000","last_fullsized":"2024-06-10T00:27:57.544968+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:17:32.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"98'1","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591500+0000","last_change":"2024-06-10T00:24:43.751399+0000","last_active":"2024-06-10T00:27:57.591500+0000","last_peered":"2024-06-10T00:27:57.591500+0000","last_clean":"2024-06-10T00:27:57.591500+0000","last_became_active":"2024-06-10T00:24:43.750987+0000","last_became_peered":"2024-06-10T00:24:43.750987+0000","last_unstale":"2024-06-10T00:27:57.591500+0000","last_undegraded":"2024-06-10T00:27:57.591500+0000","last_fullsized":"2024-06-10T00:27:57.591500+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:14:47.902017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570338+0000","last_change":"2024-06-10T00:23:57.560213+0000","last_active":"2024-06-10T00:27:57.570338+0000","last_peered":"2024-06-10T00:27:57.570338+0000","last_clean":"2024-06-10T00:27:57.570338+0000","last_became_active":"2024-06-10T00:23:57.559925+0000","last_became_peered":"2024-06-10T00:23:57.559925+0000","last_unstale":"2024-06-10T00:27:57.570338+0000","last_undegraded":"2024-06-10T00:27:57.570338+0000","last_fullsized":"2024-06-10T00:27:57.570338+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:00:37.403087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545095+0000","last_change":"2024-06-10T00:24:09.264758+0000","last_active":"2024-06-10T00:27:57.545095+0000","last_peered":"2024-06-10T00:27:57.545095+0000","last_clean":"2024-06-10T00:27:57.545095+0000","last_became_active":"2024-06-10T00:24:09.263928+0000","last_became_peered":"2024-06-10T00:24:09.263928+0000","last_unstale":"2024-06-10T00:27:57.545095+0000","last_undegraded":"2024-06-10T00:27:57.545095+0000","last_fullsized":"2024-06-10T00:27:57.545095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:23:41.083091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570103+0000","last_change":"2024-06-10T00:23:54.428553+0000","last_active":"2024-06-10T00:27:57.570103+0000","last_peered":"2024-06-10T00:27:57.570103+0000","last_clean":"2024-06-10T00:27:57.570103+0000","last_became_active":"2024-06-10T00:23:54.428108+0000","last_became_peered":"2024-06-10T00:23:54.428108+0000","last_unstale":"2024-06-10T00:27:57.570103+0000","last_undegraded":"2024-06-10T00:27:57.570103+0000","last_fullsized":"2024-06-10T00:27:57.570103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:57:26.207915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591119+0000","last_change":"2024-06-10T00:23:57.560686+0000","last_active":"2024-06-10T00:27:57.591119+0000","last_peered":"2024-06-10T00:27:57.591119+0000","last_clean":"2024-06-10T00:27:57.591119+0000","last_became_active":"2024-06-10T00:23:57.560326+0000","last_became_peered":"2024-06-10T00:23:57.560326+0000","last_unstale":"2024-06-10T00:27:57.591119+0000","last_undegraded":"2024-06-10T00:27:57.591119+0000","last_fullsized":"2024-06-10T00:27:57.591119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:08:39.506567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591281+0000","last_change":"2024-06-10T00:24:09.166947+0000","last_active":"2024-06-10T00:27:57.591281+0000","last_peered":"2024-06-10T00:27:57.591281+0000","last_clean":"2024-06-10T00:27:57.591281+0000","last_became_active":"2024-06-10T00:24:09.166503+0000","last_became_peered":"2024-06-10T00:24:09.166503+0000","last_unstale":"2024-06-10T00:27:57.591281+0000","last_undegraded":"2024-06-10T00:27:57.591281+0000","last_fullsized":"2024-06-10T00:27:57.591281+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:16:45.094642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591369+0000","last_change":"2024-06-10T00:23:54.823522+0000","last_active":"2024-06-10T00:27:57.591369+0000","last_peered":"2024-06-10T00:27:57.591369+0000","last_clean":"2024-06-10T00:27:57.591369+0000","last_became_active":"2024-06-10T00:23:54.823142+0000","last_became_peered":"2024-06-10T00:23:54.823142+0000","last_unstale":"2024-06-10T00:27:57.591369+0000","last_undegraded":"2024-06-10T00:27:57.591369+0000","last_fullsized":"2024-06-10T00:27:57.591369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:03.836850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592288+0000","last_change":"2024-06-10T00:23:57.563710+0000","last_active":"2024-06-10T00:27:57.592288+0000","last_peered":"2024-06-10T00:27:57.592288+0000","last_clean":"2024-06-10T00:27:57.592288+0000","last_became_active":"2024-06-10T00:23:57.563331+0000","last_became_peered":"2024-06-10T00:23:57.563331+0000","last_unstale":"2024-06-10T00:27:57.592288+0000","last_undegraded":"2024-06-10T00:27:57.592288+0000","last_fullsized":"2024-06-10T00:27:57.592288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:08:48.061885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592820+0000","last_change":"2024-06-10T00:24:09.171203+0000","last_active":"2024-06-10T00:27:57.592820+0000","last_peered":"2024-06-10T00:27:57.592820+0000","last_clean":"2024-06-10T00:27:57.592820+0000","last_became_active":"2024-06-10T00:24:09.170824+0000","last_became_peered":"2024-06-10T00:24:09.170824+0000","last_unstale":"2024-06-10T00:27:57.592820+0000","last_undegraded":"2024-06-10T00:27:57.592820+0000","last_fullsized":"2024-06-10T00:27:57.592820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:47:28.045602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"101'39","reported_seq":94,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.686846+0000","last_change":"2024-06-10T00:23:54.421307+0000","last_active":"2024-06-10T00:27:57.686846+0000","last_peered":"2024-06-10T00:27:57.686846+0000","last_clean":"2024-06-10T00:27:57.686846+0000","last_became_active":"2024-06-10T00:23:54.420928+0000","last_became_peered":"2024-06-10T00:23:54.420928+0000","last_unstale":"2024-06-10T00:27:57.686846+0000","last_undegraded":"2024-06-10T00:27:57.686846+0000","last_fullsized":"2024-06-10T00:27:57.686846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:59:41.433605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591623+0000","last_change":"2024-06-10T00:23:57.556494+0000","last_active":"2024-06-10T00:27:57.591623+0000","last_peered":"2024-06-10T00:27:57.591623+0000","last_clean":"2024-06-10T00:27:57.591623+0000","last_became_active":"2024-06-10T00:23:57.556088+0000","last_became_peered":"2024-06-10T00:23:57.556088+0000","last_unstale":"2024-06-10T00:27:57.591623+0000","last_undegraded":"2024-06-10T00:27:57.591623+0000","last_fullsized":"2024-06-10T00:27:57.591623+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:06:23.873281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461068+0000","last_change":"2024-06-10T00:24:09.166734+0000","last_active":"2024-06-10T00:27:57.461068+0000","last_peered":"2024-06-10T00:27:57.461068+0000","last_clean":"2024-06-10T00:27:57.461068+0000","last_became_active":"2024-06-10T00:24:09.166270+0000","last_became_peered":"2024-06-10T00:24:09.166270+0000","last_unstale":"2024-06-10T00:27:57.461068+0000","last_undegraded":"2024-06-10T00:27:57.461068+0000","last_fullsized":"2024-06-10T00:27:57.461068+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:38.039633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543256+0000","last_change":"2024-06-10T00:23:54.426226+0000","last_active":"2024-06-10T00:27:57.543256+0000","last_peered":"2024-06-10T00:27:57.543256+0000","last_clean":"2024-06-10T00:27:57.543256+0000","last_became_active":"2024-06-10T00:23:54.425879+0000","last_became_peered":"2024-06-10T00:23:54.425879+0000","last_unstale":"2024-06-10T00:27:57.543256+0000","last_undegraded":"2024-06-10T00:27:57.543256+0000","last_fullsized":"2024-06-10T00:27:57.543256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:15:32.936556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592043+0000","last_change":"2024-06-10T00:23:57.553814+0000","last_active":"2024-06-10T00:27:57.592043+0000","last_peered":"2024-06-10T00:27:57.592043+0000","last_clean":"2024-06-10T00:27:57.592043+0000","last_became_active":"2024-06-10T00:23:57.553478+0000","last_became_peered":"2024-06-10T00:23:57.553478+0000","last_unstale":"2024-06-10T00:27:57.592043+0000","last_undegraded":"2024-06-10T00:27:57.592043+0000","last_fullsized":"2024-06-10T00:27:57.592043+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:54:39.264526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"100'9","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591390+0000","last_change":"2024-06-10T00:23:57.556676+0000","last_active":"2024-06-10T00:27:57.591390+0000","last_peered":"2024-06-10T00:27:57.591390+0000","last_clean":"2024-06-10T00:27:57.591390+0000","last_became_active":"2024-06-10T00:23:57.555946+0000","last_became_peered":"2024-06-10T00:23:57.555946+0000","last_unstale":"2024-06-10T00:27:57.591390+0000","last_undegraded":"2024-06-10T00:27:57.591390+0000","last_fullsized":"2024-06-10T00:27:57.591390+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:10:10.868799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"100'12","reported_seq":53,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590743+0000","last_change":"2024-06-10T00:23:57.565423+0000","last_active":"2024-06-10T00:27:57.590743+0000","last_peered":"2024-06-10T00:27:57.590743+0000","last_clean":"2024-06-10T00:27:57.590743+0000","last_became_active":"2024-06-10T00:23:57.564830+0000","last_became_peered":"2024-06-10T00:23:57.564830+0000","last_unstale":"2024-06-10T00:27:57.590743+0000","last_undegraded":"2024-06-10T00:27:57.590743+0000","last_fullsized":"2024-06-10T00:27:57.590743+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:11:06.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590982+0000","last_change":"2024-06-10T00:23:57.558971+0000","last_active":"2024-06-10T00:27:57.590982+0000","last_peered":"2024-06-10T00:27:57.590982+0000","last_clean":"2024-06-10T00:27:57.590982+0000","last_became_active":"2024-06-10T00:23:57.558620+0000","last_became_peered":"2024-06-10T00:23:57.558620+0000","last_unstale":"2024-06-10T00:27:57.590982+0000","last_undegraded":"2024-06-10T00:27:57.590982+0000","last_fullsized":"2024-06-10T00:27:57.590982+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:45:43.308008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591491+0000","last_change":"2024-06-10T00:23:57.552341+0000","last_active":"2024-06-10T00:27:57.591491+0000","last_peered":"2024-06-10T00:27:57.591491+0000","last_clean":"2024-06-10T00:27:57.591491+0000","last_became_active":"2024-06-10T00:23:57.551464+0000","last_became_peered":"2024-06-10T00:23:57.551464+0000","last_unstale":"2024-06-10T00:27:57.591491+0000","last_undegraded":"2024-06-10T00:27:57.591491+0000","last_fullsized":"2024-06-10T00:27:57.591491+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:44:41.063516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"100'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543209+0000","last_change":"2024-06-10T00:24:43.768961+0000","last_active":"2024-06-10T00:27:57.543209+0000","last_peered":"2024-06-10T00:27:57.543209+0000","last_clean":"2024-06-10T00:27:57.543209+0000","last_became_active":"2024-06-10T00:24:43.768542+0000","last_became_peered":"2024-06-10T00:24:43.768542+0000","last_unstale":"2024-06-10T00:27:57.543209+0000","last_undegraded":"2024-06-10T00:27:57.543209+0000","last_fullsized":"2024-06-10T00:27:57.543209+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:23:58.412054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"100'6","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591613+0000","last_change":"2024-06-10T00:23:57.554924+0000","last_active":"2024-06-10T00:27:57.591613+0000","last_peered":"2024-06-10T00:27:57.591613+0000","last_clean":"2024-06-10T00:27:57.591613+0000","last_became_active":"2024-06-10T00:23:57.554515+0000","last_became_peered":"2024-06-10T00:23:57.554515+0000","last_unstale":"2024-06-10T00:27:57.591613+0000","last_undegraded":"2024-06-10T00:27:57.591613+0000","last_fullsized":"2024-06-10T00:27:57.591613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:58:15.624775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"100'15","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591485+0000","last_change":"2024-06-10T00:23:57.561940+0000","last_active":"2024-06-10T00:27:57.591485+0000","last_peered":"2024-06-10T00:27:57.591485+0000","last_clean":"2024-06-10T00:27:57.591485+0000","last_became_active":"2024-06-10T00:23:57.561512+0000","last_became_peered":"2024-06-10T00:23:57.561512+0000","last_unstale":"2024-06-10T00:27:57.591485+0000","last_undegraded":"2024-06-10T00:27:57.591485+0000","last_fullsized":"2024-06-10T00:27:57.591485+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:57:57.147311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"100'2","reported_seq":43,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570333+0000","last_change":"2024-06-10T00:23:57.557229+0000","last_active":"2024-06-10T00:27:57.570333+0000","last_peered":"2024-06-10T00:27:57.570333+0000","last_clean":"2024-06-10T00:27:57.570333+0000","last_became_active":"2024-06-10T00:23:57.556905+0000","last_became_peered":"2024-06-10T00:23:57.556905+0000","last_unstale":"2024-06-10T00:27:57.570333+0000","last_undegraded":"2024-06-10T00:27:57.570333+0000","last_fullsized":"2024-06-10T00:27:57.570333+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:30.513517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"100'6","reported_seq":24,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592589+0000","last_change":"2024-06-10T00:24:43.768509+0000","last_active":"2024-06-10T00:27:57.592589+0000","last_peered":"2024-06-10T00:27:57.592589+0000","last_clean":"2024-06-10T00:27:57.592589+0000","last_became_active":"2024-06-10T00:24:43.768038+0000","last_became_peered":"2024-06-10T00:24:43.768038+0000","last_unstale":"2024-06-10T00:27:57.592589+0000","last_undegraded":"2024-06-10T00:27:57.592589+0000","last_fullsized":"2024-06-10T00:27:57.592589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:38.344375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571244+0000","last_change":"2024-06-10T00:23:57.557830+0000","last_active":"2024-06-10T00:27:57.571244+0000","last_peered":"2024-06-10T00:27:57.571244+0000","last_clean":"2024-06-10T00:27:57.571244+0000","last_became_active":"2024-06-10T00:23:57.556986+0000","last_became_peered":"2024-06-10T00:23:57.556986+0000","last_unstale":"2024-06-10T00:27:57.571244+0000","last_undegraded":"2024-06-10T00:27:57.571244+0000","last_fullsized":"2024-06-10T00:27:57.571244+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:53.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590760+0000","last_change":"2024-06-10T00:23:57.565602+0000","last_active":"2024-06-10T00:27:57.590760+0000","last_peered":"2024-06-10T00:27:57.590760+0000","last_clean":"2024-06-10T00:27:57.590760+0000","last_became_active":"2024-06-10T00:23:57.565264+0000","last_became_peered":"2024-06-10T00:23:57.565264+0000","last_unstale":"2024-06-10T00:27:57.590760+0000","last_undegraded":"2024-06-10T00:27:57.590760+0000","last_fullsized":"2024-06-10T00:27:57.590760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:09:12.773087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542211+0000","last_change":"2024-06-10T00:23:57.548516+0000","last_active":"2024-06-10T00:27:57.542211+0000","last_peered":"2024-06-10T00:27:57.542211+0000","last_clean":"2024-06-10T00:27:57.542211+0000","last_became_active":"2024-06-10T00:23:57.548202+0000","last_became_peered":"2024-06-10T00:23:57.548202+0000","last_unstale":"2024-06-10T00:27:57.542211+0000","last_undegraded":"2024-06-10T00:27:57.542211+0000","last_fullsized":"2024-06-10T00:27:57.542211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:13.901478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591566+0000","last_change":"2024-06-10T00:23:57.565900+0000","last_active":"2024-06-10T00:27:57.591566+0000","last_peered":"2024-06-10T00:27:57.591566+0000","last_clean":"2024-06-10T00:27:57.591566+0000","last_became_active":"2024-06-10T00:23:57.565573+0000","last_became_peered":"2024-06-10T00:23:57.565573+0000","last_unstale":"2024-06-10T00:27:57.591566+0000","last_undegraded":"2024-06-10T00:27:57.591566+0000","last_fullsized":"2024-06-10T00:27:57.591566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:08.255905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107100,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599800,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1163,"num_read_kb":693,"num_write":299,"num_write_kb":1447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":1799400,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351040,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1118208,"data_stored":582820,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612677,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32432,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712720,"statfs":{"total":95995035648,"available":95961825280,"internally_reserved":0,"allocated":1286144,"data_stored":1371811,"data_compressed":77135,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776203,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1159168,"data_stored":980346,"data_compressed":54425,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037839,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28144,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717008,"statfs":{"total":95995035648,"available":95966216192,"internally_reserved":0,"allocated":1150976,"data_stored":976148,"data_compressed":54425,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299476,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":1179311,"data_compressed":70979,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463001,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1110016,"data_stored":578210,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1183744,"data_stored":647485,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":579598,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411438,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36504,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708648,"statfs":{"total":95995035648,"available":95957655552,"internally_reserved":0,"allocated":1196032,"data_stored":638999,"data_compressed":32301,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28168,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716984,"statfs":{"total":95995035648,"available":95966191616,"internally_reserved":0,"allocated":1167360,"data_stored":1103220,"data_compressed":60581,"data_compressed_allocated":81920,"data_compressed_original":614400,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888163,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":583640,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724638,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32356,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712796,"statfs":{"total":95995035648,"available":95961903104,"internally_reserved":0,"allocated":1208320,"data_stored":1306889,"data_compressed":77135,"data_compressed_allocated":102400,"data_compressed_original":811008,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134305,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":203323,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-10T00:28:01.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:28:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi135 to 176.2M 2024-06-10T00:28:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: Unable to set osd_memory_target on smithi135 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-06-10T00:28:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-06-10T00:28:02.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: Unable to set osd_memory_target on smithi193 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-06-10T00:28:02.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:02.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:01 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi135 to 176.2M 2024-06-10T00:28:02.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: Unable to set osd_memory_target on smithi135 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-06-10T00:28:02.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:02.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-06-10T00:28:02.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: Unable to set osd_memory_target on smithi193 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-06-10T00:28:02.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:02.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:01 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:02.187 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:28:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-06-10T00:28:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:03.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:03.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:03 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:28:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-06-10T00:28:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:03.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:03 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:28:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-06-10T00:28:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:28:03.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:03.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:03 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:04.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.152+0000 7f5c6ed19700 1 -- 172.21.15.121:0/414540681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c681010b0 msgr2=0x7f5c68101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:04.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.152+0000 7f5c6ed19700 1 --2- 172.21.15.121:0/414540681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c681010b0 0x7f5c68101490 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400aaa0 tx=0x7f5c6400adb0 comp rx=0 tx=0).stop 2024-06-10T00:28:04.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.152+0000 7f5c6ed19700 1 -- 172.21.15.121:0/414540681 shutdown_connections 2024-06-10T00:28:04.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.152+0000 7f5c6ed19700 1 --2- 172.21.15.121:0/414540681 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c68105f50 0x7f5c6810da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:04.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.152+0000 7f5c6ed19700 1 --2- 172.21.15.121:0/414540681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c681010b0 0x7f5c68101490 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:04.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.152+0000 7f5c6ed19700 1 --2- 172.21.15.121:0/414540681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c68101a60 0x7f5c68105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:04.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.152+0000 7f5c6ed19700 1 -- 172.21.15.121:0/414540681 >> 172.21.15.121:0/414540681 conn(0x7f5c680766b0 msgr2=0x7f5c68076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:04.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.152+0000 7f5c6ed19700 1 -- 172.21.15.121:0/414540681 shutdown_connections 2024-06-10T00:28:04.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.152+0000 7f5c6ed19700 1 -- 172.21.15.121:0/414540681 wait complete. 2024-06-10T00:28:04.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6ed19700 1 Processor -- start 2024-06-10T00:28:04.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6ed19700 1 -- start start 2024-06-10T00:28:04.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6ed19700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c681010b0 0x7f5c6819e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:04.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6ed19700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c68101a60 0x7f5c6819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:04.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6ed19700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c68105f50 0x7f5c68198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:04.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6ed19700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5c68110650 con 0x7f5c68105f50 2024-06-10T00:28:04.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6ed19700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5c681104d0 con 0x7f5c68101a60 2024-06-10T00:28:04.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6ed19700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5c681107d0 con 0x7f5c681010b0 2024-06-10T00:28:04.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6d2b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c68105f50 0x7f5c68198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:04.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6cab5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c681010b0 0x7f5c6819e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:04.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6d2b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c68105f50 0x7f5c68198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41484/0 (socket says 172.21.15.121:41484) 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6d2b6700 1 -- 172.21.15.121:0/1605229263 learned_addr learned my addr 172.21.15.121:0/1605229263 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c5ffff700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c68101a60 0x7f5c6819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.153+0000 7f5c6d2b6700 1 -- 172.21.15.121:0/1605229263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c681010b0 msgr2=0x7f5c6819e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.154+0000 7f5c6d2b6700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c681010b0 0x7f5c6819e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.154+0000 7f5c6d2b6700 1 -- 172.21.15.121:0/1605229263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c68101a60 msgr2=0x7f5c6819edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.154+0000 7f5c6d2b6700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c68101a60 0x7f5c6819edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.154+0000 7f5c6d2b6700 1 -- 172.21.15.121:0/1605229263 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c681991d0 con 0x7f5c68105f50 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.154+0000 7f5c6d2b6700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c68105f50 0x7f5c68198a70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000ef30 tx=0x7f5c6000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.154+0000 7f5c5dffb700 1 -- 172.21.15.121:0/1605229263 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c600092e0 con 0x7f5c68105f50 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.154+0000 7f5c5dffb700 1 -- 172.21.15.121:0/1605229263 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c60004bb0 con 0x7f5c68105f50 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.154+0000 7f5c6ed19700 1 -- 172.21.15.121:0/1605229263 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c68199460 con 0x7f5c68105f50 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.154+0000 7f5c5dffb700 1 -- 172.21.15.121:0/1605229263 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c60004280 con 0x7f5c68105f50 2024-06-10T00:28:04.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.154+0000 7f5c6ed19700 1 -- 172.21.15.121:0/1605229263 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c681a5780 con 0x7f5c68105f50 2024-06-10T00:28:04.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.155+0000 7f5c6ed19700 1 -- 172.21.15.121:0/1605229263 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c68069180 con 0x7f5c68105f50 2024-06-10T00:28:04.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.156+0000 7f5c5dffb700 1 -- 172.21.15.121:0/1605229263 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f5c6001b030 con 0x7f5c68105f50 2024-06-10T00:28:04.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.157+0000 7f5c5dffb700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5c500a2e20 0x7f5c500a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:04.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.157+0000 7f5c6cab5700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5c500a2e20 0x7f5c500a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:04.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.157+0000 7f5c5dffb700 1 -- 172.21.15.121:0/1605229263 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f5c600cf3e0 con 0x7f5c68105f50 2024-06-10T00:28:04.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.157+0000 7f5c6cab5700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5c500a2e20 0x7f5c500a52e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400c6d0 tx=0x7f5c64005960 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:04.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.161+0000 7f5c5dffb700 1 -- 172.21.15.121:0/1605229263 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5c60022020 con 0x7f5c68105f50 2024-06-10T00:28:04.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.401+0000 7f5c6ed19700 1 -- 172.21.15.121:0/1605229263 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c6802dca0 con 0x7f5c68105f50 2024-06-10T00:28:04.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.402+0000 7f5c5dffb700 1 -- 172.21.15.121:0/1605229263 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f5c60010040 con 0x7f5c68105f50 2024-06-10T00:28:04.402 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:04.402 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:27:55.370804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:28:04.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 -- 172.21.15.121:0/1605229263 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5c500a2e20 msgr2=0x7f5c500a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:04.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5c500a2e20 0x7f5c500a52e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400c6d0 tx=0x7f5c64005960 comp rx=0 tx=0).stop 2024-06-10T00:28:04.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 -- 172.21.15.121:0/1605229263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c68105f50 msgr2=0x7f5c68198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:04.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c68105f50 0x7f5c68198a70 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000ef30 tx=0x7f5c6000c3a0 comp rx=0 tx=0).stop 2024-06-10T00:28:04.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 -- 172.21.15.121:0/1605229263 shutdown_connections 2024-06-10T00:28:04.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5c500a2e20 0x7f5c500a52e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:04.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c68105f50 0x7f5c68198a70 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:04.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c681010b0 0x7f5c6819e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:04.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 --2- 172.21.15.121:0/1605229263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5c68101a60 0x7f5c6819edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:04.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 -- 172.21.15.121:0/1605229263 >> 172.21.15.121:0/1605229263 conn(0x7f5c680766b0 msgr2=0x7f5c680ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:04.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 -- 172.21.15.121:0/1605229263 shutdown_connections 2024-06-10T00:28:04.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:04.404+0000 7f5c6ed19700 1 -- 172.21.15.121:0/1605229263 wait complete. 2024-06-10T00:28:04.405 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-06-10T00:28:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi121 to 176.1M 2024-06-10T00:28:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: Unable to set osd_memory_target on smithi121 to 184674918: error parsing value: Value '184674918' is below minimum 939524096 2024-06-10T00:28:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: Updating smithi121:/etc/ceph/ceph.conf 2024-06-10T00:28:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: Updating smithi135:/etc/ceph/ceph.conf 2024-06-10T00:28:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: Updating smithi193:/etc/ceph/ceph.conf 2024-06-10T00:28:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: Updating smithi121:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: Updating smithi193:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: Updating smithi135:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:04.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:04.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:04.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:04 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi121 to 176.1M 2024-06-10T00:28:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: Unable to set osd_memory_target on smithi121 to 184674918: error parsing value: Value '184674918' is below minimum 939524096 2024-06-10T00:28:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: Updating smithi121:/etc/ceph/ceph.conf 2024-06-10T00:28:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: Updating smithi135:/etc/ceph/ceph.conf 2024-06-10T00:28:04.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: Updating smithi193:/etc/ceph/ceph.conf 2024-06-10T00:28:04.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: Updating smithi121:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:28:04.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: Updating smithi193:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:28:04.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: Updating smithi135:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:28:04.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:28:04.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:28:04.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:28:04.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:04.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:04.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:04.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:04 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi121 to 176.1M 2024-06-10T00:28:04.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: Unable to set osd_memory_target on smithi121 to 184674918: error parsing value: Value '184674918' is below minimum 939524096 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: Updating smithi121:/etc/ceph/ceph.conf 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: Updating smithi135:/etc/ceph/ceph.conf 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: Updating smithi193:/etc/ceph/ceph.conf 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: Updating smithi121:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: Updating smithi193:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: Updating smithi135:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: Updating smithi135:/etc/ceph/ceph.client.admin.keyring 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:04.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:04.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:04.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:04.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:04 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:05.118 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:28:05.360 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:05 smithi121 ceph-mon[32378]: Updating smithi121:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:28:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:05 smithi121 ceph-mon[32378]: Updating smithi135:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:28:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:05 smithi121 ceph-mon[32378]: Updating smithi193:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:28:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:05 smithi121 ceph-mon[32378]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-06-10T00:28:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:05 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1605229263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:05.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:05 smithi135 ceph-mon[36424]: Updating smithi121:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:28:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:05 smithi135 ceph-mon[36424]: Updating smithi135:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:28:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:05 smithi135 ceph-mon[36424]: Updating smithi193:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:28:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:05 smithi135 ceph-mon[36424]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-06-10T00:28:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:05 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1605229263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:05 smithi193 ceph-mon[36716]: Updating smithi121:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:28:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:05 smithi193 ceph-mon[36716]: Updating smithi135:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:28:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:05 smithi193 ceph-mon[36716]: Updating smithi193:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-10T00:28:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:05 smithi193 ceph-mon[36716]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 4 op/s 2024-06-10T00:28:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:05 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1605229263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:07.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.138+0000 7f4186c2b700 1 -- 172.21.15.121:0/446263901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4180101610 msgr2=0x7f41801019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:07.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.138+0000 7f4186c2b700 1 --2- 172.21.15.121:0/446263901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4180101610 0x7f41801019f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f41740099c0 tx=0x7f4174009cd0 comp rx=0 tx=0).stop 2024-06-10T00:28:07.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.138+0000 7f4186c2b700 1 -- 172.21.15.121:0/446263901 shutdown_connections 2024-06-10T00:28:07.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.138+0000 7f4186c2b700 1 --2- 172.21.15.121:0/446263901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41801066f0 0x7f418010e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:07.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.138+0000 7f4186c2b700 1 --2- 172.21.15.121:0/446263901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4180101610 0x7f41801019f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:07.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.138+0000 7f4186c2b700 1 --2- 172.21.15.121:0/446263901 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4180101fc0 0x7f4180106010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:07.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.138+0000 7f4186c2b700 1 -- 172.21.15.121:0/446263901 >> 172.21.15.121:0/446263901 conn(0x7f4180078430 msgr2=0x7f4180078840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:07.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.138+0000 7f4186c2b700 1 -- 172.21.15.121:0/446263901 shutdown_connections 2024-06-10T00:28:07.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.138+0000 7f4186c2b700 1 -- 172.21.15.121:0/446263901 wait complete. 2024-06-10T00:28:07.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.139+0000 7f4186c2b700 1 Processor -- start 2024-06-10T00:28:07.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.139+0000 7f4186c2b700 1 -- start start 2024-06-10T00:28:07.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.139+0000 7f4186c2b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4180101610 0x7f4180199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:07.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.139+0000 7f4186c2b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4180101fc0 0x7f418019a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:07.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.139+0000 7f4186c2b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41801066f0 0x7f41801a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:07.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.139+0000 7f4186c2b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4180111700 con 0x7f4180101610 2024-06-10T00:28:07.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.139+0000 7f4186c2b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4180111580 con 0x7f41801066f0 2024-06-10T00:28:07.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.139+0000 7f4186c2b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4180111880 con 0x7f4180101fc0 2024-06-10T00:28:07.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f41849c7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4180101610 0x7f4180199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:07.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f41849c7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4180101610 0x7f4180199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41506/0 (socket says 172.21.15.121:41506) 2024-06-10T00:28:07.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f41849c7700 1 -- 172.21.15.121:0/307228012 learned_addr learned my addr 172.21.15.121:0/307228012 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:07.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f41851c8700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41801066f0 0x7f41801a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:07.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f41849c7700 1 -- 172.21.15.121:0/307228012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4180101fc0 msgr2=0x7f418019a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:07.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f41849c7700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4180101fc0 0x7f418019a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:07.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f41849c7700 1 -- 172.21.15.121:0/307228012 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41801066f0 msgr2=0x7f41801a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:07.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f41849c7700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41801066f0 0x7f41801a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:07.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f41849c7700 1 -- 172.21.15.121:0/307228012 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41801a50a0 con 0x7f4180101610 2024-06-10T00:28:07.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f41849c7700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4180101610 0x7f4180199ae0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4174006010 tx=0x7f4174009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:07.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f417dffb700 1 -- 172.21.15.121:0/307228012 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174027070 con 0x7f4180101610 2024-06-10T00:28:07.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f417dffb700 1 -- 172.21.15.121:0/307228012 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4174004050 con 0x7f4180101610 2024-06-10T00:28:07.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f4186c2b700 1 -- 172.21.15.121:0/307228012 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41801a5270 con 0x7f4180101610 2024-06-10T00:28:07.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f417dffb700 1 -- 172.21.15.121:0/307228012 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174022e10 con 0x7f4180101610 2024-06-10T00:28:07.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.140+0000 7f4186c2b700 1 -- 172.21.15.121:0/307228012 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41801a56c0 con 0x7f4180101610 2024-06-10T00:28:07.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.141+0000 7f4186c2b700 1 -- 172.21.15.121:0/307228012 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4180069180 con 0x7f4180101610 2024-06-10T00:28:07.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.143+0000 7f417dffb700 1 -- 172.21.15.121:0/307228012 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f417402a070 con 0x7f4180101610 2024-06-10T00:28:07.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.143+0000 7f417dffb700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41700a2e20 0x7f41700a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:07.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.143+0000 7f417ffff700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41700a2e20 0x7f41700a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:07.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.143+0000 7f417dffb700 1 -- 172.21.15.121:0/307228012 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f41740d0bd0 con 0x7f4180101610 2024-06-10T00:28:07.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.144+0000 7f417ffff700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41700a2e20 0x7f41700a52e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f416c005950 tx=0x7f416c0058e0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:07.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.147+0000 7f417dffb700 1 -- 172.21.15.121:0/307228012 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4174031070 con 0x7f4180101610 2024-06-10T00:28:07.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.346+0000 7f4186c2b700 1 -- 172.21.15.121:0/307228012 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f418002de60 con 0x7f4180101610 2024-06-10T00:28:07.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.347+0000 7f417dffb700 1 -- 172.21.15.121:0/307228012 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+25301 (secure 0 0 0) 0x7f4174013070 con 0x7f4180101610 2024-06-10T00:28:07.348 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:07.348 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":101,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:27:57.451584+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/4266501776":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/4078460254":"2024-06-11T00:27:57.451553+0000","172.21.15.121:6801/3506007074":"2024-06-11T00:27:57.451553+0000","172.21.15.121:6800/3506007074":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/58990727":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/1002791322":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:28:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 -- 172.21.15.121:0/307228012 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41700a2e20 msgr2=0x7f41700a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41700a2e20 0x7f41700a52e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f416c005950 tx=0x7f416c0058e0 comp rx=0 tx=0).stop 2024-06-10T00:28:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 -- 172.21.15.121:0/307228012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4180101610 msgr2=0x7f4180199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4180101610 0x7f4180199ae0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4174006010 tx=0x7f4174009f80 comp rx=0 tx=0).stop 2024-06-10T00:28:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 -- 172.21.15.121:0/307228012 shutdown_connections 2024-06-10T00:28:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41700a2e20 0x7f41700a52e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41801066f0 0x7f41801a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4180101610 0x7f4180199ae0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 --2- 172.21.15.121:0/307228012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4180101fc0 0x7f418019a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 -- 172.21.15.121:0/307228012 >> 172.21.15.121:0/307228012 conn(0x7f4180078430 msgr2=0x7f41800ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:07.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 -- 172.21.15.121:0/307228012 shutdown_connections 2024-06-10T00:28:07.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:07.350+0000 7f4186c2b700 1 -- 172.21.15.121:0/307228012 wait complete. 2024-06-10T00:28:07.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:07 smithi135 ceph-mon[36424]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 146 B/s wr, 22 op/s 2024-06-10T00:28:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:07 smithi193 ceph-mon[36716]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 146 B/s wr, 22 op/s 2024-06-10T00:28:07.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:07 smithi121 ceph-mon[32378]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 146 B/s wr, 22 op/s 2024-06-10T00:28:08.038 DEBUG:tasks.fwd_scrub:Yielding 2024-06-10T00:28:08.039 INFO:teuthology.run_tasks:Running task exec... 2024-06-10T00:28:08.049 INFO:teuthology.task.exec:Executing custom commands... 2024-06-10T00:28:08.049 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi121.front.sepia.ceph.com 2024-06-10T00:28:08.049 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-06-10T00:28:08.050 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-06-10T00:28:08.050 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-10T00:28:08.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-10T00:28:08.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.160+0000 7fc5e66d6640 1 -- 172.21.15.121:0/350744281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5e0106890 msgr2=0x7fc5e0106c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.160+0000 7fc5e66d6640 1 --2- 172.21.15.121:0/350744281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5e0106890 0x7fc5e0106c70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0009930 tx=0x7fc5d002a420 comp rx=0 tx=0).stop 2024-06-10T00:28:08.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.160+0000 7fc5e66d6640 1 -- 172.21.15.121:0/350744281 shutdown_connections 2024-06-10T00:28:08.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.160+0000 7fc5e66d6640 1 --2- 172.21.15.121:0/350744281 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e0101290 0x7fc5e010d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.160+0000 7fc5e66d6640 1 --2- 172.21.15.121:0/350744281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5e01008d0 0x7fc5e0100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.160+0000 7fc5e66d6640 1 --2- 172.21.15.121:0/350744281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5e0106890 0x7fc5e0106c70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.160+0000 7fc5e66d6640 1 -- 172.21.15.121:0/350744281 >> 172.21.15.121:0/350744281 conn(0x7fc5e0077a20 msgr2=0x7fc5e0077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.160+0000 7fc5e66d6640 1 -- 172.21.15.121:0/350744281 shutdown_connections 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.160+0000 7fc5e66d6640 1 -- 172.21.15.121:0/350744281 wait complete. 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e66d6640 1 Processor -- start 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e66d6640 1 -- start start 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e66d6640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5e01008d0 0x7fc5e0199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e66d6640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e0101290 0x7fc5e01995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e66d6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5e0106890 0x7fc5e01a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e66d6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e0110ca0 con 0x7fc5e0106890 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e66d6640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e0110b20 con 0x7fc5e0101290 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e66d6640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e0110e20 con 0x7fc5e01008d0 2024-06-10T00:28:08.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e4c73640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5e0106890 0x7fc5e01a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e4c73640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5e0106890 0x7fc5e01a40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41530/0 (socket says 172.21.15.121:41530) 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e4c73640 1 -- 172.21.15.121:0/1504341316 learned_addr learned my addr 172.21.15.121:0/1504341316 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5dffff640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5e01008d0 0x7fc5e0199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e4c73640 1 -- 172.21.15.121:0/1504341316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5e01008d0 msgr2=0x7fc5e0199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e4c73640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5e01008d0 0x7fc5e0199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e4c73640 1 -- 172.21.15.121:0/1504341316 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e0101290 msgr2=0x7fc5e01995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5d7fff640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e0101290 0x7fc5e01995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e4c73640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e0101290 0x7fc5e01995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.161+0000 7fc5e4c73640 1 -- 172.21.15.121:0/1504341316 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e01a4620 con 0x7fc5e0106890 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.162+0000 7fc5d7fff640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e0101290 0x7fc5e01995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.162+0000 7fc5e4c73640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5e0106890 0x7fc5e01a40e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009fd0 tx=0x7fc5d800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.162+0000 7fc5ddffb640 1 -- 172.21.15.121:0/1504341316 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8017070 con 0x7fc5e0106890 2024-06-10T00:28:08.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.162+0000 7fc5e66d6640 1 -- 172.21.15.121:0/1504341316 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e01a47f0 con 0x7fc5e0106890 2024-06-10T00:28:08.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.162+0000 7fc5ddffb640 1 -- 172.21.15.121:0/1504341316 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5d8003e50 con 0x7fc5e0106890 2024-06-10T00:28:08.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.162+0000 7fc5ddffb640 1 -- 172.21.15.121:0/1504341316 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d80106c0 con 0x7fc5e0106890 2024-06-10T00:28:08.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.162+0000 7fc5e66d6640 1 -- 172.21.15.121:0/1504341316 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e01a4d10 con 0x7fc5e0106890 2024-06-10T00:28:08.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.163+0000 7fc5e66d6640 1 -- 172.21.15.121:0/1504341316 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5e004f760 con 0x7fc5e0106890 2024-06-10T00:28:08.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.164+0000 7fc5ddffb640 1 -- 172.21.15.121:0/1504341316 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fc5d8003970 con 0x7fc5e0106890 2024-06-10T00:28:08.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.165+0000 7fc5ddffb640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc5cc0a2e40 0x7fc5cc0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.165+0000 7fc5dffff640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc5cc0a2e40 0x7fc5cc0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.165+0000 7fc5ddffb640 1 -- 172.21.15.121:0/1504341316 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fc5d80cf6d0 con 0x7fc5e0106890 2024-06-10T00:28:08.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.165+0000 7fc5dffff640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc5cc0a2e40 0x7fc5cc0a5300 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00096c0 tx=0x7fc5d0000bc0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:08.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.168+0000 7fc5ddffb640 1 -- 172.21.15.121:0/1504341316 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc5d809bdf0 con 0x7fc5e0106890 2024-06-10T00:28:08.302 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:08.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.365+0000 7fc5e66d6640 1 -- 172.21.15.121:0/1504341316 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7fc5e0108860 con 0x7fc5e0106890 2024-06-10T00:28:08.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.369+0000 7fc5ddffb640 1 -- 172.21.15.121:0/1504341316 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v28)=0 v28) ==== 129+0+0 (secure 0 0 0) 0x7fc5d809b6f0 con 0x7fc5e0106890 2024-06-10T00:28:08.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 -- 172.21.15.121:0/1504341316 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc5cc0a2e40 msgr2=0x7fc5cc0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc5cc0a2e40 0x7fc5cc0a5300 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00096c0 tx=0x7fc5d0000bc0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 -- 172.21.15.121:0/1504341316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5e0106890 msgr2=0x7fc5e01a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5e0106890 0x7fc5e01a40e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009fd0 tx=0x7fc5d800b260 comp rx=0 tx=0).stop 2024-06-10T00:28:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 -- 172.21.15.121:0/1504341316 shutdown_connections 2024-06-10T00:28:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5e0106890 0x7fc5e01a40e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc5e0101290 0x7fc5e01995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc5cc0a2e40 0x7fc5cc0a5300 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 --2- 172.21.15.121:0/1504341316 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5e01008d0 0x7fc5e0199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 -- 172.21.15.121:0/1504341316 >> 172.21.15.121:0/1504341316 conn(0x7fc5e0077a20 msgr2=0x7fc5e01033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 -- 172.21.15.121:0/1504341316 shutdown_connections 2024-06-10T00:28:08.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.371+0000 7fc5e66d6640 1 -- 172.21.15.121:0/1504341316 wait complete. 2024-06-10T00:28:08.383 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-06-10T00:28:08.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.491+0000 7f1ad6177640 1 -- 172.21.15.121:0/1046186584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ad01008d0 msgr2=0x7f1ad0100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.491+0000 7f1ad6177640 1 --2- 172.21.15.121:0/1046186584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ad01008d0 0x7f1ad0100d50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8009930 tx=0x7f1ab802a420 comp rx=0 tx=0).stop 2024-06-10T00:28:08.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.491+0000 7f1ad6177640 1 -- 172.21.15.121:0/1046186584 shutdown_connections 2024-06-10T00:28:08.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.491+0000 7f1ad6177640 1 --2- 172.21.15.121:0/1046186584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad0101290 0x7f1ad010d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.491+0000 7f1ad6177640 1 --2- 172.21.15.121:0/1046186584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ad01008d0 0x7f1ad0100d50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.491+0000 7f1ad6177640 1 --2- 172.21.15.121:0/1046186584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad0106890 0x7f1ad0106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.491+0000 7f1ad6177640 1 -- 172.21.15.121:0/1046186584 >> 172.21.15.121:0/1046186584 conn(0x7f1ad0077a20 msgr2=0x7f1ad0077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.491+0000 7f1ad6177640 1 -- 172.21.15.121:0/1046186584 shutdown_connections 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.491+0000 7f1ad6177640 1 -- 172.21.15.121:0/1046186584 wait complete. 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.491+0000 7f1ad6177640 1 Processor -- start 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.491+0000 7f1ad6177640 1 -- start start 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1ad6177640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad01008d0 0x7f1ad01990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1ad6177640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad0101290 0x7f1ad0199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1ad6177640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ad0106890 0x7f1ad0199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1ad6177640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad0110ca0 con 0x7f1ad0106890 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1ad6177640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad0110b20 con 0x7f1ad0101290 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1ad6177640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad0110e20 con 0x7f1ad01008d0 2024-06-10T00:28:08.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ad0106890 0x7f1ad0199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ad0106890 0x7f1ad0199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41548/0 (socket says 172.21.15.121:41548) 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acffff640 1 -- 172.21.15.121:0/742619734 learned_addr learned my addr 172.21.15.121:0/742619734 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acf7fe640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad01008d0 0x7f1ad01990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1ac7fff640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad0101290 0x7f1ad0199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acffff640 1 -- 172.21.15.121:0/742619734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad01008d0 msgr2=0x7f1ad01990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acffff640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad01008d0 0x7f1ad01990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acffff640 1 -- 172.21.15.121:0/742619734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad0101290 msgr2=0x7f1ad0199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acffff640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad0101290 0x7f1ad0199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acffff640 1 -- 172.21.15.121:0/742619734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad01a44c0 con 0x7f1ad0106890 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1ac7fff640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad0101290 0x7f1ad0199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acffff640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ad0106890 0x7f1ad0199b80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800eaa0 tx=0x7f1ac800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acd7fa640 1 -- 172.21.15.121:0/742619734 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac800cdb0 con 0x7f1ad0106890 2024-06-10T00:28:08.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acd7fa640 1 -- 172.21.15.121:0/742619734 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ac8005020 con 0x7f1ad0106890 2024-06-10T00:28:08.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1acd7fa640 1 -- 172.21.15.121:0/742619734 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac80116b0 con 0x7f1ad0106890 2024-06-10T00:28:08.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1ad6177640 1 -- 172.21.15.121:0/742619734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad01a47b0 con 0x7f1ad0106890 2024-06-10T00:28:08.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.492+0000 7f1ad6177640 1 -- 172.21.15.121:0/742619734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad01a4d00 con 0x7f1ad0106890 2024-06-10T00:28:08.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.493+0000 7f1ad6177640 1 -- 172.21.15.121:0/742619734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad004f760 con 0x7f1ad0106890 2024-06-10T00:28:08.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.494+0000 7f1acd7fa640 1 -- 172.21.15.121:0/742619734 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f1ac8005560 con 0x7f1ad0106890 2024-06-10T00:28:08.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.495+0000 7f1acd7fa640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1abc0a2e40 0x7f1abc0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.495+0000 7f1acf7fe640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1abc0a2e40 0x7f1abc0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.495+0000 7f1acd7fa640 1 -- 172.21.15.121:0/742619734 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f1ac80cfbb0 con 0x7f1ad0106890 2024-06-10T00:28:08.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.496+0000 7f1acf7fe640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1abc0a2e40 0x7f1abc0a5300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009b50 tx=0x7f1ac0005f90 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:08.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.499+0000 7f1acd7fa640 1 -- 172.21.15.121:0/742619734 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1ac8016030 con 0x7f1ad0106890 2024-06-10T00:28:08.588 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:08 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/307228012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:28:08.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:08 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/307228012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:28:08.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.682+0000 7f1ad6177640 1 -- 172.21.15.121:0/742619734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f1ad002d2e0 con 0x7f1ad0106890 2024-06-10T00:28:08.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.686+0000 7f1acd7fa640 1 -- 172.21.15.121:0/742619734 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f1ac809c2d0 con 0x7f1ad0106890 2024-06-10T00:28:08.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 -- 172.21.15.121:0/742619734 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1abc0a2e40 msgr2=0x7f1abc0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1abc0a2e40 0x7f1abc0a5300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009b50 tx=0x7f1ac0005f90 comp rx=0 tx=0).stop 2024-06-10T00:28:08.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 -- 172.21.15.121:0/742619734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ad0106890 msgr2=0x7f1ad0199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ad0106890 0x7f1ad0199b80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800eaa0 tx=0x7f1ac800ef60 comp rx=0 tx=0).stop 2024-06-10T00:28:08.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 -- 172.21.15.121:0/742619734 shutdown_connections 2024-06-10T00:28:08.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ad0106890 0x7f1ad0199b80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ad0101290 0x7f1ad0199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1abc0a2e40 0x7f1abc0a5300 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 --2- 172.21.15.121:0/742619734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad01008d0 0x7f1ad01990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 -- 172.21.15.121:0/742619734 >> 172.21.15.121:0/742619734 conn(0x7f1ad0077a20 msgr2=0x7f1ad01033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:08.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 -- 172.21.15.121:0/742619734 shutdown_connections 2024-06-10T00:28:08.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.689+0000 7f1ad6177640 1 -- 172.21.15.121:0/742619734 wait complete. 2024-06-10T00:28:08.701 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-06-10T00:28:08.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:08 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/307228012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:28:08.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.812+0000 7fea38568640 1 -- 172.21.15.121:0/3786877999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea301008d0 msgr2=0x7fea30100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 --2- 172.21.15.121:0/3786877999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea301008d0 0x7fea30100d50 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009930 tx=0x7fea2c02a420 comp rx=0 tx=0).stop 2024-06-10T00:28:08.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 -- 172.21.15.121:0/3786877999 shutdown_connections 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 --2- 172.21.15.121:0/3786877999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea30101290 0x7fea3010d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 --2- 172.21.15.121:0/3786877999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea301008d0 0x7fea30100d50 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 --2- 172.21.15.121:0/3786877999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea30106890 0x7fea30106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 -- 172.21.15.121:0/3786877999 >> 172.21.15.121:0/3786877999 conn(0x7fea30077a20 msgr2=0x7fea30077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 -- 172.21.15.121:0/3786877999 shutdown_connections 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 -- 172.21.15.121:0/3786877999 wait complete. 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 Processor -- start 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 -- start start 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea301008d0 0x7fea301970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea30101290 0x7fea30197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea30106890 0x7fea3019b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fea30110ca0 con 0x7fea301008d0 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fea30110b20 con 0x7fea30106890 2024-06-10T00:28:08.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.813+0000 7fea38568640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea30110e20 con 0x7fea30101290 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea36304640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea301008d0 0x7fea301970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea36304640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea301008d0 0x7fea301970e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41576/0 (socket says 172.21.15.121:41576) 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea36304640 1 -- 172.21.15.121:0/1079830241 learned_addr learned my addr 172.21.15.121:0/1079830241 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea35b03640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea30101290 0x7fea30197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea36b05640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea30106890 0x7fea3019b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea36304640 1 -- 172.21.15.121:0/1079830241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea30101290 msgr2=0x7fea30197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea36304640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea30101290 0x7fea30197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea36304640 1 -- 172.21.15.121:0/1079830241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea30106890 msgr2=0x7fea3019b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea36304640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea30106890 0x7fea3019b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea36304640 1 -- 172.21.15.121:0/1079830241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea3019c0d0 con 0x7fea301008d0 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea35b03640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea30101290 0x7fea30197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:08.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea36304640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea301008d0 0x7fea301970e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fea2400b4a0 tx=0x7fea2400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:08.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea237fe640 1 -- 172.21.15.121:0/1079830241 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea24014070 con 0x7fea301008d0 2024-06-10T00:28:08.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea237fe640 1 -- 172.21.15.121:0/1079830241 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea24010af0 con 0x7fea301008d0 2024-06-10T00:28:08.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea38568640 1 -- 172.21.15.121:0/1079830241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea3019c3c0 con 0x7fea301008d0 2024-06-10T00:28:08.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea237fe640 1 -- 172.21.15.121:0/1079830241 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea24017d10 con 0x7fea301008d0 2024-06-10T00:28:08.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.814+0000 7fea38568640 1 -- 172.21.15.121:0/1079830241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea301a3c50 con 0x7fea301008d0 2024-06-10T00:28:08.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.815+0000 7fea38568640 1 -- 172.21.15.121:0/1079830241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea3004f760 con 0x7fea301008d0 2024-06-10T00:28:08.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.816+0000 7fea237fe640 1 -- 172.21.15.121:0/1079830241 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fea24017070 con 0x7fea301008d0 2024-06-10T00:28:08.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.817+0000 7fea237fe640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea1c0a2e40 0x7fea1c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:08.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.817+0000 7fea35b03640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea1c0a2e40 0x7fea1c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:08.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.817+0000 7fea237fe640 1 -- 172.21.15.121:0/1079830241 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fea240cec40 con 0x7fea301008d0 2024-06-10T00:28:08.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.817+0000 7fea35b03640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea1c0a2e40 0x7fea1c0a5300 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fea2c000c00 tx=0x7fea2c005a20 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:08.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:08.821+0000 7fea237fe640 1 -- 172.21.15.121:0/1079830241 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fea2409b470 con 0x7fea301008d0 2024-06-10T00:28:09.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.002+0000 7fea38568640 1 -- 172.21.15.121:0/1079830241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fea30198430 con 0x7fea301008d0 2024-06-10T00:28:09.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.006+0000 7fea237fe640 1 -- 172.21.15.121:0/1079830241 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7fea2409ad70 con 0x7fea301008d0 2024-06-10T00:28:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 -- 172.21.15.121:0/1079830241 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea1c0a2e40 msgr2=0x7fea1c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea1c0a2e40 0x7fea1c0a5300 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fea2c000c00 tx=0x7fea2c005a20 comp rx=0 tx=0).stop 2024-06-10T00:28:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 -- 172.21.15.121:0/1079830241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea301008d0 msgr2=0x7fea301970e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea301008d0 0x7fea301970e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fea2400b4a0 tx=0x7fea2400b960 comp rx=0 tx=0).stop 2024-06-10T00:28:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 -- 172.21.15.121:0/1079830241 shutdown_connections 2024-06-10T00:28:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea30106890 0x7fea3019b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea1c0a2e40 0x7fea1c0a5300 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea30101290 0x7fea30197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 --2- 172.21.15.121:0/1079830241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea301008d0 0x7fea301970e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 -- 172.21.15.121:0/1079830241 >> 172.21.15.121:0/1079830241 conn(0x7fea30077a20 msgr2=0x7fea30103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 -- 172.21.15.121:0/1079830241 shutdown_connections 2024-06-10T00:28:09.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.008+0000 7fea38568640 1 -- 172.21.15.121:0/1079830241 wait complete. 2024-06-10T00:28:09.019 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-06-10T00:28:09.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.126+0000 7f796939e640 1 -- 172.21.15.121:0/2088009445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79641019d0 msgr2=0x7f79641052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.126+0000 7f796939e640 1 --2- 172.21.15.121:0/2088009445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79641019d0 0x7f79641052a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7958009930 tx=0x7f795802a420 comp rx=0 tx=0).stop 2024-06-10T00:28:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.126+0000 7f796939e640 1 -- 172.21.15.121:0/2088009445 shutdown_connections 2024-06-10T00:28:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.126+0000 7f796939e640 1 --2- 172.21.15.121:0/2088009445 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79641059d0 0x7f796410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.126+0000 7f796939e640 1 --2- 172.21.15.121:0/2088009445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79641019d0 0x7f79641052a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.126+0000 7f796939e640 1 --2- 172.21.15.121:0/2088009445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7964101020 0x7f7964101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.126+0000 7f796939e640 1 -- 172.21.15.121:0/2088009445 >> 172.21.15.121:0/2088009445 conn(0x7f7964076c20 msgr2=0x7f79640fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.126+0000 7f796939e640 1 -- 172.21.15.121:0/2088009445 shutdown_connections 2024-06-10T00:28:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.126+0000 7f796939e640 1 -- 172.21.15.121:0/2088009445 wait complete. 2024-06-10T00:28:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.127+0000 7f796939e640 1 Processor -- start 2024-06-10T00:28:09.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.127+0000 7f796939e640 1 -- start start 2024-06-10T00:28:09.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.127+0000 7f796939e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7964101020 0x7f7964196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.127+0000 7f796939e640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79641059d0 0x7f7964197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.127+0000 7f796939e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7964197780 0x7f79641a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.127+0000 7f796939e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7964110a90 con 0x7f7964101020 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.127+0000 7f796939e640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f7964110910 con 0x7f79641059d0 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.127+0000 7f796939e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7964110c10 con 0x7f7964197780 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f7963fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7964101020 0x7f7964196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f7963fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7964101020 0x7f7964196d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34650/0 (socket says 172.21.15.121:34650) 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f7963fff640 1 -- 172.21.15.121:0/1888592234 learned_addr learned my addr 172.21.15.121:0/1888592234 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f79637fe640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79641059d0 0x7f7964197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f7968b9d640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7964197780 0x7f79641a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f7963fff640 1 -- 172.21.15.121:0/1888592234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7964197780 msgr2=0x7f79641a6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f7963fff640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7964197780 0x7f79641a6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f7963fff640 1 -- 172.21.15.121:0/1888592234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79641059d0 msgr2=0x7f7964197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f7963fff640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79641059d0 0x7f7964197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f7963fff640 1 -- 172.21.15.121:0/1888592234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79641a6940 con 0x7f7964101020 2024-06-10T00:28:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f79637fe640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79641059d0 0x7f7964197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f7963fff640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7964101020 0x7f7964196d00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f795400ea20 tx=0x7f795400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f79617fa640 1 -- 172.21.15.121:0/1888592234 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795400cd20 con 0x7f7964101020 2024-06-10T00:28:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f79617fa640 1 -- 172.21.15.121:0/1888592234 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7954005500 con 0x7f7964101020 2024-06-10T00:28:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f796939e640 1 -- 172.21.15.121:0/1888592234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79641a6c30 con 0x7f7964101020 2024-06-10T00:28:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f79617fa640 1 -- 172.21.15.121:0/1888592234 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954010690 con 0x7f7964101020 2024-06-10T00:28:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.128+0000 7f796939e640 1 -- 172.21.15.121:0/1888592234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79641a7100 con 0x7f7964101020 2024-06-10T00:28:09.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.129+0000 7f796939e640 1 -- 172.21.15.121:0/1888592234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f796404ffb0 con 0x7f7964101020 2024-06-10T00:28:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.130+0000 7f79617fa640 1 -- 172.21.15.121:0/1888592234 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f7954004e50 con 0x7f7964101020 2024-06-10T00:28:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.131+0000 7f79617fa640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f794c0a2d80 0x7f794c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.131+0000 7f79637fe640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f794c0a2d80 0x7f794c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.132+0000 7f79617fa640 1 -- 172.21.15.121:0/1888592234 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f79540cfed0 con 0x7f7964101020 2024-06-10T00:28:09.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.132+0000 7f79637fe640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f794c0a2d80 0x7f794c0a5240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7958006010 tx=0x7f7958005a20 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:09.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.136+0000 7f79617fa640 1 -- 172.21.15.121:0/1888592234 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f795409c5f0 con 0x7f7964101020 2024-06-10T00:28:09.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.341+0000 7f796939e640 1 -- 172.21.15.121:0/1888592234 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f79641019d0 con 0x7f794c0a2d80 2024-06-10T00:28:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.405+0000 7f79617fa640 1 -- 172.21.15.121:0/1888592234 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f79641019d0 con 0x7f794c0a2d80 2024-06-10T00:28:09.405 INFO:teuthology.orchestra.run.smithi121.stdout:Schedule set for path / 2024-06-10T00:28:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.407+0000 7f796939e640 1 -- 172.21.15.121:0/1888592234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f794c0a2d80 msgr2=0x7f794c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.407+0000 7f796939e640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f794c0a2d80 0x7f794c0a5240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7958006010 tx=0x7f7958005a20 comp rx=0 tx=0).stop 2024-06-10T00:28:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.407+0000 7f796939e640 1 -- 172.21.15.121:0/1888592234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7964101020 msgr2=0x7f7964196d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.407+0000 7f796939e640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7964101020 0x7f7964196d00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f795400ea20 tx=0x7f795400ee20 comp rx=0 tx=0).stop 2024-06-10T00:28:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.408+0000 7f796939e640 1 -- 172.21.15.121:0/1888592234 shutdown_connections 2024-06-10T00:28:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.408+0000 7f796939e640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f794c0a2d80 0x7f794c0a5240 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.408+0000 7f796939e640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7964197780 0x7f79641a6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.408+0000 7f796939e640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f79641059d0 0x7f7964197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.408+0000 7f796939e640 1 --2- 172.21.15.121:0/1888592234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7964101020 0x7f7964196d00 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.408+0000 7f796939e640 1 -- 172.21.15.121:0/1888592234 >> 172.21.15.121:0/1888592234 conn(0x7f7964076c20 msgr2=0x7f79640fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.408+0000 7f796939e640 1 -- 172.21.15.121:0/1888592234 shutdown_connections 2024-06-10T00:28:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.408+0000 7f796939e640 1 -- 172.21.15.121:0/1888592234 wait complete. 2024-06-10T00:28:09.417 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 113 B/s wr, 17 op/s 2024-06-10T00:28:09.417 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1504341316' entity='client.admin' 2024-06-10T00:28:09.417 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:09.417 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:09.417 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:09.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:09.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/742619734' entity='client.admin' 2024-06-10T00:28:09.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:09.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:09.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:09.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:09.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1079830241' entity='client.admin' 2024-06-10T00:28:09.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:09.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:09.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:09.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:09.419 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-06-10T00:28:09.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.529+0000 7f588149e640 1 -- 172.21.15.121:0/2501346701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f587c106890 msgr2=0x7f587c106c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.529+0000 7f588149e640 1 --2- 172.21.15.121:0/2501346701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f587c106890 0x7f587c106c70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5874009cd0 tx=0x7f587402a8e0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.529+0000 7f588149e640 1 -- 172.21.15.121:0/2501346701 shutdown_connections 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.529+0000 7f588149e640 1 --2- 172.21.15.121:0/2501346701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f587c101290 0x7f587c10d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.529+0000 7f588149e640 1 --2- 172.21.15.121:0/2501346701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c1008d0 0x7f587c100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.529+0000 7f588149e640 1 --2- 172.21.15.121:0/2501346701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f587c106890 0x7f587c106c70 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.529+0000 7f588149e640 1 -- 172.21.15.121:0/2501346701 >> 172.21.15.121:0/2501346701 conn(0x7f587c077a20 msgr2=0x7f587c077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.529+0000 7f588149e640 1 -- 172.21.15.121:0/2501346701 shutdown_connections 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.529+0000 7f588149e640 1 -- 172.21.15.121:0/2501346701 wait complete. 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.530+0000 7f588149e640 1 Processor -- start 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.530+0000 7f588149e640 1 -- start start 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.530+0000 7f588149e640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f587c1008d0 0x7f587c1990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.530+0000 7f588149e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f587c101290 0x7f587c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.530+0000 7f588149e640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c106890 0x7f587c1a4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.530+0000 7f588149e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f587c110ca0 con 0x7f587c101290 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.530+0000 7f588149e640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f587c110b20 con 0x7f587c106890 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.530+0000 7f588149e640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f587c110e20 con 0x7f587c1008d0 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.530+0000 7f587a7fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f587c101290 0x7f587c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.530+0000 7f587a7fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f587c101290 0x7f587c1995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34662/0 (socket says 172.21.15.121:34662) 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.530+0000 7f587a7fc640 1 -- 172.21.15.121:0/3476625487 learned_addr learned my addr 172.21.15.121:0/3476625487 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f587affd640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f587c1008d0 0x7f587c1990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f587a7fc640 1 -- 172.21.15.121:0/3476625487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f587c1008d0 msgr2=0x7f587c1990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f587a7fc640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f587c1008d0 0x7f587c1990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f587a7fc640 1 -- 172.21.15.121:0/3476625487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c106890 msgr2=0x7f587c1a4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f587b7fe640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c106890 0x7f587c1a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f587a7fc640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c106890 0x7f587c1a4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f587a7fc640 1 -- 172.21.15.121:0/3476625487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f587c1a4660 con 0x7f587c101290 2024-06-10T00:28:09.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f587b7fe640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c106890 0x7f587c1a4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:09.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f587a7fc640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f587c101290 0x7f587c1995e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f586400d950 tx=0x7f586400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:09.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f5873fff640 1 -- 172.21.15.121:0/3476625487 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864017070 con 0x7f587c101290 2024-06-10T00:28:09.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f5873fff640 1 -- 172.21.15.121:0/3476625487 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5864009c30 con 0x7f587c101290 2024-06-10T00:28:09.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f5873fff640 1 -- 172.21.15.121:0/3476625487 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586400f6b0 con 0x7f587c101290 2024-06-10T00:28:09.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f588149e640 1 -- 172.21.15.121:0/3476625487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f587c1a4830 con 0x7f587c101290 2024-06-10T00:28:09.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.531+0000 7f588149e640 1 -- 172.21.15.121:0/3476625487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f587c1a4d30 con 0x7f587c101290 2024-06-10T00:28:09.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.532+0000 7f588149e640 1 -- 172.21.15.121:0/3476625487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f587c04f760 con 0x7f587c101290 2024-06-10T00:28:09.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.533+0000 7f5873fff640 1 -- 172.21.15.121:0/3476625487 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f58640109d0 con 0x7f587c101290 2024-06-10T00:28:09.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.534+0000 7f5873fff640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f58680a2e40 0x7f58680a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.535+0000 7f587affd640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f58680a2e40 0x7f58680a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.535+0000 7f5873fff640 1 -- 172.21.15.121:0/3476625487 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f58640cf6c0 con 0x7f587c101290 2024-06-10T00:28:09.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.535+0000 7f587affd640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f58680a2e40 0x7f58680a5300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5874006010 tx=0x7f58740093f0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:09.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.539+0000 7f5873fff640 1 -- 172.21.15.121:0/3476625487 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5864009de0 con 0x7f587c101290 2024-06-10T00:28:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 113 B/s wr, 17 op/s 2024-06-10T00:28:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1504341316' entity='client.admin' 2024-06-10T00:28:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/742619734' entity='client.admin' 2024-06-10T00:28:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1079830241' entity='client.admin' 2024-06-10T00:28:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:09.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.746+0000 7f588149e640 1 -- 172.21.15.121:0/3476625487 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f587c061de0 con 0x7f58680a2e40 2024-06-10T00:28:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 113 B/s wr, 17 op/s 2024-06-10T00:28:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1504341316' entity='client.admin' 2024-06-10T00:28:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/742619734' entity='client.admin' 2024-06-10T00:28:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:09.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1079830241' entity='client.admin' 2024-06-10T00:28:09.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:28:09.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:28:09.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:28:09.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:28:09.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.806+0000 7f5873fff640 1 -- 172.21.15.121:0/3476625487 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f587c061de0 con 0x7f58680a2e40 2024-06-10T00:28:09.806 INFO:teuthology.orchestra.run.smithi121.stdout:Retention added to path / 2024-06-10T00:28:09.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 -- 172.21.15.121:0/3476625487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f58680a2e40 msgr2=0x7f58680a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f58680a2e40 0x7f58680a5300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5874006010 tx=0x7f58740093f0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 -- 172.21.15.121:0/3476625487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f587c101290 msgr2=0x7f587c1995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f587c101290 0x7f587c1995e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f586400d950 tx=0x7f586400de10 comp rx=0 tx=0).stop 2024-06-10T00:28:09.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 -- 172.21.15.121:0/3476625487 shutdown_connections 2024-06-10T00:28:09.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f587c106890 0x7f587c1a4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f587c101290 0x7f587c1995e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f58680a2e40 0x7f58680a5300 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 --2- 172.21.15.121:0/3476625487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f587c1008d0 0x7f587c1990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 -- 172.21.15.121:0/3476625487 >> 172.21.15.121:0/3476625487 conn(0x7f587c077a20 msgr2=0x7f587c103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:09.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 -- 172.21.15.121:0/3476625487 shutdown_connections 2024-06-10T00:28:09.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.808+0000 7f588149e640 1 -- 172.21.15.121:0/3476625487 wait complete. 2024-06-10T00:28:09.819 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-06-10T00:28:09.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.923+0000 7f6b1a888640 1 -- 172.21.15.121:0/3809958774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b1410e240 msgr2=0x7f6b14110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.923+0000 7f6b1a888640 1 --2- 172.21.15.121:0/3809958774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b1410e240 0x7f6b14110630 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c009ca0 tx=0x7f6b0c02a950 comp rx=0 tx=0).stop 2024-06-10T00:28:09.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.923+0000 7f6b1a888640 1 -- 172.21.15.121:0/3809958774 shutdown_connections 2024-06-10T00:28:09.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.923+0000 7f6b1a888640 1 --2- 172.21.15.121:0/3809958774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b1410e240 0x7f6b14110630 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.923+0000 7f6b1a888640 1 --2- 172.21.15.121:0/3809958774 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b14074670 0x7f6b1410dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.923+0000 7f6b1a888640 1 --2- 172.21.15.121:0/3809958774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b14073d50 0x7f6b14074130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.923+0000 7f6b1a888640 1 -- 172.21.15.121:0/3809958774 >> 172.21.15.121:0/3809958774 conn(0x7f6b140fd450 msgr2=0x7f6b140ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.923+0000 7f6b1a888640 1 -- 172.21.15.121:0/3809958774 shutdown_connections 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.923+0000 7f6b1a888640 1 -- 172.21.15.121:0/3809958774 wait complete. 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.923+0000 7f6b1a888640 1 Processor -- start 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b1a888640 1 -- start start 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b1a888640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b14073d50 0x7f6b1419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b1a888640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b14074670 0x7f6b1419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b1a888640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b1410e240 0x7f6b1419ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b1a888640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6b14113f60 con 0x7f6b14073d50 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b1a888640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6b14113de0 con 0x7f6b14074670 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b1a888640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6b141140e0 con 0x7f6b1410e240 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b13fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b14073d50 0x7f6b1419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b13fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b14073d50 0x7f6b1419a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34692/0 (socket says 172.21.15.121:34692) 2024-06-10T00:28:09.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b13fff640 1 -- 172.21.15.121:0/2052226635 learned_addr learned my addr 172.21.15.121:0/2052226635 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b18e25640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b1410e240 0x7f6b1419ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b137fe640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b14074670 0x7f6b1419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b13fff640 1 -- 172.21.15.121:0/2052226635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b1410e240 msgr2=0x7f6b1419ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b13fff640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b1410e240 0x7f6b1419ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b13fff640 1 -- 172.21.15.121:0/2052226635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b14074670 msgr2=0x7f6b1419a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b13fff640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b14074670 0x7f6b1419a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b13fff640 1 -- 172.21.15.121:0/2052226635 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1419f440 con 0x7f6b14073d50 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b137fe640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b14074670 0x7f6b1419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.924+0000 7f6b13fff640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b14073d50 0x7f6b1419a4a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6b0400e9f0 tx=0x7f6b0400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.925+0000 7f6b117fa640 1 -- 172.21.15.121:0/2052226635 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0400cd80 con 0x7f6b14073d50 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.925+0000 7f6b1a888640 1 -- 172.21.15.121:0/2052226635 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b1419f730 con 0x7f6b14073d50 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.925+0000 7f6b117fa640 1 -- 172.21.15.121:0/2052226635 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b04005500 con 0x7f6b14073d50 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.925+0000 7f6b117fa640 1 -- 172.21.15.121:0/2052226635 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b04010690 con 0x7f6b14073d50 2024-06-10T00:28:09.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.925+0000 7f6b1a888640 1 -- 172.21.15.121:0/2052226635 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b141a6fc0 con 0x7f6b14073d50 2024-06-10T00:28:09.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.926+0000 7f6b1a888640 1 -- 172.21.15.121:0/2052226635 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b1404f760 con 0x7f6b14073d50 2024-06-10T00:28:09.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.927+0000 7f6b117fa640 1 -- 172.21.15.121:0/2052226635 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f6b040046c0 con 0x7f6b14073d50 2024-06-10T00:28:09.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.927+0000 7f6b117fa640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6afc0a2e40 0x7f6afc0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:09.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.928+0000 7f6b117fa640 1 -- 172.21.15.121:0/2052226635 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f6b040cfec0 con 0x7f6b14073d50 2024-06-10T00:28:09.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.928+0000 7f6b137fe640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6afc0a2e40 0x7f6afc0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:09.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.928+0000 7f6b137fe640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6afc0a2e40 0x7f6afc0a5300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6b1410f960 tx=0x7f6b08005f90 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:09.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:09.932+0000 7f6b117fa640 1 -- 172.21.15.121:0/2052226635 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6b0409c5e0 con 0x7f6b14073d50 2024-06-10T00:28:10.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.137+0000 7f6b1a888640 1 -- 172.21.15.121:0/2052226635 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f6b14061de0 con 0x7f6afc0a2e40 2024-06-10T00:28:10.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.144+0000 7f6b117fa640 1 -- 172.21.15.121:0/2052226635 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f6b14061de0 con 0x7f6afc0a2e40 2024-06-10T00:28:10.144 INFO:teuthology.orchestra.run.smithi121.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-06-10T00:00:00", "created": "2024-06-10T00:28:09", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.146+0000 7f6b1a888640 1 -- 172.21.15.121:0/2052226635 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6afc0a2e40 msgr2=0x7f6afc0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.146+0000 7f6b1a888640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6afc0a2e40 0x7f6afc0a5300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6b1410f960 tx=0x7f6b08005f90 comp rx=0 tx=0).stop 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.146+0000 7f6b1a888640 1 -- 172.21.15.121:0/2052226635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b14073d50 msgr2=0x7f6b1419a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.146+0000 7f6b1a888640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b14073d50 0x7f6b1419a4a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6b0400e9f0 tx=0x7f6b0400edf0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.147+0000 7f6b1a888640 1 -- 172.21.15.121:0/2052226635 shutdown_connections 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.147+0000 7f6b1a888640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6afc0a2e40 0x7f6afc0a5300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.147+0000 7f6b1a888640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b1410e240 0x7f6b1419ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.147+0000 7f6b1a888640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b14074670 0x7f6b1419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.147+0000 7f6b1a888640 1 --2- 172.21.15.121:0/2052226635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b14073d50 0x7f6b1419a4a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.147+0000 7f6b1a888640 1 -- 172.21.15.121:0/2052226635 >> 172.21.15.121:0/2052226635 conn(0x7f6b140fd450 msgr2=0x7f6b140fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.147+0000 7f6b1a888640 1 -- 172.21.15.121:0/2052226635 shutdown_connections 2024-06-10T00:28:10.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.147+0000 7f6b1a888640 1 -- 172.21.15.121:0/2052226635 wait complete. 2024-06-10T00:28:10.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-06-10T00:28:10.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.201+0000 7ff1f7920700 1 -- 172.21.15.121:0/1983471656 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f0101810 msgr2=0x7ff1f010f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.201+0000 7ff1f7920700 1 --2- 172.21.15.121:0/1983471656 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f0101810 0x7ff1f010f280 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800b0a0 tx=0x7ff1e800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.201+0000 7ff1f7920700 1 -- 172.21.15.121:0/1983471656 shutdown_connections 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.201+0000 7ff1f7920700 1 --2- 172.21.15.121:0/1983471656 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f0101810 0x7ff1f010f280 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.201+0000 7ff1f7920700 1 --2- 172.21.15.121:0/1983471656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1f0107570 0x7ff1f0107950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.201+0000 7ff1f7920700 1 --2- 172.21.15.121:0/1983471656 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1f0100e50 0x7ff1f01012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.201+0000 7ff1f7920700 1 -- 172.21.15.121:0/1983471656 >> 172.21.15.121:0/1983471656 conn(0x7ff1f00766b0 msgr2=0x7ff1f0076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.201+0000 7ff1f7920700 1 -- 172.21.15.121:0/1983471656 shutdown_connections 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.201+0000 7ff1f7920700 1 -- 172.21.15.121:0/1983471656 wait complete. 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.201+0000 7ff1f7920700 1 Processor -- start 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f7920700 1 -- start start 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f7920700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1f0100e50 0x7ff1f019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f7920700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f0101810 0x7ff1f019ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f7920700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1f0107570 0x7ff1f0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f7920700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01127d0 con 0x7ff1f0101810 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f7920700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0112650 con 0x7ff1f0107570 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f7920700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f0112950 con 0x7ff1f0100e50 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f4ebb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f0101810 0x7ff1f019ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f4ebb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f0101810 0x7ff1f019ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34718/0 (socket says 172.21.15.121:34718) 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f4ebb700 1 -- 172.21.15.121:0/874796484 learned_addr learned my addr 172.21.15.121:0/874796484 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1f0100e50 0x7ff1f019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f5ebd700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1f0107570 0x7ff1f0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f4ebb700 1 -- 172.21.15.121:0/874796484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1f0100e50 msgr2=0x7ff1f019e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f4ebb700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1f0100e50 0x7ff1f019e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f4ebb700 1 -- 172.21.15.121:0/874796484 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1f0107570 msgr2=0x7ff1f0198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f4ebb700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1f0107570 0x7ff1f0198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f4ebb700 1 -- 172.21.15.121:0/874796484 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f01990d0 con 0x7ff1f0101810 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f5ebd700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1f0107570 0x7ff1f0198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.202+0000 7ff1f4ebb700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f0101810 0x7ff1f019ecf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000d950 tx=0x7ff1e000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:10.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.203+0000 7ff1e67fc700 1 -- 172.21.15.121:0/874796484 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0016070 con 0x7ff1f0101810 2024-06-10T00:28:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.203+0000 7ff1f7920700 1 -- 172.21.15.121:0/874796484 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f01993c0 con 0x7ff1f0101810 2024-06-10T00:28:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.203+0000 7ff1e67fc700 1 -- 172.21.15.121:0/874796484 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1e0009e20 con 0x7ff1f0101810 2024-06-10T00:28:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.203+0000 7ff1e67fc700 1 -- 172.21.15.121:0/874796484 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e000f620 con 0x7ff1f0101810 2024-06-10T00:28:10.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.203+0000 7ff1f7920700 1 -- 172.21.15.121:0/874796484 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f01a5680 con 0x7ff1f0101810 2024-06-10T00:28:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.205+0000 7ff1e67fc700 1 -- 172.21.15.121:0/874796484 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7ff1e0019070 con 0x7ff1f0101810 2024-06-10T00:28:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.205+0000 7ff1e67fc700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff1dc0a2e20 0x7ff1dc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.205+0000 7ff1e67fc700 1 -- 172.21.15.121:0/874796484 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff1e00cf550 con 0x7ff1f0101810 2024-06-10T00:28:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.206+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff1dc0a2e20 0x7ff1dc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.206+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff1dc0a2e20 0x7ff1dc0a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0199ab0 tx=0x7ff1ec009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:10.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.206+0000 7ff2021a3b80 1 Processor -- start 2024-06-10T00:28:10.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.206+0000 7ff2021a3b80 1 -- start start 2024-06-10T00:28:10.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf20536c0 0x55abf2053aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf2053fe0 0x55abf204da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf204dfa0 0x55abf204e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55abf1f9f890 con 0x55abf20536c0 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55abf1f477a0 con 0x55abf2053fe0 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55abf1f9b850 con 0x55abf204dfa0 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1f56bc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf20536c0 0x55abf2053aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1f56bc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf20536c0 0x55abf2053aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34720/0 (socket says 172.21.15.121:34720) 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1f56bc700 1 -- 172.21.15.121:0/3892494459 learned_addr learned my addr 172.21.15.121:0/3892494459 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1f4ebb700 1 --2- 172.21.15.121:0/3892494459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf2053fe0 0x55abf204da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1f5ebd700 1 --2- 172.21.15.121:0/3892494459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf204dfa0 0x55abf204e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1f56bc700 1 -- 172.21.15.121:0/3892494459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf204dfa0 msgr2=0x55abf204e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:10.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/3892494459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf204dfa0 0x55abf204e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1f56bc700 1 -- 172.21.15.121:0/3892494459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf2053fe0 msgr2=0x55abf204da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/3892494459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf2053fe0 0x55abf204da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1f56bc700 1 -- 172.21.15.121:0/3892494459 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abf1f3dfd0 con 0x55abf20536c0 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/3892494459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf20536c0 0x55abf2053aa0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec011d30 tx=0x7ff1ec014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad534b09b999468 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.207+0000 7ff1db7fe700 1 -- 172.21.15.121:0/3892494459 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec00eb20 con 0x55abf20536c0 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.208+0000 7ff1db7fe700 1 -- 172.21.15.121:0/3892494459 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1ec00ecc0 con 0x55abf20536c0 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.208+0000 7ff1db7fe700 1 -- 172.21.15.121:0/3892494459 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec01d780 con 0x55abf20536c0 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.208+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3892494459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf20536c0 msgr2=0x55abf2053aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.208+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3892494459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf20536c0 0x55abf2053aa0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec011d30 tx=0x7ff1ec014000 comp rx=0 tx=0).stop 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.208+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3892494459 shutdown_connections 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.208+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3892494459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf204dfa0 0x55abf204e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.208+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3892494459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf20536c0 0x55abf2053aa0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.208+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3892494459 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf2053fe0 0x55abf204da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.208+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3892494459 >> 172.21.15.121:0/3892494459 conn(0x55abf1f78d40 msgr2=0x55abf20488d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:10.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.208+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3892494459 shutdown_connections 2024-06-10T00:28:10.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.209+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3892494459 wait complete. 2024-06-10T00:28:10.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.209+0000 7ff2021a3b80 1 -- start start 2024-06-10T00:28:10.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.209+0000 7ff2021a3b80 1 Processor -- start 2024-06-10T00:28:10.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.209+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf204dfa0 0x55abf2059df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.209+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf20536c0 0x55abf2057140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.209+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf2057680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.209+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55abf205aa50 con 0x55abf2053fe0 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.209+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55abf1f3fc70 con 0x55abf20536c0 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.209+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55abf205abd0 con 0x55abf204dfa0 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f5ebd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf2057680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff2021a3b80 10 client.0 _ll_register_callbacks cb 0x55abf1f399a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f5ebd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf2057680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34722/0 (socket says 172.21.15.121:34722) 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f5ebd700 1 -- 172.21.15.121:0/3367813973 learned_addr learned my addr 172.21.15.121:0/3367813973 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f4ebb700 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf20536c0 0x55abf2057140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf204dfa0 0x55abf2059df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f4ebb700 1 -- 172.21.15.121:0/3367813973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf204dfa0 msgr2=0x55abf2059df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f4ebb700 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf204dfa0 0x55abf2059df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f4ebb700 1 -- 172.21.15.121:0/3367813973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 msgr2=0x55abf2057680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f4ebb700 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf2057680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f4ebb700 1 -- 172.21.15.121:0/3367813973 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abf1eee950 con 0x55abf20536c0 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f5ebd700 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf2057680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1f4ebb700 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf20536c0 0x55abf2057140 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00cc850 tx=0x7ff1e009aed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=594417caa5dd23d8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1d8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3367813973 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0020050 con 0x55abf20536c0 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3367813973 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1ec00ecc0 con 0x55abf20536c0 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.210+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3367813973 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff1e00cf140 con 0x55abf20536c0 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3367813973 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff1e00ccd00 con 0x55abf20536c0 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff2021a3b80 10 client.25028 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3367813973 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55abf2065460 con 0x55abf20536c0 2024-06-10T00:28:10.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3367813973 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1ec00ecc0 con 0x55abf20536c0 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3367813973 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7ff1e00a3d50 con 0x55abf20536c0 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff2021a3b80 10 client.25028 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff2021a3b80 10 client.25028 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff2021a3b80 20 client.25028 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55abf2067af0 0x55abf2069ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff2021a3b80 4 client.25028 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3367813973 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abf1facd90 con 0x55abf2067af0 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.211+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55abf2067af0 0x55abf2069ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.212+0000 7ff1d8ff9700 10 client.25028 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.212+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55abf2067af0 0x55abf2069ee0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e8c3eb158a0bddc server_cookie=9c83c21f2a6ea7b2 in_seq=0 out_seq=0 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3367813973 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55abf1facd90 con 0x55abf2067af0 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff1d8ff9700 10 client.25028 handle_command_reply: tid=0 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 1 client.25028 shutdown 2024-06-10T00:28:10.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3367813973 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf20536c0 msgr2=0x55abf2057140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf20536c0 0x55abf2057140 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00cc850 tx=0x7ff1e009aed0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3367813973 shutdown_connections 2024-06-10T00:28:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55abf2067af0 0x55abf2069ee0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf2057680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff1d8ff9700 0 client.25028 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf204dfa0 0x55abf2059df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3367813973 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf20536c0 0x55abf2057140 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3367813973 >> 172.21.15.121:0/3367813973 conn(0x55abf1f78d40 msgr2=0x55abf2048670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3367813973 shutdown_connections 2024-06-10T00:28:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3367813973 wait complete. 2024-06-10T00:28:10.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.213+0000 7ff2021a3b80 20 client.25028 trim_cache size 0 max 16384 2024-06-10T00:28:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff2021a3b80 1 Processor -- start 2024-06-10T00:28:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff2021a3b80 1 -- start start 2024-06-10T00:28:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf204dfa0 0x55abf204da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf20536c0 0x55abf2056140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf206f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01b0da0 con 0x55abf2053fe0 2024-06-10T00:28:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01ae630 con 0x55abf204dfa0 2024-06-10T00:28:10.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01ac480 con 0x55abf20536c0 2024-06-10T00:28:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff1f5ebd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf206f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff1f5ebd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf206f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34732/0 (socket says 172.21.15.121:34732) 2024-06-10T00:28:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff1f5ebd700 1 -- 172.21.15.121:0/4020868241 learned_addr learned my addr 172.21.15.121:0/4020868241 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.234+0000 7ff1f4ebb700 1 --2- 172.21.15.121:0/4020868241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf20536c0 0x55abf2056140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.235+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/4020868241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf204dfa0 0x55abf204da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.235+0000 7ff1f5ebd700 1 -- 172.21.15.121:0/4020868241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf20536c0 msgr2=0x55abf2056140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.235+0000 7ff1f5ebd700 1 --2- 172.21.15.121:0/4020868241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf20536c0 0x55abf2056140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.235+0000 7ff1f5ebd700 1 -- 172.21.15.121:0/4020868241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf204dfa0 msgr2=0x55abf204da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.235+0000 7ff1f5ebd700 1 --2- 172.21.15.121:0/4020868241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf204dfa0 0x55abf204da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.235+0000 7ff1f5ebd700 1 -- 172.21.15.121:0/4020868241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abf2053e00 con 0x55abf2053fe0 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.235+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/4020868241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf204dfa0 0x55abf204da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.235+0000 7ff1f5ebd700 1 --2- 172.21.15.121:0/4020868241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf206f770 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800bd20 tx=0x7ff1e800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae33493a9966305 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.235+0000 7ff1daffd700 1 -- 172.21.15.121:0/4020868241 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8004c50 con 0x55abf2053fe0 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.235+0000 7ff1daffd700 1 -- 172.21.15.121:0/4020868241 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1e8009cc0 con 0x55abf2053fe0 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.235+0000 7ff1daffd700 1 -- 172.21.15.121:0/4020868241 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e80053c0 con 0x55abf2053fe0 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.236+0000 7ff2021a3b80 1 -- 172.21.15.121:0/4020868241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 msgr2=0x55abf206f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.236+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/4020868241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf206f770 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800bd20 tx=0x7ff1e800bd50 comp rx=0 tx=0).stop 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.236+0000 7ff2021a3b80 1 -- 172.21.15.121:0/4020868241 shutdown_connections 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.236+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/4020868241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf206f770 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.236+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/4020868241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf204dfa0 0x55abf204da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.236+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/4020868241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf20536c0 0x55abf2056140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.236+0000 7ff2021a3b80 1 -- 172.21.15.121:0/4020868241 >> 172.21.15.121:0/4020868241 conn(0x55abf1f78d40 msgr2=0x55abf2055a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:10.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.236+0000 7ff2021a3b80 1 -- 172.21.15.121:0/4020868241 shutdown_connections 2024-06-10T00:28:10.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.236+0000 7ff2021a3b80 1 -- 172.21.15.121:0/4020868241 wait complete. 2024-06-10T00:28:10.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.236+0000 7ff2021a3b80 1 -- start start 2024-06-10T00:28:10.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.236+0000 7ff2021a3b80 1 Processor -- start 2024-06-10T00:28:10.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf204dfa0 0x55abf20caa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf20536c0 0x55abf20caf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff2021a3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf205c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01acaf0 con 0x55abf2053fe0 2024-06-10T00:28:10.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01adf60 con 0x55abf204dfa0 2024-06-10T00:28:10.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff2021a3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f01ad1c0 con 0x55abf20536c0 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff2021a3b80 10 client.0 _ll_register_callbacks cb 0x55abf1f399a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff1f5ebd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf205c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff1f5ebd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf205c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34734/0 (socket says 172.21.15.121:34734) 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff1f5ebd700 1 -- 172.21.15.121:0/3650161571 learned_addr learned my addr 172.21.15.121:0/3650161571 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff1f4ebb700 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf20536c0 0x55abf20caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf204dfa0 0x55abf20caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff1f5ebd700 1 -- 172.21.15.121:0/3650161571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf20536c0 msgr2=0x55abf20caf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff1f5ebd700 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf20536c0 0x55abf20caf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff1f5ebd700 1 -- 172.21.15.121:0/3650161571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf204dfa0 msgr2=0x55abf20caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff1f5ebd700 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf204dfa0 0x55abf20caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff1f5ebd700 1 -- 172.21.15.121:0/3650161571 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abf205ccc0 con 0x55abf2053fe0 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.237+0000 7ff1f4ebb700 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf20536c0 0x55abf20caf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:10.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff1f5ebd700 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf205c590 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800b3f0 tx=0x7ff1e80076d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1055cfd0a229ad5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff1d8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3650161571 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8034060 con 0x55abf2053fe0 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3650161571 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff1e8007d80 con 0x55abf2053fe0 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3650161571 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1e8009cc0 con 0x55abf2053fe0 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3650161571 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff1e802f450 con 0x55abf2053fe0 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff2021a3b80 10 client.15390 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3650161571 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55abf20d3390 con 0x55abf2053fe0 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3650161571 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1e8009cc0 con 0x55abf2053fe0 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3650161571 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7ff1e8023070 con 0x55abf2053fe0 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff2021a3b80 10 client.15390 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff2021a3b80 10 client.15390 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff2021a3b80 20 client.15390 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55abf20d3930 0x55abf20d5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff2021a3b80 4 client.15390 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-10T00:28:10.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.238+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3650161571 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55abf1fa7310 con 0x55abf20d3930 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.239+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55abf20d3930 0x55abf20d5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.239+0000 7ff1d8ff9700 10 client.15390 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.239+0000 7ff1f56bc700 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55abf20d3930 0x55abf20d5d20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ee3b934559abe32 server_cookie=72c564b9af9bc817 in_seq=0 out_seq=0 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.240+0000 7ff1d8ff9700 1 -- 172.21.15.121:0/3650161571 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55abf1fa7310 con 0x55abf20d3930 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.240+0000 7ff1d8ff9700 10 client.15390 handle_command_reply: tid=0 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.240+0000 7ff2021a3b80 1 client.15390 shutdown 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.240+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3650161571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 msgr2=0x55abf205c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.240+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf205c590 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800b3f0 tx=0x7ff1e80076d0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.240+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3650161571 shutdown_connections 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.240+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55abf20d3930 0x55abf20d5d20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.240+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55abf2053fe0 0x55abf205c590 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.240+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55abf204dfa0 0x55abf20caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.240+0000 7ff2021a3b80 1 --2- 172.21.15.121:0/3650161571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abf20536c0 0x55abf20caf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff1d8ff9700 0 client.15390 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:10.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3650161571 >> 172.21.15.121:0/3650161571 conn(0x55abf1f78d40 msgr2=0x55abf20591b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:10.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3650161571 shutdown_connections 2024-06-10T00:28:10.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff2021a3b80 1 -- 172.21.15.121:0/3650161571 wait complete. 2024-06-10T00:28:10.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff2021a3b80 20 client.15390 trim_cache size 0 max 16384 2024-06-10T00:28:10.245 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:28:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "return_code": 0, 2024-06-10T00:28:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_tag": "e0e5b8ec-b893-4254-a3bb-325bcb943ccd", 2024-06-10T00:28:10.245 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": "asynchronous" 2024-06-10T00:28:10.246 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff1f7920700 1 -- 172.21.15.121:0/874796484 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff1dc0a2e20 msgr2=0x7ff1dc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff1f7920700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff1dc0a2e20 0x7ff1dc0a52e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0199ab0 tx=0x7ff1ec009260 comp rx=0 tx=0).stop 2024-06-10T00:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff1f7920700 1 -- 172.21.15.121:0/874796484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f0101810 msgr2=0x7ff1f019ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff1f7920700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f0101810 0x7ff1f019ecf0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff1e000d950 tx=0x7ff1e000dc60 comp rx=0 tx=0).stop 2024-06-10T00:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff1f7920700 1 -- 172.21.15.121:0/874796484 shutdown_connections 2024-06-10T00:28:10.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff1f7920700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff1dc0a2e20 0x7ff1dc0a52e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff1f7920700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff1f0107570 0x7ff1f0198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff1f7920700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1f0100e50 0x7ff1f019e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff1f7920700 1 --2- 172.21.15.121:0/874796484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1f0101810 0x7ff1f019ecf0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.241+0000 7ff1f7920700 1 -- 172.21.15.121:0/874796484 >> 172.21.15.121:0/874796484 conn(0x7ff1f00766b0 msgr2=0x7ff1f00ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:10.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.242+0000 7ff1f7920700 1 -- 172.21.15.121:0/874796484 shutdown_connections 2024-06-10T00:28:10.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.242+0000 7ff1f7920700 1 -- 172.21.15.121:0/874796484 wait complete. 2024-06-10T00:28:10.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.273+0000 7f75adf14640 1 -- 172.21.15.121:0/245248327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75a81079c0 msgr2=0x7f75a8107da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.273+0000 7f75adf14640 1 --2- 172.21.15.121:0/245248327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75a81079c0 0x7f75a8107da0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7598009860 tx=0x7f759802a340 comp rx=0 tx=0).stop 2024-06-10T00:28:10.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.273+0000 7f75adf14640 1 -- 172.21.15.121:0/245248327 shutdown_connections 2024-06-10T00:28:10.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.273+0000 7f75adf14640 1 --2- 172.21.15.121:0/245248327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75a8102370 0x7f75a810e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.273+0000 7f75adf14640 1 --2- 172.21.15.121:0/245248327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a81019b0 0x7f75a8101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.273+0000 7f75adf14640 1 --2- 172.21.15.121:0/245248327 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75a81079c0 0x7f75a8107da0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.273+0000 7f75adf14640 1 -- 172.21.15.121:0/245248327 >> 172.21.15.121:0/245248327 conn(0x7f75a8078e30 msgr2=0x7f75a8079240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:10.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.273+0000 7f75adf14640 1 -- 172.21.15.121:0/245248327 shutdown_connections 2024-06-10T00:28:10.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.273+0000 7f75adf14640 1 -- 172.21.15.121:0/245248327 wait complete. 2024-06-10T00:28:10.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75adf14640 1 Processor -- start 2024-06-10T00:28:10.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75adf14640 1 -- start start 2024-06-10T00:28:10.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75adf14640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75a81019b0 0x7f75a8198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75adf14640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75a8102370 0x7f75a8198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75adf14640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a81079c0 0x7f75a819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75adf14640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f75a8111da0 con 0x7f75a81079c0 2024-06-10T00:28:10.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75adf14640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f75a8111c20 con 0x7f75a81019b0 2024-06-10T00:28:10.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75adf14640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f75a8111f20 con 0x7f75a8102370 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75a7fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a81079c0 0x7f75a819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75a7fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a81079c0 0x7f75a819ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34750/0 (socket says 172.21.15.121:34750) 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75a7fff640 1 -- 172.21.15.121:0/730530923 learned_addr learned my addr 172.21.15.121:0/730530923 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75a77fe640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75a81019b0 0x7f75a8198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.274+0000 7f75a6ffd640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75a8102370 0x7f75a8198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75a7fff640 1 -- 172.21.15.121:0/730530923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75a8102370 msgr2=0x7f75a8198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75a7fff640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75a8102370 0x7f75a8198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75a7fff640 1 -- 172.21.15.121:0/730530923 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75a81019b0 msgr2=0x7f75a8198220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75a7fff640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75a81019b0 0x7f75a8198220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75a7fff640 1 -- 172.21.15.121:0/730530923 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a819d1c0 con 0x7f75a81079c0 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75a77fe640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75a81019b0 0x7f75a8198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75a7fff640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a81079c0 0x7f75a819ca90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f75a000ec70 tx=0x7f75a000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75a4ff9640 1 -- 172.21.15.121:0/730530923 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0003b70 con 0x7f75a81079c0 2024-06-10T00:28:10.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75a4ff9640 1 -- 172.21.15.121:0/730530923 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a000ee80 con 0x7f75a81079c0 2024-06-10T00:28:10.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75adf14640 1 -- 172.21.15.121:0/730530923 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a819d4b0 con 0x7f75a81079c0 2024-06-10T00:28:10.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75a4ff9640 1 -- 172.21.15.121:0/730530923 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a00108a0 con 0x7f75a81079c0 2024-06-10T00:28:10.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.275+0000 7f75adf14640 1 -- 172.21.15.121:0/730530923 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a81a4d40 con 0x7f75a81079c0 2024-06-10T00:28:10.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.276+0000 7f75adf14640 1 -- 172.21.15.121:0/730530923 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a804f760 con 0x7f75a81079c0 2024-06-10T00:28:10.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.277+0000 7f75a4ff9640 1 -- 172.21.15.121:0/730530923 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f75a0004020 con 0x7f75a81079c0 2024-06-10T00:28:10.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.278+0000 7f75a4ff9640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f75900a2e40 0x7f75900a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:10.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.278+0000 7f75a77fe640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f75900a2e40 0x7f75900a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:10.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.278+0000 7f75a4ff9640 1 -- 172.21.15.121:0/730530923 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f75a00cfdb0 con 0x7f75a81079c0 2024-06-10T00:28:10.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.278+0000 7f75a77fe640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f75900a2e40 0x7f75900a5300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f759802a930 tx=0x7f75980056d0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:10.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.282+0000 7f75a4ff9640 1 -- 172.21.15.121:0/730530923 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f75a009c4d0 con 0x7f75a81079c0 2024-06-10T00:28:10.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.487+0000 7f75adf14640 1 -- 172.21.15.121:0/730530923 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f75a80ff870 con 0x7f75900a2e40 2024-06-10T00:28:10.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.494+0000 7f75a4ff9640 1 -- 172.21.15.121:0/730530923 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f75a80ff870 con 0x7f75900a2e40 2024-06-10T00:28:10.494 INFO:teuthology.orchestra.run.smithi121.stdout:/ 1m 6m3h 2024-06-10T00:28:10.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 -- 172.21.15.121:0/730530923 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f75900a2e40 msgr2=0x7f75900a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f75900a2e40 0x7f75900a5300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f759802a930 tx=0x7f75980056d0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 -- 172.21.15.121:0/730530923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a81079c0 msgr2=0x7f75a819ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a81079c0 0x7f75a819ca90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f75a000ec70 tx=0x7f75a000c6a0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 -- 172.21.15.121:0/730530923 shutdown_connections 2024-06-10T00:28:10.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a81079c0 0x7f75a819ca90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f75900a2e40 0x7f75900a5300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75a8102370 0x7f75a8198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 --2- 172.21.15.121:0/730530923 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75a81019b0 0x7f75a8198220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 -- 172.21.15.121:0/730530923 >> 172.21.15.121:0/730530923 conn(0x7f75a8078e30 msgr2=0x7f75a80ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 -- 172.21.15.121:0/730530923 shutdown_connections 2024-06-10T00:28:10.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:10.496+0000 7f75adf14640 1 -- 172.21.15.121:0/730530923 wait complete. 2024-06-10T00:28:10.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-06-10T00:28:10.574 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-06-10T00:28:10.585 INFO:teuthology.run_tasks:Running task quiescer... 2024-06-10T00:28:10.597 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-10T00:28:10.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-10T00:28:11.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:28:11.185 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:11.307 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:11 smithi121 ceph-mon[32378]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:11.307 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:11 smithi121 ceph-mon[32378]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 93 B/s wr, 16 op/s 2024-06-10T00:28:11.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:11 smithi121 ceph-mon[32378]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:11.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:11 smithi121 ceph-mon[32378]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:11.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:11 smithi121 ceph-mon[32378]: scrub queued for path: / 2024-06-10T00:28:11.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:11 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:28:11.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:11 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:28:11.308 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:11 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:28:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:11 smithi193 ceph-mon[36716]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:11 smithi193 ceph-mon[36716]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 93 B/s wr, 16 op/s 2024-06-10T00:28:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:11 smithi193 ceph-mon[36716]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:11 smithi193 ceph-mon[36716]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:11 smithi193 ceph-mon[36716]: scrub queued for path: / 2024-06-10T00:28:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:11 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:28:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:11 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:28:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:11 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:28:11.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:11 smithi135 ceph-mon[36424]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:11.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:11 smithi135 ceph-mon[36424]: pgmap v9: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 93 B/s wr, 16 op/s 2024-06-10T00:28:11.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:11 smithi135 ceph-mon[36424]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:11.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:11 smithi135 ceph-mon[36424]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:11.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:11 smithi135 ceph-mon[36424]: scrub queued for path: / 2024-06-10T00:28:11.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:11 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:28:11.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:11 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:28:11.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:11 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:28:11.912 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:12.563 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:12 smithi121 ceph-mon[32378]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:12.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:12 smithi193 ceph-mon[36716]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:12.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:12 smithi135 ceph-mon[36424]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:28:13.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:13 smithi121 ceph-mon[32378]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 85 B/s wr, 15 op/s 2024-06-10T00:28:13.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.594+0000 7f72e816e700 1 -- 172.21.15.121:0/3364175447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72e0101fd0 msgr2=0x7f72e0105fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:13.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.594+0000 7f72e816e700 1 --2- 172.21.15.121:0/3364175447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72e0101fd0 0x7f72e0105fd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f72d00099c0 tx=0x7f72d0009cd0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.594+0000 7f72e816e700 1 -- 172.21.15.121:0/3364175447 shutdown_connections 2024-06-10T00:28:13.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.594+0000 7f72e816e700 1 --2- 172.21.15.121:0/3364175447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72e01066b0 0x7f72e010e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.594+0000 7f72e816e700 1 --2- 172.21.15.121:0/3364175447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72e0101620 0x7f72e0101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.594+0000 7f72e816e700 1 --2- 172.21.15.121:0/3364175447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72e0101fd0 0x7f72e0105fd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.594+0000 7f72e816e700 1 -- 172.21.15.121:0/3364175447 >> 172.21.15.121:0/3364175447 conn(0x7f72e0078340 msgr2=0x7f72e0078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:13.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e816e700 1 -- 172.21.15.121:0/3364175447 shutdown_connections 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e816e700 1 -- 172.21.15.121:0/3364175447 wait complete. 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e816e700 1 Processor -- start 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e816e700 1 -- start start 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e816e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72e0101620 0x7f72e0198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e816e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72e0101fd0 0x7f72e0199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e816e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72e01066b0 0x7f72e019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e816e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f72e0111670 con 0x7f72e01066b0 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e816e700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01114f0 con 0x7f72e0101620 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e816e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01117f0 con 0x7f72e0101fd0 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e670b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72e01066b0 0x7f72e019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e670b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72e01066b0 0x7f72e019d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34774/0 (socket says 172.21.15.121:34774) 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.595+0000 7f72e670b700 1 -- 172.21.15.121:0/562249180 learned_addr learned my addr 172.21.15.121:0/562249180 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e5f0a700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72e0101620 0x7f72e0198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:13.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e5709700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72e0101fd0 0x7f72e0199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e670b700 1 -- 172.21.15.121:0/562249180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72e0101fd0 msgr2=0x7f72e0199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e670b700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72e0101fd0 0x7f72e0199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e670b700 1 -- 172.21.15.121:0/562249180 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72e0101620 msgr2=0x7f72e0198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e670b700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72e0101620 0x7f72e0198d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e670b700 1 -- 172.21.15.121:0/562249180 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e019dd20 con 0x7f72e01066b0 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e5f0a700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72e0101620 0x7f72e0198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e5709700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72e0101fd0 0x7f72e0199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e670b700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72e01066b0 0x7f72e019d5e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f72d8007ee0 tx=0x7f72d800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72d6ffd700 1 -- 172.21.15.121:0/562249180 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d800dbd0 con 0x7f72e01066b0 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e816e700 1 -- 172.21.15.121:0/562249180 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e019e010 con 0x7f72e01066b0 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72d6ffd700 1 -- 172.21.15.121:0/562249180 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d800dd70 con 0x7f72e01066b0 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72d6ffd700 1 -- 172.21.15.121:0/562249180 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d80133f0 con 0x7f72e01066b0 2024-06-10T00:28:13.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.596+0000 7f72e816e700 1 -- 172.21.15.121:0/562249180 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e01a5890 con 0x7f72e01066b0 2024-06-10T00:28:13.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.597+0000 7f72e816e700 1 -- 172.21.15.121:0/562249180 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72e0069180 con 0x7f72e01066b0 2024-06-10T00:28:13.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.598+0000 7f72d6ffd700 1 -- 172.21.15.121:0/562249180 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f72d801a020 con 0x7f72e01066b0 2024-06-10T00:28:13.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.599+0000 7f72d6ffd700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f72cc0a2e20 0x7f72cc0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:13.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.599+0000 7f72e5f0a700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f72cc0a2e20 0x7f72cc0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:13.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.599+0000 7f72d6ffd700 1 -- 172.21.15.121:0/562249180 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f72d80cf7c0 con 0x7f72e01066b0 2024-06-10T00:28:13.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.599+0000 7f72e5f0a700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f72cc0a2e20 0x7f72cc0a52e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f72dc005950 tx=0x7f72dc0058e0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:13.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.602+0000 7f72d6ffd700 1 -- 172.21.15.121:0/562249180 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f72d8010070 con 0x7f72e01066b0 2024-06-10T00:28:13.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:13 smithi193 ceph-mon[36716]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 85 B/s wr, 15 op/s 2024-06-10T00:28:13.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:13 smithi135 ceph-mon[36424]: pgmap v10: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 85 B/s wr, 15 op/s 2024-06-10T00:28:13.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.800+0000 7f72e816e700 1 -- 172.21.15.121:0/562249180 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f72e019a070 con 0x7f72cc0a2e20 2024-06-10T00:28:13.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.810+0000 7f72d6ffd700 1 -- 172.21.15.121:0/562249180 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339906 (secure 0 0 0) 0x7f72e019a070 con 0x7f72cc0a2e20 2024-06-10T00:28:13.814 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:13.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 -- 172.21.15.121:0/562249180 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f72cc0a2e20 msgr2=0x7f72cc0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:13.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f72cc0a2e20 0x7f72cc0a52e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f72dc005950 tx=0x7f72dc0058e0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 -- 172.21.15.121:0/562249180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72e01066b0 msgr2=0x7f72e019d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:13.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72e01066b0 0x7f72e019d5e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f72d8007ee0 tx=0x7f72d800d3b0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 -- 172.21.15.121:0/562249180 shutdown_connections 2024-06-10T00:28:13.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f72cc0a2e20 0x7f72cc0a52e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72e01066b0 0x7f72e019d5e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f72e0101620 0x7f72e0198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 --2- 172.21.15.121:0/562249180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72e0101fd0 0x7f72e0199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:13.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 -- 172.21.15.121:0/562249180 >> 172.21.15.121:0/562249180 conn(0x7f72e0078340 msgr2=0x7f72e006aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:13.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 -- 172.21.15.121:0/562249180 shutdown_connections 2024-06-10T00:28:13.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:13.813+0000 7f72e816e700 1 -- 172.21.15.121:0/562249180 wait complete. 2024-06-10T00:28:13.818 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-06-10T00:28:14.933 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-06-10T00:28:13.475688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1387034,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1748,"num_read_kb":1303,"num_write":630,"num_write_kb":5302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367136,"kb_used_data":13772,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574688,"statfs":{"total":1151940427776,"available":1151564480512,"internally_reserved":0,"allocated":14102528,"data_stored":11130608,"data_compressed":633915,"data_compressed_allocated":860160,"data_compressed_original":5050368,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":449,"num_read_kb":491,"num_write":148,"num_write_kb":915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002880"},"pg_stats":[{"pgid":"3.3f","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592226+0000","last_change":"2024-06-10T00:23:57.564073+0000","last_active":"2024-06-10T00:27:57.592226+0000","last_peered":"2024-06-10T00:27:57.592226+0000","last_clean":"2024-06-10T00:27:57.592226+0000","last_became_active":"2024-06-10T00:23:57.563618+0000","last_became_peered":"2024-06-10T00:23:57.563618+0000","last_unstale":"2024-06-10T00:27:57.592226+0000","last_undegraded":"2024-06-10T00:27:57.592226+0000","last_fullsized":"2024-06-10T00:27:57.592226+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:26:25.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"100'4","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591431+0000","last_change":"2024-06-10T00:23:57.560148+0000","last_active":"2024-06-10T00:27:57.591431+0000","last_peered":"2024-06-10T00:27:57.591431+0000","last_clean":"2024-06-10T00:27:57.591431+0000","last_became_active":"2024-06-10T00:23:57.559794+0000","last_became_peered":"2024-06-10T00:23:57.559794+0000","last_unstale":"2024-06-10T00:27:57.591431+0000","last_undegraded":"2024-06-10T00:27:57.591431+0000","last_fullsized":"2024-06-10T00:27:57.591431+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:59:50.624542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"100'7","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593168+0000","last_change":"2024-06-10T00:23:57.563292+0000","last_active":"2024-06-10T00:27:57.593168+0000","last_peered":"2024-06-10T00:27:57.593168+0000","last_clean":"2024-06-10T00:27:57.593168+0000","last_became_active":"2024-06-10T00:23:57.563007+0000","last_became_peered":"2024-06-10T00:23:57.563007+0000","last_unstale":"2024-06-10T00:27:57.593168+0000","last_undegraded":"2024-06-10T00:27:57.593168+0000","last_fullsized":"2024-06-10T00:27:57.593168+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:27:44.897770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545262+0000","last_change":"2024-06-10T00:23:57.558789+0000","last_active":"2024-06-10T00:27:57.545262+0000","last_peered":"2024-06-10T00:27:57.545262+0000","last_clean":"2024-06-10T00:27:57.545262+0000","last_became_active":"2024-06-10T00:23:57.557943+0000","last_became_peered":"2024-06-10T00:23:57.557943+0000","last_unstale":"2024-06-10T00:27:57.545262+0000","last_undegraded":"2024-06-10T00:27:57.545262+0000","last_fullsized":"2024-06-10T00:27:57.545262+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:10:40.081996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569868+0000","last_change":"2024-06-10T00:23:57.550806+0000","last_active":"2024-06-10T00:27:57.569868+0000","last_peered":"2024-06-10T00:27:57.569868+0000","last_clean":"2024-06-10T00:27:57.569868+0000","last_became_active":"2024-06-10T00:23:57.550453+0000","last_became_peered":"2024-06-10T00:23:57.550453+0000","last_unstale":"2024-06-10T00:27:57.569868+0000","last_undegraded":"2024-06-10T00:27:57.569868+0000","last_fullsized":"2024-06-10T00:27:57.569868+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:11:58.765099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458657+0000","last_change":"2024-06-10T00:23:57.567307+0000","last_active":"2024-06-10T00:27:57.458657+0000","last_peered":"2024-06-10T00:27:57.458657+0000","last_clean":"2024-06-10T00:27:57.458657+0000","last_became_active":"2024-06-10T00:23:57.566973+0000","last_became_peered":"2024-06-10T00:23:57.566973+0000","last_unstale":"2024-06-10T00:27:57.458657+0000","last_undegraded":"2024-06-10T00:27:57.458657+0000","last_fullsized":"2024-06-10T00:27:57.458657+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:27:25.135046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591279+0000","last_change":"2024-06-10T00:23:57.547998+0000","last_active":"2024-06-10T00:27:57.591279+0000","last_peered":"2024-06-10T00:27:57.591279+0000","last_clean":"2024-06-10T00:27:57.591279+0000","last_became_active":"2024-06-10T00:23:57.547593+0000","last_became_peered":"2024-06-10T00:23:57.547593+0000","last_unstale":"2024-06-10T00:27:57.591279+0000","last_undegraded":"2024-06-10T00:27:57.591279+0000","last_fullsized":"2024-06-10T00:27:57.591279+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:34:04.219399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570182+0000","last_change":"2024-06-10T00:23:57.560252+0000","last_active":"2024-06-10T00:27:57.570182+0000","last_peered":"2024-06-10T00:27:57.570182+0000","last_clean":"2024-06-10T00:27:57.570182+0000","last_became_active":"2024-06-10T00:23:57.559924+0000","last_became_peered":"2024-06-10T00:23:57.559924+0000","last_unstale":"2024-06-10T00:27:57.570182+0000","last_undegraded":"2024-06-10T00:27:57.570182+0000","last_fullsized":"2024-06-10T00:27:57.570182+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:01:43.684262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590876+0000","last_change":"2024-06-10T00:23:57.564031+0000","last_active":"2024-06-10T00:27:57.590876+0000","last_peered":"2024-06-10T00:27:57.590876+0000","last_clean":"2024-06-10T00:27:57.590876+0000","last_became_active":"2024-06-10T00:23:57.563642+0000","last_became_peered":"2024-06-10T00:23:57.563642+0000","last_unstale":"2024-06-10T00:27:57.590876+0000","last_undegraded":"2024-06-10T00:27:57.590876+0000","last_fullsized":"2024-06-10T00:27:57.590876+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:16:23.003152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"100'16","reported_seq":57,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459536+0000","last_change":"2024-06-10T00:23:57.566877+0000","last_active":"2024-06-10T00:27:57.459536+0000","last_peered":"2024-06-10T00:27:57.459536+0000","last_clean":"2024-06-10T00:27:57.459536+0000","last_became_active":"2024-06-10T00:23:57.566529+0000","last_became_peered":"2024-06-10T00:23:57.566529+0000","last_unstale":"2024-06-10T00:27:57.459536+0000","last_undegraded":"2024-06-10T00:27:57.459536+0000","last_fullsized":"2024-06-10T00:27:57.459536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:30:43.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592151+0000","last_change":"2024-06-10T00:23:57.564979+0000","last_active":"2024-06-10T00:27:57.592151+0000","last_peered":"2024-06-10T00:27:57.592151+0000","last_clean":"2024-06-10T00:27:57.592151+0000","last_became_active":"2024-06-10T00:23:57.564578+0000","last_became_peered":"2024-06-10T00:23:57.564578+0000","last_unstale":"2024-06-10T00:27:57.592151+0000","last_undegraded":"2024-06-10T00:27:57.592151+0000","last_fullsized":"2024-06-10T00:27:57.592151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:40:53.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"100'7","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545061+0000","last_change":"2024-06-10T00:23:57.559126+0000","last_active":"2024-06-10T00:27:57.545061+0000","last_peered":"2024-06-10T00:27:57.545061+0000","last_clean":"2024-06-10T00:27:57.545061+0000","last_became_active":"2024-06-10T00:23:57.558401+0000","last_became_peered":"2024-06-10T00:23:57.558401+0000","last_unstale":"2024-06-10T00:27:57.545061+0000","last_undegraded":"2024-06-10T00:27:57.545061+0000","last_fullsized":"2024-06-10T00:27:57.545061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:57.894137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542053+0000","last_change":"2024-06-10T00:23:57.559272+0000","last_active":"2024-06-10T00:27:57.542053+0000","last_peered":"2024-06-10T00:27:57.542053+0000","last_clean":"2024-06-10T00:27:57.542053+0000","last_became_active":"2024-06-10T00:23:57.558941+0000","last_became_peered":"2024-06-10T00:23:57.558941+0000","last_unstale":"2024-06-10T00:27:57.542053+0000","last_undegraded":"2024-06-10T00:27:57.542053+0000","last_fullsized":"2024-06-10T00:27:57.542053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:36:48.088347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591361+0000","last_change":"2024-06-10T00:23:57.555875+0000","last_active":"2024-06-10T00:27:57.591361+0000","last_peered":"2024-06-10T00:27:57.591361+0000","last_clean":"2024-06-10T00:27:57.591361+0000","last_became_active":"2024-06-10T00:23:57.555492+0000","last_became_peered":"2024-06-10T00:23:57.555492+0000","last_unstale":"2024-06-10T00:27:57.591361+0000","last_undegraded":"2024-06-10T00:27:57.591361+0000","last_fullsized":"2024-06-10T00:27:57.591361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:49:10.827458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591360+0000","last_change":"2024-06-10T00:23:57.555316+0000","last_active":"2024-06-10T00:27:57.591360+0000","last_peered":"2024-06-10T00:27:57.591360+0000","last_clean":"2024-06-10T00:27:57.591360+0000","last_became_active":"2024-06-10T00:23:57.554952+0000","last_became_peered":"2024-06-10T00:23:57.554952+0000","last_unstale":"2024-06-10T00:27:57.591360+0000","last_undegraded":"2024-06-10T00:27:57.591360+0000","last_fullsized":"2024-06-10T00:27:57.591360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:03:57.557330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591053+0000","last_change":"2024-06-10T00:23:57.562442+0000","last_active":"2024-06-10T00:27:57.591053+0000","last_peered":"2024-06-10T00:27:57.591053+0000","last_clean":"2024-06-10T00:27:57.591053+0000","last_became_active":"2024-06-10T00:23:57.561965+0000","last_became_peered":"2024-06-10T00:23:57.561965+0000","last_unstale":"2024-06-10T00:27:57.591053+0000","last_undegraded":"2024-06-10T00:27:57.591053+0000","last_fullsized":"2024-06-10T00:27:57.591053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:40:16.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591981+0000","last_change":"2024-06-10T00:23:57.564422+0000","last_active":"2024-06-10T00:27:57.591981+0000","last_peered":"2024-06-10T00:27:57.591981+0000","last_clean":"2024-06-10T00:27:57.591981+0000","last_became_active":"2024-06-10T00:23:57.564069+0000","last_became_peered":"2024-06-10T00:23:57.564069+0000","last_unstale":"2024-06-10T00:27:57.591981+0000","last_undegraded":"2024-06-10T00:27:57.591981+0000","last_fullsized":"2024-06-10T00:27:57.591981+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:07.662465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458239+0000","last_change":"2024-06-10T00:23:57.561284+0000","last_active":"2024-06-10T00:27:57.458239+0000","last_peered":"2024-06-10T00:27:57.458239+0000","last_clean":"2024-06-10T00:27:57.458239+0000","last_became_active":"2024-06-10T00:23:57.560933+0000","last_became_peered":"2024-06-10T00:23:57.560933+0000","last_unstale":"2024-06-10T00:27:57.458239+0000","last_undegraded":"2024-06-10T00:27:57.458239+0000","last_fullsized":"2024-06-10T00:27:57.458239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:58:48.511774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590777+0000","last_change":"2024-06-10T00:23:57.557752+0000","last_active":"2024-06-10T00:27:57.590777+0000","last_peered":"2024-06-10T00:27:57.590777+0000","last_clean":"2024-06-10T00:27:57.590777+0000","last_became_active":"2024-06-10T00:23:57.557419+0000","last_became_peered":"2024-06-10T00:23:57.557419+0000","last_unstale":"2024-06-10T00:27:57.590777+0000","last_undegraded":"2024-06-10T00:27:57.590777+0000","last_fullsized":"2024-06-10T00:27:57.590777+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:48.234181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591122+0000","last_change":"2024-06-10T00:24:09.177183+0000","last_active":"2024-06-10T00:27:57.591122+0000","last_peered":"2024-06-10T00:27:57.591122+0000","last_clean":"2024-06-10T00:27:57.591122+0000","last_became_active":"2024-06-10T00:24:09.176680+0000","last_became_peered":"2024-06-10T00:24:09.176680+0000","last_unstale":"2024-06-10T00:27:57.591122+0000","last_undegraded":"2024-06-10T00:27:57.591122+0000","last_fullsized":"2024-06-10T00:27:57.591122+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:17:06.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"98'9","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:03.469961+0000","last_change":"2024-06-10T00:23:54.426364+0000","last_active":"2024-06-10T00:28:03.469961+0000","last_peered":"2024-06-10T00:28:03.469961+0000","last_clean":"2024-06-10T00:28:03.469961+0000","last_became_active":"2024-06-10T00:23:54.426054+0000","last_became_peered":"2024-06-10T00:23:54.426054+0000","last_unstale":"2024-06-10T00:28:03.469961+0000","last_undegraded":"2024-06-10T00:28:03.469961+0000","last_fullsized":"2024-06-10T00:28:03.469961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:47.613402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570428+0000","last_change":"2024-06-10T00:23:57.557206+0000","last_active":"2024-06-10T00:27:57.570428+0000","last_peered":"2024-06-10T00:27:57.570428+0000","last_clean":"2024-06-10T00:27:57.570428+0000","last_became_active":"2024-06-10T00:23:57.556853+0000","last_became_peered":"2024-06-10T00:23:57.556853+0000","last_unstale":"2024-06-10T00:27:57.570428+0000","last_undegraded":"2024-06-10T00:27:57.570428+0000","last_fullsized":"2024-06-10T00:27:57.570428+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:09.686334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544782+0000","last_change":"2024-06-10T00:24:09.171047+0000","last_active":"2024-06-10T00:27:57.544782+0000","last_peered":"2024-06-10T00:27:57.544782+0000","last_clean":"2024-06-10T00:27:57.544782+0000","last_became_active":"2024-06-10T00:24:09.170501+0000","last_became_peered":"2024-06-10T00:24:09.170501+0000","last_unstale":"2024-06-10T00:27:57.544782+0000","last_undegraded":"2024-06-10T00:27:57.544782+0000","last_fullsized":"2024-06-10T00:27:57.544782+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:47:27.690439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"98'2","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571600+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:27:57.571600+0000","last_peered":"2024-06-10T00:27:57.571600+0000","last_clean":"2024-06-10T00:27:57.571600+0000","last_became_active":"2024-06-10T00:23:54.822447+0000","last_became_peered":"2024-06-10T00:23:54.822447+0000","last_unstale":"2024-06-10T00:27:57.571600+0000","last_undegraded":"2024-06-10T00:27:57.571600+0000","last_fullsized":"2024-06-10T00:27:57.571600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:45:14.956540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"100'6","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591883+0000","last_change":"2024-06-10T00:23:57.554958+0000","last_active":"2024-06-10T00:27:57.591883+0000","last_peered":"2024-06-10T00:27:57.591883+0000","last_clean":"2024-06-10T00:27:57.591883+0000","last_became_active":"2024-06-10T00:23:57.554477+0000","last_became_peered":"2024-06-10T00:23:57.554477+0000","last_unstale":"2024-06-10T00:27:57.591883+0000","last_undegraded":"2024-06-10T00:27:57.591883+0000","last_fullsized":"2024-06-10T00:27:57.591883+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:40:06.066967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544895+0000","last_change":"2024-06-10T00:24:09.169642+0000","last_active":"2024-06-10T00:27:57.544895+0000","last_peered":"2024-06-10T00:27:57.544895+0000","last_clean":"2024-06-10T00:27:57.544895+0000","last_became_active":"2024-06-10T00:24:09.168071+0000","last_became_peered":"2024-06-10T00:24:09.168071+0000","last_unstale":"2024-06-10T00:27:57.544895+0000","last_undegraded":"2024-06-10T00:27:57.544895+0000","last_fullsized":"2024-06-10T00:27:57.544895+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:44.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"98'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570379+0000","last_change":"2024-06-10T00:23:54.430543+0000","last_active":"2024-06-10T00:27:57.570379+0000","last_peered":"2024-06-10T00:27:57.570379+0000","last_clean":"2024-06-10T00:27:57.570379+0000","last_became_active":"2024-06-10T00:23:54.430138+0000","last_became_peered":"2024-06-10T00:23:54.430138+0000","last_unstale":"2024-06-10T00:27:57.570379+0000","last_undegraded":"2024-06-10T00:27:57.570379+0000","last_fullsized":"2024-06-10T00:27:57.570379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:39:53.212607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590796+0000","last_change":"2024-06-10T00:23:57.562251+0000","last_active":"2024-06-10T00:27:57.590796+0000","last_peered":"2024-06-10T00:27:57.590796+0000","last_clean":"2024-06-10T00:27:57.590796+0000","last_became_active":"2024-06-10T00:23:57.561903+0000","last_became_peered":"2024-06-10T00:23:57.561903+0000","last_unstale":"2024-06-10T00:27:57.590796+0000","last_undegraded":"2024-06-10T00:27:57.590796+0000","last_fullsized":"2024-06-10T00:27:57.590796+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:59.489475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544829+0000","last_change":"2024-06-10T00:24:09.165894+0000","last_active":"2024-06-10T00:27:57.544829+0000","last_peered":"2024-06-10T00:27:57.544829+0000","last_clean":"2024-06-10T00:27:57.544829+0000","last_became_active":"2024-06-10T00:24:09.165435+0000","last_became_peered":"2024-06-10T00:24:09.165435+0000","last_unstale":"2024-06-10T00:27:57.544829+0000","last_undegraded":"2024-06-10T00:27:57.544829+0000","last_fullsized":"2024-06-10T00:27:57.544829+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:13:36.058375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570296+0000","last_change":"2024-06-10T00:23:54.427868+0000","last_active":"2024-06-10T00:27:57.570296+0000","last_peered":"2024-06-10T00:27:57.570296+0000","last_clean":"2024-06-10T00:27:57.570296+0000","last_became_active":"2024-06-10T00:23:54.427366+0000","last_became_peered":"2024-06-10T00:23:54.427366+0000","last_unstale":"2024-06-10T00:27:57.570296+0000","last_undegraded":"2024-06-10T00:27:57.570296+0000","last_fullsized":"2024-06-10T00:27:57.570296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:15.499292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"100'15","reported_seq":62,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462390+0000","last_change":"2024-06-10T00:24:43.768556+0000","last_active":"2024-06-10T00:27:57.462390+0000","last_peered":"2024-06-10T00:27:57.462390+0000","last_clean":"2024-06-10T00:27:57.462390+0000","last_became_active":"2024-06-10T00:24:43.768090+0000","last_became_peered":"2024-06-10T00:24:43.768090+0000","last_unstale":"2024-06-10T00:27:57.462390+0000","last_undegraded":"2024-06-10T00:27:57.462390+0000","last_fullsized":"2024-06-10T00:27:57.462390+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:05:41.604184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591403+0000","last_change":"2024-06-10T00:24:09.170514+0000","last_active":"2024-06-10T00:27:57.591403+0000","last_peered":"2024-06-10T00:27:57.591403+0000","last_clean":"2024-06-10T00:27:57.591403+0000","last_became_active":"2024-06-10T00:24:09.168902+0000","last_became_peered":"2024-06-10T00:24:09.168902+0000","last_unstale":"2024-06-10T00:27:57.591403+0000","last_undegraded":"2024-06-10T00:27:57.591403+0000","last_fullsized":"2024-06-10T00:27:57.591403+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:33:10.534657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"101'9","reported_seq":250,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:11.572833+0000","last_change":"2024-06-10T00:23:54.426581+0000","last_active":"2024-06-10T00:28:11.572833+0000","last_peered":"2024-06-10T00:28:11.572833+0000","last_clean":"2024-06-10T00:28:11.572833+0000","last_became_active":"2024-06-10T00:23:54.426211+0000","last_became_peered":"2024-06-10T00:23:54.426211+0000","last_unstale":"2024-06-10T00:28:11.572833+0000","last_undegraded":"2024-06-10T00:28:11.572833+0000","last_fullsized":"2024-06-10T00:28:11.572833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:49:05.360742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":194,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569865+0000","last_change":"2024-06-10T00:23:57.552308+0000","last_active":"2024-06-10T00:27:57.569865+0000","last_peered":"2024-06-10T00:27:57.569865+0000","last_clean":"2024-06-10T00:27:57.569865+0000","last_became_active":"2024-06-10T00:23:57.551762+0000","last_became_peered":"2024-06-10T00:23:57.551762+0000","last_unstale":"2024-06-10T00:27:57.569865+0000","last_undegraded":"2024-06-10T00:27:57.569865+0000","last_fullsized":"2024-06-10T00:27:57.569865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:22:28.476019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591613+0000","last_change":"2024-06-10T00:24:09.171456+0000","last_active":"2024-06-10T00:27:57.591613+0000","last_peered":"2024-06-10T00:27:57.591613+0000","last_clean":"2024-06-10T00:27:57.591613+0000","last_became_active":"2024-06-10T00:24:09.170969+0000","last_became_peered":"2024-06-10T00:24:09.170969+0000","last_unstale":"2024-06-10T00:27:57.591613+0000","last_undegraded":"2024-06-10T00:27:57.591613+0000","last_fullsized":"2024-06-10T00:27:57.591613+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:22:46.023897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458602+0000","last_change":"2024-06-10T00:23:54.428384+0000","last_active":"2024-06-10T00:27:57.458602+0000","last_peered":"2024-06-10T00:27:57.458602+0000","last_clean":"2024-06-10T00:27:57.458602+0000","last_became_active":"2024-06-10T00:23:54.427967+0000","last_became_peered":"2024-06-10T00:23:54.427967+0000","last_unstale":"2024-06-10T00:27:57.458602+0000","last_undegraded":"2024-06-10T00:27:57.458602+0000","last_fullsized":"2024-06-10T00:27:57.458602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:54:30.408684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"100'8","reported_seq":55,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461853+0000","last_change":"2024-06-10T00:24:43.262593+0000","last_active":"2024-06-10T00:27:57.461853+0000","last_peered":"2024-06-10T00:27:57.461853+0000","last_clean":"2024-06-10T00:27:57.461853+0000","last_became_active":"2024-06-10T00:24:43.262212+0000","last_became_peered":"2024-06-10T00:24:43.262212+0000","last_unstale":"2024-06-10T00:27:57.461853+0000","last_undegraded":"2024-06-10T00:27:57.461853+0000","last_fullsized":"2024-06-10T00:27:57.461853+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:40:24.442712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591973+0000","last_change":"2024-06-10T00:24:43.752805+0000","last_active":"2024-06-10T00:27:57.591973+0000","last_peered":"2024-06-10T00:27:57.591973+0000","last_clean":"2024-06-10T00:27:57.591973+0000","last_became_active":"2024-06-10T00:24:43.752251+0000","last_became_peered":"2024-06-10T00:24:43.752251+0000","last_unstale":"2024-06-10T00:27:57.591973+0000","last_undegraded":"2024-06-10T00:27:57.591973+0000","last_fullsized":"2024-06-10T00:27:57.591973+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:27:57.617443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542566+0000","last_change":"2024-06-10T00:23:54.421493+0000","last_active":"2024-06-10T00:27:57.542566+0000","last_peered":"2024-06-10T00:27:57.542566+0000","last_clean":"2024-06-10T00:27:57.542566+0000","last_became_active":"2024-06-10T00:23:54.421151+0000","last_became_peered":"2024-06-10T00:23:54.421151+0000","last_unstale":"2024-06-10T00:27:57.542566+0000","last_undegraded":"2024-06-10T00:27:57.542566+0000","last_fullsized":"2024-06-10T00:27:57.542566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:03.995244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570099+0000","last_change":"2024-06-10T00:23:57.554020+0000","last_active":"2024-06-10T00:27:57.570099+0000","last_peered":"2024-06-10T00:27:57.570099+0000","last_clean":"2024-06-10T00:27:57.570099+0000","last_became_active":"2024-06-10T00:23:57.553690+0000","last_became_peered":"2024-06-10T00:23:57.553690+0000","last_unstale":"2024-06-10T00:27:57.570099+0000","last_undegraded":"2024-06-10T00:27:57.570099+0000","last_fullsized":"2024-06-10T00:27:57.570099+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:48.097074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593638+0000","last_change":"2024-06-10T00:24:09.166426+0000","last_active":"2024-06-10T00:27:57.593638+0000","last_peered":"2024-06-10T00:27:57.593638+0000","last_clean":"2024-06-10T00:27:57.593638+0000","last_became_active":"2024-06-10T00:24:09.166047+0000","last_became_peered":"2024-06-10T00:24:09.166047+0000","last_unstale":"2024-06-10T00:27:57.593638+0000","last_undegraded":"2024-06-10T00:27:57.593638+0000","last_fullsized":"2024-06-10T00:27:57.593638+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:56:11.608061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"98'1","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461240+0000","last_change":"2024-06-10T00:23:54.426409+0000","last_active":"2024-06-10T00:27:57.461240+0000","last_peered":"2024-06-10T00:27:57.461240+0000","last_clean":"2024-06-10T00:27:57.461240+0000","last_became_active":"2024-06-10T00:23:54.425385+0000","last_became_peered":"2024-06-10T00:23:54.425385+0000","last_unstale":"2024-06-10T00:27:57.461240+0000","last_undegraded":"2024-06-10T00:27:57.461240+0000","last_fullsized":"2024-06-10T00:27:57.461240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:27:00.379189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457389+0000","last_change":"2024-06-10T00:23:57.560132+0000","last_active":"2024-06-10T00:27:57.457389+0000","last_peered":"2024-06-10T00:27:57.457389+0000","last_clean":"2024-06-10T00:27:57.457389+0000","last_became_active":"2024-06-10T00:23:57.559803+0000","last_became_peered":"2024-06-10T00:23:57.559803+0000","last_unstale":"2024-06-10T00:27:57.457389+0000","last_undegraded":"2024-06-10T00:27:57.457389+0000","last_fullsized":"2024-06-10T00:27:57.457389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:13:16.004233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571137+0000","last_change":"2024-06-10T00:24:09.170962+0000","last_active":"2024-06-10T00:27:57.571137+0000","last_peered":"2024-06-10T00:27:57.571137+0000","last_clean":"2024-06-10T00:27:57.571137+0000","last_became_active":"2024-06-10T00:24:09.170090+0000","last_became_peered":"2024-06-10T00:24:09.170090+0000","last_unstale":"2024-06-10T00:27:57.571137+0000","last_undegraded":"2024-06-10T00:27:57.571137+0000","last_fullsized":"2024-06-10T00:27:57.571137+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:07:33.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'5","reported_seq":240,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:11.572251+0000","last_change":"2024-06-10T00:23:54.421472+0000","last_active":"2024-06-10T00:28:11.572251+0000","last_peered":"2024-06-10T00:28:11.572251+0000","last_clean":"2024-06-10T00:28:11.572251+0000","last_became_active":"2024-06-10T00:23:54.421019+0000","last_became_peered":"2024-06-10T00:23:54.421019+0000","last_unstale":"2024-06-10T00:28:11.572251+0000","last_undegraded":"2024-06-10T00:28:11.572251+0000","last_fullsized":"2024-06-10T00:28:11.572251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:28:18.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":188,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590963+0000","last_change":"2024-06-10T00:23:57.563010+0000","last_active":"2024-06-10T00:27:57.590963+0000","last_peered":"2024-06-10T00:27:57.590963+0000","last_clean":"2024-06-10T00:27:57.590963+0000","last_became_active":"2024-06-10T00:23:57.561986+0000","last_became_peered":"2024-06-10T00:23:57.561986+0000","last_unstale":"2024-06-10T00:27:57.590963+0000","last_undegraded":"2024-06-10T00:27:57.590963+0000","last_fullsized":"2024-06-10T00:27:57.590963+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:31:17.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457415+0000","last_change":"2024-06-10T00:24:09.171746+0000","last_active":"2024-06-10T00:27:57.457415+0000","last_peered":"2024-06-10T00:27:57.457415+0000","last_clean":"2024-06-10T00:27:57.457415+0000","last_became_active":"2024-06-10T00:24:09.171030+0000","last_became_peered":"2024-06-10T00:24:09.171030+0000","last_unstale":"2024-06-10T00:27:57.457415+0000","last_undegraded":"2024-06-10T00:27:57.457415+0000","last_fullsized":"2024-06-10T00:27:57.457415+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:41.669471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591812+0000","last_change":"2024-06-10T00:23:54.428917+0000","last_active":"2024-06-10T00:27:57.591812+0000","last_peered":"2024-06-10T00:27:57.591812+0000","last_clean":"2024-06-10T00:27:57.591812+0000","last_became_active":"2024-06-10T00:23:54.428585+0000","last_became_peered":"2024-06-10T00:23:54.428585+0000","last_unstale":"2024-06-10T00:27:57.591812+0000","last_undegraded":"2024-06-10T00:27:57.591812+0000","last_fullsized":"2024-06-10T00:27:57.591812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:21:58.235848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591586+0000","last_change":"2024-06-10T00:23:57.551778+0000","last_active":"2024-06-10T00:27:57.591586+0000","last_peered":"2024-06-10T00:27:57.591586+0000","last_clean":"2024-06-10T00:27:57.591586+0000","last_became_active":"2024-06-10T00:23:57.551436+0000","last_became_peered":"2024-06-10T00:23:57.551436+0000","last_unstale":"2024-06-10T00:27:57.591586+0000","last_undegraded":"2024-06-10T00:27:57.591586+0000","last_fullsized":"2024-06-10T00:27:57.591586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:30.059783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545101+0000","last_change":"2024-06-10T00:24:09.167547+0000","last_active":"2024-06-10T00:27:57.545101+0000","last_peered":"2024-06-10T00:27:57.545101+0000","last_clean":"2024-06-10T00:27:57.545101+0000","last_became_active":"2024-06-10T00:24:09.167132+0000","last_became_peered":"2024-06-10T00:24:09.167132+0000","last_unstale":"2024-06-10T00:27:57.545101+0000","last_undegraded":"2024-06-10T00:27:57.545101+0000","last_fullsized":"2024-06-10T00:27:57.545101+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:11.520330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461450+0000","last_change":"2024-06-10T00:23:54.823754+0000","last_active":"2024-06-10T00:27:57.461450+0000","last_peered":"2024-06-10T00:27:57.461450+0000","last_clean":"2024-06-10T00:27:57.461450+0000","last_became_active":"2024-06-10T00:23:54.823362+0000","last_became_peered":"2024-06-10T00:23:54.823362+0000","last_unstale":"2024-06-10T00:27:57.461450+0000","last_undegraded":"2024-06-10T00:27:57.461450+0000","last_fullsized":"2024-06-10T00:27:57.461450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:48:50.915439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590772+0000","last_change":"2024-06-10T00:23:57.563313+0000","last_active":"2024-06-10T00:27:57.590772+0000","last_peered":"2024-06-10T00:27:57.590772+0000","last_clean":"2024-06-10T00:27:57.590772+0000","last_became_active":"2024-06-10T00:23:57.562307+0000","last_became_peered":"2024-06-10T00:23:57.562307+0000","last_unstale":"2024-06-10T00:27:57.590772+0000","last_undegraded":"2024-06-10T00:27:57.590772+0000","last_fullsized":"2024-06-10T00:27:57.590772+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:23.815267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457507+0000","last_change":"2024-06-10T00:24:09.174660+0000","last_active":"2024-06-10T00:27:57.457507+0000","last_peered":"2024-06-10T00:27:57.457507+0000","last_clean":"2024-06-10T00:27:57.457507+0000","last_became_active":"2024-06-10T00:24:09.174283+0000","last_became_peered":"2024-06-10T00:24:09.174283+0000","last_unstale":"2024-06-10T00:27:57.457507+0000","last_undegraded":"2024-06-10T00:27:57.457507+0000","last_fullsized":"2024-06-10T00:27:57.457507+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:15.236146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"101'41","reported_seq":478,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:12.573920+0000","last_change":"2024-06-10T00:23:54.426621+0000","last_active":"2024-06-10T00:28:12.573920+0000","last_peered":"2024-06-10T00:28:12.573920+0000","last_clean":"2024-06-10T00:28:12.573920+0000","last_became_active":"2024-06-10T00:23:54.426131+0000","last_became_peered":"2024-06-10T00:23:54.426131+0000","last_unstale":"2024-06-10T00:28:12.573920+0000","last_undegraded":"2024-06-10T00:28:12.573920+0000","last_fullsized":"2024-06-10T00:28:12.573920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:56:14.819454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69365,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":390,"num_read_kb":264,"num_write":42,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570532+0000","last_change":"2024-06-10T00:23:57.552606+0000","last_active":"2024-06-10T00:27:57.570532+0000","last_peered":"2024-06-10T00:27:57.570532+0000","last_clean":"2024-06-10T00:27:57.570532+0000","last_became_active":"2024-06-10T00:23:57.552246+0000","last_became_peered":"2024-06-10T00:23:57.552246+0000","last_unstale":"2024-06-10T00:27:57.570532+0000","last_undegraded":"2024-06-10T00:27:57.570532+0000","last_fullsized":"2024-06-10T00:27:57.570532+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:19:21.508721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462817+0000","last_change":"2024-06-10T00:24:09.174862+0000","last_active":"2024-06-10T00:27:57.462817+0000","last_peered":"2024-06-10T00:27:57.462817+0000","last_clean":"2024-06-10T00:27:57.462817+0000","last_became_active":"2024-06-10T00:24:09.173550+0000","last_became_peered":"2024-06-10T00:24:09.173550+0000","last_unstale":"2024-06-10T00:27:57.462817+0000","last_undegraded":"2024-06-10T00:27:57.462817+0000","last_fullsized":"2024-06-10T00:27:57.462817+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:49:25.549084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"98'6","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591325+0000","last_change":"2024-06-10T00:23:54.426074+0000","last_active":"2024-06-10T00:27:57.591325+0000","last_peered":"2024-06-10T00:27:57.591325+0000","last_clean":"2024-06-10T00:27:57.591325+0000","last_became_active":"2024-06-10T00:23:54.425707+0000","last_became_peered":"2024-06-10T00:23:54.425707+0000","last_unstale":"2024-06-10T00:27:57.591325+0000","last_undegraded":"2024-06-10T00:27:57.591325+0000","last_fullsized":"2024-06-10T00:27:57.591325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:11:10.563595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542128+0000","last_change":"2024-06-10T00:23:57.549784+0000","last_active":"2024-06-10T00:27:57.542128+0000","last_peered":"2024-06-10T00:27:57.542128+0000","last_clean":"2024-06-10T00:27:57.542128+0000","last_became_active":"2024-06-10T00:23:57.549457+0000","last_became_peered":"2024-06-10T00:23:57.549457+0000","last_unstale":"2024-06-10T00:27:57.542128+0000","last_undegraded":"2024-06-10T00:27:57.542128+0000","last_fullsized":"2024-06-10T00:27:57.542128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:59:27.352927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592095+0000","last_change":"2024-06-10T00:24:09.171028+0000","last_active":"2024-06-10T00:27:57.592095+0000","last_peered":"2024-06-10T00:27:57.592095+0000","last_clean":"2024-06-10T00:27:57.592095+0000","last_became_active":"2024-06-10T00:24:09.170543+0000","last_became_peered":"2024-06-10T00:24:09.170543+0000","last_unstale":"2024-06-10T00:27:57.592095+0000","last_undegraded":"2024-06-10T00:27:57.592095+0000","last_fullsized":"2024-06-10T00:27:57.592095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:57:40.816585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461583+0000","last_change":"2024-06-10T00:23:54.423332+0000","last_active":"2024-06-10T00:27:57.461583+0000","last_peered":"2024-06-10T00:27:57.461583+0000","last_clean":"2024-06-10T00:27:57.461583+0000","last_became_active":"2024-06-10T00:23:54.422792+0000","last_became_peered":"2024-06-10T00:23:54.422792+0000","last_unstale":"2024-06-10T00:27:57.461583+0000","last_undegraded":"2024-06-10T00:27:57.461583+0000","last_fullsized":"2024-06-10T00:27:57.461583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:50:52.655641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"100'15","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591219+0000","last_change":"2024-06-10T00:23:57.551411+0000","last_active":"2024-06-10T00:27:57.591219+0000","last_peered":"2024-06-10T00:27:57.591219+0000","last_clean":"2024-06-10T00:27:57.591219+0000","last_became_active":"2024-06-10T00:23:57.551053+0000","last_became_peered":"2024-06-10T00:23:57.551053+0000","last_unstale":"2024-06-10T00:27:57.591219+0000","last_undegraded":"2024-06-10T00:27:57.591219+0000","last_fullsized":"2024-06-10T00:27:57.591219+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:16:18.082777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459282+0000","last_change":"2024-06-10T00:24:43.648767+0000","last_active":"2024-06-10T00:27:57.459282+0000","last_peered":"2024-06-10T00:27:57.459282+0000","last_clean":"2024-06-10T00:27:57.459282+0000","last_became_active":"2024-06-10T00:24:43.648267+0000","last_became_peered":"2024-06-10T00:24:43.648267+0000","last_unstale":"2024-06-10T00:27:57.459282+0000","last_undegraded":"2024-06-10T00:27:57.459282+0000","last_fullsized":"2024-06-10T00:27:57.459282+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:41:09.005202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"101'5","reported_seq":242,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:11.565030+0000","last_change":"2024-06-10T00:23:54.425963+0000","last_active":"2024-06-10T00:28:11.565030+0000","last_peered":"2024-06-10T00:28:11.565030+0000","last_clean":"2024-06-10T00:28:11.565030+0000","last_became_active":"2024-06-10T00:23:54.425581+0000","last_became_peered":"2024-06-10T00:23:54.425581+0000","last_unstale":"2024-06-10T00:28:11.565030+0000","last_undegraded":"2024-06-10T00:28:11.565030+0000","last_fullsized":"2024-06-10T00:28:11.565030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:54.760610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":192,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592340+0000","last_change":"2024-06-10T00:23:57.557464+0000","last_active":"2024-06-10T00:27:57.592340+0000","last_peered":"2024-06-10T00:27:57.592340+0000","last_clean":"2024-06-10T00:27:57.592340+0000","last_became_active":"2024-06-10T00:23:57.557130+0000","last_became_peered":"2024-06-10T00:23:57.557130+0000","last_unstale":"2024-06-10T00:27:57.592340+0000","last_undegraded":"2024-06-10T00:27:57.592340+0000","last_fullsized":"2024-06-10T00:27:57.592340+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:16:36.341210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":38,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543379+0000","last_change":"2024-06-10T00:24:43.753291+0000","last_active":"2024-06-10T00:27:57.543379+0000","last_peered":"2024-06-10T00:27:57.543379+0000","last_clean":"2024-06-10T00:27:57.543379+0000","last_became_active":"2024-06-10T00:24:43.752765+0000","last_became_peered":"2024-06-10T00:24:43.752765+0000","last_unstale":"2024-06-10T00:27:57.543379+0000","last_undegraded":"2024-06-10T00:27:57.543379+0000","last_fullsized":"2024-06-10T00:27:57.543379+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:55:06.349564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"98'15","reported_seq":63,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570040+0000","last_change":"2024-06-10T00:23:54.823489+0000","last_active":"2024-06-10T00:27:57.570040+0000","last_peered":"2024-06-10T00:27:57.570040+0000","last_clean":"2024-06-10T00:27:57.570040+0000","last_became_active":"2024-06-10T00:23:54.823073+0000","last_became_peered":"2024-06-10T00:23:54.823073+0000","last_unstale":"2024-06-10T00:27:57.570040+0000","last_undegraded":"2024-06-10T00:27:57.570040+0000","last_fullsized":"2024-06-10T00:27:57.570040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:07:52.563009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461389+0000","last_change":"2024-06-10T00:23:57.563388+0000","last_active":"2024-06-10T00:27:57.461389+0000","last_peered":"2024-06-10T00:27:57.461389+0000","last_clean":"2024-06-10T00:27:57.461389+0000","last_became_active":"2024-06-10T00:23:57.563004+0000","last_became_peered":"2024-06-10T00:23:57.563004+0000","last_unstale":"2024-06-10T00:27:57.461389+0000","last_undegraded":"2024-06-10T00:27:57.461389+0000","last_fullsized":"2024-06-10T00:27:57.461389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:51:18.805662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461226+0000","last_change":"2024-06-10T00:24:09.168447+0000","last_active":"2024-06-10T00:27:57.461226+0000","last_peered":"2024-06-10T00:27:57.461226+0000","last_clean":"2024-06-10T00:27:57.461226+0000","last_became_active":"2024-06-10T00:24:09.167439+0000","last_became_peered":"2024-06-10T00:24:09.167439+0000","last_unstale":"2024-06-10T00:27:57.461226+0000","last_undegraded":"2024-06-10T00:27:57.461226+0000","last_fullsized":"2024-06-10T00:27:57.461226+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:52.319395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"101'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570752+0000","last_change":"2024-06-10T00:23:54.427666+0000","last_active":"2024-06-10T00:27:57.570752+0000","last_peered":"2024-06-10T00:27:57.570752+0000","last_clean":"2024-06-10T00:27:57.570752+0000","last_became_active":"2024-06-10T00:23:54.427285+0000","last_became_peered":"2024-06-10T00:23:54.427285+0000","last_unstale":"2024-06-10T00:27:57.570752+0000","last_undegraded":"2024-06-10T00:27:57.570752+0000","last_fullsized":"2024-06-10T00:27:57.570752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:28:20.196168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"100'5","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.558787+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.558470+0000","last_became_peered":"2024-06-10T00:23:57.558470+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:02:14.758802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570142+0000","last_change":"2024-06-10T00:24:09.264952+0000","last_active":"2024-06-10T00:27:57.570142+0000","last_peered":"2024-06-10T00:27:57.570142+0000","last_clean":"2024-06-10T00:27:57.570142+0000","last_became_active":"2024-06-10T00:24:09.264379+0000","last_became_peered":"2024-06-10T00:24:09.264379+0000","last_unstale":"2024-06-10T00:27:57.570142+0000","last_undegraded":"2024-06-10T00:27:57.570142+0000","last_fullsized":"2024-06-10T00:27:57.570142+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:25:40.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458166+0000","last_change":"2024-06-10T00:23:54.823298+0000","last_active":"2024-06-10T00:27:57.458166+0000","last_peered":"2024-06-10T00:27:57.458166+0000","last_clean":"2024-06-10T00:27:57.458166+0000","last_became_active":"2024-06-10T00:23:54.822917+0000","last_became_peered":"2024-06-10T00:23:54.822917+0000","last_unstale":"2024-06-10T00:27:57.458166+0000","last_undegraded":"2024-06-10T00:27:57.458166+0000","last_fullsized":"2024-06-10T00:27:57.458166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:07:52.500641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"100'4","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459230+0000","last_change":"2024-06-10T00:23:57.559202+0000","last_active":"2024-06-10T00:27:57.459230+0000","last_peered":"2024-06-10T00:27:57.459230+0000","last_clean":"2024-06-10T00:27:57.459230+0000","last_became_active":"2024-06-10T00:23:57.558800+0000","last_became_peered":"2024-06-10T00:23:57.558800+0000","last_unstale":"2024-06-10T00:27:57.459230+0000","last_undegraded":"2024-06-10T00:27:57.459230+0000","last_fullsized":"2024-06-10T00:27:57.459230+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:49:51.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542486+0000","last_change":"2024-06-10T00:24:09.168603+0000","last_active":"2024-06-10T00:27:57.542486+0000","last_peered":"2024-06-10T00:27:57.542486+0000","last_clean":"2024-06-10T00:27:57.542486+0000","last_became_active":"2024-06-10T00:24:09.168233+0000","last_became_peered":"2024-06-10T00:24:09.168233+0000","last_unstale":"2024-06-10T00:27:57.542486+0000","last_undegraded":"2024-06-10T00:27:57.542486+0000","last_fullsized":"2024-06-10T00:27:57.542486+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:59.585023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'1","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592243+0000","last_change":"2024-06-10T00:23:54.425505+0000","last_active":"2024-06-10T00:27:57.592243+0000","last_peered":"2024-06-10T00:27:57.592243+0000","last_clean":"2024-06-10T00:27:57.592243+0000","last_became_active":"2024-06-10T00:23:54.425058+0000","last_became_peered":"2024-06-10T00:23:54.425058+0000","last_unstale":"2024-06-10T00:27:57.592243+0000","last_undegraded":"2024-06-10T00:27:57.592243+0000","last_fullsized":"2024-06-10T00:27:57.592243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:48:05.605947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"100'7","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458879+0000","last_change":"2024-06-10T00:23:57.566060+0000","last_active":"2024-06-10T00:27:57.458879+0000","last_peered":"2024-06-10T00:27:57.458879+0000","last_clean":"2024-06-10T00:27:57.458879+0000","last_became_active":"2024-06-10T00:23:57.565099+0000","last_became_peered":"2024-06-10T00:23:57.565099+0000","last_unstale":"2024-06-10T00:27:57.458879+0000","last_undegraded":"2024-06-10T00:27:57.458879+0000","last_fullsized":"2024-06-10T00:27:57.458879+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:24:54.204399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543284+0000","last_change":"2024-06-10T00:24:09.264741+0000","last_active":"2024-06-10T00:27:57.543284+0000","last_peered":"2024-06-10T00:27:57.543284+0000","last_clean":"2024-06-10T00:27:57.543284+0000","last_became_active":"2024-06-10T00:24:09.264267+0000","last_became_peered":"2024-06-10T00:24:09.264267+0000","last_unstale":"2024-06-10T00:27:57.543284+0000","last_undegraded":"2024-06-10T00:27:57.543284+0000","last_fullsized":"2024-06-10T00:27:57.543284+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:27.627391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457224+0000","last_change":"2024-06-10T00:25:44.346070+0000","last_active":"2024-06-10T00:27:57.457224+0000","last_peered":"2024-06-10T00:27:57.457224+0000","last_clean":"2024-06-10T00:27:57.457224+0000","last_became_active":"2024-06-10T00:25:44.345798+0000","last_became_peered":"2024-06-10T00:25:44.345798+0000","last_unstale":"2024-06-10T00:27:57.457224+0000","last_undegraded":"2024-06-10T00:27:57.457224+0000","last_fullsized":"2024-06-10T00:27:57.457224+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:04.506535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"101'5","reported_seq":241,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:11.564249+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:28:11.564249+0000","last_peered":"2024-06-10T00:28:11.564249+0000","last_clean":"2024-06-10T00:28:11.564249+0000","last_became_active":"2024-06-10T00:23:54.822442+0000","last_became_peered":"2024-06-10T00:23:54.822442+0000","last_unstale":"2024-06-10T00:28:11.564249+0000","last_undegraded":"2024-06-10T00:28:11.564249+0000","last_fullsized":"2024-06-10T00:28:11.564249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:51:10.278257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":188,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"98'1","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591953+0000","last_change":"2024-06-10T00:24:09.168073+0000","last_active":"2024-06-10T00:27:57.591953+0000","last_peered":"2024-06-10T00:27:57.591953+0000","last_clean":"2024-06-10T00:27:57.591953+0000","last_became_active":"2024-06-10T00:24:09.167657+0000","last_became_peered":"2024-06-10T00:24:09.167657+0000","last_unstale":"2024-06-10T00:27:57.591953+0000","last_undegraded":"2024-06-10T00:27:57.591953+0000","last_fullsized":"2024-06-10T00:27:57.591953+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:20:23.481685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457629+0000","last_change":"2024-06-10T00:25:44.346629+0000","last_active":"2024-06-10T00:27:57.457629+0000","last_peered":"2024-06-10T00:27:57.457629+0000","last_clean":"2024-06-10T00:27:57.457629+0000","last_became_active":"2024-06-10T00:25:44.346388+0000","last_became_peered":"2024-06-10T00:25:44.346388+0000","last_unstale":"2024-06-10T00:27:57.457629+0000","last_undegraded":"2024-06-10T00:27:57.457629+0000","last_fullsized":"2024-06-10T00:27:57.457629+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:03:19.228632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"98'3","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544980+0000","last_change":"2024-06-10T00:23:54.423507+0000","last_active":"2024-06-10T00:27:57.544980+0000","last_peered":"2024-06-10T00:27:57.544980+0000","last_clean":"2024-06-10T00:27:57.544980+0000","last_became_active":"2024-06-10T00:23:54.423092+0000","last_became_peered":"2024-06-10T00:23:54.423092+0000","last_unstale":"2024-06-10T00:27:57.544980+0000","last_undegraded":"2024-06-10T00:27:57.544980+0000","last_fullsized":"2024-06-10T00:27:57.544980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:06:15.924827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"101'148","reported_seq":424,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:59.577811+0000","last_change":"2024-06-10T00:22:28.025243+0000","last_active":"2024-06-10T00:27:59.577811+0000","last_peered":"2024-06-10T00:27:59.577811+0000","last_clean":"2024-06-10T00:27:59.577811+0000","last_became_active":"2024-06-10T00:22:28.024751+0000","last_became_peered":"2024-06-10T00:22:28.024751+0000","last_unstale":"2024-06-10T00:27:59.577811+0000","last_undegraded":"2024-06-10T00:27:59.577811+0000","last_fullsized":"2024-06-10T00:27:59.577811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_clean_scrub_stamp":"2024-06-10T00:19:05.039631+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:02:36.557931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461952+0000","last_change":"2024-06-10T00:24:09.265084+0000","last_active":"2024-06-10T00:27:57.461952+0000","last_peered":"2024-06-10T00:27:57.461952+0000","last_clean":"2024-06-10T00:27:57.461952+0000","last_became_active":"2024-06-10T00:24:09.264425+0000","last_became_peered":"2024-06-10T00:24:09.264425+0000","last_unstale":"2024-06-10T00:27:57.461952+0000","last_undegraded":"2024-06-10T00:27:57.461952+0000","last_fullsized":"2024-06-10T00:27:57.461952+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:37:31.577688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.541983+0000","last_change":"2024-06-10T00:23:57.558540+0000","last_active":"2024-06-10T00:27:57.541983+0000","last_peered":"2024-06-10T00:27:57.541983+0000","last_clean":"2024-06-10T00:27:57.541983+0000","last_became_active":"2024-06-10T00:23:57.558197+0000","last_became_peered":"2024-06-10T00:23:57.558197+0000","last_unstale":"2024-06-10T00:27:57.541983+0000","last_undegraded":"2024-06-10T00:27:57.541983+0000","last_fullsized":"2024-06-10T00:27:57.541983+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:04:25.566135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592707+0000","last_change":"2024-06-10T00:23:54.428126+0000","last_active":"2024-06-10T00:27:57.592707+0000","last_peered":"2024-06-10T00:27:57.592707+0000","last_clean":"2024-06-10T00:27:57.592707+0000","last_became_active":"2024-06-10T00:23:54.427630+0000","last_became_peered":"2024-06-10T00:23:54.427630+0000","last_unstale":"2024-06-10T00:27:57.592707+0000","last_undegraded":"2024-06-10T00:27:57.592707+0000","last_fullsized":"2024-06-10T00:27:57.592707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:05:23.679704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591957+0000","last_change":"2024-06-10T00:24:09.169984+0000","last_active":"2024-06-10T00:27:57.591957+0000","last_peered":"2024-06-10T00:27:57.591957+0000","last_clean":"2024-06-10T00:27:57.591957+0000","last_became_active":"2024-06-10T00:24:09.169556+0000","last_became_peered":"2024-06-10T00:24:09.169556+0000","last_unstale":"2024-06-10T00:27:57.591957+0000","last_undegraded":"2024-06-10T00:27:57.591957+0000","last_fullsized":"2024-06-10T00:27:57.591957+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:54:13.575207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"98'1","reported_seq":20,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591972+0000","last_change":"2024-06-10T00:24:43.767901+0000","last_active":"2024-06-10T00:27:57.591972+0000","last_peered":"2024-06-10T00:27:57.591972+0000","last_clean":"2024-06-10T00:27:57.591972+0000","last_became_active":"2024-06-10T00:24:43.767562+0000","last_became_peered":"2024-06-10T00:24:43.767562+0000","last_unstale":"2024-06-10T00:27:57.591972+0000","last_undegraded":"2024-06-10T00:27:57.591972+0000","last_fullsized":"2024-06-10T00:27:57.591972+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:46:36.099750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"100'9","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543096+0000","last_change":"2024-06-10T00:23:57.554906+0000","last_active":"2024-06-10T00:27:57.543096+0000","last_peered":"2024-06-10T00:27:57.543096+0000","last_clean":"2024-06-10T00:27:57.543096+0000","last_became_active":"2024-06-10T00:23:57.554608+0000","last_became_peered":"2024-06-10T00:23:57.554608+0000","last_unstale":"2024-06-10T00:27:57.543096+0000","last_undegraded":"2024-06-10T00:27:57.543096+0000","last_fullsized":"2024-06-10T00:27:57.543096+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:24.690594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570670+0000","last_change":"2024-06-10T00:24:09.264676+0000","last_active":"2024-06-10T00:27:57.570670+0000","last_peered":"2024-06-10T00:27:57.570670+0000","last_clean":"2024-06-10T00:27:57.570670+0000","last_became_active":"2024-06-10T00:24:09.264259+0000","last_became_peered":"2024-06-10T00:24:09.264259+0000","last_unstale":"2024-06-10T00:27:57.570670+0000","last_undegraded":"2024-06-10T00:27:57.570670+0000","last_fullsized":"2024-06-10T00:27:57.570670+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:37:53.787039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"101'102","reported_seq":242,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:10.494814+0000","last_change":"2024-06-10T00:23:54.424201+0000","last_active":"2024-06-10T00:28:10.494814+0000","last_peered":"2024-06-10T00:28:10.494814+0000","last_clean":"2024-06-10T00:28:10.494814+0000","last_became_active":"2024-06-10T00:23:54.423853+0000","last_became_peered":"2024-06-10T00:23:54.423853+0000","last_unstale":"2024-06-10T00:28:10.494814+0000","last_undegraded":"2024-06-10T00:28:10.494814+0000","last_fullsized":"2024-06-10T00:28:10.494814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:00.884041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":187,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571583+0000","last_change":"2024-06-10T00:23:57.554276+0000","last_active":"2024-06-10T00:27:57.571583+0000","last_peered":"2024-06-10T00:27:57.571583+0000","last_clean":"2024-06-10T00:27:57.571583+0000","last_became_active":"2024-06-10T00:23:57.553964+0000","last_became_peered":"2024-06-10T00:23:57.553964+0000","last_unstale":"2024-06-10T00:27:57.571583+0000","last_undegraded":"2024-06-10T00:27:57.571583+0000","last_fullsized":"2024-06-10T00:27:57.571583+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:39:51.974770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591089+0000","last_change":"2024-06-10T00:24:09.170249+0000","last_active":"2024-06-10T00:27:57.591089+0000","last_peered":"2024-06-10T00:27:57.591089+0000","last_clean":"2024-06-10T00:27:57.591089+0000","last_became_active":"2024-06-10T00:24:09.169830+0000","last_became_peered":"2024-06-10T00:24:09.169830+0000","last_unstale":"2024-06-10T00:27:57.591089+0000","last_undegraded":"2024-06-10T00:27:57.591089+0000","last_fullsized":"2024-06-10T00:27:57.591089+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.320609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"98'4","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545304+0000","last_change":"2024-06-10T00:23:54.473943+0000","last_active":"2024-06-10T00:27:57.545304+0000","last_peered":"2024-06-10T00:27:57.545304+0000","last_clean":"2024-06-10T00:27:57.545304+0000","last_became_active":"2024-06-10T00:23:54.473652+0000","last_became_peered":"2024-06-10T00:23:54.473652+0000","last_unstale":"2024-06-10T00:27:57.545304+0000","last_undegraded":"2024-06-10T00:27:57.545304+0000","last_fullsized":"2024-06-10T00:27:57.545304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:52:06.773684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"100'7","reported_seq":55,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593765+0000","last_change":"2024-06-10T00:25:44.735050+0000","last_active":"2024-06-10T00:27:57.593765+0000","last_peered":"2024-06-10T00:27:57.593765+0000","last_clean":"2024-06-10T00:27:57.593765+0000","last_became_active":"2024-06-10T00:25:44.734756+0000","last_became_peered":"2024-06-10T00:25:44.734756+0000","last_unstale":"2024-06-10T00:27:57.593765+0000","last_undegraded":"2024-06-10T00:27:57.593765+0000","last_fullsized":"2024-06-10T00:27:57.593765+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:16:03.924238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545083+0000","last_change":"2024-06-10T00:24:43.768837+0000","last_active":"2024-06-10T00:27:57.545083+0000","last_peered":"2024-06-10T00:27:57.545083+0000","last_clean":"2024-06-10T00:27:57.545083+0000","last_became_active":"2024-06-10T00:24:43.768295+0000","last_became_peered":"2024-06-10T00:24:43.768295+0000","last_unstale":"2024-06-10T00:27:57.545083+0000","last_undegraded":"2024-06-10T00:27:57.545083+0000","last_fullsized":"2024-06-10T00:27:57.545083+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:59.600864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"98'2","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591761+0000","last_change":"2024-06-10T00:23:54.823448+0000","last_active":"2024-06-10T00:27:57.591761+0000","last_peered":"2024-06-10T00:27:57.591761+0000","last_clean":"2024-06-10T00:27:57.591761+0000","last_became_active":"2024-06-10T00:23:54.823107+0000","last_became_peered":"2024-06-10T00:23:54.823107+0000","last_unstale":"2024-06-10T00:27:57.591761+0000","last_undegraded":"2024-06-10T00:27:57.591761+0000","last_fullsized":"2024-06-10T00:27:57.591761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:50.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"100'2","reported_seq":43,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569862+0000","last_change":"2024-06-10T00:23:57.556733+0000","last_active":"2024-06-10T00:27:57.569862+0000","last_peered":"2024-06-10T00:27:57.569862+0000","last_clean":"2024-06-10T00:27:57.569862+0000","last_became_active":"2024-06-10T00:23:57.556386+0000","last_became_peered":"2024-06-10T00:23:57.556386+0000","last_unstale":"2024-06-10T00:27:57.569862+0000","last_undegraded":"2024-06-10T00:27:57.569862+0000","last_fullsized":"2024-06-10T00:27:57.569862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:39:00.572311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458015+0000","last_change":"2024-06-10T00:24:09.174600+0000","last_active":"2024-06-10T00:27:57.458015+0000","last_peered":"2024-06-10T00:27:57.458015+0000","last_clean":"2024-06-10T00:27:57.458015+0000","last_became_active":"2024-06-10T00:24:09.174236+0000","last_became_peered":"2024-06-10T00:24:09.174236+0000","last_unstale":"2024-06-10T00:27:57.458015+0000","last_undegraded":"2024-06-10T00:27:57.458015+0000","last_fullsized":"2024-06-10T00:27:57.458015+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:58:18.447768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571190+0000","last_change":"2024-06-10T00:23:54.425111+0000","last_active":"2024-06-10T00:27:57.571190+0000","last_peered":"2024-06-10T00:27:57.571190+0000","last_clean":"2024-06-10T00:27:57.571190+0000","last_became_active":"2024-06-10T00:23:54.424724+0000","last_became_peered":"2024-06-10T00:23:54.424724+0000","last_unstale":"2024-06-10T00:27:57.571190+0000","last_undegraded":"2024-06-10T00:27:57.571190+0000","last_fullsized":"2024-06-10T00:27:57.571190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:54.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"100'9","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.556355+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.555479+0000","last_became_peered":"2024-06-10T00:23:57.555479+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:08.817381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"98'1","reported_seq":30,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544968+0000","last_change":"2024-06-10T00:24:09.169589+0000","last_active":"2024-06-10T00:27:57.544968+0000","last_peered":"2024-06-10T00:27:57.544968+0000","last_clean":"2024-06-10T00:27:57.544968+0000","last_became_active":"2024-06-10T00:24:09.168436+0000","last_became_peered":"2024-06-10T00:24:09.168436+0000","last_unstale":"2024-06-10T00:27:57.544968+0000","last_undegraded":"2024-06-10T00:27:57.544968+0000","last_fullsized":"2024-06-10T00:27:57.544968+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:17:32.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"98'1","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591500+0000","last_change":"2024-06-10T00:24:43.751399+0000","last_active":"2024-06-10T00:27:57.591500+0000","last_peered":"2024-06-10T00:27:57.591500+0000","last_clean":"2024-06-10T00:27:57.591500+0000","last_became_active":"2024-06-10T00:24:43.750987+0000","last_became_peered":"2024-06-10T00:24:43.750987+0000","last_unstale":"2024-06-10T00:27:57.591500+0000","last_undegraded":"2024-06-10T00:27:57.591500+0000","last_fullsized":"2024-06-10T00:27:57.591500+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:14:47.902017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570338+0000","last_change":"2024-06-10T00:23:57.560213+0000","last_active":"2024-06-10T00:27:57.570338+0000","last_peered":"2024-06-10T00:27:57.570338+0000","last_clean":"2024-06-10T00:27:57.570338+0000","last_became_active":"2024-06-10T00:23:57.559925+0000","last_became_peered":"2024-06-10T00:23:57.559925+0000","last_unstale":"2024-06-10T00:27:57.570338+0000","last_undegraded":"2024-06-10T00:27:57.570338+0000","last_fullsized":"2024-06-10T00:27:57.570338+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:00:37.403087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545095+0000","last_change":"2024-06-10T00:24:09.264758+0000","last_active":"2024-06-10T00:27:57.545095+0000","last_peered":"2024-06-10T00:27:57.545095+0000","last_clean":"2024-06-10T00:27:57.545095+0000","last_became_active":"2024-06-10T00:24:09.263928+0000","last_became_peered":"2024-06-10T00:24:09.263928+0000","last_unstale":"2024-06-10T00:27:57.545095+0000","last_undegraded":"2024-06-10T00:27:57.545095+0000","last_fullsized":"2024-06-10T00:27:57.545095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:23:41.083091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570103+0000","last_change":"2024-06-10T00:23:54.428553+0000","last_active":"2024-06-10T00:27:57.570103+0000","last_peered":"2024-06-10T00:27:57.570103+0000","last_clean":"2024-06-10T00:27:57.570103+0000","last_became_active":"2024-06-10T00:23:54.428108+0000","last_became_peered":"2024-06-10T00:23:54.428108+0000","last_unstale":"2024-06-10T00:27:57.570103+0000","last_undegraded":"2024-06-10T00:27:57.570103+0000","last_fullsized":"2024-06-10T00:27:57.570103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:57:26.207915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591119+0000","last_change":"2024-06-10T00:23:57.560686+0000","last_active":"2024-06-10T00:27:57.591119+0000","last_peered":"2024-06-10T00:27:57.591119+0000","last_clean":"2024-06-10T00:27:57.591119+0000","last_became_active":"2024-06-10T00:23:57.560326+0000","last_became_peered":"2024-06-10T00:23:57.560326+0000","last_unstale":"2024-06-10T00:27:57.591119+0000","last_undegraded":"2024-06-10T00:27:57.591119+0000","last_fullsized":"2024-06-10T00:27:57.591119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:08:39.506567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591281+0000","last_change":"2024-06-10T00:24:09.166947+0000","last_active":"2024-06-10T00:27:57.591281+0000","last_peered":"2024-06-10T00:27:57.591281+0000","last_clean":"2024-06-10T00:27:57.591281+0000","last_became_active":"2024-06-10T00:24:09.166503+0000","last_became_peered":"2024-06-10T00:24:09.166503+0000","last_unstale":"2024-06-10T00:27:57.591281+0000","last_undegraded":"2024-06-10T00:27:57.591281+0000","last_fullsized":"2024-06-10T00:27:57.591281+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:16:45.094642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591369+0000","last_change":"2024-06-10T00:23:54.823522+0000","last_active":"2024-06-10T00:27:57.591369+0000","last_peered":"2024-06-10T00:27:57.591369+0000","last_clean":"2024-06-10T00:27:57.591369+0000","last_became_active":"2024-06-10T00:23:54.823142+0000","last_became_peered":"2024-06-10T00:23:54.823142+0000","last_unstale":"2024-06-10T00:27:57.591369+0000","last_undegraded":"2024-06-10T00:27:57.591369+0000","last_fullsized":"2024-06-10T00:27:57.591369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:03.836850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592288+0000","last_change":"2024-06-10T00:23:57.563710+0000","last_active":"2024-06-10T00:27:57.592288+0000","last_peered":"2024-06-10T00:27:57.592288+0000","last_clean":"2024-06-10T00:27:57.592288+0000","last_became_active":"2024-06-10T00:23:57.563331+0000","last_became_peered":"2024-06-10T00:23:57.563331+0000","last_unstale":"2024-06-10T00:27:57.592288+0000","last_undegraded":"2024-06-10T00:27:57.592288+0000","last_fullsized":"2024-06-10T00:27:57.592288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:08:48.061885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592820+0000","last_change":"2024-06-10T00:24:09.171203+0000","last_active":"2024-06-10T00:27:57.592820+0000","last_peered":"2024-06-10T00:27:57.592820+0000","last_clean":"2024-06-10T00:27:57.592820+0000","last_became_active":"2024-06-10T00:24:09.170824+0000","last_became_peered":"2024-06-10T00:24:09.170824+0000","last_unstale":"2024-06-10T00:27:57.592820+0000","last_undegraded":"2024-06-10T00:27:57.592820+0000","last_fullsized":"2024-06-10T00:27:57.592820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:47:28.045602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"101'65","reported_seq":143,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:10.142035+0000","last_change":"2024-06-10T00:23:54.421307+0000","last_active":"2024-06-10T00:28:10.142035+0000","last_peered":"2024-06-10T00:28:10.142035+0000","last_clean":"2024-06-10T00:28:10.142035+0000","last_became_active":"2024-06-10T00:23:54.420928+0000","last_became_peered":"2024-06-10T00:23:54.420928+0000","last_unstale":"2024-06-10T00:28:10.142035+0000","last_undegraded":"2024-06-10T00:28:10.142035+0000","last_fullsized":"2024-06-10T00:28:10.142035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:59:41.433605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591623+0000","last_change":"2024-06-10T00:23:57.556494+0000","last_active":"2024-06-10T00:27:57.591623+0000","last_peered":"2024-06-10T00:27:57.591623+0000","last_clean":"2024-06-10T00:27:57.591623+0000","last_became_active":"2024-06-10T00:23:57.556088+0000","last_became_peered":"2024-06-10T00:23:57.556088+0000","last_unstale":"2024-06-10T00:27:57.591623+0000","last_undegraded":"2024-06-10T00:27:57.591623+0000","last_fullsized":"2024-06-10T00:27:57.591623+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:06:23.873281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461068+0000","last_change":"2024-06-10T00:24:09.166734+0000","last_active":"2024-06-10T00:27:57.461068+0000","last_peered":"2024-06-10T00:27:57.461068+0000","last_clean":"2024-06-10T00:27:57.461068+0000","last_became_active":"2024-06-10T00:24:09.166270+0000","last_became_peered":"2024-06-10T00:24:09.166270+0000","last_unstale":"2024-06-10T00:27:57.461068+0000","last_undegraded":"2024-06-10T00:27:57.461068+0000","last_fullsized":"2024-06-10T00:27:57.461068+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:38.039633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543256+0000","last_change":"2024-06-10T00:23:54.426226+0000","last_active":"2024-06-10T00:27:57.543256+0000","last_peered":"2024-06-10T00:27:57.543256+0000","last_clean":"2024-06-10T00:27:57.543256+0000","last_became_active":"2024-06-10T00:23:54.425879+0000","last_became_peered":"2024-06-10T00:23:54.425879+0000","last_unstale":"2024-06-10T00:27:57.543256+0000","last_undegraded":"2024-06-10T00:27:57.543256+0000","last_fullsized":"2024-06-10T00:27:57.543256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:15:32.936556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592043+0000","last_change":"2024-06-10T00:23:57.553814+0000","last_active":"2024-06-10T00:27:57.592043+0000","last_peered":"2024-06-10T00:27:57.592043+0000","last_clean":"2024-06-10T00:27:57.592043+0000","last_became_active":"2024-06-10T00:23:57.553478+0000","last_became_peered":"2024-06-10T00:23:57.553478+0000","last_unstale":"2024-06-10T00:27:57.592043+0000","last_undegraded":"2024-06-10T00:27:57.592043+0000","last_fullsized":"2024-06-10T00:27:57.592043+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:54:39.264526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591455+0000","last_change":"2024-06-10T00:23:57.556676+0000","last_active":"2024-06-10T00:27:57.591455+0000","last_peered":"2024-06-10T00:27:57.591455+0000","last_clean":"2024-06-10T00:27:57.591455+0000","last_became_active":"2024-06-10T00:23:57.555946+0000","last_became_peered":"2024-06-10T00:23:57.555946+0000","last_unstale":"2024-06-10T00:27:57.591455+0000","last_undegraded":"2024-06-10T00:27:57.591455+0000","last_fullsized":"2024-06-10T00:27:57.591455+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:10:10.868799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"100'12","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590806+0000","last_change":"2024-06-10T00:23:57.565423+0000","last_active":"2024-06-10T00:27:57.590806+0000","last_peered":"2024-06-10T00:27:57.590806+0000","last_clean":"2024-06-10T00:27:57.590806+0000","last_became_active":"2024-06-10T00:23:57.564830+0000","last_became_peered":"2024-06-10T00:23:57.564830+0000","last_unstale":"2024-06-10T00:27:57.590806+0000","last_undegraded":"2024-06-10T00:27:57.590806+0000","last_fullsized":"2024-06-10T00:27:57.590806+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:11:06.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"100'8","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591073+0000","last_change":"2024-06-10T00:23:57.558971+0000","last_active":"2024-06-10T00:27:57.591073+0000","last_peered":"2024-06-10T00:27:57.591073+0000","last_clean":"2024-06-10T00:27:57.591073+0000","last_became_active":"2024-06-10T00:23:57.558620+0000","last_became_peered":"2024-06-10T00:23:57.558620+0000","last_unstale":"2024-06-10T00:27:57.591073+0000","last_undegraded":"2024-06-10T00:27:57.591073+0000","last_fullsized":"2024-06-10T00:27:57.591073+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:45:43.308008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"100'7","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591540+0000","last_change":"2024-06-10T00:23:57.552341+0000","last_active":"2024-06-10T00:27:57.591540+0000","last_peered":"2024-06-10T00:27:57.591540+0000","last_clean":"2024-06-10T00:27:57.591540+0000","last_became_active":"2024-06-10T00:23:57.551464+0000","last_became_peered":"2024-06-10T00:23:57.551464+0000","last_unstale":"2024-06-10T00:27:57.591540+0000","last_undegraded":"2024-06-10T00:27:57.591540+0000","last_fullsized":"2024-06-10T00:27:57.591540+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:44:41.063516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"100'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543209+0000","last_change":"2024-06-10T00:24:43.768961+0000","last_active":"2024-06-10T00:27:57.543209+0000","last_peered":"2024-06-10T00:27:57.543209+0000","last_clean":"2024-06-10T00:27:57.543209+0000","last_became_active":"2024-06-10T00:24:43.768542+0000","last_became_peered":"2024-06-10T00:24:43.768542+0000","last_unstale":"2024-06-10T00:27:57.543209+0000","last_undegraded":"2024-06-10T00:27:57.543209+0000","last_fullsized":"2024-06-10T00:27:57.543209+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:23:58.412054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"100'6","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591613+0000","last_change":"2024-06-10T00:23:57.554924+0000","last_active":"2024-06-10T00:27:57.591613+0000","last_peered":"2024-06-10T00:27:57.591613+0000","last_clean":"2024-06-10T00:27:57.591613+0000","last_became_active":"2024-06-10T00:23:57.554515+0000","last_became_peered":"2024-06-10T00:23:57.554515+0000","last_unstale":"2024-06-10T00:27:57.591613+0000","last_undegraded":"2024-06-10T00:27:57.591613+0000","last_fullsized":"2024-06-10T00:27:57.591613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:58:15.624775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"100'15","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591485+0000","last_change":"2024-06-10T00:23:57.561940+0000","last_active":"2024-06-10T00:27:57.591485+0000","last_peered":"2024-06-10T00:27:57.591485+0000","last_clean":"2024-06-10T00:27:57.591485+0000","last_became_active":"2024-06-10T00:23:57.561512+0000","last_became_peered":"2024-06-10T00:23:57.561512+0000","last_unstale":"2024-06-10T00:27:57.591485+0000","last_undegraded":"2024-06-10T00:27:57.591485+0000","last_fullsized":"2024-06-10T00:27:57.591485+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:57:57.147311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"100'2","reported_seq":43,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570333+0000","last_change":"2024-06-10T00:23:57.557229+0000","last_active":"2024-06-10T00:27:57.570333+0000","last_peered":"2024-06-10T00:27:57.570333+0000","last_clean":"2024-06-10T00:27:57.570333+0000","last_became_active":"2024-06-10T00:23:57.556905+0000","last_became_peered":"2024-06-10T00:23:57.556905+0000","last_unstale":"2024-06-10T00:27:57.570333+0000","last_undegraded":"2024-06-10T00:27:57.570333+0000","last_fullsized":"2024-06-10T00:27:57.570333+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:30.513517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"100'6","reported_seq":24,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592589+0000","last_change":"2024-06-10T00:24:43.768509+0000","last_active":"2024-06-10T00:27:57.592589+0000","last_peered":"2024-06-10T00:27:57.592589+0000","last_clean":"2024-06-10T00:27:57.592589+0000","last_became_active":"2024-06-10T00:24:43.768038+0000","last_became_peered":"2024-06-10T00:24:43.768038+0000","last_unstale":"2024-06-10T00:27:57.592589+0000","last_undegraded":"2024-06-10T00:27:57.592589+0000","last_fullsized":"2024-06-10T00:27:57.592589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:38.344375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571244+0000","last_change":"2024-06-10T00:23:57.557830+0000","last_active":"2024-06-10T00:27:57.571244+0000","last_peered":"2024-06-10T00:27:57.571244+0000","last_clean":"2024-06-10T00:27:57.571244+0000","last_became_active":"2024-06-10T00:23:57.556986+0000","last_became_peered":"2024-06-10T00:23:57.556986+0000","last_unstale":"2024-06-10T00:27:57.571244+0000","last_undegraded":"2024-06-10T00:27:57.571244+0000","last_fullsized":"2024-06-10T00:27:57.571244+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:53.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590760+0000","last_change":"2024-06-10T00:23:57.565602+0000","last_active":"2024-06-10T00:27:57.590760+0000","last_peered":"2024-06-10T00:27:57.590760+0000","last_clean":"2024-06-10T00:27:57.590760+0000","last_became_active":"2024-06-10T00:23:57.565264+0000","last_became_peered":"2024-06-10T00:23:57.565264+0000","last_unstale":"2024-06-10T00:27:57.590760+0000","last_undegraded":"2024-06-10T00:27:57.590760+0000","last_fullsized":"2024-06-10T00:27:57.590760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:09:12.773087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542211+0000","last_change":"2024-06-10T00:23:57.548516+0000","last_active":"2024-06-10T00:27:57.542211+0000","last_peered":"2024-06-10T00:27:57.542211+0000","last_clean":"2024-06-10T00:27:57.542211+0000","last_became_active":"2024-06-10T00:23:57.548202+0000","last_became_peered":"2024-06-10T00:23:57.548202+0000","last_unstale":"2024-06-10T00:27:57.542211+0000","last_undegraded":"2024-06-10T00:27:57.542211+0000","last_fullsized":"2024-06-10T00:27:57.542211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:13.901478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591566+0000","last_change":"2024-06-10T00:23:57.565900+0000","last_active":"2024-06-10T00:27:57.591566+0000","last_peered":"2024-06-10T00:27:57.591566+0000","last_clean":"2024-06-10T00:27:57.591566+0000","last_became_active":"2024-06-10T00:23:57.565573+0000","last_became_peered":"2024-06-10T00:23:57.565573+0000","last_unstale":"2024-06-10T00:27:57.591566+0000","last_undegraded":"2024-06-10T00:27:57.591566+0000","last_fullsized":"2024-06-10T00:27:57.591566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:08.255905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107100,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":796432,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1511,"num_read_kb":973,"num_write":427,"num_write_kb":2362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2389296,"data_compressed":103467,"data_compressed_allocated":135168,"data_compressed_original":2101248,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351043,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1118208,"data_stored":582820,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612680,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712688,"statfs":{"total":95995035648,"available":95961792512,"internally_reserved":0,"allocated":1310720,"data_stored":1568443,"data_compressed":86181,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776206,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1159168,"data_stored":980346,"data_compressed":54566,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037842,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1150976,"data_stored":976148,"data_compressed":54566,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299479,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":1180481,"data_compressed":70979,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463004,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1110016,"data_stored":578210,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1183744,"data_stored":647485,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":579598,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411441,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36520,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708632,"statfs":{"total":95995035648,"available":95957639168,"internally_reserved":0,"allocated":1208320,"data_stored":650064,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1191936,"data_stored":1299852,"data_compressed":69768,"data_compressed_allocated":94208,"data_compressed_original":798720,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888166,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":583640,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724641,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32388,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712764,"statfs":{"total":95995035648,"available":95961870336,"internally_reserved":0,"allocated":1232896,"data_stored":1503521,"data_compressed":86181,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34489,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330937,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19287,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19287,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":399955,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-10T00:28:14.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-10T00:28:15.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.445+0000 7f474cb44700 1 -- 172.21.15.121:0/1882162413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4748108500 msgr2=0x7f47481088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:15.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.445+0000 7f474cb44700 1 --2- 172.21.15.121:0/1882162413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4748108500 0x7f47481088e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4738009990 tx=0x7f4738009ca0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.445+0000 7f474cb44700 1 -- 172.21.15.121:0/1882162413 shutdown_connections 2024-06-10T00:28:15.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.445+0000 7f474cb44700 1 --2- 172.21.15.121:0/1882162413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474806ab30 0x7f474810f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.445+0000 7f474cb44700 1 --2- 172.21.15.121:0/1882162413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4748108500 0x7f47481088e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.445+0000 7f474cb44700 1 --2- 172.21.15.121:0/1882162413 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f474806a170 0x7f474806a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.445+0000 7f474cb44700 1 -- 172.21.15.121:0/1882162413 >> 172.21.15.121:0/1882162413 conn(0x7f4748078340 msgr2=0x7f4748078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:15.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.445+0000 7f474cb44700 1 -- 172.21.15.121:0/1882162413 shutdown_connections 2024-06-10T00:28:15.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.445+0000 7f474cb44700 1 -- 172.21.15.121:0/1882162413 wait complete. 2024-06-10T00:28:15.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f474cb44700 1 Processor -- start 2024-06-10T00:28:15.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f474cb44700 1 -- start start 2024-06-10T00:28:15.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f474cb44700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474806a170 0x7f4748198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f474cb44700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474806ab30 0x7f4748199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f474cb44700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4748108500 0x7f474819d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f474cb44700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4748112860 con 0x7f474806a170 2024-06-10T00:28:15.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f474cb44700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f47481126e0 con 0x7f4748108500 2024-06-10T00:28:15.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f474cb44700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47481129e0 con 0x7f474806ab30 2024-06-10T00:28:15.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f474659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474806a170 0x7f4748198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f474659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474806a170 0x7f4748198d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34780/0 (socket says 172.21.15.121:34780) 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f474659c700 1 -- 172.21.15.121:0/3742292789 learned_addr learned my addr 172.21.15.121:0/3742292789 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f4746d9d700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4748108500 0x7f474819d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.446+0000 7f4745d9b700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474806ab30 0x7f4748199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.447+0000 7f474659c700 1 -- 172.21.15.121:0/3742292789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474806ab30 msgr2=0x7f4748199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.447+0000 7f474659c700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474806ab30 0x7f4748199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.447+0000 7f474659c700 1 -- 172.21.15.121:0/3742292789 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4748108500 msgr2=0x7f474819d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.447+0000 7f474659c700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4748108500 0x7f474819d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.447+0000 7f474659c700 1 -- 172.21.15.121:0/3742292789 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474819dd40 con 0x7f474806a170 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.447+0000 7f474659c700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474806a170 0x7f4748198d30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f473800bff0 tx=0x7f4738005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.447+0000 7f47377fe700 1 -- 172.21.15.121:0/3742292789 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738027070 con 0x7f474806a170 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.447+0000 7f47377fe700 1 -- 172.21.15.121:0/3742292789 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4738004050 con 0x7f474806a170 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.447+0000 7f474cb44700 1 -- 172.21.15.121:0/3742292789 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f474819dfd0 con 0x7f474806a170 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.447+0000 7f47377fe700 1 -- 172.21.15.121:0/3742292789 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738022d00 con 0x7f474806a170 2024-06-10T00:28:15.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.447+0000 7f474cb44700 1 -- 172.21.15.121:0/3742292789 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4748101390 con 0x7f474806a170 2024-06-10T00:28:15.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.450+0000 7f47377fe700 1 -- 172.21.15.121:0/3742292789 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f473802a070 con 0x7f474806a170 2024-06-10T00:28:15.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.450+0000 7f47377fe700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f47300a2e20 0x7f47300a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.450+0000 7f4745d9b700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f47300a2e20 0x7f47300a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.450+0000 7f47377fe700 1 -- 172.21.15.121:0/3742292789 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f47380d0800 con 0x7f474806a170 2024-06-10T00:28:15.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.451+0000 7f4745d9b700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f47300a2e20 0x7f47300a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4748078990 tx=0x7f473c008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:15.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.452+0000 7f47573c7b80 1 Processor -- start 2024-06-10T00:28:15.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.452+0000 7f47573c7b80 1 -- start start 2024-06-10T00:28:15.452 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:15.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd59640 0x5622ccd59a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd539e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd53f20 0x5622ccd54340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f47573c7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5622ccca5810 con 0x5622ccd59640 2024-06-10T00:28:15.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f47573c7b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5622ccc4d720 con 0x5622ccd53f20 2024-06-10T00:28:15.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f47573c7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5622ccca17d0 con 0x5622ccd59f60 2024-06-10T00:28:15.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f4745d9b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f474659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd59640 0x5622ccd59a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f4745d9b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:35590/0 (socket says 172.21.15.121:35590) 2024-06-10T00:28:15.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f4746d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd53f20 0x5622ccd54340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f4745d9b700 1 -- 172.21.15.121:0/580713463 learned_addr learned my addr 172.21.15.121:0/580713463 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f4745d9b700 1 -- 172.21.15.121:0/580713463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd53f20 msgr2=0x5622ccd54340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f4745d9b700 1 --2- 172.21.15.121:0/580713463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd53f20 0x5622ccd54340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f4745d9b700 1 -- 172.21.15.121:0/580713463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd59640 msgr2=0x5622ccd59a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f4745d9b700 1 --2- 172.21.15.121:0/580713463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd59640 0x5622ccd59a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f4745d9b700 1 -- 172.21.15.121:0/580713463 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622ccc43f50 con 0x5622ccd59f60 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f4746d9d700 1 --2- 172.21.15.121:0/580713463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd53f20 0x5622ccd54340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.453+0000 7f4745d9b700 1 --2- 172.21.15.121:0/580713463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd539e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f473c009cb0 tx=0x7f473c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a25a40981d2727a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.454+0000 7f4734ff9700 1 -- 172.21.15.121:0/580713463 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c02b070 con 0x5622ccd59f60 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.454+0000 7f4734ff9700 1 -- 172.21.15.121:0/580713463 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f473c01daa0 con 0x5622ccd59f60 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.454+0000 7f4734ff9700 1 -- 172.21.15.121:0/580713463 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c026ab0 con 0x5622ccd59f60 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.455+0000 7f47573c7b80 1 -- 172.21.15.121:0/580713463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 msgr2=0x5622ccd539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.455+0000 7f47573c7b80 1 --2- 172.21.15.121:0/580713463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd539e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f473c009cb0 tx=0x7f473c016000 comp rx=0 tx=0).stop 2024-06-10T00:28:15.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.455+0000 7f47573c7b80 1 -- 172.21.15.121:0/580713463 shutdown_connections 2024-06-10T00:28:15.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.455+0000 7f47573c7b80 1 --2- 172.21.15.121:0/580713463 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd53f20 0x5622ccd54340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.455+0000 7f47573c7b80 1 --2- 172.21.15.121:0/580713463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd59640 0x5622ccd59a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.455+0000 7f47573c7b80 1 --2- 172.21.15.121:0/580713463 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd539e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.455+0000 7f47573c7b80 1 -- 172.21.15.121:0/580713463 >> 172.21.15.121:0/580713463 conn(0x5622ccc7ecc0 msgr2=0x5622ccd4e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:15.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.455+0000 7f47573c7b80 1 -- 172.21.15.121:0/580713463 shutdown_connections 2024-06-10T00:28:15.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.455+0000 7f47573c7b80 1 -- 172.21.15.121:0/580713463 wait complete. 2024-06-10T00:28:15.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.455+0000 7f47573c7b80 1 -- start start 2024-06-10T00:28:15.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f47573c7b80 1 Processor -- start 2024-06-10T00:28:15.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd53f20 0x5622ccd5fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd59640 0x5622ccd5d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd5d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f47573c7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5622ccd609d0 con 0x5622ccd59640 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f47573c7b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5622ccc45bf0 con 0x5622ccd53f20 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f47573c7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5622ccd60b50 con 0x5622ccd59f60 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f47573c7b80 10 client.0 _ll_register_callbacks cb 0x5622ccc3f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f4746d9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd5d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f4746d9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd5d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:35606/0 (socket says 172.21.15.121:35606) 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f4746d9d700 1 -- 172.21.15.121:0/4249931722 learned_addr learned my addr 172.21.15.121:0/4249931722 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f4745d9b700 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd59640 0x5622ccd5d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.456+0000 7f474659c700 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd53f20 0x5622ccd5fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f4746d9d700 1 -- 172.21.15.121:0/4249931722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd53f20 msgr2=0x5622ccd5fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:15.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f4746d9d700 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd53f20 0x5622ccd5fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f4746d9d700 1 -- 172.21.15.121:0/4249931722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd59640 msgr2=0x5622ccd5d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f4746d9d700 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd59640 0x5622ccd5d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f4746d9d700 1 -- 172.21.15.121:0/4249931722 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622ccbf48d0 con 0x5622ccd59f60 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f4746d9d700 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd5d550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f474000ae40 tx=0x7f4740011490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5f11a66f838252f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f472dffb700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f472dffb700 1 -- 172.21.15.121:0/4249931722 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474001a070 con 0x5622ccd59f60 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f47573c7b80 1 -- 172.21.15.121:0/4249931722 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f473c01daa0 con 0x5622ccd59f60 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f472dffb700 1 -- 172.21.15.121:0/4249931722 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f4740011e00 con 0x5622ccd59f60 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f472dffb700 1 -- 172.21.15.121:0/4249931722 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4740026050 con 0x5622ccd59f60 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f47573c7b80 10 client.25036 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.457+0000 7f47573c7b80 1 -- 172.21.15.121:0/4249931722 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5622ccd6b3c0 con 0x5622ccd59f60 2024-06-10T00:28:15.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.458+0000 7f472dffb700 1 -- 172.21.15.121:0/4249931722 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f473c01daa0 con 0x5622ccd59f60 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.458+0000 7f472dffb700 1 -- 172.21.15.121:0/4249931722 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f474001e4f0 con 0x5622ccd59f60 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.458+0000 7f47573c7b80 10 client.25036 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.458+0000 7f47573c7b80 10 client.25036 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.458+0000 7f47573c7b80 20 client.25036 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.458+0000 7f47573c7b80 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5622ccd6d950 0x5622ccd6fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.458+0000 7f47573c7b80 4 client.25036 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.458+0000 7f47573c7b80 1 -- 172.21.15.121:0/4249931722 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622cccb2d10 con 0x5622ccd6d950 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.458+0000 7f474659c700 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5622ccd6d950 0x5622ccd6fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.458+0000 7f474659c700 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5622ccd6d950 0x5622ccd6fd40 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2555d77db8162c5f server_cookie=21d701b2ce07ca0 in_seq=0 out_seq=0 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.458+0000 7f472dffb700 10 client.25036 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f472dffb700 1 -- 172.21.15.121:0/4249931722 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5622cccb2d10 con 0x5622ccd6d950 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f472dffb700 10 client.25036 handle_command_reply: tid=0 2024-06-10T00:28:15.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f47573c7b80 1 client.25036 shutdown 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f47573c7b80 1 -- 172.21.15.121:0/4249931722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 msgr2=0x5622ccd5d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f47573c7b80 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd5d550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f474000ae40 tx=0x7f4740011490 comp rx=0 tx=0).stop 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f47573c7b80 1 -- 172.21.15.121:0/4249931722 shutdown_connections 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f47573c7b80 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5622ccd6d950 0x5622ccd6fd40 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f47573c7b80 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd5d550 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f47573c7b80 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd53f20 0x5622ccd5fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f47573c7b80 1 --2- 172.21.15.121:0/4249931722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd59640 0x5622ccd5d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f472dffb700 0 client.25036 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.459+0000 7f47573c7b80 1 -- 172.21.15.121:0/4249931722 >> 172.21.15.121:0/4249931722 conn(0x5622ccc7ecc0 msgr2=0x5622ccd529d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.460+0000 7f47573c7b80 1 -- 172.21.15.121:0/4249931722 shutdown_connections 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.460+0000 7f47573c7b80 1 -- 172.21.15.121:0/4249931722 wait complete. 2024-06-10T00:28:15.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.460+0000 7f47573c7b80 20 client.25036 trim_cache size 0 max 16384 2024-06-10T00:28:15.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.482+0000 7f47573c7b80 1 Processor -- start 2024-06-10T00:28:15.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.482+0000 7f47573c7b80 1 -- start start 2024-06-10T00:28:15.483 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:15 smithi121 ceph-mon[32378]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-06-10T00:28:15.483 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:15 smithi121 ceph-mon[32378]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd539e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd59640 0x5622ccd577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd5cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f47573c7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f47481aeca0 con 0x5622ccd53f20 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f47573c7b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f47481ac530 con 0x5622ccd59640 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f47573c7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47481aa380 con 0x5622ccd59f60 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f474659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f474659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34808/0 (socket says 172.21.15.121:34808) 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f474659c700 1 -- 172.21.15.121:0/2284621541 learned_addr learned my addr 172.21.15.121:0/2284621541 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f4745d9b700 1 --2- 172.21.15.121:0/2284621541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd59640 0x5622ccd577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f4746d9d700 1 --2- 172.21.15.121:0/2284621541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd5cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f474659c700 1 -- 172.21.15.121:0/2284621541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 msgr2=0x5622ccd5cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f474659c700 1 --2- 172.21.15.121:0/2284621541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd5cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f474659c700 1 -- 172.21.15.121:0/2284621541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd59640 msgr2=0x5622ccd577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f474659c700 1 --2- 172.21.15.121:0/2284621541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd59640 0x5622ccd577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f474659c700 1 -- 172.21.15.121:0/2284621541 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622ccd4f500 con 0x5622ccd53f20 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.483+0000 7f4745d9b700 1 --2- 172.21.15.121:0/2284621541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd59640 0x5622ccd577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.484+0000 7f474659c700 1 --2- 172.21.15.121:0/2284621541 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd539e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f473809c5b0 tx=0x7f473800b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c403d48a9ee8125 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.484+0000 7f472ffff700 1 -- 172.21.15.121:0/2284621541 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47380b5070 con 0x5622ccd53f20 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.484+0000 7f472ffff700 1 -- 172.21.15.121:0/2284621541 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f473809c8b0 con 0x5622ccd53f20 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.484+0000 7f472ffff700 1 -- 172.21.15.121:0/2284621541 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47380990b0 con 0x5622ccd53f20 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 -- 172.21.15.121:0/2284621541 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 msgr2=0x5622ccd539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 --2- 172.21.15.121:0/2284621541 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd539e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f473809c5b0 tx=0x7f473800b540 comp rx=0 tx=0).stop 2024-06-10T00:28:15.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 -- 172.21.15.121:0/2284621541 shutdown_connections 2024-06-10T00:28:15.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 --2- 172.21.15.121:0/2284621541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59f60 0x5622ccd5cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 --2- 172.21.15.121:0/2284621541 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd539e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 --2- 172.21.15.121:0/2284621541 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd59640 0x5622ccd577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 -- 172.21.15.121:0/2284621541 >> 172.21.15.121:0/2284621541 conn(0x5622ccc7ecc0 msgr2=0x5622ccd570c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:15.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 -- 172.21.15.121:0/2284621541 shutdown_connections 2024-06-10T00:28:15.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 -- 172.21.15.121:0/2284621541 wait complete. 2024-06-10T00:28:15.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 -- start start 2024-06-10T00:28:15.487 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 Processor -- start 2024-06-10T00:28:15.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd661b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59640 0x5622ccd666f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd59f60 0x5622ccd74d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f47481aa9f0 con 0x5622ccd53f20 2024-06-10T00:28:15.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f47481abe60 con 0x5622ccd59f60 2024-06-10T00:28:15.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47481ab0c0 con 0x5622ccd59640 2024-06-10T00:28:15.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.485+0000 7f47573c7b80 10 client.0 _ll_register_callbacks cb 0x5622ccc3f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:15.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f474659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f474659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd661b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34820/0 (socket says 172.21.15.121:34820) 2024-06-10T00:28:15.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f474659c700 1 -- 172.21.15.121:0/2249245136 learned_addr learned my addr 172.21.15.121:0/2249245136 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f4745d9b700 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59640 0x5622ccd666f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f474659c700 1 -- 172.21.15.121:0/2249245136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59640 msgr2=0x5622ccd666f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f474659c700 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59640 0x5622ccd666f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f474659c700 1 -- 172.21.15.121:0/2249245136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd59f60 msgr2=0x5622ccd74d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f474659c700 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd59f60 0x5622ccd74d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f474659c700 1 -- 172.21.15.121:0/2249245136 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622ccd75580 con 0x5622ccd53f20 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f4745d9b700 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59640 0x5622ccd666f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f474659c700 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd661b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4738005830 tx=0x7f473809c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bae3bf8090f9882 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f472dffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f472dffb700 1 -- 172.21.15.121:0/2249245136 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47380b5040 con 0x5622ccd53f20 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f47573c7b80 1 -- 172.21.15.121:0/2249245136 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f473809c8b0 con 0x5622ccd53f20 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f472dffb700 1 -- 172.21.15.121:0/2249245136 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f473809abb0 con 0x5622ccd53f20 2024-06-10T00:28:15.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f472dffb700 1 -- 172.21.15.121:0/2249245136 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f47380a5490 con 0x5622ccd53f20 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f47573c7b80 10 client.15420 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.486+0000 7f47573c7b80 1 -- 172.21.15.121:0/2249245136 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5622ccdd82d0 con 0x5622ccd53f20 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.487+0000 7f472dffb700 1 -- 172.21.15.121:0/2249245136 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f473809c8b0 con 0x5622ccd53f20 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.487+0000 7f472dffb700 1 -- 172.21.15.121:0/2249245136 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f47380cde20 con 0x5622ccd53f20 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.487+0000 7f47573c7b80 10 client.15420 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.487+0000 7f47573c7b80 10 client.15420 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.487+0000 7f47573c7b80 20 client.15420 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.487+0000 7f47573c7b80 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5622ccdd8700 0x5622ccddab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.487+0000 7f47573c7b80 4 client.15420 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.487+0000 7f47573c7b80 1 -- 172.21.15.121:0/2249245136 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5622cccad290 con 0x5622ccdd8700 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.487+0000 7f4745d9b700 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5622ccdd8700 0x5622ccddab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f472dffb700 10 client.15420 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f4745d9b700 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5622ccdd8700 0x5622ccddab20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9d8a381144d121c server_cookie=47881e5354de8cf6 in_seq=0 out_seq=0 2024-06-10T00:28:15.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f472dffb700 1 -- 172.21.15.121:0/2249245136 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5622cccad290 con 0x5622ccdd8700 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f472dffb700 10 client.15420 handle_command_reply: tid=0 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 1 client.15420 shutdown 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 1 -- 172.21.15.121:0/2249245136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 msgr2=0x5622ccd661b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd661b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4738005830 tx=0x7f473809c4d0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 1 -- 172.21.15.121:0/2249245136 shutdown_connections 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5622ccdd8700 0x5622ccddab20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5622ccd59f60 0x5622ccd74d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f472dffb700 0 client.15420 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5622ccd53f20 0x5622ccd661b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 1 --2- 172.21.15.121:0/2249245136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622ccd59640 0x5622ccd666f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 1 -- 172.21.15.121:0/2249245136 >> 172.21.15.121:0/2249245136 conn(0x5622ccc7ecc0 msgr2=0x5622ccd5cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 1 -- 172.21.15.121:0/2249245136 shutdown_connections 2024-06-10T00:28:15.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 1 -- 172.21.15.121:0/2249245136 wait complete. 2024-06-10T00:28:15.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.488+0000 7f47573c7b80 20 client.15420 trim_cache size 0 max 16384 2024-06-10T00:28:15.494 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:28:15.494 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-10T00:28:15.494 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": { 2024-06-10T00:28:15.494 INFO:teuthology.orchestra.run.smithi121.stdout: "e0e5b8ec-b893-4254-a3bb-325bcb943ccd": { 2024-06-10T00:28:15.494 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/", 2024-06-10T00:28:15.494 INFO:teuthology.orchestra.run.smithi121.stdout: "tag": "e0e5b8ec-b893-4254-a3bb-325bcb943ccd", 2024-06-10T00:28:15.494 INFO:teuthology.orchestra.run.smithi121.stdout: "options": "recursive,force" 2024-06-10T00:28:15.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:28:15.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:28:15.495 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 -- 172.21.15.121:0/3742292789 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f47300a2e20 msgr2=0x7f47300a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f47300a2e20 0x7f47300a52e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4748078990 tx=0x7f473c008040 comp rx=0 tx=0).stop 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 -- 172.21.15.121:0/3742292789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474806a170 msgr2=0x7f4748198d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474806a170 0x7f4748198d30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f473800bff0 tx=0x7f4738005590 comp rx=0 tx=0).stop 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 -- 172.21.15.121:0/3742292789 shutdown_connections 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f47300a2e20 0x7f47300a52e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4748108500 0x7f474819d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f474806a170 0x7f4748198d30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 --2- 172.21.15.121:0/3742292789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474806ab30 0x7f4748199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 -- 172.21.15.121:0/3742292789 >> 172.21.15.121:0/3742292789 conn(0x7f4748078340 msgr2=0x7f47480ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 -- 172.21.15.121:0/3742292789 shutdown_connections 2024-06-10T00:28:15.496 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:15.489+0000 7f474cb44700 1 -- 172.21.15.121:0/3742292789 wait complete. 2024-06-10T00:28:15.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:15 smithi193 ceph-mon[36716]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-06-10T00:28:15.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:15 smithi193 ceph-mon[36716]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:15.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:15 smithi135 ceph-mon[36424]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 76 KiB/s wr, 49 op/s 2024-06-10T00:28:15.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:15 smithi135 ceph-mon[36424]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:16.863 INFO:tasks.cephfs.filesystem:scrub status for tag:e0e5b8ec-b893-4254-a3bb-325bcb943ccd - {'path': '/', 'tag': 'e0e5b8ec-b893-4254-a3bb-325bcb943ccd', 'options': 'recursive,force'} 2024-06-10T00:28:17.563 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:17 smithi121 ceph-mon[32378]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 47 op/s 2024-06-10T00:28:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:17 smithi193 ceph-mon[36716]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 47 op/s 2024-06-10T00:28:17.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:17 smithi135 ceph-mon[36424]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 76 KiB/s wr, 47 op/s 2024-06-10T00:28:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.900+0000 7f083f340700 1 -- 172.21.15.121:0/3717993436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08381010b0 msgr2=0x7f0838101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.900+0000 7f083f340700 1 --2- 172.21.15.121:0/3717993436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08381010b0 0x7f0838101490 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f083400aaa0 tx=0x7f083400adb0 comp rx=0 tx=0).stop 2024-06-10T00:28:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.900+0000 7f083f340700 1 -- 172.21.15.121:0/3717993436 shutdown_connections 2024-06-10T00:28:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.901+0000 7f083f340700 1 --2- 172.21.15.121:0/3717993436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0838105f50 0x7f083810da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.901+0000 7f083f340700 1 --2- 172.21.15.121:0/3717993436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08381010b0 0x7f0838101490 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.901+0000 7f083f340700 1 --2- 172.21.15.121:0/3717993436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0838101a60 0x7f0838105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.901+0000 7f083f340700 1 -- 172.21.15.121:0/3717993436 >> 172.21.15.121:0/3717993436 conn(0x7f08380766b0 msgr2=0x7f0838076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.901+0000 7f083f340700 1 -- 172.21.15.121:0/3717993436 shutdown_connections 2024-06-10T00:28:17.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.901+0000 7f083f340700 1 -- 172.21.15.121:0/3717993436 wait complete. 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.901+0000 7f083f340700 1 Processor -- start 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.901+0000 7f083f340700 1 -- start start 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083f340700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08381010b0 0x7f0838198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083f340700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0838101a60 0x7f0838199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083f340700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0838105f50 0x7f083819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083f340700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f08381106d0 con 0x7f08381010b0 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083f340700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f0838110550 con 0x7f0838101a60 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083f340700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0838110850 con 0x7f0838105f50 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083d0dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08381010b0 0x7f0838198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083d0dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08381010b0 0x7f0838198d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:34850/0 (socket says 172.21.15.121:34850) 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083d0dc700 1 -- 172.21.15.121:0/3423446958 learned_addr learned my addr 172.21.15.121:0/3423446958 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083c8db700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0838101a60 0x7f0838199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:17.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083d0dc700 1 -- 172.21.15.121:0/3423446958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0838105f50 msgr2=0x7f083819d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083d8dd700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0838105f50 0x7f083819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083d0dc700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0838105f50 0x7f083819d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083d0dc700 1 -- 172.21.15.121:0/3423446958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0838101a60 msgr2=0x7f0838199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083d0dc700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0838101a60 0x7f0838199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083d0dc700 1 -- 172.21.15.121:0/3423446958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f083819dd20 con 0x7f08381010b0 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083c8db700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0838101a60 0x7f0838199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083d8dd700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0838105f50 0x7f083819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083d0dc700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08381010b0 0x7f0838198d10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f08340042a0 tx=0x7f0834004380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f082e7fc700 1 -- 172.21.15.121:0/3423446958 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834003ad0 con 0x7f08381010b0 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f082e7fc700 1 -- 172.21.15.121:0/3423446958 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0834024070 con 0x7f08381010b0 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f082e7fc700 1 -- 172.21.15.121:0/3423446958 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083400e450 con 0x7f08381010b0 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.902+0000 7f083f340700 1 -- 172.21.15.121:0/3423446958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f083819dfb0 con 0x7f08381010b0 2024-06-10T00:28:17.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.903+0000 7f083f340700 1 -- 172.21.15.121:0/3423446958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08381a5890 con 0x7f08381010b0 2024-06-10T00:28:17.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.904+0000 7f083f340700 1 -- 172.21.15.121:0/3423446958 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0838069180 con 0x7f08381010b0 2024-06-10T00:28:17.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.905+0000 7f082e7fc700 1 -- 172.21.15.121:0/3423446958 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f083401e070 con 0x7f08381010b0 2024-06-10T00:28:17.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.905+0000 7f082e7fc700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f08240a2ea0 0x7f08240a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:17.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.905+0000 7f083c8db700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f08240a2ea0 0x7f08240a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:17.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.905+0000 7f082e7fc700 1 -- 172.21.15.121:0/3423446958 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f08340d0c30 con 0x7f08381010b0 2024-06-10T00:28:17.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.906+0000 7f083c8db700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f08240a2ea0 0x7f08240a5360 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0838100ba0 tx=0x7f0828008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:17.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:17.910+0000 7f082e7fc700 1 -- 172.21.15.121:0/3423446958 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f083409d3e0 con 0x7f08381010b0 2024-06-10T00:28:18.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.100+0000 7f083f340700 1 -- 172.21.15.121:0/3423446958 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f08380628d0 con 0x7f08240a2ea0 2024-06-10T00:28:18.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.109+0000 7f082e7fc700 1 -- 172.21.15.121:0/3423446958 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339906 (secure 0 0 0) 0x7f08380628d0 con 0x7f08240a2ea0 2024-06-10T00:28:18.114 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:18.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 -- 172.21.15.121:0/3423446958 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f08240a2ea0 msgr2=0x7f08240a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:18.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f08240a2ea0 0x7f08240a5360 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0838100ba0 tx=0x7f0828008040 comp rx=0 tx=0).stop 2024-06-10T00:28:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 -- 172.21.15.121:0/3423446958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08381010b0 msgr2=0x7f0838198d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08381010b0 0x7f0838198d10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f08340042a0 tx=0x7f0834004380 comp rx=0 tx=0).stop 2024-06-10T00:28:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 -- 172.21.15.121:0/3423446958 shutdown_connections 2024-06-10T00:28:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f08240a2ea0 0x7f08240a5360 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0838105f50 0x7f083819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08381010b0 0x7f0838198d10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 --2- 172.21.15.121:0/3423446958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0838101a60 0x7f0838199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 -- 172.21.15.121:0/3423446958 >> 172.21.15.121:0/3423446958 conn(0x7f08380766b0 msgr2=0x7f08380ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 -- 172.21.15.121:0/3423446958 shutdown_connections 2024-06-10T00:28:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:18.112+0000 7f083f340700 1 -- 172.21.15.121:0/3423446958 wait complete. 2024-06-10T00:28:18.117 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-06-10T00:28:18.920 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-06-10T00:28:17.476651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1387034,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1748,"num_read_kb":1303,"num_write":630,"num_write_kb":5302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367136,"kb_used_data":13772,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574688,"statfs":{"total":1151940427776,"available":1151564480512,"internally_reserved":0,"allocated":14102528,"data_stored":11130608,"data_compressed":633915,"data_compressed_allocated":860160,"data_compressed_original":5050368,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":315,"num_read_kb":262,"num_write":126,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002886"},"pg_stats":[{"pgid":"3.3f","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592226+0000","last_change":"2024-06-10T00:23:57.564073+0000","last_active":"2024-06-10T00:27:57.592226+0000","last_peered":"2024-06-10T00:27:57.592226+0000","last_clean":"2024-06-10T00:27:57.592226+0000","last_became_active":"2024-06-10T00:23:57.563618+0000","last_became_peered":"2024-06-10T00:23:57.563618+0000","last_unstale":"2024-06-10T00:27:57.592226+0000","last_undegraded":"2024-06-10T00:27:57.592226+0000","last_fullsized":"2024-06-10T00:27:57.592226+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:26:25.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"100'4","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591431+0000","last_change":"2024-06-10T00:23:57.560148+0000","last_active":"2024-06-10T00:27:57.591431+0000","last_peered":"2024-06-10T00:27:57.591431+0000","last_clean":"2024-06-10T00:27:57.591431+0000","last_became_active":"2024-06-10T00:23:57.559794+0000","last_became_peered":"2024-06-10T00:23:57.559794+0000","last_unstale":"2024-06-10T00:27:57.591431+0000","last_undegraded":"2024-06-10T00:27:57.591431+0000","last_fullsized":"2024-06-10T00:27:57.591431+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:59:50.624542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"100'7","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593168+0000","last_change":"2024-06-10T00:23:57.563292+0000","last_active":"2024-06-10T00:27:57.593168+0000","last_peered":"2024-06-10T00:27:57.593168+0000","last_clean":"2024-06-10T00:27:57.593168+0000","last_became_active":"2024-06-10T00:23:57.563007+0000","last_became_peered":"2024-06-10T00:23:57.563007+0000","last_unstale":"2024-06-10T00:27:57.593168+0000","last_undegraded":"2024-06-10T00:27:57.593168+0000","last_fullsized":"2024-06-10T00:27:57.593168+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:27:44.897770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545262+0000","last_change":"2024-06-10T00:23:57.558789+0000","last_active":"2024-06-10T00:27:57.545262+0000","last_peered":"2024-06-10T00:27:57.545262+0000","last_clean":"2024-06-10T00:27:57.545262+0000","last_became_active":"2024-06-10T00:23:57.557943+0000","last_became_peered":"2024-06-10T00:23:57.557943+0000","last_unstale":"2024-06-10T00:27:57.545262+0000","last_undegraded":"2024-06-10T00:27:57.545262+0000","last_fullsized":"2024-06-10T00:27:57.545262+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:10:40.081996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569868+0000","last_change":"2024-06-10T00:23:57.550806+0000","last_active":"2024-06-10T00:27:57.569868+0000","last_peered":"2024-06-10T00:27:57.569868+0000","last_clean":"2024-06-10T00:27:57.569868+0000","last_became_active":"2024-06-10T00:23:57.550453+0000","last_became_peered":"2024-06-10T00:23:57.550453+0000","last_unstale":"2024-06-10T00:27:57.569868+0000","last_undegraded":"2024-06-10T00:27:57.569868+0000","last_fullsized":"2024-06-10T00:27:57.569868+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:11:58.765099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458657+0000","last_change":"2024-06-10T00:23:57.567307+0000","last_active":"2024-06-10T00:27:57.458657+0000","last_peered":"2024-06-10T00:27:57.458657+0000","last_clean":"2024-06-10T00:27:57.458657+0000","last_became_active":"2024-06-10T00:23:57.566973+0000","last_became_peered":"2024-06-10T00:23:57.566973+0000","last_unstale":"2024-06-10T00:27:57.458657+0000","last_undegraded":"2024-06-10T00:27:57.458657+0000","last_fullsized":"2024-06-10T00:27:57.458657+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:27:25.135046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591279+0000","last_change":"2024-06-10T00:23:57.547998+0000","last_active":"2024-06-10T00:27:57.591279+0000","last_peered":"2024-06-10T00:27:57.591279+0000","last_clean":"2024-06-10T00:27:57.591279+0000","last_became_active":"2024-06-10T00:23:57.547593+0000","last_became_peered":"2024-06-10T00:23:57.547593+0000","last_unstale":"2024-06-10T00:27:57.591279+0000","last_undegraded":"2024-06-10T00:27:57.591279+0000","last_fullsized":"2024-06-10T00:27:57.591279+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:34:04.219399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570182+0000","last_change":"2024-06-10T00:23:57.560252+0000","last_active":"2024-06-10T00:27:57.570182+0000","last_peered":"2024-06-10T00:27:57.570182+0000","last_clean":"2024-06-10T00:27:57.570182+0000","last_became_active":"2024-06-10T00:23:57.559924+0000","last_became_peered":"2024-06-10T00:23:57.559924+0000","last_unstale":"2024-06-10T00:27:57.570182+0000","last_undegraded":"2024-06-10T00:27:57.570182+0000","last_fullsized":"2024-06-10T00:27:57.570182+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:01:43.684262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590876+0000","last_change":"2024-06-10T00:23:57.564031+0000","last_active":"2024-06-10T00:27:57.590876+0000","last_peered":"2024-06-10T00:27:57.590876+0000","last_clean":"2024-06-10T00:27:57.590876+0000","last_became_active":"2024-06-10T00:23:57.563642+0000","last_became_peered":"2024-06-10T00:23:57.563642+0000","last_unstale":"2024-06-10T00:27:57.590876+0000","last_undegraded":"2024-06-10T00:27:57.590876+0000","last_fullsized":"2024-06-10T00:27:57.590876+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:16:23.003152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"100'16","reported_seq":57,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459536+0000","last_change":"2024-06-10T00:23:57.566877+0000","last_active":"2024-06-10T00:27:57.459536+0000","last_peered":"2024-06-10T00:27:57.459536+0000","last_clean":"2024-06-10T00:27:57.459536+0000","last_became_active":"2024-06-10T00:23:57.566529+0000","last_became_peered":"2024-06-10T00:23:57.566529+0000","last_unstale":"2024-06-10T00:27:57.459536+0000","last_undegraded":"2024-06-10T00:27:57.459536+0000","last_fullsized":"2024-06-10T00:27:57.459536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:30:43.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592151+0000","last_change":"2024-06-10T00:23:57.564979+0000","last_active":"2024-06-10T00:27:57.592151+0000","last_peered":"2024-06-10T00:27:57.592151+0000","last_clean":"2024-06-10T00:27:57.592151+0000","last_became_active":"2024-06-10T00:23:57.564578+0000","last_became_peered":"2024-06-10T00:23:57.564578+0000","last_unstale":"2024-06-10T00:27:57.592151+0000","last_undegraded":"2024-06-10T00:27:57.592151+0000","last_fullsized":"2024-06-10T00:27:57.592151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:40:53.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"100'7","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545061+0000","last_change":"2024-06-10T00:23:57.559126+0000","last_active":"2024-06-10T00:27:57.545061+0000","last_peered":"2024-06-10T00:27:57.545061+0000","last_clean":"2024-06-10T00:27:57.545061+0000","last_became_active":"2024-06-10T00:23:57.558401+0000","last_became_peered":"2024-06-10T00:23:57.558401+0000","last_unstale":"2024-06-10T00:27:57.545061+0000","last_undegraded":"2024-06-10T00:27:57.545061+0000","last_fullsized":"2024-06-10T00:27:57.545061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:57.894137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542053+0000","last_change":"2024-06-10T00:23:57.559272+0000","last_active":"2024-06-10T00:27:57.542053+0000","last_peered":"2024-06-10T00:27:57.542053+0000","last_clean":"2024-06-10T00:27:57.542053+0000","last_became_active":"2024-06-10T00:23:57.558941+0000","last_became_peered":"2024-06-10T00:23:57.558941+0000","last_unstale":"2024-06-10T00:27:57.542053+0000","last_undegraded":"2024-06-10T00:27:57.542053+0000","last_fullsized":"2024-06-10T00:27:57.542053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:36:48.088347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591361+0000","last_change":"2024-06-10T00:23:57.555875+0000","last_active":"2024-06-10T00:27:57.591361+0000","last_peered":"2024-06-10T00:27:57.591361+0000","last_clean":"2024-06-10T00:27:57.591361+0000","last_became_active":"2024-06-10T00:23:57.555492+0000","last_became_peered":"2024-06-10T00:23:57.555492+0000","last_unstale":"2024-06-10T00:27:57.591361+0000","last_undegraded":"2024-06-10T00:27:57.591361+0000","last_fullsized":"2024-06-10T00:27:57.591361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:49:10.827458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591360+0000","last_change":"2024-06-10T00:23:57.555316+0000","last_active":"2024-06-10T00:27:57.591360+0000","last_peered":"2024-06-10T00:27:57.591360+0000","last_clean":"2024-06-10T00:27:57.591360+0000","last_became_active":"2024-06-10T00:23:57.554952+0000","last_became_peered":"2024-06-10T00:23:57.554952+0000","last_unstale":"2024-06-10T00:27:57.591360+0000","last_undegraded":"2024-06-10T00:27:57.591360+0000","last_fullsized":"2024-06-10T00:27:57.591360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:03:57.557330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591053+0000","last_change":"2024-06-10T00:23:57.562442+0000","last_active":"2024-06-10T00:27:57.591053+0000","last_peered":"2024-06-10T00:27:57.591053+0000","last_clean":"2024-06-10T00:27:57.591053+0000","last_became_active":"2024-06-10T00:23:57.561965+0000","last_became_peered":"2024-06-10T00:23:57.561965+0000","last_unstale":"2024-06-10T00:27:57.591053+0000","last_undegraded":"2024-06-10T00:27:57.591053+0000","last_fullsized":"2024-06-10T00:27:57.591053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:40:16.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591981+0000","last_change":"2024-06-10T00:23:57.564422+0000","last_active":"2024-06-10T00:27:57.591981+0000","last_peered":"2024-06-10T00:27:57.591981+0000","last_clean":"2024-06-10T00:27:57.591981+0000","last_became_active":"2024-06-10T00:23:57.564069+0000","last_became_peered":"2024-06-10T00:23:57.564069+0000","last_unstale":"2024-06-10T00:27:57.591981+0000","last_undegraded":"2024-06-10T00:27:57.591981+0000","last_fullsized":"2024-06-10T00:27:57.591981+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:07.662465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458239+0000","last_change":"2024-06-10T00:23:57.561284+0000","last_active":"2024-06-10T00:27:57.458239+0000","last_peered":"2024-06-10T00:27:57.458239+0000","last_clean":"2024-06-10T00:27:57.458239+0000","last_became_active":"2024-06-10T00:23:57.560933+0000","last_became_peered":"2024-06-10T00:23:57.560933+0000","last_unstale":"2024-06-10T00:27:57.458239+0000","last_undegraded":"2024-06-10T00:27:57.458239+0000","last_fullsized":"2024-06-10T00:27:57.458239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:58:48.511774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590777+0000","last_change":"2024-06-10T00:23:57.557752+0000","last_active":"2024-06-10T00:27:57.590777+0000","last_peered":"2024-06-10T00:27:57.590777+0000","last_clean":"2024-06-10T00:27:57.590777+0000","last_became_active":"2024-06-10T00:23:57.557419+0000","last_became_peered":"2024-06-10T00:23:57.557419+0000","last_unstale":"2024-06-10T00:27:57.590777+0000","last_undegraded":"2024-06-10T00:27:57.590777+0000","last_fullsized":"2024-06-10T00:27:57.590777+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:48.234181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591122+0000","last_change":"2024-06-10T00:24:09.177183+0000","last_active":"2024-06-10T00:27:57.591122+0000","last_peered":"2024-06-10T00:27:57.591122+0000","last_clean":"2024-06-10T00:27:57.591122+0000","last_became_active":"2024-06-10T00:24:09.176680+0000","last_became_peered":"2024-06-10T00:24:09.176680+0000","last_unstale":"2024-06-10T00:27:57.591122+0000","last_undegraded":"2024-06-10T00:27:57.591122+0000","last_fullsized":"2024-06-10T00:27:57.591122+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:17:06.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"98'9","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:03.469961+0000","last_change":"2024-06-10T00:23:54.426364+0000","last_active":"2024-06-10T00:28:03.469961+0000","last_peered":"2024-06-10T00:28:03.469961+0000","last_clean":"2024-06-10T00:28:03.469961+0000","last_became_active":"2024-06-10T00:23:54.426054+0000","last_became_peered":"2024-06-10T00:23:54.426054+0000","last_unstale":"2024-06-10T00:28:03.469961+0000","last_undegraded":"2024-06-10T00:28:03.469961+0000","last_fullsized":"2024-06-10T00:28:03.469961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:47.613402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570428+0000","last_change":"2024-06-10T00:23:57.557206+0000","last_active":"2024-06-10T00:27:57.570428+0000","last_peered":"2024-06-10T00:27:57.570428+0000","last_clean":"2024-06-10T00:27:57.570428+0000","last_became_active":"2024-06-10T00:23:57.556853+0000","last_became_peered":"2024-06-10T00:23:57.556853+0000","last_unstale":"2024-06-10T00:27:57.570428+0000","last_undegraded":"2024-06-10T00:27:57.570428+0000","last_fullsized":"2024-06-10T00:27:57.570428+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:09.686334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544782+0000","last_change":"2024-06-10T00:24:09.171047+0000","last_active":"2024-06-10T00:27:57.544782+0000","last_peered":"2024-06-10T00:27:57.544782+0000","last_clean":"2024-06-10T00:27:57.544782+0000","last_became_active":"2024-06-10T00:24:09.170501+0000","last_became_peered":"2024-06-10T00:24:09.170501+0000","last_unstale":"2024-06-10T00:27:57.544782+0000","last_undegraded":"2024-06-10T00:27:57.544782+0000","last_fullsized":"2024-06-10T00:27:57.544782+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:47:27.690439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"98'2","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571600+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:27:57.571600+0000","last_peered":"2024-06-10T00:27:57.571600+0000","last_clean":"2024-06-10T00:27:57.571600+0000","last_became_active":"2024-06-10T00:23:54.822447+0000","last_became_peered":"2024-06-10T00:23:54.822447+0000","last_unstale":"2024-06-10T00:27:57.571600+0000","last_undegraded":"2024-06-10T00:27:57.571600+0000","last_fullsized":"2024-06-10T00:27:57.571600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:45:14.956540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"100'6","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591883+0000","last_change":"2024-06-10T00:23:57.554958+0000","last_active":"2024-06-10T00:27:57.591883+0000","last_peered":"2024-06-10T00:27:57.591883+0000","last_clean":"2024-06-10T00:27:57.591883+0000","last_became_active":"2024-06-10T00:23:57.554477+0000","last_became_peered":"2024-06-10T00:23:57.554477+0000","last_unstale":"2024-06-10T00:27:57.591883+0000","last_undegraded":"2024-06-10T00:27:57.591883+0000","last_fullsized":"2024-06-10T00:27:57.591883+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:40:06.066967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544895+0000","last_change":"2024-06-10T00:24:09.169642+0000","last_active":"2024-06-10T00:27:57.544895+0000","last_peered":"2024-06-10T00:27:57.544895+0000","last_clean":"2024-06-10T00:27:57.544895+0000","last_became_active":"2024-06-10T00:24:09.168071+0000","last_became_peered":"2024-06-10T00:24:09.168071+0000","last_unstale":"2024-06-10T00:27:57.544895+0000","last_undegraded":"2024-06-10T00:27:57.544895+0000","last_fullsized":"2024-06-10T00:27:57.544895+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:44.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"98'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570379+0000","last_change":"2024-06-10T00:23:54.430543+0000","last_active":"2024-06-10T00:27:57.570379+0000","last_peered":"2024-06-10T00:27:57.570379+0000","last_clean":"2024-06-10T00:27:57.570379+0000","last_became_active":"2024-06-10T00:23:54.430138+0000","last_became_peered":"2024-06-10T00:23:54.430138+0000","last_unstale":"2024-06-10T00:27:57.570379+0000","last_undegraded":"2024-06-10T00:27:57.570379+0000","last_fullsized":"2024-06-10T00:27:57.570379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:39:53.212607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590796+0000","last_change":"2024-06-10T00:23:57.562251+0000","last_active":"2024-06-10T00:27:57.590796+0000","last_peered":"2024-06-10T00:27:57.590796+0000","last_clean":"2024-06-10T00:27:57.590796+0000","last_became_active":"2024-06-10T00:23:57.561903+0000","last_became_peered":"2024-06-10T00:23:57.561903+0000","last_unstale":"2024-06-10T00:27:57.590796+0000","last_undegraded":"2024-06-10T00:27:57.590796+0000","last_fullsized":"2024-06-10T00:27:57.590796+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:59.489475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544829+0000","last_change":"2024-06-10T00:24:09.165894+0000","last_active":"2024-06-10T00:27:57.544829+0000","last_peered":"2024-06-10T00:27:57.544829+0000","last_clean":"2024-06-10T00:27:57.544829+0000","last_became_active":"2024-06-10T00:24:09.165435+0000","last_became_peered":"2024-06-10T00:24:09.165435+0000","last_unstale":"2024-06-10T00:27:57.544829+0000","last_undegraded":"2024-06-10T00:27:57.544829+0000","last_fullsized":"2024-06-10T00:27:57.544829+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:13:36.058375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570296+0000","last_change":"2024-06-10T00:23:54.427868+0000","last_active":"2024-06-10T00:27:57.570296+0000","last_peered":"2024-06-10T00:27:57.570296+0000","last_clean":"2024-06-10T00:27:57.570296+0000","last_became_active":"2024-06-10T00:23:54.427366+0000","last_became_peered":"2024-06-10T00:23:54.427366+0000","last_unstale":"2024-06-10T00:27:57.570296+0000","last_undegraded":"2024-06-10T00:27:57.570296+0000","last_fullsized":"2024-06-10T00:27:57.570296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:15.499292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"100'15","reported_seq":62,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462390+0000","last_change":"2024-06-10T00:24:43.768556+0000","last_active":"2024-06-10T00:27:57.462390+0000","last_peered":"2024-06-10T00:27:57.462390+0000","last_clean":"2024-06-10T00:27:57.462390+0000","last_became_active":"2024-06-10T00:24:43.768090+0000","last_became_peered":"2024-06-10T00:24:43.768090+0000","last_unstale":"2024-06-10T00:27:57.462390+0000","last_undegraded":"2024-06-10T00:27:57.462390+0000","last_fullsized":"2024-06-10T00:27:57.462390+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:05:41.604184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591403+0000","last_change":"2024-06-10T00:24:09.170514+0000","last_active":"2024-06-10T00:27:57.591403+0000","last_peered":"2024-06-10T00:27:57.591403+0000","last_clean":"2024-06-10T00:27:57.591403+0000","last_became_active":"2024-06-10T00:24:09.168902+0000","last_became_peered":"2024-06-10T00:24:09.168902+0000","last_unstale":"2024-06-10T00:27:57.591403+0000","last_undegraded":"2024-06-10T00:27:57.591403+0000","last_fullsized":"2024-06-10T00:27:57.591403+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:33:10.534657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"101'9","reported_seq":250,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:11.572833+0000","last_change":"2024-06-10T00:23:54.426581+0000","last_active":"2024-06-10T00:28:11.572833+0000","last_peered":"2024-06-10T00:28:11.572833+0000","last_clean":"2024-06-10T00:28:11.572833+0000","last_became_active":"2024-06-10T00:23:54.426211+0000","last_became_peered":"2024-06-10T00:23:54.426211+0000","last_unstale":"2024-06-10T00:28:11.572833+0000","last_undegraded":"2024-06-10T00:28:11.572833+0000","last_fullsized":"2024-06-10T00:28:11.572833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:49:05.360742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":194,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569865+0000","last_change":"2024-06-10T00:23:57.552308+0000","last_active":"2024-06-10T00:27:57.569865+0000","last_peered":"2024-06-10T00:27:57.569865+0000","last_clean":"2024-06-10T00:27:57.569865+0000","last_became_active":"2024-06-10T00:23:57.551762+0000","last_became_peered":"2024-06-10T00:23:57.551762+0000","last_unstale":"2024-06-10T00:27:57.569865+0000","last_undegraded":"2024-06-10T00:27:57.569865+0000","last_fullsized":"2024-06-10T00:27:57.569865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:22:28.476019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591613+0000","last_change":"2024-06-10T00:24:09.171456+0000","last_active":"2024-06-10T00:27:57.591613+0000","last_peered":"2024-06-10T00:27:57.591613+0000","last_clean":"2024-06-10T00:27:57.591613+0000","last_became_active":"2024-06-10T00:24:09.170969+0000","last_became_peered":"2024-06-10T00:24:09.170969+0000","last_unstale":"2024-06-10T00:27:57.591613+0000","last_undegraded":"2024-06-10T00:27:57.591613+0000","last_fullsized":"2024-06-10T00:27:57.591613+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:22:46.023897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458602+0000","last_change":"2024-06-10T00:23:54.428384+0000","last_active":"2024-06-10T00:27:57.458602+0000","last_peered":"2024-06-10T00:27:57.458602+0000","last_clean":"2024-06-10T00:27:57.458602+0000","last_became_active":"2024-06-10T00:23:54.427967+0000","last_became_peered":"2024-06-10T00:23:54.427967+0000","last_unstale":"2024-06-10T00:27:57.458602+0000","last_undegraded":"2024-06-10T00:27:57.458602+0000","last_fullsized":"2024-06-10T00:27:57.458602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:54:30.408684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"100'8","reported_seq":55,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461853+0000","last_change":"2024-06-10T00:24:43.262593+0000","last_active":"2024-06-10T00:27:57.461853+0000","last_peered":"2024-06-10T00:27:57.461853+0000","last_clean":"2024-06-10T00:27:57.461853+0000","last_became_active":"2024-06-10T00:24:43.262212+0000","last_became_peered":"2024-06-10T00:24:43.262212+0000","last_unstale":"2024-06-10T00:27:57.461853+0000","last_undegraded":"2024-06-10T00:27:57.461853+0000","last_fullsized":"2024-06-10T00:27:57.461853+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:40:24.442712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591973+0000","last_change":"2024-06-10T00:24:43.752805+0000","last_active":"2024-06-10T00:27:57.591973+0000","last_peered":"2024-06-10T00:27:57.591973+0000","last_clean":"2024-06-10T00:27:57.591973+0000","last_became_active":"2024-06-10T00:24:43.752251+0000","last_became_peered":"2024-06-10T00:24:43.752251+0000","last_unstale":"2024-06-10T00:27:57.591973+0000","last_undegraded":"2024-06-10T00:27:57.591973+0000","last_fullsized":"2024-06-10T00:27:57.591973+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:27:57.617443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542566+0000","last_change":"2024-06-10T00:23:54.421493+0000","last_active":"2024-06-10T00:27:57.542566+0000","last_peered":"2024-06-10T00:27:57.542566+0000","last_clean":"2024-06-10T00:27:57.542566+0000","last_became_active":"2024-06-10T00:23:54.421151+0000","last_became_peered":"2024-06-10T00:23:54.421151+0000","last_unstale":"2024-06-10T00:27:57.542566+0000","last_undegraded":"2024-06-10T00:27:57.542566+0000","last_fullsized":"2024-06-10T00:27:57.542566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:03.995244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570099+0000","last_change":"2024-06-10T00:23:57.554020+0000","last_active":"2024-06-10T00:27:57.570099+0000","last_peered":"2024-06-10T00:27:57.570099+0000","last_clean":"2024-06-10T00:27:57.570099+0000","last_became_active":"2024-06-10T00:23:57.553690+0000","last_became_peered":"2024-06-10T00:23:57.553690+0000","last_unstale":"2024-06-10T00:27:57.570099+0000","last_undegraded":"2024-06-10T00:27:57.570099+0000","last_fullsized":"2024-06-10T00:27:57.570099+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:48.097074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593638+0000","last_change":"2024-06-10T00:24:09.166426+0000","last_active":"2024-06-10T00:27:57.593638+0000","last_peered":"2024-06-10T00:27:57.593638+0000","last_clean":"2024-06-10T00:27:57.593638+0000","last_became_active":"2024-06-10T00:24:09.166047+0000","last_became_peered":"2024-06-10T00:24:09.166047+0000","last_unstale":"2024-06-10T00:27:57.593638+0000","last_undegraded":"2024-06-10T00:27:57.593638+0000","last_fullsized":"2024-06-10T00:27:57.593638+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:56:11.608061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"98'1","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461240+0000","last_change":"2024-06-10T00:23:54.426409+0000","last_active":"2024-06-10T00:27:57.461240+0000","last_peered":"2024-06-10T00:27:57.461240+0000","last_clean":"2024-06-10T00:27:57.461240+0000","last_became_active":"2024-06-10T00:23:54.425385+0000","last_became_peered":"2024-06-10T00:23:54.425385+0000","last_unstale":"2024-06-10T00:27:57.461240+0000","last_undegraded":"2024-06-10T00:27:57.461240+0000","last_fullsized":"2024-06-10T00:27:57.461240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:27:00.379189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457389+0000","last_change":"2024-06-10T00:23:57.560132+0000","last_active":"2024-06-10T00:27:57.457389+0000","last_peered":"2024-06-10T00:27:57.457389+0000","last_clean":"2024-06-10T00:27:57.457389+0000","last_became_active":"2024-06-10T00:23:57.559803+0000","last_became_peered":"2024-06-10T00:23:57.559803+0000","last_unstale":"2024-06-10T00:27:57.457389+0000","last_undegraded":"2024-06-10T00:27:57.457389+0000","last_fullsized":"2024-06-10T00:27:57.457389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:13:16.004233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571137+0000","last_change":"2024-06-10T00:24:09.170962+0000","last_active":"2024-06-10T00:27:57.571137+0000","last_peered":"2024-06-10T00:27:57.571137+0000","last_clean":"2024-06-10T00:27:57.571137+0000","last_became_active":"2024-06-10T00:24:09.170090+0000","last_became_peered":"2024-06-10T00:24:09.170090+0000","last_unstale":"2024-06-10T00:27:57.571137+0000","last_undegraded":"2024-06-10T00:27:57.571137+0000","last_fullsized":"2024-06-10T00:27:57.571137+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:07:33.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'5","reported_seq":240,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:11.572251+0000","last_change":"2024-06-10T00:23:54.421472+0000","last_active":"2024-06-10T00:28:11.572251+0000","last_peered":"2024-06-10T00:28:11.572251+0000","last_clean":"2024-06-10T00:28:11.572251+0000","last_became_active":"2024-06-10T00:23:54.421019+0000","last_became_peered":"2024-06-10T00:23:54.421019+0000","last_unstale":"2024-06-10T00:28:11.572251+0000","last_undegraded":"2024-06-10T00:28:11.572251+0000","last_fullsized":"2024-06-10T00:28:11.572251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:28:18.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":188,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590963+0000","last_change":"2024-06-10T00:23:57.563010+0000","last_active":"2024-06-10T00:27:57.590963+0000","last_peered":"2024-06-10T00:27:57.590963+0000","last_clean":"2024-06-10T00:27:57.590963+0000","last_became_active":"2024-06-10T00:23:57.561986+0000","last_became_peered":"2024-06-10T00:23:57.561986+0000","last_unstale":"2024-06-10T00:27:57.590963+0000","last_undegraded":"2024-06-10T00:27:57.590963+0000","last_fullsized":"2024-06-10T00:27:57.590963+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:31:17.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457415+0000","last_change":"2024-06-10T00:24:09.171746+0000","last_active":"2024-06-10T00:27:57.457415+0000","last_peered":"2024-06-10T00:27:57.457415+0000","last_clean":"2024-06-10T00:27:57.457415+0000","last_became_active":"2024-06-10T00:24:09.171030+0000","last_became_peered":"2024-06-10T00:24:09.171030+0000","last_unstale":"2024-06-10T00:27:57.457415+0000","last_undegraded":"2024-06-10T00:27:57.457415+0000","last_fullsized":"2024-06-10T00:27:57.457415+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:41.669471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591812+0000","last_change":"2024-06-10T00:23:54.428917+0000","last_active":"2024-06-10T00:27:57.591812+0000","last_peered":"2024-06-10T00:27:57.591812+0000","last_clean":"2024-06-10T00:27:57.591812+0000","last_became_active":"2024-06-10T00:23:54.428585+0000","last_became_peered":"2024-06-10T00:23:54.428585+0000","last_unstale":"2024-06-10T00:27:57.591812+0000","last_undegraded":"2024-06-10T00:27:57.591812+0000","last_fullsized":"2024-06-10T00:27:57.591812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:21:58.235848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591586+0000","last_change":"2024-06-10T00:23:57.551778+0000","last_active":"2024-06-10T00:27:57.591586+0000","last_peered":"2024-06-10T00:27:57.591586+0000","last_clean":"2024-06-10T00:27:57.591586+0000","last_became_active":"2024-06-10T00:23:57.551436+0000","last_became_peered":"2024-06-10T00:23:57.551436+0000","last_unstale":"2024-06-10T00:27:57.591586+0000","last_undegraded":"2024-06-10T00:27:57.591586+0000","last_fullsized":"2024-06-10T00:27:57.591586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:30.059783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545101+0000","last_change":"2024-06-10T00:24:09.167547+0000","last_active":"2024-06-10T00:27:57.545101+0000","last_peered":"2024-06-10T00:27:57.545101+0000","last_clean":"2024-06-10T00:27:57.545101+0000","last_became_active":"2024-06-10T00:24:09.167132+0000","last_became_peered":"2024-06-10T00:24:09.167132+0000","last_unstale":"2024-06-10T00:27:57.545101+0000","last_undegraded":"2024-06-10T00:27:57.545101+0000","last_fullsized":"2024-06-10T00:27:57.545101+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:11.520330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461450+0000","last_change":"2024-06-10T00:23:54.823754+0000","last_active":"2024-06-10T00:27:57.461450+0000","last_peered":"2024-06-10T00:27:57.461450+0000","last_clean":"2024-06-10T00:27:57.461450+0000","last_became_active":"2024-06-10T00:23:54.823362+0000","last_became_peered":"2024-06-10T00:23:54.823362+0000","last_unstale":"2024-06-10T00:27:57.461450+0000","last_undegraded":"2024-06-10T00:27:57.461450+0000","last_fullsized":"2024-06-10T00:27:57.461450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:48:50.915439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590772+0000","last_change":"2024-06-10T00:23:57.563313+0000","last_active":"2024-06-10T00:27:57.590772+0000","last_peered":"2024-06-10T00:27:57.590772+0000","last_clean":"2024-06-10T00:27:57.590772+0000","last_became_active":"2024-06-10T00:23:57.562307+0000","last_became_peered":"2024-06-10T00:23:57.562307+0000","last_unstale":"2024-06-10T00:27:57.590772+0000","last_undegraded":"2024-06-10T00:27:57.590772+0000","last_fullsized":"2024-06-10T00:27:57.590772+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:23.815267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457507+0000","last_change":"2024-06-10T00:24:09.174660+0000","last_active":"2024-06-10T00:27:57.457507+0000","last_peered":"2024-06-10T00:27:57.457507+0000","last_clean":"2024-06-10T00:27:57.457507+0000","last_became_active":"2024-06-10T00:24:09.174283+0000","last_became_peered":"2024-06-10T00:24:09.174283+0000","last_unstale":"2024-06-10T00:27:57.457507+0000","last_undegraded":"2024-06-10T00:27:57.457507+0000","last_fullsized":"2024-06-10T00:27:57.457507+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:15.236146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"101'41","reported_seq":478,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:12.573920+0000","last_change":"2024-06-10T00:23:54.426621+0000","last_active":"2024-06-10T00:28:12.573920+0000","last_peered":"2024-06-10T00:28:12.573920+0000","last_clean":"2024-06-10T00:28:12.573920+0000","last_became_active":"2024-06-10T00:23:54.426131+0000","last_became_peered":"2024-06-10T00:23:54.426131+0000","last_unstale":"2024-06-10T00:28:12.573920+0000","last_undegraded":"2024-06-10T00:28:12.573920+0000","last_fullsized":"2024-06-10T00:28:12.573920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:56:14.819454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69365,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":390,"num_read_kb":264,"num_write":42,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570532+0000","last_change":"2024-06-10T00:23:57.552606+0000","last_active":"2024-06-10T00:27:57.570532+0000","last_peered":"2024-06-10T00:27:57.570532+0000","last_clean":"2024-06-10T00:27:57.570532+0000","last_became_active":"2024-06-10T00:23:57.552246+0000","last_became_peered":"2024-06-10T00:23:57.552246+0000","last_unstale":"2024-06-10T00:27:57.570532+0000","last_undegraded":"2024-06-10T00:27:57.570532+0000","last_fullsized":"2024-06-10T00:27:57.570532+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:19:21.508721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462817+0000","last_change":"2024-06-10T00:24:09.174862+0000","last_active":"2024-06-10T00:27:57.462817+0000","last_peered":"2024-06-10T00:27:57.462817+0000","last_clean":"2024-06-10T00:27:57.462817+0000","last_became_active":"2024-06-10T00:24:09.173550+0000","last_became_peered":"2024-06-10T00:24:09.173550+0000","last_unstale":"2024-06-10T00:27:57.462817+0000","last_undegraded":"2024-06-10T00:27:57.462817+0000","last_fullsized":"2024-06-10T00:27:57.462817+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:49:25.549084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"98'6","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591325+0000","last_change":"2024-06-10T00:23:54.426074+0000","last_active":"2024-06-10T00:27:57.591325+0000","last_peered":"2024-06-10T00:27:57.591325+0000","last_clean":"2024-06-10T00:27:57.591325+0000","last_became_active":"2024-06-10T00:23:54.425707+0000","last_became_peered":"2024-06-10T00:23:54.425707+0000","last_unstale":"2024-06-10T00:27:57.591325+0000","last_undegraded":"2024-06-10T00:27:57.591325+0000","last_fullsized":"2024-06-10T00:27:57.591325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:11:10.563595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542128+0000","last_change":"2024-06-10T00:23:57.549784+0000","last_active":"2024-06-10T00:27:57.542128+0000","last_peered":"2024-06-10T00:27:57.542128+0000","last_clean":"2024-06-10T00:27:57.542128+0000","last_became_active":"2024-06-10T00:23:57.549457+0000","last_became_peered":"2024-06-10T00:23:57.549457+0000","last_unstale":"2024-06-10T00:27:57.542128+0000","last_undegraded":"2024-06-10T00:27:57.542128+0000","last_fullsized":"2024-06-10T00:27:57.542128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:59:27.352927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592095+0000","last_change":"2024-06-10T00:24:09.171028+0000","last_active":"2024-06-10T00:27:57.592095+0000","last_peered":"2024-06-10T00:27:57.592095+0000","last_clean":"2024-06-10T00:27:57.592095+0000","last_became_active":"2024-06-10T00:24:09.170543+0000","last_became_peered":"2024-06-10T00:24:09.170543+0000","last_unstale":"2024-06-10T00:27:57.592095+0000","last_undegraded":"2024-06-10T00:27:57.592095+0000","last_fullsized":"2024-06-10T00:27:57.592095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:57:40.816585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461583+0000","last_change":"2024-06-10T00:23:54.423332+0000","last_active":"2024-06-10T00:27:57.461583+0000","last_peered":"2024-06-10T00:27:57.461583+0000","last_clean":"2024-06-10T00:27:57.461583+0000","last_became_active":"2024-06-10T00:23:54.422792+0000","last_became_peered":"2024-06-10T00:23:54.422792+0000","last_unstale":"2024-06-10T00:27:57.461583+0000","last_undegraded":"2024-06-10T00:27:57.461583+0000","last_fullsized":"2024-06-10T00:27:57.461583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:50:52.655641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"100'15","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591219+0000","last_change":"2024-06-10T00:23:57.551411+0000","last_active":"2024-06-10T00:27:57.591219+0000","last_peered":"2024-06-10T00:27:57.591219+0000","last_clean":"2024-06-10T00:27:57.591219+0000","last_became_active":"2024-06-10T00:23:57.551053+0000","last_became_peered":"2024-06-10T00:23:57.551053+0000","last_unstale":"2024-06-10T00:27:57.591219+0000","last_undegraded":"2024-06-10T00:27:57.591219+0000","last_fullsized":"2024-06-10T00:27:57.591219+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:16:18.082777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459282+0000","last_change":"2024-06-10T00:24:43.648767+0000","last_active":"2024-06-10T00:27:57.459282+0000","last_peered":"2024-06-10T00:27:57.459282+0000","last_clean":"2024-06-10T00:27:57.459282+0000","last_became_active":"2024-06-10T00:24:43.648267+0000","last_became_peered":"2024-06-10T00:24:43.648267+0000","last_unstale":"2024-06-10T00:27:57.459282+0000","last_undegraded":"2024-06-10T00:27:57.459282+0000","last_fullsized":"2024-06-10T00:27:57.459282+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:41:09.005202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"101'5","reported_seq":242,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:11.565030+0000","last_change":"2024-06-10T00:23:54.425963+0000","last_active":"2024-06-10T00:28:11.565030+0000","last_peered":"2024-06-10T00:28:11.565030+0000","last_clean":"2024-06-10T00:28:11.565030+0000","last_became_active":"2024-06-10T00:23:54.425581+0000","last_became_peered":"2024-06-10T00:23:54.425581+0000","last_unstale":"2024-06-10T00:28:11.565030+0000","last_undegraded":"2024-06-10T00:28:11.565030+0000","last_fullsized":"2024-06-10T00:28:11.565030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:54.760610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":192,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592340+0000","last_change":"2024-06-10T00:23:57.557464+0000","last_active":"2024-06-10T00:27:57.592340+0000","last_peered":"2024-06-10T00:27:57.592340+0000","last_clean":"2024-06-10T00:27:57.592340+0000","last_became_active":"2024-06-10T00:23:57.557130+0000","last_became_peered":"2024-06-10T00:23:57.557130+0000","last_unstale":"2024-06-10T00:27:57.592340+0000","last_undegraded":"2024-06-10T00:27:57.592340+0000","last_fullsized":"2024-06-10T00:27:57.592340+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:16:36.341210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":38,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543379+0000","last_change":"2024-06-10T00:24:43.753291+0000","last_active":"2024-06-10T00:27:57.543379+0000","last_peered":"2024-06-10T00:27:57.543379+0000","last_clean":"2024-06-10T00:27:57.543379+0000","last_became_active":"2024-06-10T00:24:43.752765+0000","last_became_peered":"2024-06-10T00:24:43.752765+0000","last_unstale":"2024-06-10T00:27:57.543379+0000","last_undegraded":"2024-06-10T00:27:57.543379+0000","last_fullsized":"2024-06-10T00:27:57.543379+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:55:06.349564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"98'15","reported_seq":63,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570040+0000","last_change":"2024-06-10T00:23:54.823489+0000","last_active":"2024-06-10T00:27:57.570040+0000","last_peered":"2024-06-10T00:27:57.570040+0000","last_clean":"2024-06-10T00:27:57.570040+0000","last_became_active":"2024-06-10T00:23:54.823073+0000","last_became_peered":"2024-06-10T00:23:54.823073+0000","last_unstale":"2024-06-10T00:27:57.570040+0000","last_undegraded":"2024-06-10T00:27:57.570040+0000","last_fullsized":"2024-06-10T00:27:57.570040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:07:52.563009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461389+0000","last_change":"2024-06-10T00:23:57.563388+0000","last_active":"2024-06-10T00:27:57.461389+0000","last_peered":"2024-06-10T00:27:57.461389+0000","last_clean":"2024-06-10T00:27:57.461389+0000","last_became_active":"2024-06-10T00:23:57.563004+0000","last_became_peered":"2024-06-10T00:23:57.563004+0000","last_unstale":"2024-06-10T00:27:57.461389+0000","last_undegraded":"2024-06-10T00:27:57.461389+0000","last_fullsized":"2024-06-10T00:27:57.461389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:51:18.805662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461226+0000","last_change":"2024-06-10T00:24:09.168447+0000","last_active":"2024-06-10T00:27:57.461226+0000","last_peered":"2024-06-10T00:27:57.461226+0000","last_clean":"2024-06-10T00:27:57.461226+0000","last_became_active":"2024-06-10T00:24:09.167439+0000","last_became_peered":"2024-06-10T00:24:09.167439+0000","last_unstale":"2024-06-10T00:27:57.461226+0000","last_undegraded":"2024-06-10T00:27:57.461226+0000","last_fullsized":"2024-06-10T00:27:57.461226+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:52.319395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"101'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570752+0000","last_change":"2024-06-10T00:23:54.427666+0000","last_active":"2024-06-10T00:27:57.570752+0000","last_peered":"2024-06-10T00:27:57.570752+0000","last_clean":"2024-06-10T00:27:57.570752+0000","last_became_active":"2024-06-10T00:23:54.427285+0000","last_became_peered":"2024-06-10T00:23:54.427285+0000","last_unstale":"2024-06-10T00:27:57.570752+0000","last_undegraded":"2024-06-10T00:27:57.570752+0000","last_fullsized":"2024-06-10T00:27:57.570752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:28:20.196168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"100'5","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.558787+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.558470+0000","last_became_peered":"2024-06-10T00:23:57.558470+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:02:14.758802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570142+0000","last_change":"2024-06-10T00:24:09.264952+0000","last_active":"2024-06-10T00:27:57.570142+0000","last_peered":"2024-06-10T00:27:57.570142+0000","last_clean":"2024-06-10T00:27:57.570142+0000","last_became_active":"2024-06-10T00:24:09.264379+0000","last_became_peered":"2024-06-10T00:24:09.264379+0000","last_unstale":"2024-06-10T00:27:57.570142+0000","last_undegraded":"2024-06-10T00:27:57.570142+0000","last_fullsized":"2024-06-10T00:27:57.570142+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:25:40.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458166+0000","last_change":"2024-06-10T00:23:54.823298+0000","last_active":"2024-06-10T00:27:57.458166+0000","last_peered":"2024-06-10T00:27:57.458166+0000","last_clean":"2024-06-10T00:27:57.458166+0000","last_became_active":"2024-06-10T00:23:54.822917+0000","last_became_peered":"2024-06-10T00:23:54.822917+0000","last_unstale":"2024-06-10T00:27:57.458166+0000","last_undegraded":"2024-06-10T00:27:57.458166+0000","last_fullsized":"2024-06-10T00:27:57.458166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:07:52.500641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"100'4","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459230+0000","last_change":"2024-06-10T00:23:57.559202+0000","last_active":"2024-06-10T00:27:57.459230+0000","last_peered":"2024-06-10T00:27:57.459230+0000","last_clean":"2024-06-10T00:27:57.459230+0000","last_became_active":"2024-06-10T00:23:57.558800+0000","last_became_peered":"2024-06-10T00:23:57.558800+0000","last_unstale":"2024-06-10T00:27:57.459230+0000","last_undegraded":"2024-06-10T00:27:57.459230+0000","last_fullsized":"2024-06-10T00:27:57.459230+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:49:51.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542486+0000","last_change":"2024-06-10T00:24:09.168603+0000","last_active":"2024-06-10T00:27:57.542486+0000","last_peered":"2024-06-10T00:27:57.542486+0000","last_clean":"2024-06-10T00:27:57.542486+0000","last_became_active":"2024-06-10T00:24:09.168233+0000","last_became_peered":"2024-06-10T00:24:09.168233+0000","last_unstale":"2024-06-10T00:27:57.542486+0000","last_undegraded":"2024-06-10T00:27:57.542486+0000","last_fullsized":"2024-06-10T00:27:57.542486+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:59.585023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'1","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592243+0000","last_change":"2024-06-10T00:23:54.425505+0000","last_active":"2024-06-10T00:27:57.592243+0000","last_peered":"2024-06-10T00:27:57.592243+0000","last_clean":"2024-06-10T00:27:57.592243+0000","last_became_active":"2024-06-10T00:23:54.425058+0000","last_became_peered":"2024-06-10T00:23:54.425058+0000","last_unstale":"2024-06-10T00:27:57.592243+0000","last_undegraded":"2024-06-10T00:27:57.592243+0000","last_fullsized":"2024-06-10T00:27:57.592243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:48:05.605947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"100'7","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458879+0000","last_change":"2024-06-10T00:23:57.566060+0000","last_active":"2024-06-10T00:27:57.458879+0000","last_peered":"2024-06-10T00:27:57.458879+0000","last_clean":"2024-06-10T00:27:57.458879+0000","last_became_active":"2024-06-10T00:23:57.565099+0000","last_became_peered":"2024-06-10T00:23:57.565099+0000","last_unstale":"2024-06-10T00:27:57.458879+0000","last_undegraded":"2024-06-10T00:27:57.458879+0000","last_fullsized":"2024-06-10T00:27:57.458879+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:24:54.204399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543284+0000","last_change":"2024-06-10T00:24:09.264741+0000","last_active":"2024-06-10T00:27:57.543284+0000","last_peered":"2024-06-10T00:27:57.543284+0000","last_clean":"2024-06-10T00:27:57.543284+0000","last_became_active":"2024-06-10T00:24:09.264267+0000","last_became_peered":"2024-06-10T00:24:09.264267+0000","last_unstale":"2024-06-10T00:27:57.543284+0000","last_undegraded":"2024-06-10T00:27:57.543284+0000","last_fullsized":"2024-06-10T00:27:57.543284+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:27.627391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457224+0000","last_change":"2024-06-10T00:25:44.346070+0000","last_active":"2024-06-10T00:27:57.457224+0000","last_peered":"2024-06-10T00:27:57.457224+0000","last_clean":"2024-06-10T00:27:57.457224+0000","last_became_active":"2024-06-10T00:25:44.345798+0000","last_became_peered":"2024-06-10T00:25:44.345798+0000","last_unstale":"2024-06-10T00:27:57.457224+0000","last_undegraded":"2024-06-10T00:27:57.457224+0000","last_fullsized":"2024-06-10T00:27:57.457224+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:04.506535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"101'5","reported_seq":241,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:11.564249+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:28:11.564249+0000","last_peered":"2024-06-10T00:28:11.564249+0000","last_clean":"2024-06-10T00:28:11.564249+0000","last_became_active":"2024-06-10T00:23:54.822442+0000","last_became_peered":"2024-06-10T00:23:54.822442+0000","last_unstale":"2024-06-10T00:28:11.564249+0000","last_undegraded":"2024-06-10T00:28:11.564249+0000","last_fullsized":"2024-06-10T00:28:11.564249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:51:10.278257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":188,"num_read_kb":188,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"98'1","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591953+0000","last_change":"2024-06-10T00:24:09.168073+0000","last_active":"2024-06-10T00:27:57.591953+0000","last_peered":"2024-06-10T00:27:57.591953+0000","last_clean":"2024-06-10T00:27:57.591953+0000","last_became_active":"2024-06-10T00:24:09.167657+0000","last_became_peered":"2024-06-10T00:24:09.167657+0000","last_unstale":"2024-06-10T00:27:57.591953+0000","last_undegraded":"2024-06-10T00:27:57.591953+0000","last_fullsized":"2024-06-10T00:27:57.591953+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:20:23.481685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457629+0000","last_change":"2024-06-10T00:25:44.346629+0000","last_active":"2024-06-10T00:27:57.457629+0000","last_peered":"2024-06-10T00:27:57.457629+0000","last_clean":"2024-06-10T00:27:57.457629+0000","last_became_active":"2024-06-10T00:25:44.346388+0000","last_became_peered":"2024-06-10T00:25:44.346388+0000","last_unstale":"2024-06-10T00:27:57.457629+0000","last_undegraded":"2024-06-10T00:27:57.457629+0000","last_fullsized":"2024-06-10T00:27:57.457629+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:03:19.228632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"98'3","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544980+0000","last_change":"2024-06-10T00:23:54.423507+0000","last_active":"2024-06-10T00:27:57.544980+0000","last_peered":"2024-06-10T00:27:57.544980+0000","last_clean":"2024-06-10T00:27:57.544980+0000","last_became_active":"2024-06-10T00:23:54.423092+0000","last_became_peered":"2024-06-10T00:23:54.423092+0000","last_unstale":"2024-06-10T00:27:57.544980+0000","last_undegraded":"2024-06-10T00:27:57.544980+0000","last_fullsized":"2024-06-10T00:27:57.544980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:06:15.924827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"101'148","reported_seq":424,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:59.577811+0000","last_change":"2024-06-10T00:22:28.025243+0000","last_active":"2024-06-10T00:27:59.577811+0000","last_peered":"2024-06-10T00:27:59.577811+0000","last_clean":"2024-06-10T00:27:59.577811+0000","last_became_active":"2024-06-10T00:22:28.024751+0000","last_became_peered":"2024-06-10T00:22:28.024751+0000","last_unstale":"2024-06-10T00:27:59.577811+0000","last_undegraded":"2024-06-10T00:27:59.577811+0000","last_fullsized":"2024-06-10T00:27:59.577811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_clean_scrub_stamp":"2024-06-10T00:19:05.039631+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:02:36.557931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461952+0000","last_change":"2024-06-10T00:24:09.265084+0000","last_active":"2024-06-10T00:27:57.461952+0000","last_peered":"2024-06-10T00:27:57.461952+0000","last_clean":"2024-06-10T00:27:57.461952+0000","last_became_active":"2024-06-10T00:24:09.264425+0000","last_became_peered":"2024-06-10T00:24:09.264425+0000","last_unstale":"2024-06-10T00:27:57.461952+0000","last_undegraded":"2024-06-10T00:27:57.461952+0000","last_fullsized":"2024-06-10T00:27:57.461952+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:37:31.577688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.541983+0000","last_change":"2024-06-10T00:23:57.558540+0000","last_active":"2024-06-10T00:27:57.541983+0000","last_peered":"2024-06-10T00:27:57.541983+0000","last_clean":"2024-06-10T00:27:57.541983+0000","last_became_active":"2024-06-10T00:23:57.558197+0000","last_became_peered":"2024-06-10T00:23:57.558197+0000","last_unstale":"2024-06-10T00:27:57.541983+0000","last_undegraded":"2024-06-10T00:27:57.541983+0000","last_fullsized":"2024-06-10T00:27:57.541983+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:04:25.566135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592707+0000","last_change":"2024-06-10T00:23:54.428126+0000","last_active":"2024-06-10T00:27:57.592707+0000","last_peered":"2024-06-10T00:27:57.592707+0000","last_clean":"2024-06-10T00:27:57.592707+0000","last_became_active":"2024-06-10T00:23:54.427630+0000","last_became_peered":"2024-06-10T00:23:54.427630+0000","last_unstale":"2024-06-10T00:27:57.592707+0000","last_undegraded":"2024-06-10T00:27:57.592707+0000","last_fullsized":"2024-06-10T00:27:57.592707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:05:23.679704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591957+0000","last_change":"2024-06-10T00:24:09.169984+0000","last_active":"2024-06-10T00:27:57.591957+0000","last_peered":"2024-06-10T00:27:57.591957+0000","last_clean":"2024-06-10T00:27:57.591957+0000","last_became_active":"2024-06-10T00:24:09.169556+0000","last_became_peered":"2024-06-10T00:24:09.169556+0000","last_unstale":"2024-06-10T00:27:57.591957+0000","last_undegraded":"2024-06-10T00:27:57.591957+0000","last_fullsized":"2024-06-10T00:27:57.591957+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:54:13.575207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"98'1","reported_seq":20,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591972+0000","last_change":"2024-06-10T00:24:43.767901+0000","last_active":"2024-06-10T00:27:57.591972+0000","last_peered":"2024-06-10T00:27:57.591972+0000","last_clean":"2024-06-10T00:27:57.591972+0000","last_became_active":"2024-06-10T00:24:43.767562+0000","last_became_peered":"2024-06-10T00:24:43.767562+0000","last_unstale":"2024-06-10T00:27:57.591972+0000","last_undegraded":"2024-06-10T00:27:57.591972+0000","last_fullsized":"2024-06-10T00:27:57.591972+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:46:36.099750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"100'9","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543096+0000","last_change":"2024-06-10T00:23:57.554906+0000","last_active":"2024-06-10T00:27:57.543096+0000","last_peered":"2024-06-10T00:27:57.543096+0000","last_clean":"2024-06-10T00:27:57.543096+0000","last_became_active":"2024-06-10T00:23:57.554608+0000","last_became_peered":"2024-06-10T00:23:57.554608+0000","last_unstale":"2024-06-10T00:27:57.543096+0000","last_undegraded":"2024-06-10T00:27:57.543096+0000","last_fullsized":"2024-06-10T00:27:57.543096+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:24.690594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570670+0000","last_change":"2024-06-10T00:24:09.264676+0000","last_active":"2024-06-10T00:27:57.570670+0000","last_peered":"2024-06-10T00:27:57.570670+0000","last_clean":"2024-06-10T00:27:57.570670+0000","last_became_active":"2024-06-10T00:24:09.264259+0000","last_became_peered":"2024-06-10T00:24:09.264259+0000","last_unstale":"2024-06-10T00:27:57.570670+0000","last_undegraded":"2024-06-10T00:27:57.570670+0000","last_fullsized":"2024-06-10T00:27:57.570670+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:37:53.787039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"101'102","reported_seq":242,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:10.494814+0000","last_change":"2024-06-10T00:23:54.424201+0000","last_active":"2024-06-10T00:28:10.494814+0000","last_peered":"2024-06-10T00:28:10.494814+0000","last_clean":"2024-06-10T00:28:10.494814+0000","last_became_active":"2024-06-10T00:23:54.423853+0000","last_became_peered":"2024-06-10T00:23:54.423853+0000","last_unstale":"2024-06-10T00:28:10.494814+0000","last_undegraded":"2024-06-10T00:28:10.494814+0000","last_fullsized":"2024-06-10T00:28:10.494814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:00.884041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":187,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571583+0000","last_change":"2024-06-10T00:23:57.554276+0000","last_active":"2024-06-10T00:27:57.571583+0000","last_peered":"2024-06-10T00:27:57.571583+0000","last_clean":"2024-06-10T00:27:57.571583+0000","last_became_active":"2024-06-10T00:23:57.553964+0000","last_became_peered":"2024-06-10T00:23:57.553964+0000","last_unstale":"2024-06-10T00:27:57.571583+0000","last_undegraded":"2024-06-10T00:27:57.571583+0000","last_fullsized":"2024-06-10T00:27:57.571583+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:39:51.974770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591089+0000","last_change":"2024-06-10T00:24:09.170249+0000","last_active":"2024-06-10T00:27:57.591089+0000","last_peered":"2024-06-10T00:27:57.591089+0000","last_clean":"2024-06-10T00:27:57.591089+0000","last_became_active":"2024-06-10T00:24:09.169830+0000","last_became_peered":"2024-06-10T00:24:09.169830+0000","last_unstale":"2024-06-10T00:27:57.591089+0000","last_undegraded":"2024-06-10T00:27:57.591089+0000","last_fullsized":"2024-06-10T00:27:57.591089+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.320609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"98'4","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545304+0000","last_change":"2024-06-10T00:23:54.473943+0000","last_active":"2024-06-10T00:27:57.545304+0000","last_peered":"2024-06-10T00:27:57.545304+0000","last_clean":"2024-06-10T00:27:57.545304+0000","last_became_active":"2024-06-10T00:23:54.473652+0000","last_became_peered":"2024-06-10T00:23:54.473652+0000","last_unstale":"2024-06-10T00:27:57.545304+0000","last_undegraded":"2024-06-10T00:27:57.545304+0000","last_fullsized":"2024-06-10T00:27:57.545304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:52:06.773684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"100'7","reported_seq":55,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593765+0000","last_change":"2024-06-10T00:25:44.735050+0000","last_active":"2024-06-10T00:27:57.593765+0000","last_peered":"2024-06-10T00:27:57.593765+0000","last_clean":"2024-06-10T00:27:57.593765+0000","last_became_active":"2024-06-10T00:25:44.734756+0000","last_became_peered":"2024-06-10T00:25:44.734756+0000","last_unstale":"2024-06-10T00:27:57.593765+0000","last_undegraded":"2024-06-10T00:27:57.593765+0000","last_fullsized":"2024-06-10T00:27:57.593765+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:16:03.924238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545083+0000","last_change":"2024-06-10T00:24:43.768837+0000","last_active":"2024-06-10T00:27:57.545083+0000","last_peered":"2024-06-10T00:27:57.545083+0000","last_clean":"2024-06-10T00:27:57.545083+0000","last_became_active":"2024-06-10T00:24:43.768295+0000","last_became_peered":"2024-06-10T00:24:43.768295+0000","last_unstale":"2024-06-10T00:27:57.545083+0000","last_undegraded":"2024-06-10T00:27:57.545083+0000","last_fullsized":"2024-06-10T00:27:57.545083+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:59.600864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"98'2","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591761+0000","last_change":"2024-06-10T00:23:54.823448+0000","last_active":"2024-06-10T00:27:57.591761+0000","last_peered":"2024-06-10T00:27:57.591761+0000","last_clean":"2024-06-10T00:27:57.591761+0000","last_became_active":"2024-06-10T00:23:54.823107+0000","last_became_peered":"2024-06-10T00:23:54.823107+0000","last_unstale":"2024-06-10T00:27:57.591761+0000","last_undegraded":"2024-06-10T00:27:57.591761+0000","last_fullsized":"2024-06-10T00:27:57.591761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:50.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"100'2","reported_seq":43,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569862+0000","last_change":"2024-06-10T00:23:57.556733+0000","last_active":"2024-06-10T00:27:57.569862+0000","last_peered":"2024-06-10T00:27:57.569862+0000","last_clean":"2024-06-10T00:27:57.569862+0000","last_became_active":"2024-06-10T00:23:57.556386+0000","last_became_peered":"2024-06-10T00:23:57.556386+0000","last_unstale":"2024-06-10T00:27:57.569862+0000","last_undegraded":"2024-06-10T00:27:57.569862+0000","last_fullsized":"2024-06-10T00:27:57.569862+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:39:00.572311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458015+0000","last_change":"2024-06-10T00:24:09.174600+0000","last_active":"2024-06-10T00:27:57.458015+0000","last_peered":"2024-06-10T00:27:57.458015+0000","last_clean":"2024-06-10T00:27:57.458015+0000","last_became_active":"2024-06-10T00:24:09.174236+0000","last_became_peered":"2024-06-10T00:24:09.174236+0000","last_unstale":"2024-06-10T00:27:57.458015+0000","last_undegraded":"2024-06-10T00:27:57.458015+0000","last_fullsized":"2024-06-10T00:27:57.458015+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:58:18.447768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571190+0000","last_change":"2024-06-10T00:23:54.425111+0000","last_active":"2024-06-10T00:27:57.571190+0000","last_peered":"2024-06-10T00:27:57.571190+0000","last_clean":"2024-06-10T00:27:57.571190+0000","last_became_active":"2024-06-10T00:23:54.424724+0000","last_became_peered":"2024-06-10T00:23:54.424724+0000","last_unstale":"2024-06-10T00:27:57.571190+0000","last_undegraded":"2024-06-10T00:27:57.571190+0000","last_fullsized":"2024-06-10T00:27:57.571190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:54.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"100'9","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.556355+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.555479+0000","last_became_peered":"2024-06-10T00:23:57.555479+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:08.817381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"98'1","reported_seq":30,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544968+0000","last_change":"2024-06-10T00:24:09.169589+0000","last_active":"2024-06-10T00:27:57.544968+0000","last_peered":"2024-06-10T00:27:57.544968+0000","last_clean":"2024-06-10T00:27:57.544968+0000","last_became_active":"2024-06-10T00:24:09.168436+0000","last_became_peered":"2024-06-10T00:24:09.168436+0000","last_unstale":"2024-06-10T00:27:57.544968+0000","last_undegraded":"2024-06-10T00:27:57.544968+0000","last_fullsized":"2024-06-10T00:27:57.544968+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:17:32.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"98'1","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591500+0000","last_change":"2024-06-10T00:24:43.751399+0000","last_active":"2024-06-10T00:27:57.591500+0000","last_peered":"2024-06-10T00:27:57.591500+0000","last_clean":"2024-06-10T00:27:57.591500+0000","last_became_active":"2024-06-10T00:24:43.750987+0000","last_became_peered":"2024-06-10T00:24:43.750987+0000","last_unstale":"2024-06-10T00:27:57.591500+0000","last_undegraded":"2024-06-10T00:27:57.591500+0000","last_fullsized":"2024-06-10T00:27:57.591500+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:14:47.902017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570338+0000","last_change":"2024-06-10T00:23:57.560213+0000","last_active":"2024-06-10T00:27:57.570338+0000","last_peered":"2024-06-10T00:27:57.570338+0000","last_clean":"2024-06-10T00:27:57.570338+0000","last_became_active":"2024-06-10T00:23:57.559925+0000","last_became_peered":"2024-06-10T00:23:57.559925+0000","last_unstale":"2024-06-10T00:27:57.570338+0000","last_undegraded":"2024-06-10T00:27:57.570338+0000","last_fullsized":"2024-06-10T00:27:57.570338+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:00:37.403087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545095+0000","last_change":"2024-06-10T00:24:09.264758+0000","last_active":"2024-06-10T00:27:57.545095+0000","last_peered":"2024-06-10T00:27:57.545095+0000","last_clean":"2024-06-10T00:27:57.545095+0000","last_became_active":"2024-06-10T00:24:09.263928+0000","last_became_peered":"2024-06-10T00:24:09.263928+0000","last_unstale":"2024-06-10T00:27:57.545095+0000","last_undegraded":"2024-06-10T00:27:57.545095+0000","last_fullsized":"2024-06-10T00:27:57.545095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:23:41.083091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570103+0000","last_change":"2024-06-10T00:23:54.428553+0000","last_active":"2024-06-10T00:27:57.570103+0000","last_peered":"2024-06-10T00:27:57.570103+0000","last_clean":"2024-06-10T00:27:57.570103+0000","last_became_active":"2024-06-10T00:23:54.428108+0000","last_became_peered":"2024-06-10T00:23:54.428108+0000","last_unstale":"2024-06-10T00:27:57.570103+0000","last_undegraded":"2024-06-10T00:27:57.570103+0000","last_fullsized":"2024-06-10T00:27:57.570103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:57:26.207915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591119+0000","last_change":"2024-06-10T00:23:57.560686+0000","last_active":"2024-06-10T00:27:57.591119+0000","last_peered":"2024-06-10T00:27:57.591119+0000","last_clean":"2024-06-10T00:27:57.591119+0000","last_became_active":"2024-06-10T00:23:57.560326+0000","last_became_peered":"2024-06-10T00:23:57.560326+0000","last_unstale":"2024-06-10T00:27:57.591119+0000","last_undegraded":"2024-06-10T00:27:57.591119+0000","last_fullsized":"2024-06-10T00:27:57.591119+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:08:39.506567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591281+0000","last_change":"2024-06-10T00:24:09.166947+0000","last_active":"2024-06-10T00:27:57.591281+0000","last_peered":"2024-06-10T00:27:57.591281+0000","last_clean":"2024-06-10T00:27:57.591281+0000","last_became_active":"2024-06-10T00:24:09.166503+0000","last_became_peered":"2024-06-10T00:24:09.166503+0000","last_unstale":"2024-06-10T00:27:57.591281+0000","last_undegraded":"2024-06-10T00:27:57.591281+0000","last_fullsized":"2024-06-10T00:27:57.591281+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:16:45.094642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591369+0000","last_change":"2024-06-10T00:23:54.823522+0000","last_active":"2024-06-10T00:27:57.591369+0000","last_peered":"2024-06-10T00:27:57.591369+0000","last_clean":"2024-06-10T00:27:57.591369+0000","last_became_active":"2024-06-10T00:23:54.823142+0000","last_became_peered":"2024-06-10T00:23:54.823142+0000","last_unstale":"2024-06-10T00:27:57.591369+0000","last_undegraded":"2024-06-10T00:27:57.591369+0000","last_fullsized":"2024-06-10T00:27:57.591369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:03.836850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592288+0000","last_change":"2024-06-10T00:23:57.563710+0000","last_active":"2024-06-10T00:27:57.592288+0000","last_peered":"2024-06-10T00:27:57.592288+0000","last_clean":"2024-06-10T00:27:57.592288+0000","last_became_active":"2024-06-10T00:23:57.563331+0000","last_became_peered":"2024-06-10T00:23:57.563331+0000","last_unstale":"2024-06-10T00:27:57.592288+0000","last_undegraded":"2024-06-10T00:27:57.592288+0000","last_fullsized":"2024-06-10T00:27:57.592288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:08:48.061885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592820+0000","last_change":"2024-06-10T00:24:09.171203+0000","last_active":"2024-06-10T00:27:57.592820+0000","last_peered":"2024-06-10T00:27:57.592820+0000","last_clean":"2024-06-10T00:27:57.592820+0000","last_became_active":"2024-06-10T00:24:09.170824+0000","last_became_peered":"2024-06-10T00:24:09.170824+0000","last_unstale":"2024-06-10T00:27:57.592820+0000","last_undegraded":"2024-06-10T00:27:57.592820+0000","last_fullsized":"2024-06-10T00:27:57.592820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:47:28.045602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"101'65","reported_seq":143,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:10.142035+0000","last_change":"2024-06-10T00:23:54.421307+0000","last_active":"2024-06-10T00:28:10.142035+0000","last_peered":"2024-06-10T00:28:10.142035+0000","last_clean":"2024-06-10T00:28:10.142035+0000","last_became_active":"2024-06-10T00:23:54.420928+0000","last_became_peered":"2024-06-10T00:23:54.420928+0000","last_unstale":"2024-06-10T00:28:10.142035+0000","last_undegraded":"2024-06-10T00:28:10.142035+0000","last_fullsized":"2024-06-10T00:28:10.142035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:59:41.433605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591623+0000","last_change":"2024-06-10T00:23:57.556494+0000","last_active":"2024-06-10T00:27:57.591623+0000","last_peered":"2024-06-10T00:27:57.591623+0000","last_clean":"2024-06-10T00:27:57.591623+0000","last_became_active":"2024-06-10T00:23:57.556088+0000","last_became_peered":"2024-06-10T00:23:57.556088+0000","last_unstale":"2024-06-10T00:27:57.591623+0000","last_undegraded":"2024-06-10T00:27:57.591623+0000","last_fullsized":"2024-06-10T00:27:57.591623+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:06:23.873281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461068+0000","last_change":"2024-06-10T00:24:09.166734+0000","last_active":"2024-06-10T00:27:57.461068+0000","last_peered":"2024-06-10T00:27:57.461068+0000","last_clean":"2024-06-10T00:27:57.461068+0000","last_became_active":"2024-06-10T00:24:09.166270+0000","last_became_peered":"2024-06-10T00:24:09.166270+0000","last_unstale":"2024-06-10T00:27:57.461068+0000","last_undegraded":"2024-06-10T00:27:57.461068+0000","last_fullsized":"2024-06-10T00:27:57.461068+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:38.039633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543256+0000","last_change":"2024-06-10T00:23:54.426226+0000","last_active":"2024-06-10T00:27:57.543256+0000","last_peered":"2024-06-10T00:27:57.543256+0000","last_clean":"2024-06-10T00:27:57.543256+0000","last_became_active":"2024-06-10T00:23:54.425879+0000","last_became_peered":"2024-06-10T00:23:54.425879+0000","last_unstale":"2024-06-10T00:27:57.543256+0000","last_undegraded":"2024-06-10T00:27:57.543256+0000","last_fullsized":"2024-06-10T00:27:57.543256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:15:32.936556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592043+0000","last_change":"2024-06-10T00:23:57.553814+0000","last_active":"2024-06-10T00:27:57.592043+0000","last_peered":"2024-06-10T00:27:57.592043+0000","last_clean":"2024-06-10T00:27:57.592043+0000","last_became_active":"2024-06-10T00:23:57.553478+0000","last_became_peered":"2024-06-10T00:23:57.553478+0000","last_unstale":"2024-06-10T00:27:57.592043+0000","last_undegraded":"2024-06-10T00:27:57.592043+0000","last_fullsized":"2024-06-10T00:27:57.592043+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:54:39.264526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591455+0000","last_change":"2024-06-10T00:23:57.556676+0000","last_active":"2024-06-10T00:27:57.591455+0000","last_peered":"2024-06-10T00:27:57.591455+0000","last_clean":"2024-06-10T00:27:57.591455+0000","last_became_active":"2024-06-10T00:23:57.555946+0000","last_became_peered":"2024-06-10T00:23:57.555946+0000","last_unstale":"2024-06-10T00:27:57.591455+0000","last_undegraded":"2024-06-10T00:27:57.591455+0000","last_fullsized":"2024-06-10T00:27:57.591455+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:10:10.868799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"100'12","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590806+0000","last_change":"2024-06-10T00:23:57.565423+0000","last_active":"2024-06-10T00:27:57.590806+0000","last_peered":"2024-06-10T00:27:57.590806+0000","last_clean":"2024-06-10T00:27:57.590806+0000","last_became_active":"2024-06-10T00:23:57.564830+0000","last_became_peered":"2024-06-10T00:23:57.564830+0000","last_unstale":"2024-06-10T00:27:57.590806+0000","last_undegraded":"2024-06-10T00:27:57.590806+0000","last_fullsized":"2024-06-10T00:27:57.590806+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:11:06.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"100'8","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591073+0000","last_change":"2024-06-10T00:23:57.558971+0000","last_active":"2024-06-10T00:27:57.591073+0000","last_peered":"2024-06-10T00:27:57.591073+0000","last_clean":"2024-06-10T00:27:57.591073+0000","last_became_active":"2024-06-10T00:23:57.558620+0000","last_became_peered":"2024-06-10T00:23:57.558620+0000","last_unstale":"2024-06-10T00:27:57.591073+0000","last_undegraded":"2024-06-10T00:27:57.591073+0000","last_fullsized":"2024-06-10T00:27:57.591073+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:45:43.308008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"100'7","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591540+0000","last_change":"2024-06-10T00:23:57.552341+0000","last_active":"2024-06-10T00:27:57.591540+0000","last_peered":"2024-06-10T00:27:57.591540+0000","last_clean":"2024-06-10T00:27:57.591540+0000","last_became_active":"2024-06-10T00:23:57.551464+0000","last_became_peered":"2024-06-10T00:23:57.551464+0000","last_unstale":"2024-06-10T00:27:57.591540+0000","last_undegraded":"2024-06-10T00:27:57.591540+0000","last_fullsized":"2024-06-10T00:27:57.591540+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:44:41.063516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"100'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543209+0000","last_change":"2024-06-10T00:24:43.768961+0000","last_active":"2024-06-10T00:27:57.543209+0000","last_peered":"2024-06-10T00:27:57.543209+0000","last_clean":"2024-06-10T00:27:57.543209+0000","last_became_active":"2024-06-10T00:24:43.768542+0000","last_became_peered":"2024-06-10T00:24:43.768542+0000","last_unstale":"2024-06-10T00:27:57.543209+0000","last_undegraded":"2024-06-10T00:27:57.543209+0000","last_fullsized":"2024-06-10T00:27:57.543209+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:23:58.412054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"100'6","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591613+0000","last_change":"2024-06-10T00:23:57.554924+0000","last_active":"2024-06-10T00:27:57.591613+0000","last_peered":"2024-06-10T00:27:57.591613+0000","last_clean":"2024-06-10T00:27:57.591613+0000","last_became_active":"2024-06-10T00:23:57.554515+0000","last_became_peered":"2024-06-10T00:23:57.554515+0000","last_unstale":"2024-06-10T00:27:57.591613+0000","last_undegraded":"2024-06-10T00:27:57.591613+0000","last_fullsized":"2024-06-10T00:27:57.591613+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:58:15.624775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"100'15","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591485+0000","last_change":"2024-06-10T00:23:57.561940+0000","last_active":"2024-06-10T00:27:57.591485+0000","last_peered":"2024-06-10T00:27:57.591485+0000","last_clean":"2024-06-10T00:27:57.591485+0000","last_became_active":"2024-06-10T00:23:57.561512+0000","last_became_peered":"2024-06-10T00:23:57.561512+0000","last_unstale":"2024-06-10T00:27:57.591485+0000","last_undegraded":"2024-06-10T00:27:57.591485+0000","last_fullsized":"2024-06-10T00:27:57.591485+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:57:57.147311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"100'2","reported_seq":43,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570333+0000","last_change":"2024-06-10T00:23:57.557229+0000","last_active":"2024-06-10T00:27:57.570333+0000","last_peered":"2024-06-10T00:27:57.570333+0000","last_clean":"2024-06-10T00:27:57.570333+0000","last_became_active":"2024-06-10T00:23:57.556905+0000","last_became_peered":"2024-06-10T00:23:57.556905+0000","last_unstale":"2024-06-10T00:27:57.570333+0000","last_undegraded":"2024-06-10T00:27:57.570333+0000","last_fullsized":"2024-06-10T00:27:57.570333+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:30.513517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"100'6","reported_seq":24,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592589+0000","last_change":"2024-06-10T00:24:43.768509+0000","last_active":"2024-06-10T00:27:57.592589+0000","last_peered":"2024-06-10T00:27:57.592589+0000","last_clean":"2024-06-10T00:27:57.592589+0000","last_became_active":"2024-06-10T00:24:43.768038+0000","last_became_peered":"2024-06-10T00:24:43.768038+0000","last_unstale":"2024-06-10T00:27:57.592589+0000","last_undegraded":"2024-06-10T00:27:57.592589+0000","last_fullsized":"2024-06-10T00:27:57.592589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:38.344375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571244+0000","last_change":"2024-06-10T00:23:57.557830+0000","last_active":"2024-06-10T00:27:57.571244+0000","last_peered":"2024-06-10T00:27:57.571244+0000","last_clean":"2024-06-10T00:27:57.571244+0000","last_became_active":"2024-06-10T00:23:57.556986+0000","last_became_peered":"2024-06-10T00:23:57.556986+0000","last_unstale":"2024-06-10T00:27:57.571244+0000","last_undegraded":"2024-06-10T00:27:57.571244+0000","last_fullsized":"2024-06-10T00:27:57.571244+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:53.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590760+0000","last_change":"2024-06-10T00:23:57.565602+0000","last_active":"2024-06-10T00:27:57.590760+0000","last_peered":"2024-06-10T00:27:57.590760+0000","last_clean":"2024-06-10T00:27:57.590760+0000","last_became_active":"2024-06-10T00:23:57.565264+0000","last_became_peered":"2024-06-10T00:23:57.565264+0000","last_unstale":"2024-06-10T00:27:57.590760+0000","last_undegraded":"2024-06-10T00:27:57.590760+0000","last_fullsized":"2024-06-10T00:27:57.590760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:09:12.773087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542211+0000","last_change":"2024-06-10T00:23:57.548516+0000","last_active":"2024-06-10T00:27:57.542211+0000","last_peered":"2024-06-10T00:27:57.542211+0000","last_clean":"2024-06-10T00:27:57.542211+0000","last_became_active":"2024-06-10T00:23:57.548202+0000","last_became_peered":"2024-06-10T00:23:57.548202+0000","last_unstale":"2024-06-10T00:27:57.542211+0000","last_undegraded":"2024-06-10T00:27:57.542211+0000","last_fullsized":"2024-06-10T00:27:57.542211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:13.901478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591566+0000","last_change":"2024-06-10T00:23:57.565900+0000","last_active":"2024-06-10T00:27:57.591566+0000","last_peered":"2024-06-10T00:27:57.591566+0000","last_clean":"2024-06-10T00:27:57.591566+0000","last_became_active":"2024-06-10T00:23:57.565573+0000","last_became_peered":"2024-06-10T00:23:57.565573+0000","last_unstale":"2024-06-10T00:27:57.591566+0000","last_undegraded":"2024-06-10T00:27:57.591566+0000","last_fullsized":"2024-06-10T00:27:57.591566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:08.255905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107100,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":796432,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1511,"num_read_kb":973,"num_write":427,"num_write_kb":2362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2389296,"data_compressed":103467,"data_compressed_allocated":135168,"data_compressed_original":2101248,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351043,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1118208,"data_stored":582820,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612680,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712688,"statfs":{"total":95995035648,"available":95961792512,"internally_reserved":0,"allocated":1310720,"data_stored":1568443,"data_compressed":86181,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776206,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1159168,"data_stored":980346,"data_compressed":54566,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037842,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1150976,"data_stored":976148,"data_compressed":54566,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299479,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":1180481,"data_compressed":70979,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463004,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1110016,"data_stored":578210,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1183744,"data_stored":647485,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":579598,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411441,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36520,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708632,"statfs":{"total":95995035648,"available":95957639168,"internally_reserved":0,"allocated":1208320,"data_stored":650064,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1191936,"data_stored":1299852,"data_compressed":69768,"data_compressed_allocated":94208,"data_compressed_original":798720,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888166,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":583640,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724641,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32388,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712764,"statfs":{"total":95995035648,"available":95961870336,"internally_reserved":0,"allocated":1232896,"data_stored":1503521,"data_compressed":86181,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34489,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330937,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19287,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19287,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":399955,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-10T00:28:18.926 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-10T00:28:18.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-10T00:28:19.170 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:19.576 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:19 smithi121 ceph-mon[32378]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-06-10T00:28:19.576 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:19 smithi121 ceph-mon[32378]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:19.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:19 smithi193 ceph-mon[36716]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-06-10T00:28:19.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:19 smithi193 ceph-mon[36716]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:19.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:19 smithi135 ceph-mon[36424]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-06-10T00:28:19.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:19 smithi135 ceph-mon[36424]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:20.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.870+0000 7fc05da4b700 1 -- 172.21.15.121:0/4122458625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc058100b50 msgr2=0x7fc058100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:20.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.870+0000 7fc05da4b700 1 --2- 172.21.15.121:0/4122458625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc058100b50 0x7fc058100fd0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc04c009990 tx=0x7fc04c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:28:20.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.870+0000 7fc05da4b700 1 -- 172.21.15.121:0/4122458625 shutdown_connections 2024-06-10T00:28:20.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.870+0000 7fc05da4b700 1 --2- 172.21.15.121:0/4122458625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc058101510 0x7fc05810d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:20.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.870+0000 7fc05da4b700 1 --2- 172.21.15.121:0/4122458625 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc058106b90 0x7fc058106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:20.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.870+0000 7fc05da4b700 1 --2- 172.21.15.121:0/4122458625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc058100b50 0x7fc058100fd0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:20.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.870+0000 7fc05da4b700 1 -- 172.21.15.121:0/4122458625 >> 172.21.15.121:0/4122458625 conn(0x7fc058077a00 msgr2=0x7fc0580fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:20.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.870+0000 7fc05da4b700 1 -- 172.21.15.121:0/4122458625 shutdown_connections 2024-06-10T00:28:20.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.870+0000 7fc05da4b700 1 -- 172.21.15.121:0/4122458625 wait complete. 2024-06-10T00:28:20.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.870+0000 7fc05da4b700 1 Processor -- start 2024-06-10T00:28:20.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc05da4b700 1 -- start start 2024-06-10T00:28:20.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc05da4b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc058100b50 0x7fc058199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:20.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc05da4b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc058101510 0x7fc058199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:20.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc05da4b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc058106b90 0x7fc0581a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:20.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc05da4b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc0581116f0 con 0x7fc058101510 2024-06-10T00:28:20.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc05da4b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc058111570 con 0x7fc058100b50 2024-06-10T00:28:20.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc05da4b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc058111870 con 0x7fc058106b90 2024-06-10T00:28:20.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc0567fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc058101510 0x7fc058199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:20.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc0567fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc058101510 0x7fc058199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46710/0 (socket says 172.21.15.121:46710) 2024-06-10T00:28:20.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc0567fc700 1 -- 172.21.15.121:0/3342337914 learned_addr learned my addr 172.21.15.121:0/3342337914 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:20.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc056ffd700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc058100b50 0x7fc058199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:20.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc0577fe700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc058106b90 0x7fc0581a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:20.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc0567fc700 1 -- 172.21.15.121:0/3342337914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc058106b90 msgr2=0x7fc0581a4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:20.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc0567fc700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc058106b90 0x7fc0581a4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:20.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc0567fc700 1 -- 172.21.15.121:0/3342337914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc058100b50 msgr2=0x7fc058199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:20.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc0567fc700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc058100b50 0x7fc058199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:20.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.871+0000 7fc0567fc700 1 -- 172.21.15.121:0/3342337914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0581a5070 con 0x7fc058101510 2024-06-10T00:28:20.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.872+0000 7fc0567fc700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc058101510 0x7fc058199ff0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0038b0 tx=0x7fc04c009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:20.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.872+0000 7fc05ca49700 1 -- 172.21.15.121:0/3342337914 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c027070 con 0x7fc058101510 2024-06-10T00:28:20.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.872+0000 7fc05ca49700 1 -- 172.21.15.121:0/3342337914 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc04c004050 con 0x7fc058101510 2024-06-10T00:28:20.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.872+0000 7fc05da4b700 1 -- 172.21.15.121:0/3342337914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0581a5240 con 0x7fc058101510 2024-06-10T00:28:20.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.872+0000 7fc05ca49700 1 -- 172.21.15.121:0/3342337914 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc04c022cd0 con 0x7fc058101510 2024-06-10T00:28:20.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.872+0000 7fc05da4b700 1 -- 172.21.15.121:0/3342337914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0581a56f0 con 0x7fc058101510 2024-06-10T00:28:20.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.873+0000 7fc05da4b700 1 -- 172.21.15.121:0/3342337914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc058069180 con 0x7fc058101510 2024-06-10T00:28:20.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.874+0000 7fc05ca49700 1 -- 172.21.15.121:0/3342337914 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fc04c02a070 con 0x7fc058101510 2024-06-10T00:28:20.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.875+0000 7fc05ca49700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc0400a2e20 0x7fc0400a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:20.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.875+0000 7fc056ffd700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc0400a2e20 0x7fc0400a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:20.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.875+0000 7fc05ca49700 1 -- 172.21.15.121:0/3342337914 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fc04c0d0aa0 con 0x7fc058101510 2024-06-10T00:28:20.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.875+0000 7fc056ffd700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc0400a2e20 0x7fc0400a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc048007b60 tx=0x7fc0480073d0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:20.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:20.879+0000 7fc05ca49700 1 -- 172.21.15.121:0/3342337914 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc04c013070 con 0x7fc058101510 2024-06-10T00:28:21.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.070+0000 7fc05da4b700 1 -- 172.21.15.121:0/3342337914 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc05810e980 con 0x7fc0400a2e20 2024-06-10T00:28:21.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.079+0000 7fc05ca49700 1 -- 172.21.15.121:0/3342337914 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339906 (secure 0 0 0) 0x7fc05810e980 con 0x7fc0400a2e20 2024-06-10T00:28:21.084 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.082+0000 7fc05da4b700 1 -- 172.21.15.121:0/3342337914 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc0400a2e20 msgr2=0x7fc0400a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.082+0000 7fc05da4b700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc0400a2e20 0x7fc0400a52e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc048007b60 tx=0x7fc0480073d0 comp rx=0 tx=0).stop 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.082+0000 7fc05da4b700 1 -- 172.21.15.121:0/3342337914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc058101510 msgr2=0x7fc058199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.082+0000 7fc05da4b700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc058101510 0x7fc058199ff0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0038b0 tx=0x7fc04c009f50 comp rx=0 tx=0).stop 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.082+0000 7fc05da4b700 1 -- 172.21.15.121:0/3342337914 shutdown_connections 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.082+0000 7fc05da4b700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc0400a2e20 0x7fc0400a52e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.082+0000 7fc05da4b700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc058106b90 0x7fc0581a4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.082+0000 7fc05da4b700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc058100b50 0x7fc058199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.082+0000 7fc05da4b700 1 --2- 172.21.15.121:0/3342337914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc058101510 0x7fc058199ff0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.082+0000 7fc05da4b700 1 -- 172.21.15.121:0/3342337914 >> 172.21.15.121:0/3342337914 conn(0x7fc058077a00 msgr2=0x7fc058103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.083+0000 7fc05da4b700 1 -- 172.21.15.121:0/3342337914 shutdown_connections 2024-06-10T00:28:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:21.083+0000 7fc05da4b700 1 -- 172.21.15.121:0/3342337914 wait complete. 2024-06-10T00:28:21.088 INFO:teuthology.orchestra.run.smithi121.stderr:dumped all 2024-06-10T00:28:21.548 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:21 smithi121 ceph-mon[32378]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-06-10T00:28:21.560 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:21 smithi135 ceph-mon[36424]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-06-10T00:28:21.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:21 smithi193 ceph-mon[36716]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-06-10T00:28:21.964 INFO:teuthology.orchestra.run.smithi121.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-06-10T00:28:19.477138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1387034,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1778,"num_read_kb":1318,"num_write":630,"num_write_kb":5302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":965,"ondisk_log_size":965,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367136,"kb_used_data":13772,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574688,"statfs":{"total":1151940427776,"available":1151564480512,"internally_reserved":0,"allocated":14102528,"data_stored":11130608,"data_compressed":633915,"data_compressed_allocated":860160,"data_compressed_original":5050368,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":196632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":345,"num_read_kb":277,"num_write":126,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002776"},"pg_stats":[{"pgid":"3.3f","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592226+0000","last_change":"2024-06-10T00:23:57.564073+0000","last_active":"2024-06-10T00:27:57.592226+0000","last_peered":"2024-06-10T00:27:57.592226+0000","last_clean":"2024-06-10T00:27:57.592226+0000","last_became_active":"2024-06-10T00:23:57.563618+0000","last_became_peered":"2024-06-10T00:23:57.563618+0000","last_unstale":"2024-06-10T00:27:57.592226+0000","last_undegraded":"2024-06-10T00:27:57.592226+0000","last_fullsized":"2024-06-10T00:27:57.592226+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:26:25.839941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"100'4","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591431+0000","last_change":"2024-06-10T00:23:57.560148+0000","last_active":"2024-06-10T00:27:57.591431+0000","last_peered":"2024-06-10T00:27:57.591431+0000","last_clean":"2024-06-10T00:27:57.591431+0000","last_became_active":"2024-06-10T00:23:57.559794+0000","last_became_peered":"2024-06-10T00:23:57.559794+0000","last_unstale":"2024-06-10T00:27:57.591431+0000","last_undegraded":"2024-06-10T00:27:57.591431+0000","last_fullsized":"2024-06-10T00:27:57.591431+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:59:50.624542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"100'7","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593168+0000","last_change":"2024-06-10T00:23:57.563292+0000","last_active":"2024-06-10T00:27:57.593168+0000","last_peered":"2024-06-10T00:27:57.593168+0000","last_clean":"2024-06-10T00:27:57.593168+0000","last_became_active":"2024-06-10T00:23:57.563007+0000","last_became_peered":"2024-06-10T00:23:57.563007+0000","last_unstale":"2024-06-10T00:27:57.593168+0000","last_undegraded":"2024-06-10T00:27:57.593168+0000","last_fullsized":"2024-06-10T00:27:57.593168+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:27:44.897770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545262+0000","last_change":"2024-06-10T00:23:57.558789+0000","last_active":"2024-06-10T00:27:57.545262+0000","last_peered":"2024-06-10T00:27:57.545262+0000","last_clean":"2024-06-10T00:27:57.545262+0000","last_became_active":"2024-06-10T00:23:57.557943+0000","last_became_peered":"2024-06-10T00:23:57.557943+0000","last_unstale":"2024-06-10T00:27:57.545262+0000","last_undegraded":"2024-06-10T00:27:57.545262+0000","last_fullsized":"2024-06-10T00:27:57.545262+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:10:40.081996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"100'4","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569868+0000","last_change":"2024-06-10T00:23:57.550806+0000","last_active":"2024-06-10T00:27:57.569868+0000","last_peered":"2024-06-10T00:27:57.569868+0000","last_clean":"2024-06-10T00:27:57.569868+0000","last_became_active":"2024-06-10T00:23:57.550453+0000","last_became_peered":"2024-06-10T00:23:57.550453+0000","last_unstale":"2024-06-10T00:27:57.569868+0000","last_undegraded":"2024-06-10T00:27:57.569868+0000","last_fullsized":"2024-06-10T00:27:57.569868+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:11:58.765099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458657+0000","last_change":"2024-06-10T00:23:57.567307+0000","last_active":"2024-06-10T00:27:57.458657+0000","last_peered":"2024-06-10T00:27:57.458657+0000","last_clean":"2024-06-10T00:27:57.458657+0000","last_became_active":"2024-06-10T00:23:57.566973+0000","last_became_peered":"2024-06-10T00:23:57.566973+0000","last_unstale":"2024-06-10T00:27:57.458657+0000","last_undegraded":"2024-06-10T00:27:57.458657+0000","last_fullsized":"2024-06-10T00:27:57.458657+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:27:25.135046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"100'5","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591279+0000","last_change":"2024-06-10T00:23:57.547998+0000","last_active":"2024-06-10T00:27:57.591279+0000","last_peered":"2024-06-10T00:27:57.591279+0000","last_clean":"2024-06-10T00:27:57.591279+0000","last_became_active":"2024-06-10T00:23:57.547593+0000","last_became_peered":"2024-06-10T00:23:57.547593+0000","last_unstale":"2024-06-10T00:27:57.591279+0000","last_undegraded":"2024-06-10T00:27:57.591279+0000","last_fullsized":"2024-06-10T00:27:57.591279+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:34:04.219399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570182+0000","last_change":"2024-06-10T00:23:57.560252+0000","last_active":"2024-06-10T00:27:57.570182+0000","last_peered":"2024-06-10T00:27:57.570182+0000","last_clean":"2024-06-10T00:27:57.570182+0000","last_became_active":"2024-06-10T00:23:57.559924+0000","last_became_peered":"2024-06-10T00:23:57.559924+0000","last_unstale":"2024-06-10T00:27:57.570182+0000","last_undegraded":"2024-06-10T00:27:57.570182+0000","last_fullsized":"2024-06-10T00:27:57.570182+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:01:43.684262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590876+0000","last_change":"2024-06-10T00:23:57.564031+0000","last_active":"2024-06-10T00:27:57.590876+0000","last_peered":"2024-06-10T00:27:57.590876+0000","last_clean":"2024-06-10T00:27:57.590876+0000","last_became_active":"2024-06-10T00:23:57.563642+0000","last_became_peered":"2024-06-10T00:23:57.563642+0000","last_unstale":"2024-06-10T00:27:57.590876+0000","last_undegraded":"2024-06-10T00:27:57.590876+0000","last_fullsized":"2024-06-10T00:27:57.590876+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:16:23.003152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"100'16","reported_seq":57,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459536+0000","last_change":"2024-06-10T00:23:57.566877+0000","last_active":"2024-06-10T00:27:57.459536+0000","last_peered":"2024-06-10T00:27:57.459536+0000","last_clean":"2024-06-10T00:27:57.459536+0000","last_became_active":"2024-06-10T00:23:57.566529+0000","last_became_peered":"2024-06-10T00:23:57.566529+0000","last_unstale":"2024-06-10T00:27:57.459536+0000","last_undegraded":"2024-06-10T00:27:57.459536+0000","last_fullsized":"2024-06-10T00:27:57.459536+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:30:43.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592151+0000","last_change":"2024-06-10T00:23:57.564979+0000","last_active":"2024-06-10T00:27:57.592151+0000","last_peered":"2024-06-10T00:27:57.592151+0000","last_clean":"2024-06-10T00:27:57.592151+0000","last_became_active":"2024-06-10T00:23:57.564578+0000","last_became_peered":"2024-06-10T00:23:57.564578+0000","last_unstale":"2024-06-10T00:27:57.592151+0000","last_undegraded":"2024-06-10T00:27:57.592151+0000","last_fullsized":"2024-06-10T00:27:57.592151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:40:53.414777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"100'7","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545061+0000","last_change":"2024-06-10T00:23:57.559126+0000","last_active":"2024-06-10T00:27:57.545061+0000","last_peered":"2024-06-10T00:27:57.545061+0000","last_clean":"2024-06-10T00:27:57.545061+0000","last_became_active":"2024-06-10T00:23:57.558401+0000","last_became_peered":"2024-06-10T00:23:57.558401+0000","last_unstale":"2024-06-10T00:27:57.545061+0000","last_undegraded":"2024-06-10T00:27:57.545061+0000","last_fullsized":"2024-06-10T00:27:57.545061+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:57.894137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542053+0000","last_change":"2024-06-10T00:23:57.559272+0000","last_active":"2024-06-10T00:27:57.542053+0000","last_peered":"2024-06-10T00:27:57.542053+0000","last_clean":"2024-06-10T00:27:57.542053+0000","last_became_active":"2024-06-10T00:23:57.558941+0000","last_became_peered":"2024-06-10T00:23:57.558941+0000","last_unstale":"2024-06-10T00:27:57.542053+0000","last_undegraded":"2024-06-10T00:27:57.542053+0000","last_fullsized":"2024-06-10T00:27:57.542053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:36:48.088347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591361+0000","last_change":"2024-06-10T00:23:57.555875+0000","last_active":"2024-06-10T00:27:57.591361+0000","last_peered":"2024-06-10T00:27:57.591361+0000","last_clean":"2024-06-10T00:27:57.591361+0000","last_became_active":"2024-06-10T00:23:57.555492+0000","last_became_peered":"2024-06-10T00:23:57.555492+0000","last_unstale":"2024-06-10T00:27:57.591361+0000","last_undegraded":"2024-06-10T00:27:57.591361+0000","last_fullsized":"2024-06-10T00:27:57.591361+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:49:10.827458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591360+0000","last_change":"2024-06-10T00:23:57.555316+0000","last_active":"2024-06-10T00:27:57.591360+0000","last_peered":"2024-06-10T00:27:57.591360+0000","last_clean":"2024-06-10T00:27:57.591360+0000","last_became_active":"2024-06-10T00:23:57.554952+0000","last_became_peered":"2024-06-10T00:23:57.554952+0000","last_unstale":"2024-06-10T00:27:57.591360+0000","last_undegraded":"2024-06-10T00:27:57.591360+0000","last_fullsized":"2024-06-10T00:27:57.591360+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:03:57.557330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591053+0000","last_change":"2024-06-10T00:23:57.562442+0000","last_active":"2024-06-10T00:27:57.591053+0000","last_peered":"2024-06-10T00:27:57.591053+0000","last_clean":"2024-06-10T00:27:57.591053+0000","last_became_active":"2024-06-10T00:23:57.561965+0000","last_became_peered":"2024-06-10T00:23:57.561965+0000","last_unstale":"2024-06-10T00:27:57.591053+0000","last_undegraded":"2024-06-10T00:27:57.591053+0000","last_fullsized":"2024-06-10T00:27:57.591053+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:40:16.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591981+0000","last_change":"2024-06-10T00:23:57.564422+0000","last_active":"2024-06-10T00:27:57.591981+0000","last_peered":"2024-06-10T00:27:57.591981+0000","last_clean":"2024-06-10T00:27:57.591981+0000","last_became_active":"2024-06-10T00:23:57.564069+0000","last_became_peered":"2024-06-10T00:23:57.564069+0000","last_unstale":"2024-06-10T00:27:57.591981+0000","last_undegraded":"2024-06-10T00:27:57.591981+0000","last_fullsized":"2024-06-10T00:27:57.591981+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:07.662465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458239+0000","last_change":"2024-06-10T00:23:57.561284+0000","last_active":"2024-06-10T00:27:57.458239+0000","last_peered":"2024-06-10T00:27:57.458239+0000","last_clean":"2024-06-10T00:27:57.458239+0000","last_became_active":"2024-06-10T00:23:57.560933+0000","last_became_peered":"2024-06-10T00:23:57.560933+0000","last_unstale":"2024-06-10T00:27:57.458239+0000","last_undegraded":"2024-06-10T00:27:57.458239+0000","last_fullsized":"2024-06-10T00:27:57.458239+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:58:48.511774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590777+0000","last_change":"2024-06-10T00:23:57.557752+0000","last_active":"2024-06-10T00:27:57.590777+0000","last_peered":"2024-06-10T00:27:57.590777+0000","last_clean":"2024-06-10T00:27:57.590777+0000","last_became_active":"2024-06-10T00:23:57.557419+0000","last_became_peered":"2024-06-10T00:23:57.557419+0000","last_unstale":"2024-06-10T00:27:57.590777+0000","last_undegraded":"2024-06-10T00:27:57.590777+0000","last_fullsized":"2024-06-10T00:27:57.590777+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:13:48.234181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591122+0000","last_change":"2024-06-10T00:24:09.177183+0000","last_active":"2024-06-10T00:27:57.591122+0000","last_peered":"2024-06-10T00:27:57.591122+0000","last_clean":"2024-06-10T00:27:57.591122+0000","last_became_active":"2024-06-10T00:24:09.176680+0000","last_became_peered":"2024-06-10T00:24:09.176680+0000","last_unstale":"2024-06-10T00:27:57.591122+0000","last_undegraded":"2024-06-10T00:27:57.591122+0000","last_fullsized":"2024-06-10T00:27:57.591122+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:17:06.458164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"98'9","reported_seq":59,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:03.469961+0000","last_change":"2024-06-10T00:23:54.426364+0000","last_active":"2024-06-10T00:28:03.469961+0000","last_peered":"2024-06-10T00:28:03.469961+0000","last_clean":"2024-06-10T00:28:03.469961+0000","last_became_active":"2024-06-10T00:23:54.426054+0000","last_became_peered":"2024-06-10T00:23:54.426054+0000","last_unstale":"2024-06-10T00:28:03.469961+0000","last_undegraded":"2024-06-10T00:28:03.469961+0000","last_fullsized":"2024-06-10T00:28:03.469961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:04:47.613402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570428+0000","last_change":"2024-06-10T00:23:57.557206+0000","last_active":"2024-06-10T00:27:57.570428+0000","last_peered":"2024-06-10T00:27:57.570428+0000","last_clean":"2024-06-10T00:27:57.570428+0000","last_became_active":"2024-06-10T00:23:57.556853+0000","last_became_peered":"2024-06-10T00:23:57.556853+0000","last_unstale":"2024-06-10T00:27:57.570428+0000","last_undegraded":"2024-06-10T00:27:57.570428+0000","last_fullsized":"2024-06-10T00:27:57.570428+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:33:09.686334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544782+0000","last_change":"2024-06-10T00:24:09.171047+0000","last_active":"2024-06-10T00:27:57.544782+0000","last_peered":"2024-06-10T00:27:57.544782+0000","last_clean":"2024-06-10T00:27:57.544782+0000","last_became_active":"2024-06-10T00:24:09.170501+0000","last_became_peered":"2024-06-10T00:24:09.170501+0000","last_unstale":"2024-06-10T00:27:57.544782+0000","last_undegraded":"2024-06-10T00:27:57.544782+0000","last_fullsized":"2024-06-10T00:27:57.544782+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:47:27.690439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"98'2","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571600+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:27:57.571600+0000","last_peered":"2024-06-10T00:27:57.571600+0000","last_clean":"2024-06-10T00:27:57.571600+0000","last_became_active":"2024-06-10T00:23:54.822447+0000","last_became_peered":"2024-06-10T00:23:54.822447+0000","last_unstale":"2024-06-10T00:27:57.571600+0000","last_undegraded":"2024-06-10T00:27:57.571600+0000","last_fullsized":"2024-06-10T00:27:57.571600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:45:14.956540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"100'6","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591883+0000","last_change":"2024-06-10T00:23:57.554958+0000","last_active":"2024-06-10T00:27:57.591883+0000","last_peered":"2024-06-10T00:27:57.591883+0000","last_clean":"2024-06-10T00:27:57.591883+0000","last_became_active":"2024-06-10T00:23:57.554477+0000","last_became_peered":"2024-06-10T00:23:57.554477+0000","last_unstale":"2024-06-10T00:27:57.591883+0000","last_undegraded":"2024-06-10T00:27:57.591883+0000","last_fullsized":"2024-06-10T00:27:57.591883+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:40:06.066967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544895+0000","last_change":"2024-06-10T00:24:09.169642+0000","last_active":"2024-06-10T00:27:57.544895+0000","last_peered":"2024-06-10T00:27:57.544895+0000","last_clean":"2024-06-10T00:27:57.544895+0000","last_became_active":"2024-06-10T00:24:09.168071+0000","last_became_peered":"2024-06-10T00:24:09.168071+0000","last_unstale":"2024-06-10T00:27:57.544895+0000","last_undegraded":"2024-06-10T00:27:57.544895+0000","last_fullsized":"2024-06-10T00:27:57.544895+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:44.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"98'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570379+0000","last_change":"2024-06-10T00:23:54.430543+0000","last_active":"2024-06-10T00:27:57.570379+0000","last_peered":"2024-06-10T00:27:57.570379+0000","last_clean":"2024-06-10T00:27:57.570379+0000","last_became_active":"2024-06-10T00:23:54.430138+0000","last_became_peered":"2024-06-10T00:23:54.430138+0000","last_unstale":"2024-06-10T00:27:57.570379+0000","last_undegraded":"2024-06-10T00:27:57.570379+0000","last_fullsized":"2024-06-10T00:27:57.570379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:39:53.212607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590796+0000","last_change":"2024-06-10T00:23:57.562251+0000","last_active":"2024-06-10T00:27:57.590796+0000","last_peered":"2024-06-10T00:27:57.590796+0000","last_clean":"2024-06-10T00:27:57.590796+0000","last_became_active":"2024-06-10T00:23:57.561903+0000","last_became_peered":"2024-06-10T00:23:57.561903+0000","last_unstale":"2024-06-10T00:27:57.590796+0000","last_undegraded":"2024-06-10T00:27:57.590796+0000","last_fullsized":"2024-06-10T00:27:57.590796+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:59.489475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544829+0000","last_change":"2024-06-10T00:24:09.165894+0000","last_active":"2024-06-10T00:27:57.544829+0000","last_peered":"2024-06-10T00:27:57.544829+0000","last_clean":"2024-06-10T00:27:57.544829+0000","last_became_active":"2024-06-10T00:24:09.165435+0000","last_became_peered":"2024-06-10T00:24:09.165435+0000","last_unstale":"2024-06-10T00:27:57.544829+0000","last_undegraded":"2024-06-10T00:27:57.544829+0000","last_fullsized":"2024-06-10T00:27:57.544829+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:13:36.058375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570296+0000","last_change":"2024-06-10T00:23:54.427868+0000","last_active":"2024-06-10T00:27:57.570296+0000","last_peered":"2024-06-10T00:27:57.570296+0000","last_clean":"2024-06-10T00:27:57.570296+0000","last_became_active":"2024-06-10T00:23:54.427366+0000","last_became_peered":"2024-06-10T00:23:54.427366+0000","last_unstale":"2024-06-10T00:27:57.570296+0000","last_undegraded":"2024-06-10T00:27:57.570296+0000","last_fullsized":"2024-06-10T00:27:57.570296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:18:15.499292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"100'15","reported_seq":62,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462390+0000","last_change":"2024-06-10T00:24:43.768556+0000","last_active":"2024-06-10T00:27:57.462390+0000","last_peered":"2024-06-10T00:27:57.462390+0000","last_clean":"2024-06-10T00:27:57.462390+0000","last_became_active":"2024-06-10T00:24:43.768090+0000","last_became_peered":"2024-06-10T00:24:43.768090+0000","last_unstale":"2024-06-10T00:27:57.462390+0000","last_undegraded":"2024-06-10T00:27:57.462390+0000","last_fullsized":"2024-06-10T00:27:57.462390+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:05:41.604184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591403+0000","last_change":"2024-06-10T00:24:09.170514+0000","last_active":"2024-06-10T00:27:57.591403+0000","last_peered":"2024-06-10T00:27:57.591403+0000","last_clean":"2024-06-10T00:27:57.591403+0000","last_became_active":"2024-06-10T00:24:09.168902+0000","last_became_peered":"2024-06-10T00:24:09.168902+0000","last_unstale":"2024-06-10T00:27:57.591403+0000","last_undegraded":"2024-06-10T00:27:57.591403+0000","last_fullsized":"2024-06-10T00:27:57.591403+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:33:10.534657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"101'9","reported_seq":255,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:16.580088+0000","last_change":"2024-06-10T00:23:54.426581+0000","last_active":"2024-06-10T00:28:16.580088+0000","last_peered":"2024-06-10T00:28:16.580088+0000","last_clean":"2024-06-10T00:28:16.580088+0000","last_became_active":"2024-06-10T00:23:54.426211+0000","last_became_peered":"2024-06-10T00:23:54.426211+0000","last_unstale":"2024-06-10T00:28:16.580088+0000","last_undegraded":"2024-06-10T00:28:16.580088+0000","last_fullsized":"2024-06-10T00:28:16.580088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:49:05.360742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1851,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":199,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569865+0000","last_change":"2024-06-10T00:23:57.552308+0000","last_active":"2024-06-10T00:27:57.569865+0000","last_peered":"2024-06-10T00:27:57.569865+0000","last_clean":"2024-06-10T00:27:57.569865+0000","last_became_active":"2024-06-10T00:23:57.551762+0000","last_became_peered":"2024-06-10T00:23:57.551762+0000","last_unstale":"2024-06-10T00:27:57.569865+0000","last_undegraded":"2024-06-10T00:27:57.569865+0000","last_fullsized":"2024-06-10T00:27:57.569865+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:22:28.476019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591613+0000","last_change":"2024-06-10T00:24:09.171456+0000","last_active":"2024-06-10T00:27:57.591613+0000","last_peered":"2024-06-10T00:27:57.591613+0000","last_clean":"2024-06-10T00:27:57.591613+0000","last_became_active":"2024-06-10T00:24:09.170969+0000","last_became_peered":"2024-06-10T00:24:09.170969+0000","last_unstale":"2024-06-10T00:27:57.591613+0000","last_undegraded":"2024-06-10T00:27:57.591613+0000","last_fullsized":"2024-06-10T00:27:57.591613+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:22:46.023897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458602+0000","last_change":"2024-06-10T00:23:54.428384+0000","last_active":"2024-06-10T00:27:57.458602+0000","last_peered":"2024-06-10T00:27:57.458602+0000","last_clean":"2024-06-10T00:27:57.458602+0000","last_became_active":"2024-06-10T00:23:54.427967+0000","last_became_peered":"2024-06-10T00:23:54.427967+0000","last_unstale":"2024-06-10T00:27:57.458602+0000","last_undegraded":"2024-06-10T00:27:57.458602+0000","last_fullsized":"2024-06-10T00:27:57.458602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:54:30.408684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"100'8","reported_seq":55,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461853+0000","last_change":"2024-06-10T00:24:43.262593+0000","last_active":"2024-06-10T00:27:57.461853+0000","last_peered":"2024-06-10T00:27:57.461853+0000","last_clean":"2024-06-10T00:27:57.461853+0000","last_became_active":"2024-06-10T00:24:43.262212+0000","last_became_peered":"2024-06-10T00:24:43.262212+0000","last_unstale":"2024-06-10T00:27:57.461853+0000","last_undegraded":"2024-06-10T00:27:57.461853+0000","last_fullsized":"2024-06-10T00:27:57.461853+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:40:24.442712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":37,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592034+0000","last_change":"2024-06-10T00:24:43.752805+0000","last_active":"2024-06-10T00:27:57.592034+0000","last_peered":"2024-06-10T00:27:57.592034+0000","last_clean":"2024-06-10T00:27:57.592034+0000","last_became_active":"2024-06-10T00:24:43.752251+0000","last_became_peered":"2024-06-10T00:24:43.752251+0000","last_unstale":"2024-06-10T00:27:57.592034+0000","last_undegraded":"2024-06-10T00:27:57.592034+0000","last_fullsized":"2024-06-10T00:27:57.592034+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:27:57.617443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542566+0000","last_change":"2024-06-10T00:23:54.421493+0000","last_active":"2024-06-10T00:27:57.542566+0000","last_peered":"2024-06-10T00:27:57.542566+0000","last_clean":"2024-06-10T00:27:57.542566+0000","last_became_active":"2024-06-10T00:23:54.421151+0000","last_became_peered":"2024-06-10T00:23:54.421151+0000","last_unstale":"2024-06-10T00:27:57.542566+0000","last_undegraded":"2024-06-10T00:27:57.542566+0000","last_fullsized":"2024-06-10T00:27:57.542566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:35:03.995244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570099+0000","last_change":"2024-06-10T00:23:57.554020+0000","last_active":"2024-06-10T00:27:57.570099+0000","last_peered":"2024-06-10T00:27:57.570099+0000","last_clean":"2024-06-10T00:27:57.570099+0000","last_became_active":"2024-06-10T00:23:57.553690+0000","last_became_peered":"2024-06-10T00:23:57.553690+0000","last_unstale":"2024-06-10T00:27:57.570099+0000","last_undegraded":"2024-06-10T00:27:57.570099+0000","last_fullsized":"2024-06-10T00:27:57.570099+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:24:48.097074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593638+0000","last_change":"2024-06-10T00:24:09.166426+0000","last_active":"2024-06-10T00:27:57.593638+0000","last_peered":"2024-06-10T00:27:57.593638+0000","last_clean":"2024-06-10T00:27:57.593638+0000","last_became_active":"2024-06-10T00:24:09.166047+0000","last_became_peered":"2024-06-10T00:24:09.166047+0000","last_unstale":"2024-06-10T00:27:57.593638+0000","last_undegraded":"2024-06-10T00:27:57.593638+0000","last_fullsized":"2024-06-10T00:27:57.593638+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:56:11.608061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"98'1","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461240+0000","last_change":"2024-06-10T00:23:54.426409+0000","last_active":"2024-06-10T00:27:57.461240+0000","last_peered":"2024-06-10T00:27:57.461240+0000","last_clean":"2024-06-10T00:27:57.461240+0000","last_became_active":"2024-06-10T00:23:54.425385+0000","last_became_peered":"2024-06-10T00:23:54.425385+0000","last_unstale":"2024-06-10T00:27:57.461240+0000","last_undegraded":"2024-06-10T00:27:57.461240+0000","last_fullsized":"2024-06-10T00:27:57.461240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:27:00.379189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457389+0000","last_change":"2024-06-10T00:23:57.560132+0000","last_active":"2024-06-10T00:27:57.457389+0000","last_peered":"2024-06-10T00:27:57.457389+0000","last_clean":"2024-06-10T00:27:57.457389+0000","last_became_active":"2024-06-10T00:23:57.559803+0000","last_became_peered":"2024-06-10T00:23:57.559803+0000","last_unstale":"2024-06-10T00:27:57.457389+0000","last_undegraded":"2024-06-10T00:27:57.457389+0000","last_fullsized":"2024-06-10T00:27:57.457389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:13:16.004233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571137+0000","last_change":"2024-06-10T00:24:09.170962+0000","last_active":"2024-06-10T00:27:57.571137+0000","last_peered":"2024-06-10T00:27:57.571137+0000","last_clean":"2024-06-10T00:27:57.571137+0000","last_became_active":"2024-06-10T00:24:09.170090+0000","last_became_peered":"2024-06-10T00:24:09.170090+0000","last_unstale":"2024-06-10T00:27:57.571137+0000","last_undegraded":"2024-06-10T00:27:57.571137+0000","last_fullsized":"2024-06-10T00:27:57.571137+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:07:33.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"100'5","reported_seq":245,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:16.579292+0000","last_change":"2024-06-10T00:23:54.421472+0000","last_active":"2024-06-10T00:28:16.579292+0000","last_peered":"2024-06-10T00:28:16.579292+0000","last_clean":"2024-06-10T00:28:16.579292+0000","last_became_active":"2024-06-10T00:23:54.421019+0000","last_became_peered":"2024-06-10T00:23:54.421019+0000","last_unstale":"2024-06-10T00:28:16.579292+0000","last_undegraded":"2024-06-10T00:28:16.579292+0000","last_fullsized":"2024-06-10T00:28:16.579292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:28:18.367953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":193,"num_read_kb":193,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590963+0000","last_change":"2024-06-10T00:23:57.563010+0000","last_active":"2024-06-10T00:27:57.590963+0000","last_peered":"2024-06-10T00:27:57.590963+0000","last_clean":"2024-06-10T00:27:57.590963+0000","last_became_active":"2024-06-10T00:23:57.561986+0000","last_became_peered":"2024-06-10T00:23:57.561986+0000","last_unstale":"2024-06-10T00:27:57.590963+0000","last_undegraded":"2024-06-10T00:27:57.590963+0000","last_fullsized":"2024-06-10T00:27:57.590963+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:31:17.308276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457415+0000","last_change":"2024-06-10T00:24:09.171746+0000","last_active":"2024-06-10T00:27:57.457415+0000","last_peered":"2024-06-10T00:27:57.457415+0000","last_clean":"2024-06-10T00:27:57.457415+0000","last_became_active":"2024-06-10T00:24:09.171030+0000","last_became_peered":"2024-06-10T00:24:09.171030+0000","last_unstale":"2024-06-10T00:27:57.457415+0000","last_undegraded":"2024-06-10T00:27:57.457415+0000","last_fullsized":"2024-06-10T00:27:57.457415+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:41.669471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"98'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591812+0000","last_change":"2024-06-10T00:23:54.428917+0000","last_active":"2024-06-10T00:27:57.591812+0000","last_peered":"2024-06-10T00:27:57.591812+0000","last_clean":"2024-06-10T00:27:57.591812+0000","last_became_active":"2024-06-10T00:23:54.428585+0000","last_became_peered":"2024-06-10T00:23:54.428585+0000","last_unstale":"2024-06-10T00:27:57.591812+0000","last_undegraded":"2024-06-10T00:27:57.591812+0000","last_fullsized":"2024-06-10T00:27:57.591812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:21:58.235848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591586+0000","last_change":"2024-06-10T00:23:57.551778+0000","last_active":"2024-06-10T00:27:57.591586+0000","last_peered":"2024-06-10T00:27:57.591586+0000","last_clean":"2024-06-10T00:27:57.591586+0000","last_became_active":"2024-06-10T00:23:57.551436+0000","last_became_peered":"2024-06-10T00:23:57.551436+0000","last_unstale":"2024-06-10T00:27:57.591586+0000","last_undegraded":"2024-06-10T00:27:57.591586+0000","last_fullsized":"2024-06-10T00:27:57.591586+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:03:30.059783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545101+0000","last_change":"2024-06-10T00:24:09.167547+0000","last_active":"2024-06-10T00:27:57.545101+0000","last_peered":"2024-06-10T00:27:57.545101+0000","last_clean":"2024-06-10T00:27:57.545101+0000","last_became_active":"2024-06-10T00:24:09.167132+0000","last_became_peered":"2024-06-10T00:24:09.167132+0000","last_unstale":"2024-06-10T00:27:57.545101+0000","last_undegraded":"2024-06-10T00:27:57.545101+0000","last_fullsized":"2024-06-10T00:27:57.545101+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:20:11.520330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461450+0000","last_change":"2024-06-10T00:23:54.823754+0000","last_active":"2024-06-10T00:27:57.461450+0000","last_peered":"2024-06-10T00:27:57.461450+0000","last_clean":"2024-06-10T00:27:57.461450+0000","last_became_active":"2024-06-10T00:23:54.823362+0000","last_became_peered":"2024-06-10T00:23:54.823362+0000","last_unstale":"2024-06-10T00:27:57.461450+0000","last_undegraded":"2024-06-10T00:27:57.461450+0000","last_fullsized":"2024-06-10T00:27:57.461450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:48:50.915439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590772+0000","last_change":"2024-06-10T00:23:57.563313+0000","last_active":"2024-06-10T00:27:57.590772+0000","last_peered":"2024-06-10T00:27:57.590772+0000","last_clean":"2024-06-10T00:27:57.590772+0000","last_became_active":"2024-06-10T00:23:57.562307+0000","last_became_peered":"2024-06-10T00:23:57.562307+0000","last_unstale":"2024-06-10T00:27:57.590772+0000","last_undegraded":"2024-06-10T00:27:57.590772+0000","last_fullsized":"2024-06-10T00:27:57.590772+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:01:23.815267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457507+0000","last_change":"2024-06-10T00:24:09.174660+0000","last_active":"2024-06-10T00:27:57.457507+0000","last_peered":"2024-06-10T00:27:57.457507+0000","last_clean":"2024-06-10T00:27:57.457507+0000","last_became_active":"2024-06-10T00:24:09.174283+0000","last_became_peered":"2024-06-10T00:24:09.174283+0000","last_unstale":"2024-06-10T00:27:57.457507+0000","last_undegraded":"2024-06-10T00:27:57.457507+0000","last_fullsized":"2024-06-10T00:27:57.457507+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:24:15.236146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"101'41","reported_seq":488,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:17.580668+0000","last_change":"2024-06-10T00:23:54.426621+0000","last_active":"2024-06-10T00:28:17.580668+0000","last_peered":"2024-06-10T00:28:17.580668+0000","last_clean":"2024-06-10T00:28:17.580668+0000","last_became_active":"2024-06-10T00:23:54.426131+0000","last_became_peered":"2024-06-10T00:23:54.426131+0000","last_unstale":"2024-06-10T00:28:17.580668+0000","last_undegraded":"2024-06-10T00:28:17.580668+0000","last_fullsized":"2024-06-10T00:28:17.580668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:56:14.819454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69365,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":400,"num_read_kb":269,"num_write":42,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570532+0000","last_change":"2024-06-10T00:23:57.552606+0000","last_active":"2024-06-10T00:27:57.570532+0000","last_peered":"2024-06-10T00:27:57.570532+0000","last_clean":"2024-06-10T00:27:57.570532+0000","last_became_active":"2024-06-10T00:23:57.552246+0000","last_became_peered":"2024-06-10T00:23:57.552246+0000","last_unstale":"2024-06-10T00:27:57.570532+0000","last_undegraded":"2024-06-10T00:27:57.570532+0000","last_fullsized":"2024-06-10T00:27:57.570532+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:19:21.508721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.462817+0000","last_change":"2024-06-10T00:24:09.174862+0000","last_active":"2024-06-10T00:27:57.462817+0000","last_peered":"2024-06-10T00:27:57.462817+0000","last_clean":"2024-06-10T00:27:57.462817+0000","last_became_active":"2024-06-10T00:24:09.173550+0000","last_became_peered":"2024-06-10T00:24:09.173550+0000","last_unstale":"2024-06-10T00:27:57.462817+0000","last_undegraded":"2024-06-10T00:27:57.462817+0000","last_fullsized":"2024-06-10T00:27:57.462817+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:49:25.549084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"98'6","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591325+0000","last_change":"2024-06-10T00:23:54.426074+0000","last_active":"2024-06-10T00:27:57.591325+0000","last_peered":"2024-06-10T00:27:57.591325+0000","last_clean":"2024-06-10T00:27:57.591325+0000","last_became_active":"2024-06-10T00:23:54.425707+0000","last_became_peered":"2024-06-10T00:23:54.425707+0000","last_unstale":"2024-06-10T00:27:57.591325+0000","last_undegraded":"2024-06-10T00:27:57.591325+0000","last_fullsized":"2024-06-10T00:27:57.591325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:11:10.563595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542128+0000","last_change":"2024-06-10T00:23:57.549784+0000","last_active":"2024-06-10T00:27:57.542128+0000","last_peered":"2024-06-10T00:27:57.542128+0000","last_clean":"2024-06-10T00:27:57.542128+0000","last_became_active":"2024-06-10T00:23:57.549457+0000","last_became_peered":"2024-06-10T00:23:57.549457+0000","last_unstale":"2024-06-10T00:27:57.542128+0000","last_undegraded":"2024-06-10T00:27:57.542128+0000","last_fullsized":"2024-06-10T00:27:57.542128+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:59:27.352927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592095+0000","last_change":"2024-06-10T00:24:09.171028+0000","last_active":"2024-06-10T00:27:57.592095+0000","last_peered":"2024-06-10T00:27:57.592095+0000","last_clean":"2024-06-10T00:27:57.592095+0000","last_became_active":"2024-06-10T00:24:09.170543+0000","last_became_peered":"2024-06-10T00:24:09.170543+0000","last_unstale":"2024-06-10T00:27:57.592095+0000","last_undegraded":"2024-06-10T00:27:57.592095+0000","last_fullsized":"2024-06-10T00:27:57.592095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:57:40.816585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"98'3","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461583+0000","last_change":"2024-06-10T00:23:54.423332+0000","last_active":"2024-06-10T00:27:57.461583+0000","last_peered":"2024-06-10T00:27:57.461583+0000","last_clean":"2024-06-10T00:27:57.461583+0000","last_became_active":"2024-06-10T00:23:54.422792+0000","last_became_peered":"2024-06-10T00:23:54.422792+0000","last_unstale":"2024-06-10T00:27:57.461583+0000","last_undegraded":"2024-06-10T00:27:57.461583+0000","last_fullsized":"2024-06-10T00:27:57.461583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:50:52.655641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"100'15","reported_seq":56,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591219+0000","last_change":"2024-06-10T00:23:57.551411+0000","last_active":"2024-06-10T00:27:57.591219+0000","last_peered":"2024-06-10T00:27:57.591219+0000","last_clean":"2024-06-10T00:27:57.591219+0000","last_became_active":"2024-06-10T00:23:57.551053+0000","last_became_peered":"2024-06-10T00:23:57.551053+0000","last_unstale":"2024-06-10T00:27:57.591219+0000","last_undegraded":"2024-06-10T00:27:57.591219+0000","last_fullsized":"2024-06-10T00:27:57.591219+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:16:18.082777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459282+0000","last_change":"2024-06-10T00:24:43.648767+0000","last_active":"2024-06-10T00:27:57.459282+0000","last_peered":"2024-06-10T00:27:57.459282+0000","last_clean":"2024-06-10T00:27:57.459282+0000","last_became_active":"2024-06-10T00:24:43.648267+0000","last_became_peered":"2024-06-10T00:24:43.648267+0000","last_unstale":"2024-06-10T00:27:57.459282+0000","last_undegraded":"2024-06-10T00:27:57.459282+0000","last_fullsized":"2024-06-10T00:27:57.459282+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:41:09.005202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"101'5","reported_seq":247,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:16.571822+0000","last_change":"2024-06-10T00:23:54.425963+0000","last_active":"2024-06-10T00:28:16.571822+0000","last_peered":"2024-06-10T00:28:16.571822+0000","last_clean":"2024-06-10T00:28:16.571822+0000","last_became_active":"2024-06-10T00:23:54.425581+0000","last_became_peered":"2024-06-10T00:23:54.425581+0000","last_unstale":"2024-06-10T00:28:16.571822+0000","last_undegraded":"2024-06-10T00:28:16.571822+0000","last_fullsized":"2024-06-10T00:28:16.571822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:50:54.760610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":197,"num_read_kb":4,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592340+0000","last_change":"2024-06-10T00:23:57.557464+0000","last_active":"2024-06-10T00:27:57.592340+0000","last_peered":"2024-06-10T00:27:57.592340+0000","last_clean":"2024-06-10T00:27:57.592340+0000","last_became_active":"2024-06-10T00:23:57.557130+0000","last_became_peered":"2024-06-10T00:23:57.557130+0000","last_unstale":"2024-06-10T00:27:57.592340+0000","last_undegraded":"2024-06-10T00:27:57.592340+0000","last_fullsized":"2024-06-10T00:27:57.592340+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:16:36.341210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":38,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543379+0000","last_change":"2024-06-10T00:24:43.753291+0000","last_active":"2024-06-10T00:27:57.543379+0000","last_peered":"2024-06-10T00:27:57.543379+0000","last_clean":"2024-06-10T00:27:57.543379+0000","last_became_active":"2024-06-10T00:24:43.752765+0000","last_became_peered":"2024-06-10T00:24:43.752765+0000","last_unstale":"2024-06-10T00:27:57.543379+0000","last_undegraded":"2024-06-10T00:27:57.543379+0000","last_fullsized":"2024-06-10T00:27:57.543379+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:55:06.349564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"98'15","reported_seq":63,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570040+0000","last_change":"2024-06-10T00:23:54.823489+0000","last_active":"2024-06-10T00:27:57.570040+0000","last_peered":"2024-06-10T00:27:57.570040+0000","last_clean":"2024-06-10T00:27:57.570040+0000","last_became_active":"2024-06-10T00:23:54.823073+0000","last_became_peered":"2024-06-10T00:23:54.823073+0000","last_unstale":"2024-06-10T00:27:57.570040+0000","last_undegraded":"2024-06-10T00:27:57.570040+0000","last_fullsized":"2024-06-10T00:27:57.570040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:07:52.563009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461389+0000","last_change":"2024-06-10T00:23:57.563388+0000","last_active":"2024-06-10T00:27:57.461389+0000","last_peered":"2024-06-10T00:27:57.461389+0000","last_clean":"2024-06-10T00:27:57.461389+0000","last_became_active":"2024-06-10T00:23:57.563004+0000","last_became_peered":"2024-06-10T00:23:57.563004+0000","last_unstale":"2024-06-10T00:27:57.461389+0000","last_undegraded":"2024-06-10T00:27:57.461389+0000","last_fullsized":"2024-06-10T00:27:57.461389+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:51:18.805662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461226+0000","last_change":"2024-06-10T00:24:09.168447+0000","last_active":"2024-06-10T00:27:57.461226+0000","last_peered":"2024-06-10T00:27:57.461226+0000","last_clean":"2024-06-10T00:27:57.461226+0000","last_became_active":"2024-06-10T00:24:09.167439+0000","last_became_peered":"2024-06-10T00:24:09.167439+0000","last_unstale":"2024-06-10T00:27:57.461226+0000","last_undegraded":"2024-06-10T00:27:57.461226+0000","last_fullsized":"2024-06-10T00:27:57.461226+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:50:52.319395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"101'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570752+0000","last_change":"2024-06-10T00:23:54.427666+0000","last_active":"2024-06-10T00:27:57.570752+0000","last_peered":"2024-06-10T00:27:57.570752+0000","last_clean":"2024-06-10T00:27:57.570752+0000","last_became_active":"2024-06-10T00:23:54.427285+0000","last_became_peered":"2024-06-10T00:23:54.427285+0000","last_unstale":"2024-06-10T00:27:57.570752+0000","last_undegraded":"2024-06-10T00:27:57.570752+0000","last_fullsized":"2024-06-10T00:27:57.570752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:28:20.196168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"100'5","reported_seq":45,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.558787+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.558470+0000","last_became_peered":"2024-06-10T00:23:57.558470+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:02:14.758802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570192+0000","last_change":"2024-06-10T00:24:09.264952+0000","last_active":"2024-06-10T00:27:57.570192+0000","last_peered":"2024-06-10T00:27:57.570192+0000","last_clean":"2024-06-10T00:27:57.570192+0000","last_became_active":"2024-06-10T00:24:09.264379+0000","last_became_peered":"2024-06-10T00:24:09.264379+0000","last_unstale":"2024-06-10T00:27:57.570192+0000","last_undegraded":"2024-06-10T00:27:57.570192+0000","last_fullsized":"2024-06-10T00:27:57.570192+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:25:40.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458166+0000","last_change":"2024-06-10T00:23:54.823298+0000","last_active":"2024-06-10T00:27:57.458166+0000","last_peered":"2024-06-10T00:27:57.458166+0000","last_clean":"2024-06-10T00:27:57.458166+0000","last_became_active":"2024-06-10T00:23:54.822917+0000","last_became_peered":"2024-06-10T00:23:54.822917+0000","last_unstale":"2024-06-10T00:27:57.458166+0000","last_undegraded":"2024-06-10T00:27:57.458166+0000","last_fullsized":"2024-06-10T00:27:57.458166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:07:52.500641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"100'4","reported_seq":46,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.459230+0000","last_change":"2024-06-10T00:23:57.559202+0000","last_active":"2024-06-10T00:27:57.459230+0000","last_peered":"2024-06-10T00:27:57.459230+0000","last_clean":"2024-06-10T00:27:57.459230+0000","last_became_active":"2024-06-10T00:23:57.558800+0000","last_became_peered":"2024-06-10T00:23:57.558800+0000","last_unstale":"2024-06-10T00:27:57.459230+0000","last_undegraded":"2024-06-10T00:27:57.459230+0000","last_fullsized":"2024-06-10T00:27:57.459230+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:49:51.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542486+0000","last_change":"2024-06-10T00:24:09.168603+0000","last_active":"2024-06-10T00:27:57.542486+0000","last_peered":"2024-06-10T00:27:57.542486+0000","last_clean":"2024-06-10T00:27:57.542486+0000","last_became_active":"2024-06-10T00:24:09.168233+0000","last_became_peered":"2024-06-10T00:24:09.168233+0000","last_unstale":"2024-06-10T00:27:57.542486+0000","last_undegraded":"2024-06-10T00:27:57.542486+0000","last_fullsized":"2024-06-10T00:27:57.542486+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:59.585023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"98'1","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592243+0000","last_change":"2024-06-10T00:23:54.425505+0000","last_active":"2024-06-10T00:27:57.592243+0000","last_peered":"2024-06-10T00:27:57.592243+0000","last_clean":"2024-06-10T00:27:57.592243+0000","last_became_active":"2024-06-10T00:23:54.425058+0000","last_became_peered":"2024-06-10T00:23:54.425058+0000","last_unstale":"2024-06-10T00:27:57.592243+0000","last_undegraded":"2024-06-10T00:27:57.592243+0000","last_fullsized":"2024-06-10T00:27:57.592243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:48:05.605947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"100'7","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458879+0000","last_change":"2024-06-10T00:23:57.566060+0000","last_active":"2024-06-10T00:27:57.458879+0000","last_peered":"2024-06-10T00:27:57.458879+0000","last_clean":"2024-06-10T00:27:57.458879+0000","last_became_active":"2024-06-10T00:23:57.565099+0000","last_became_peered":"2024-06-10T00:23:57.565099+0000","last_unstale":"2024-06-10T00:27:57.458879+0000","last_undegraded":"2024-06-10T00:27:57.458879+0000","last_fullsized":"2024-06-10T00:27:57.458879+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:24:54.204399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543284+0000","last_change":"2024-06-10T00:24:09.264741+0000","last_active":"2024-06-10T00:27:57.543284+0000","last_peered":"2024-06-10T00:27:57.543284+0000","last_clean":"2024-06-10T00:27:57.543284+0000","last_became_active":"2024-06-10T00:24:09.264267+0000","last_became_peered":"2024-06-10T00:24:09.264267+0000","last_unstale":"2024-06-10T00:27:57.543284+0000","last_undegraded":"2024-06-10T00:27:57.543284+0000","last_fullsized":"2024-06-10T00:27:57.543284+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:11:27.627391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457224+0000","last_change":"2024-06-10T00:25:44.346070+0000","last_active":"2024-06-10T00:27:57.457224+0000","last_peered":"2024-06-10T00:27:57.457224+0000","last_clean":"2024-06-10T00:27:57.457224+0000","last_became_active":"2024-06-10T00:25:44.345798+0000","last_became_peered":"2024-06-10T00:25:44.345798+0000","last_unstale":"2024-06-10T00:27:57.457224+0000","last_undegraded":"2024-06-10T00:27:57.457224+0000","last_fullsized":"2024-06-10T00:27:57.457224+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:04.506535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"101'5","reported_seq":246,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:16.571072+0000","last_change":"2024-06-10T00:23:54.822809+0000","last_active":"2024-06-10T00:28:16.571072+0000","last_peered":"2024-06-10T00:28:16.571072+0000","last_clean":"2024-06-10T00:28:16.571072+0000","last_became_active":"2024-06-10T00:23:54.822442+0000","last_became_peered":"2024-06-10T00:23:54.822442+0000","last_unstale":"2024-06-10T00:28:16.571072+0000","last_undegraded":"2024-06-10T00:28:16.571072+0000","last_fullsized":"2024-06-10T00:28:16.571072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:51:10.278257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":193,"num_read_kb":193,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"98'1","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591953+0000","last_change":"2024-06-10T00:24:09.168073+0000","last_active":"2024-06-10T00:27:57.591953+0000","last_peered":"2024-06-10T00:27:57.591953+0000","last_clean":"2024-06-10T00:27:57.591953+0000","last_became_active":"2024-06-10T00:24:09.167657+0000","last_became_peered":"2024-06-10T00:24:09.167657+0000","last_unstale":"2024-06-10T00:27:57.591953+0000","last_undegraded":"2024-06-10T00:27:57.591953+0000","last_fullsized":"2024-06-10T00:27:57.591953+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T05:20:23.481685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"100'8","reported_seq":22,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.457629+0000","last_change":"2024-06-10T00:25:44.346629+0000","last_active":"2024-06-10T00:27:57.457629+0000","last_peered":"2024-06-10T00:27:57.457629+0000","last_clean":"2024-06-10T00:27:57.457629+0000","last_became_active":"2024-06-10T00:25:44.346388+0000","last_became_peered":"2024-06-10T00:25:44.346388+0000","last_unstale":"2024-06-10T00:27:57.457629+0000","last_undegraded":"2024-06-10T00:27:57.457629+0000","last_fullsized":"2024-06-10T00:27:57.457629+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T12:03:19.228632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"98'3","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544980+0000","last_change":"2024-06-10T00:23:54.423507+0000","last_active":"2024-06-10T00:27:57.544980+0000","last_peered":"2024-06-10T00:27:57.544980+0000","last_clean":"2024-06-10T00:27:57.544980+0000","last_became_active":"2024-06-10T00:23:54.423092+0000","last_became_peered":"2024-06-10T00:23:54.423092+0000","last_unstale":"2024-06-10T00:27:57.544980+0000","last_undegraded":"2024-06-10T00:27:57.544980+0000","last_fullsized":"2024-06-10T00:27:57.544980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:06:15.924827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"101'148","reported_seq":424,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:59.577811+0000","last_change":"2024-06-10T00:22:28.025243+0000","last_active":"2024-06-10T00:27:59.577811+0000","last_peered":"2024-06-10T00:27:59.577811+0000","last_clean":"2024-06-10T00:27:59.577811+0000","last_became_active":"2024-06-10T00:22:28.024751+0000","last_became_peered":"2024-06-10T00:22:28.024751+0000","last_unstale":"2024-06-10T00:27:59.577811+0000","last_undegraded":"2024-06-10T00:27:59.577811+0000","last_fullsized":"2024-06-10T00:27:59.577811+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:19:05.039631+0000","last_clean_scrub_stamp":"2024-06-10T00:19:05.039631+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:02:36.557931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461952+0000","last_change":"2024-06-10T00:24:09.265084+0000","last_active":"2024-06-10T00:27:57.461952+0000","last_peered":"2024-06-10T00:27:57.461952+0000","last_clean":"2024-06-10T00:27:57.461952+0000","last_became_active":"2024-06-10T00:24:09.264425+0000","last_became_peered":"2024-06-10T00:24:09.264425+0000","last_unstale":"2024-06-10T00:27:57.461952+0000","last_undegraded":"2024-06-10T00:27:57.461952+0000","last_fullsized":"2024-06-10T00:27:57.461952+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:37:31.577688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.541983+0000","last_change":"2024-06-10T00:23:57.558540+0000","last_active":"2024-06-10T00:27:57.541983+0000","last_peered":"2024-06-10T00:27:57.541983+0000","last_clean":"2024-06-10T00:27:57.541983+0000","last_became_active":"2024-06-10T00:23:57.558197+0000","last_became_peered":"2024-06-10T00:23:57.558197+0000","last_unstale":"2024-06-10T00:27:57.541983+0000","last_undegraded":"2024-06-10T00:27:57.541983+0000","last_fullsized":"2024-06-10T00:27:57.541983+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:04:25.566135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592707+0000","last_change":"2024-06-10T00:23:54.428126+0000","last_active":"2024-06-10T00:27:57.592707+0000","last_peered":"2024-06-10T00:27:57.592707+0000","last_clean":"2024-06-10T00:27:57.592707+0000","last_became_active":"2024-06-10T00:23:54.427630+0000","last_became_peered":"2024-06-10T00:23:54.427630+0000","last_unstale":"2024-06-10T00:27:57.592707+0000","last_undegraded":"2024-06-10T00:27:57.592707+0000","last_fullsized":"2024-06-10T00:27:57.592707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:05:23.679704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591957+0000","last_change":"2024-06-10T00:24:09.169984+0000","last_active":"2024-06-10T00:27:57.591957+0000","last_peered":"2024-06-10T00:27:57.591957+0000","last_clean":"2024-06-10T00:27:57.591957+0000","last_became_active":"2024-06-10T00:24:09.169556+0000","last_became_peered":"2024-06-10T00:24:09.169556+0000","last_unstale":"2024-06-10T00:27:57.591957+0000","last_undegraded":"2024-06-10T00:27:57.591957+0000","last_fullsized":"2024-06-10T00:27:57.591957+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:54:13.575207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"98'1","reported_seq":20,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591972+0000","last_change":"2024-06-10T00:24:43.767901+0000","last_active":"2024-06-10T00:27:57.591972+0000","last_peered":"2024-06-10T00:27:57.591972+0000","last_clean":"2024-06-10T00:27:57.591972+0000","last_became_active":"2024-06-10T00:24:43.767562+0000","last_became_peered":"2024-06-10T00:24:43.767562+0000","last_unstale":"2024-06-10T00:27:57.591972+0000","last_undegraded":"2024-06-10T00:27:57.591972+0000","last_fullsized":"2024-06-10T00:27:57.591972+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:46:36.099750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"100'9","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543096+0000","last_change":"2024-06-10T00:23:57.554906+0000","last_active":"2024-06-10T00:27:57.543096+0000","last_peered":"2024-06-10T00:27:57.543096+0000","last_clean":"2024-06-10T00:27:57.543096+0000","last_became_active":"2024-06-10T00:23:57.554608+0000","last_became_peered":"2024-06-10T00:23:57.554608+0000","last_unstale":"2024-06-10T00:27:57.543096+0000","last_undegraded":"2024-06-10T00:27:57.543096+0000","last_fullsized":"2024-06-10T00:27:57.543096+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:18:24.690594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570670+0000","last_change":"2024-06-10T00:24:09.264676+0000","last_active":"2024-06-10T00:27:57.570670+0000","last_peered":"2024-06-10T00:27:57.570670+0000","last_clean":"2024-06-10T00:27:57.570670+0000","last_became_active":"2024-06-10T00:24:09.264259+0000","last_became_peered":"2024-06-10T00:24:09.264259+0000","last_unstale":"2024-06-10T00:27:57.570670+0000","last_undegraded":"2024-06-10T00:27:57.570670+0000","last_fullsized":"2024-06-10T00:27:57.570670+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:37:53.787039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"101'102","reported_seq":242,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:10.494814+0000","last_change":"2024-06-10T00:23:54.424201+0000","last_active":"2024-06-10T00:28:10.494814+0000","last_peered":"2024-06-10T00:28:10.494814+0000","last_clean":"2024-06-10T00:28:10.494814+0000","last_became_active":"2024-06-10T00:23:54.423853+0000","last_became_peered":"2024-06-10T00:23:54.423853+0000","last_unstale":"2024-06-10T00:28:10.494814+0000","last_undegraded":"2024-06-10T00:28:10.494814+0000","last_fullsized":"2024-06-10T00:28:10.494814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:33:00.884041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":312,"num_read_kb":277,"num_write":187,"num_write_kb":1287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571583+0000","last_change":"2024-06-10T00:23:57.554276+0000","last_active":"2024-06-10T00:27:57.571583+0000","last_peered":"2024-06-10T00:27:57.571583+0000","last_clean":"2024-06-10T00:27:57.571583+0000","last_became_active":"2024-06-10T00:23:57.553964+0000","last_became_peered":"2024-06-10T00:23:57.553964+0000","last_unstale":"2024-06-10T00:27:57.571583+0000","last_undegraded":"2024-06-10T00:27:57.571583+0000","last_fullsized":"2024-06-10T00:27:57.571583+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:39:51.974770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591148+0000","last_change":"2024-06-10T00:24:09.170249+0000","last_active":"2024-06-10T00:27:57.591148+0000","last_peered":"2024-06-10T00:27:57.591148+0000","last_clean":"2024-06-10T00:27:57.591148+0000","last_became_active":"2024-06-10T00:24:09.169830+0000","last_became_peered":"2024-06-10T00:24:09.169830+0000","last_unstale":"2024-06-10T00:27:57.591148+0000","last_undegraded":"2024-06-10T00:27:57.591148+0000","last_fullsized":"2024-06-10T00:27:57.591148+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:54:11.320609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"98'4","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545359+0000","last_change":"2024-06-10T00:23:54.473943+0000","last_active":"2024-06-10T00:27:57.545359+0000","last_peered":"2024-06-10T00:27:57.545359+0000","last_clean":"2024-06-10T00:27:57.545359+0000","last_became_active":"2024-06-10T00:23:54.473652+0000","last_became_peered":"2024-06-10T00:23:54.473652+0000","last_unstale":"2024-06-10T00:27:57.545359+0000","last_undegraded":"2024-06-10T00:27:57.545359+0000","last_fullsized":"2024-06-10T00:27:57.545359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:52:06.773684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"100'7","reported_seq":55,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.593765+0000","last_change":"2024-06-10T00:25:44.735050+0000","last_active":"2024-06-10T00:27:57.593765+0000","last_peered":"2024-06-10T00:27:57.593765+0000","last_clean":"2024-06-10T00:27:57.593765+0000","last_became_active":"2024-06-10T00:25:44.734756+0000","last_became_peered":"2024-06-10T00:25:44.734756+0000","last_unstale":"2024-06-10T00:27:57.593765+0000","last_undegraded":"2024-06-10T00:27:57.593765+0000","last_fullsized":"2024-06-10T00:27:57.593765+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:16:03.924238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545083+0000","last_change":"2024-06-10T00:24:43.768837+0000","last_active":"2024-06-10T00:27:57.545083+0000","last_peered":"2024-06-10T00:27:57.545083+0000","last_clean":"2024-06-10T00:27:57.545083+0000","last_became_active":"2024-06-10T00:24:43.768295+0000","last_became_peered":"2024-06-10T00:24:43.768295+0000","last_unstale":"2024-06-10T00:27:57.545083+0000","last_undegraded":"2024-06-10T00:27:57.545083+0000","last_fullsized":"2024-06-10T00:27:57.545083+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:23:59.600864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"98'2","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591761+0000","last_change":"2024-06-10T00:23:54.823448+0000","last_active":"2024-06-10T00:27:57.591761+0000","last_peered":"2024-06-10T00:27:57.591761+0000","last_clean":"2024-06-10T00:27:57.591761+0000","last_became_active":"2024-06-10T00:23:54.823107+0000","last_became_peered":"2024-06-10T00:23:54.823107+0000","last_unstale":"2024-06-10T00:27:57.591761+0000","last_undegraded":"2024-06-10T00:27:57.591761+0000","last_fullsized":"2024-06-10T00:27:57.591761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:53:50.966048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"100'2","reported_seq":44,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.569941+0000","last_change":"2024-06-10T00:23:57.556733+0000","last_active":"2024-06-10T00:27:57.569941+0000","last_peered":"2024-06-10T00:27:57.569941+0000","last_clean":"2024-06-10T00:27:57.569941+0000","last_became_active":"2024-06-10T00:23:57.556386+0000","last_became_peered":"2024-06-10T00:23:57.556386+0000","last_unstale":"2024-06-10T00:27:57.569941+0000","last_undegraded":"2024-06-10T00:27:57.569941+0000","last_fullsized":"2024-06-10T00:27:57.569941+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:39:00.572311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.458015+0000","last_change":"2024-06-10T00:24:09.174600+0000","last_active":"2024-06-10T00:27:57.458015+0000","last_peered":"2024-06-10T00:27:57.458015+0000","last_clean":"2024-06-10T00:27:57.458015+0000","last_became_active":"2024-06-10T00:24:09.174236+0000","last_became_peered":"2024-06-10T00:24:09.174236+0000","last_unstale":"2024-06-10T00:27:57.458015+0000","last_undegraded":"2024-06-10T00:27:57.458015+0000","last_fullsized":"2024-06-10T00:27:57.458015+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:58:18.447768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571238+0000","last_change":"2024-06-10T00:23:54.425111+0000","last_active":"2024-06-10T00:27:57.571238+0000","last_peered":"2024-06-10T00:27:57.571238+0000","last_clean":"2024-06-10T00:27:57.571238+0000","last_became_active":"2024-06-10T00:23:54.424724+0000","last_became_peered":"2024-06-10T00:23:54.424724+0000","last_unstale":"2024-06-10T00:27:57.571238+0000","last_undegraded":"2024-06-10T00:27:57.571238+0000","last_fullsized":"2024-06-10T00:27:57.571238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:54.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"100'9","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591671+0000","last_change":"2024-06-10T00:23:57.556355+0000","last_active":"2024-06-10T00:27:57.591671+0000","last_peered":"2024-06-10T00:27:57.591671+0000","last_clean":"2024-06-10T00:27:57.591671+0000","last_became_active":"2024-06-10T00:23:57.555479+0000","last_became_peered":"2024-06-10T00:23:57.555479+0000","last_unstale":"2024-06-10T00:27:57.591671+0000","last_undegraded":"2024-06-10T00:27:57.591671+0000","last_fullsized":"2024-06-10T00:27:57.591671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T11:41:08.817381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"98'1","reported_seq":30,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.544968+0000","last_change":"2024-06-10T00:24:09.169589+0000","last_active":"2024-06-10T00:27:57.544968+0000","last_peered":"2024-06-10T00:27:57.544968+0000","last_clean":"2024-06-10T00:27:57.544968+0000","last_became_active":"2024-06-10T00:24:09.168436+0000","last_became_peered":"2024-06-10T00:24:09.168436+0000","last_unstale":"2024-06-10T00:27:57.544968+0000","last_undegraded":"2024-06-10T00:27:57.544968+0000","last_fullsized":"2024-06-10T00:27:57.544968+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:17:32.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"98'1","reported_seq":19,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591500+0000","last_change":"2024-06-10T00:24:43.751399+0000","last_active":"2024-06-10T00:27:57.591500+0000","last_peered":"2024-06-10T00:27:57.591500+0000","last_clean":"2024-06-10T00:27:57.591500+0000","last_became_active":"2024-06-10T00:24:43.750987+0000","last_became_peered":"2024-06-10T00:24:43.750987+0000","last_unstale":"2024-06-10T00:27:57.591500+0000","last_undegraded":"2024-06-10T00:27:57.591500+0000","last_fullsized":"2024-06-10T00:27:57.591500+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:14:47.902017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570338+0000","last_change":"2024-06-10T00:23:57.560213+0000","last_active":"2024-06-10T00:27:57.570338+0000","last_peered":"2024-06-10T00:27:57.570338+0000","last_clean":"2024-06-10T00:27:57.570338+0000","last_became_active":"2024-06-10T00:23:57.559925+0000","last_became_peered":"2024-06-10T00:23:57.559925+0000","last_unstale":"2024-06-10T00:27:57.570338+0000","last_undegraded":"2024-06-10T00:27:57.570338+0000","last_fullsized":"2024-06-10T00:27:57.570338+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:00:37.403087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.545095+0000","last_change":"2024-06-10T00:24:09.264758+0000","last_active":"2024-06-10T00:27:57.545095+0000","last_peered":"2024-06-10T00:27:57.545095+0000","last_clean":"2024-06-10T00:27:57.545095+0000","last_became_active":"2024-06-10T00:24:09.263928+0000","last_became_peered":"2024-06-10T00:24:09.263928+0000","last_unstale":"2024-06-10T00:27:57.545095+0000","last_undegraded":"2024-06-10T00:27:57.545095+0000","last_fullsized":"2024-06-10T00:27:57.545095+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:23:41.083091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"98'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570103+0000","last_change":"2024-06-10T00:23:54.428553+0000","last_active":"2024-06-10T00:27:57.570103+0000","last_peered":"2024-06-10T00:27:57.570103+0000","last_clean":"2024-06-10T00:27:57.570103+0000","last_became_active":"2024-06-10T00:23:54.428108+0000","last_became_peered":"2024-06-10T00:23:54.428108+0000","last_unstale":"2024-06-10T00:27:57.570103+0000","last_undegraded":"2024-06-10T00:27:57.570103+0000","last_fullsized":"2024-06-10T00:27:57.570103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T09:57:26.207915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"100'5","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591181+0000","last_change":"2024-06-10T00:23:57.560686+0000","last_active":"2024-06-10T00:27:57.591181+0000","last_peered":"2024-06-10T00:27:57.591181+0000","last_clean":"2024-06-10T00:27:57.591181+0000","last_became_active":"2024-06-10T00:23:57.560326+0000","last_became_peered":"2024-06-10T00:23:57.560326+0000","last_unstale":"2024-06-10T00:27:57.591181+0000","last_undegraded":"2024-06-10T00:27:57.591181+0000","last_fullsized":"2024-06-10T00:27:57.591181+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:08:39.506567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591281+0000","last_change":"2024-06-10T00:24:09.166947+0000","last_active":"2024-06-10T00:27:57.591281+0000","last_peered":"2024-06-10T00:27:57.591281+0000","last_clean":"2024-06-10T00:27:57.591281+0000","last_became_active":"2024-06-10T00:24:09.166503+0000","last_became_peered":"2024-06-10T00:24:09.166503+0000","last_unstale":"2024-06-10T00:27:57.591281+0000","last_undegraded":"2024-06-10T00:27:57.591281+0000","last_fullsized":"2024-06-10T00:27:57.591281+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:16:45.094642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"98'3","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591369+0000","last_change":"2024-06-10T00:23:54.823522+0000","last_active":"2024-06-10T00:27:57.591369+0000","last_peered":"2024-06-10T00:27:57.591369+0000","last_clean":"2024-06-10T00:27:57.591369+0000","last_became_active":"2024-06-10T00:23:54.823142+0000","last_became_peered":"2024-06-10T00:23:54.823142+0000","last_unstale":"2024-06-10T00:27:57.591369+0000","last_undegraded":"2024-06-10T00:27:57.591369+0000","last_fullsized":"2024-06-10T00:27:57.591369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:13:03.836850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"100'7","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592288+0000","last_change":"2024-06-10T00:23:57.563710+0000","last_active":"2024-06-10T00:27:57.592288+0000","last_peered":"2024-06-10T00:27:57.592288+0000","last_clean":"2024-06-10T00:27:57.592288+0000","last_became_active":"2024-06-10T00:23:57.563331+0000","last_became_peered":"2024-06-10T00:23:57.563331+0000","last_unstale":"2024-06-10T00:27:57.592288+0000","last_undegraded":"2024-06-10T00:27:57.592288+0000","last_fullsized":"2024-06-10T00:27:57.592288+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:08:48.061885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592820+0000","last_change":"2024-06-10T00:24:09.171203+0000","last_active":"2024-06-10T00:27:57.592820+0000","last_peered":"2024-06-10T00:27:57.592820+0000","last_clean":"2024-06-10T00:27:57.592820+0000","last_became_active":"2024-06-10T00:24:09.170824+0000","last_became_peered":"2024-06-10T00:24:09.170824+0000","last_unstale":"2024-06-10T00:27:57.592820+0000","last_undegraded":"2024-06-10T00:27:57.592820+0000","last_fullsized":"2024-06-10T00:27:57.592820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:47:28.045602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"101'65","reported_seq":143,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:28:10.142035+0000","last_change":"2024-06-10T00:23:54.421307+0000","last_active":"2024-06-10T00:28:10.142035+0000","last_peered":"2024-06-10T00:28:10.142035+0000","last_clean":"2024-06-10T00:28:10.142035+0000","last_became_active":"2024-06-10T00:23:54.420928+0000","last_became_peered":"2024-06-10T00:23:54.420928+0000","last_unstale":"2024-06-10T00:28:10.142035+0000","last_undegraded":"2024-06-10T00:28:10.142035+0000","last_fullsized":"2024-06-10T00:28:10.142035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:59:41.433605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328302,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":33,"num_write":76,"num_write_kb":941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591623+0000","last_change":"2024-06-10T00:23:57.556494+0000","last_active":"2024-06-10T00:27:57.591623+0000","last_peered":"2024-06-10T00:27:57.591623+0000","last_clean":"2024-06-10T00:27:57.591623+0000","last_became_active":"2024-06-10T00:23:57.556088+0000","last_became_peered":"2024-06-10T00:23:57.556088+0000","last_unstale":"2024-06-10T00:27:57.591623+0000","last_undegraded":"2024-06-10T00:27:57.591623+0000","last_fullsized":"2024-06-10T00:27:57.591623+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:06:23.873281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.461068+0000","last_change":"2024-06-10T00:24:09.166734+0000","last_active":"2024-06-10T00:27:57.461068+0000","last_peered":"2024-06-10T00:27:57.461068+0000","last_clean":"2024-06-10T00:27:57.461068+0000","last_became_active":"2024-06-10T00:24:09.166270+0000","last_became_peered":"2024-06-10T00:24:09.166270+0000","last_unstale":"2024-06-10T00:27:57.461068+0000","last_undegraded":"2024-06-10T00:27:57.461068+0000","last_fullsized":"2024-06-10T00:27:57.461068+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:24:08.149424+0000","last_clean_scrub_stamp":"2024-06-10T00:24:08.149424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T06:25:38.039633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'2","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543256+0000","last_change":"2024-06-10T00:23:54.426226+0000","last_active":"2024-06-10T00:27:57.543256+0000","last_peered":"2024-06-10T00:27:57.543256+0000","last_clean":"2024-06-10T00:27:57.543256+0000","last_became_active":"2024-06-10T00:23:54.425879+0000","last_became_peered":"2024-06-10T00:23:54.425879+0000","last_unstale":"2024-06-10T00:27:57.543256+0000","last_undegraded":"2024-06-10T00:27:57.543256+0000","last_fullsized":"2024-06-10T00:27:57.543256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:53.407411+0000","last_clean_scrub_stamp":"2024-06-10T00:23:53.407411+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T07:15:32.936556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592043+0000","last_change":"2024-06-10T00:23:57.553814+0000","last_active":"2024-06-10T00:27:57.592043+0000","last_peered":"2024-06-10T00:27:57.592043+0000","last_clean":"2024-06-10T00:27:57.592043+0000","last_became_active":"2024-06-10T00:23:57.553478+0000","last_became_peered":"2024-06-10T00:23:57.553478+0000","last_unstale":"2024-06-10T00:27:57.592043+0000","last_undegraded":"2024-06-10T00:27:57.592043+0000","last_fullsized":"2024-06-10T00:27:57.592043+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:54:39.264526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"100'9","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591455+0000","last_change":"2024-06-10T00:23:57.556676+0000","last_active":"2024-06-10T00:27:57.591455+0000","last_peered":"2024-06-10T00:27:57.591455+0000","last_clean":"2024-06-10T00:27:57.591455+0000","last_became_active":"2024-06-10T00:23:57.555946+0000","last_became_peered":"2024-06-10T00:23:57.555946+0000","last_unstale":"2024-06-10T00:27:57.591455+0000","last_undegraded":"2024-06-10T00:27:57.591455+0000","last_fullsized":"2024-06-10T00:27:57.591455+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:10:10.868799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"100'12","reported_seq":54,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590806+0000","last_change":"2024-06-10T00:23:57.565423+0000","last_active":"2024-06-10T00:27:57.590806+0000","last_peered":"2024-06-10T00:27:57.590806+0000","last_clean":"2024-06-10T00:27:57.590806+0000","last_became_active":"2024-06-10T00:23:57.564830+0000","last_became_peered":"2024-06-10T00:23:57.564830+0000","last_unstale":"2024-06-10T00:27:57.590806+0000","last_undegraded":"2024-06-10T00:27:57.590806+0000","last_fullsized":"2024-06-10T00:27:57.590806+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:11:06.843500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"100'8","reported_seq":50,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591073+0000","last_change":"2024-06-10T00:23:57.558971+0000","last_active":"2024-06-10T00:27:57.591073+0000","last_peered":"2024-06-10T00:27:57.591073+0000","last_clean":"2024-06-10T00:27:57.591073+0000","last_became_active":"2024-06-10T00:23:57.558620+0000","last_became_peered":"2024-06-10T00:23:57.558620+0000","last_unstale":"2024-06-10T00:27:57.591073+0000","last_undegraded":"2024-06-10T00:27:57.591073+0000","last_fullsized":"2024-06-10T00:27:57.591073+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T04:45:43.308008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"100'7","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591540+0000","last_change":"2024-06-10T00:23:57.552341+0000","last_active":"2024-06-10T00:27:57.591540+0000","last_peered":"2024-06-10T00:27:57.591540+0000","last_clean":"2024-06-10T00:27:57.591540+0000","last_became_active":"2024-06-10T00:23:57.551464+0000","last_became_peered":"2024-06-10T00:23:57.551464+0000","last_unstale":"2024-06-10T00:27:57.591540+0000","last_undegraded":"2024-06-10T00:27:57.591540+0000","last_fullsized":"2024-06-10T00:27:57.591540+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:44:41.063516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"100'4","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.543258+0000","last_change":"2024-06-10T00:24:43.768961+0000","last_active":"2024-06-10T00:27:57.543258+0000","last_peered":"2024-06-10T00:27:57.543258+0000","last_clean":"2024-06-10T00:27:57.543258+0000","last_became_active":"2024-06-10T00:24:43.768542+0000","last_became_peered":"2024-06-10T00:24:43.768542+0000","last_unstale":"2024-06-10T00:27:57.543258+0000","last_undegraded":"2024-06-10T00:27:57.543258+0000","last_fullsized":"2024-06-10T00:27:57.543258+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:23:58.412054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"100'6","reported_seq":47,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591674+0000","last_change":"2024-06-10T00:23:57.554924+0000","last_active":"2024-06-10T00:27:57.591674+0000","last_peered":"2024-06-10T00:27:57.591674+0000","last_clean":"2024-06-10T00:27:57.591674+0000","last_became_active":"2024-06-10T00:23:57.554515+0000","last_became_peered":"2024-06-10T00:23:57.554515+0000","last_unstale":"2024-06-10T00:27:57.591674+0000","last_undegraded":"2024-06-10T00:27:57.591674+0000","last_fullsized":"2024-06-10T00:27:57.591674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T00:58:15.624775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"100'15","reported_seq":57,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591537+0000","last_change":"2024-06-10T00:23:57.561940+0000","last_active":"2024-06-10T00:27:57.591537+0000","last_peered":"2024-06-10T00:27:57.591537+0000","last_clean":"2024-06-10T00:27:57.591537+0000","last_became_active":"2024-06-10T00:23:57.561512+0000","last_became_peered":"2024-06-10T00:23:57.561512+0000","last_unstale":"2024-06-10T00:27:57.591537+0000","last_undegraded":"2024-06-10T00:27:57.591537+0000","last_fullsized":"2024-06-10T00:27:57.591537+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T03:57:57.147311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"100'2","reported_seq":44,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.570391+0000","last_change":"2024-06-10T00:23:57.557229+0000","last_active":"2024-06-10T00:27:57.570391+0000","last_peered":"2024-06-10T00:27:57.570391+0000","last_clean":"2024-06-10T00:27:57.570391+0000","last_became_active":"2024-06-10T00:23:57.556905+0000","last_became_peered":"2024-06-10T00:23:57.556905+0000","last_unstale":"2024-06-10T00:27:57.570391+0000","last_undegraded":"2024-06-10T00:27:57.570391+0000","last_fullsized":"2024-06-10T00:27:57.570391+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T01:30:30.513517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"100'6","reported_seq":24,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.592589+0000","last_change":"2024-06-10T00:24:43.768509+0000","last_active":"2024-06-10T00:27:57.592589+0000","last_peered":"2024-06-10T00:27:57.592589+0000","last_clean":"2024-06-10T00:27:57.592589+0000","last_became_active":"2024-06-10T00:24:43.768038+0000","last_became_peered":"2024-06-10T00:24:43.768038+0000","last_unstale":"2024-06-10T00:27:57.592589+0000","last_undegraded":"2024-06-10T00:27:57.592589+0000","last_fullsized":"2024-06-10T00:27:57.592589+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:38.344375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,3],"acting":[7,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.29","version":"100'8","reported_seq":49,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.571244+0000","last_change":"2024-06-10T00:23:57.557830+0000","last_active":"2024-06-10T00:27:57.571244+0000","last_peered":"2024-06-10T00:27:57.571244+0000","last_clean":"2024-06-10T00:27:57.571244+0000","last_became_active":"2024-06-10T00:23:57.556986+0000","last_became_peered":"2024-06-10T00:23:57.556986+0000","last_unstale":"2024-06-10T00:27:57.571244+0000","last_undegraded":"2024-06-10T00:27:57.571244+0000","last_fullsized":"2024-06-10T00:27:57.571244+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:12:53.522004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"100'10","reported_seq":51,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.590760+0000","last_change":"2024-06-10T00:23:57.565602+0000","last_active":"2024-06-10T00:27:57.590760+0000","last_peered":"2024-06-10T00:27:57.590760+0000","last_clean":"2024-06-10T00:27:57.590760+0000","last_became_active":"2024-06-10T00:23:57.565264+0000","last_became_peered":"2024-06-10T00:23:57.565264+0000","last_unstale":"2024-06-10T00:27:57.590760+0000","last_undegraded":"2024-06-10T00:27:57.590760+0000","last_fullsized":"2024-06-10T00:27:57.590760+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T10:09:12.773087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"100'11","reported_seq":52,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.542211+0000","last_change":"2024-06-10T00:23:57.548516+0000","last_active":"2024-06-10T00:27:57.542211+0000","last_peered":"2024-06-10T00:27:57.542211+0000","last_clean":"2024-06-10T00:27:57.542211+0000","last_became_active":"2024-06-10T00:23:57.548202+0000","last_became_peered":"2024-06-10T00:23:57.548202+0000","last_unstale":"2024-06-10T00:27:57.542211+0000","last_undegraded":"2024-06-10T00:27:57.542211+0000","last_fullsized":"2024-06-10T00:27:57.542211+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T02:15:13.901478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"100'8","reported_seq":48,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-06-10T00:27:57.591566+0000","last_change":"2024-06-10T00:23:57.565900+0000","last_active":"2024-06-10T00:27:57.591566+0000","last_peered":"2024-06-10T00:27:57.591566+0000","last_clean":"2024-06-10T00:27:57.591566+0000","last_became_active":"2024-06-10T00:23:57.565573+0000","last_became_peered":"2024-06-10T00:23:57.565573+0000","last_unstale":"2024-06-10T00:27:57.591566+0000","last_undegraded":"2024-06-10T00:27:57.591566+0000","last_fullsized":"2024-06-10T00:27:57.591566+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-10T00:23:56.533288+0000","last_clean_scrub_stamp":"2024-06-10T00:23:56.533288+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-11T08:46:08.255905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":237,"num_read_kb":330,"num_write":201,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107100,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":796432,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1541,"num_read_kb":988,"num_write":427,"num_write_kb":2362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":2389296,"data_compressed":103467,"data_compressed_allocated":135168,"data_compressed_original":2101248,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351044,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1118208,"data_stored":582820,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612681,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712688,"statfs":{"total":95995035648,"available":95961792512,"internally_reserved":0,"allocated":1310720,"data_stored":1568443,"data_compressed":86181,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776207,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1159168,"data_stored":980346,"data_compressed":54566,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037843,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28140,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717012,"statfs":{"total":95995035648,"available":95966220288,"internally_reserved":0,"allocated":1150976,"data_stored":976148,"data_compressed":54566,"data_compressed_allocated":73728,"data_compressed_original":491520,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299480,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":1180481,"data_compressed":70979,"data_compressed_allocated":94208,"data_compressed_original":688128,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863463005,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1110016,"data_stored":578210,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1183744,"data_stored":647485,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1134592,"data_stored":579598,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36520,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708632,"statfs":{"total":95995035648,"available":95957639168,"internally_reserved":0,"allocated":1208320,"data_stored":650064,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1191936,"data_stored":1299852,"data_compressed":69768,"data_compressed_allocated":94208,"data_compressed_original":798720,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888167,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28116,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717036,"statfs":{"total":95995035648,"available":95966244864,"internally_reserved":0,"allocated":1122304,"data_stored":583640,"data_compressed":35279,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724642,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32388,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712764,"statfs":{"total":95995035648,"available":95961870336,"internally_reserved":0,"allocated":1232896,"data_stored":1503521,"data_compressed":86181,"data_compressed_allocated":114688,"data_compressed_original":995328,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35700,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":721732,"data_compressed":34489,"data_compressed_allocated":45056,"data_compressed_original":700416,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":330937,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":393932,"data_compressed":19287,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19287,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":399955,"data_compressed":15202,"data_compressed_allocated":20480,"data_compressed_original":307200,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-10T00:28:21.969 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:28:22.215 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:22.580 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:22 smithi135 ceph-mon[36424]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:22.580 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:22 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:28:22.611 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:22 smithi121 ceph-mon[32378]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:22.611 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:22 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:28:22.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:22 smithi193 ceph-mon[36716]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-10T00:28:22.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:22 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:28:23.562 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:23 smithi121 ceph-mon[32378]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-06-10T00:28:23.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:23 smithi193 ceph-mon[36716]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-06-10T00:28:23.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:23 smithi135 ceph-mon[36424]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 76 KiB/s wr, 36 op/s 2024-06-10T00:28:23.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.964+0000 7fbded5f6700 1 -- 172.21.15.121:0/1456317738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde81015f0 msgr2=0x7fbde8107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:23.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.964+0000 7fbded5f6700 1 --2- 172.21.15.121:0/1456317738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde81015f0 0x7fbde8107150 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbddc0099c0 tx=0x7fbddc009cd0 comp rx=0 tx=0).stop 2024-06-10T00:28:23.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.964+0000 7fbded5f6700 1 -- 172.21.15.121:0/1456317738 shutdown_connections 2024-06-10T00:28:23.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.964+0000 7fbded5f6700 1 --2- 172.21.15.121:0/1456317738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde8107880 0x7fbde810f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:23.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.964+0000 7fbded5f6700 1 --2- 172.21.15.121:0/1456317738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde8100c40 0x7fbde8101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:23.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.964+0000 7fbded5f6700 1 --2- 172.21.15.121:0/1456317738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde81015f0 0x7fbde8107150 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:23.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.964+0000 7fbded5f6700 1 -- 172.21.15.121:0/1456317738 >> 172.21.15.121:0/1456317738 conn(0x7fbde8077990 msgr2=0x7fbde80fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:23.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.964+0000 7fbded5f6700 1 -- 172.21.15.121:0/1456317738 shutdown_connections 2024-06-10T00:28:23.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.964+0000 7fbded5f6700 1 -- 172.21.15.121:0/1456317738 wait complete. 2024-06-10T00:28:23.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.964+0000 7fbded5f6700 1 Processor -- start 2024-06-10T00:28:23.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbded5f6700 1 -- start start 2024-06-10T00:28:23.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbded5f6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde8100c40 0x7fbde819e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:23.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbded5f6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde81015f0 0x7fbde819ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:23.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbded5f6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde8107880 0x7fbde8198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:23.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbded5f6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbde81127b0 con 0x7fbde8107880 2024-06-10T00:28:23.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbded5f6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fbde8112630 con 0x7fbde81015f0 2024-06-10T00:28:23.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbded5f6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbde8112930 con 0x7fbde8100c40 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde8107880 0x7fbde8198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde8107880 0x7fbde8198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46718/0 (socket says 172.21.15.121:46718) 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde77fe700 1 -- 172.21.15.121:0/3738612014 learned_addr learned my addr 172.21.15.121:0/3738612014 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde67fc700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde81015f0 0x7fbde819ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde6ffd700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde8100c40 0x7fbde819e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde77fe700 1 -- 172.21.15.121:0/3738612014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde8100c40 msgr2=0x7fbde819e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde77fe700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde8100c40 0x7fbde819e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde77fe700 1 -- 172.21.15.121:0/3738612014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde81015f0 msgr2=0x7fbde819ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde77fe700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde81015f0 0x7fbde819ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde77fe700 1 -- 172.21.15.121:0/3738612014 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde8199170 con 0x7fbde8107880 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde67fc700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde81015f0 0x7fbde819ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:28:23.968 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.965+0000 7fbde77fe700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde8107880 0x7fbde8198a10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbde000efc0 tx=0x7fbde000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:23.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.966+0000 7fbdcffff700 1 -- 172.21.15.121:0/3738612014 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde00092e0 con 0x7fbde8107880 2024-06-10T00:28:23.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.966+0000 7fbded5f6700 1 -- 172.21.15.121:0/3738612014 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde8199400 con 0x7fbde8107880 2024-06-10T00:28:23.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.966+0000 7fbdcffff700 1 -- 172.21.15.121:0/3738612014 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbde0004bb0 con 0x7fbde8107880 2024-06-10T00:28:23.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.966+0000 7fbdcffff700 1 -- 172.21.15.121:0/3738612014 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0004020 con 0x7fbde8107880 2024-06-10T00:28:23.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.966+0000 7fbded5f6700 1 -- 172.21.15.121:0/3738612014 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde81a5720 con 0x7fbde8107880 2024-06-10T00:28:23.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.967+0000 7fbded5f6700 1 -- 172.21.15.121:0/3738612014 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde810aa50 con 0x7fbde8107880 2024-06-10T00:28:23.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.969+0000 7fbdcffff700 1 -- 172.21.15.121:0/3738612014 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fbde001b030 con 0x7fbde8107880 2024-06-10T00:28:23.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.969+0000 7fbdcffff700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdd00a2e20 0x7fbdd00a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:23.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.969+0000 7fbde6ffd700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdd00a2e20 0x7fbdd00a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:23.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.969+0000 7fbdcffff700 1 -- 172.21.15.121:0/3738612014 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fbde00cfc60 con 0x7fbde8107880 2024-06-10T00:28:23.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.970+0000 7fbde6ffd700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdd00a2e20 0x7fbdd00a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8005950 tx=0x7fbdd800a300 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:23.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:23.973+0000 7fbdcffff700 1 -- 172.21.15.121:0/3738612014 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbde0022020 con 0x7fbde8107880 2024-06-10T00:28:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.212+0000 7fbded5f6700 1 -- 172.21.15.121:0/3738612014 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbde8199f40 con 0x7fbde8107880 2024-06-10T00:28:24.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.213+0000 7fbdcffff700 1 -- 172.21.15.121:0/3738612014 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7fbde0010040 con 0x7fbde8107880 2024-06-10T00:28:24.213 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:24.213 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:27:55.370804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:28:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.215+0000 7fbded5f6700 1 -- 172.21.15.121:0/3738612014 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdd00a2e20 msgr2=0x7fbdd00a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.215+0000 7fbded5f6700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdd00a2e20 0x7fbdd00a52e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8005950 tx=0x7fbdd800a300 comp rx=0 tx=0).stop 2024-06-10T00:28:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.215+0000 7fbded5f6700 1 -- 172.21.15.121:0/3738612014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde8107880 msgr2=0x7fbde8198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.215+0000 7fbded5f6700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde8107880 0x7fbde8198a10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbde000efc0 tx=0x7fbde000c3a0 comp rx=0 tx=0).stop 2024-06-10T00:28:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.215+0000 7fbded5f6700 1 -- 172.21.15.121:0/3738612014 shutdown_connections 2024-06-10T00:28:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.215+0000 7fbded5f6700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdd00a2e20 0x7fbdd00a52e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.215+0000 7fbded5f6700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde8107880 0x7fbde8198a10 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.215+0000 7fbded5f6700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde8100c40 0x7fbde819e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.215+0000 7fbded5f6700 1 --2- 172.21.15.121:0/3738612014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde81015f0 0x7fbde819ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.215+0000 7fbded5f6700 1 -- 172.21.15.121:0/3738612014 >> 172.21.15.121:0/3738612014 conn(0x7fbde8077990 msgr2=0x7fbde80fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.216+0000 7fbded5f6700 1 -- 172.21.15.121:0/3738612014 shutdown_connections 2024-06-10T00:28:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:24.216+0000 7fbded5f6700 1 -- 172.21.15.121:0/3738612014 wait complete. 2024-06-10T00:28:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-06-10T00:28:24.541 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:24 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3738612014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:24.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:24 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3738612014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:24.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:24 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3738612014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:24.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:28:25.137 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:25.520 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:25 smithi121 ceph-mon[32378]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-06-10T00:28:25.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:25 smithi193 ceph-mon[36716]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-06-10T00:28:25.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:25 smithi135 ceph-mon[36424]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 76 KiB/s wr, 39 op/s 2024-06-10T00:28:27.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.009+0000 7f0977d4f700 1 -- 172.21.15.121:0/2420958055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09701071a0 msgr2=0x7f0970107620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:27.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.009+0000 7f0977d4f700 1 --2- 172.21.15.121:0/2420958055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09701071a0 0x7f0970107620 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f09640099c0 tx=0x7f0964009cd0 comp rx=0 tx=0).stop 2024-06-10T00:28:27.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.009+0000 7f0977d4f700 1 -- 172.21.15.121:0/2420958055 shutdown_connections 2024-06-10T00:28:27.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.009+0000 7f0977d4f700 1 --2- 172.21.15.121:0/2420958055 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0970107b60 0x7f097010e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:27.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.009+0000 7f0977d4f700 1 --2- 172.21.15.121:0/2420958055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0970105330 0x7f0970105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:27.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.009+0000 7f0977d4f700 1 --2- 172.21.15.121:0/2420958055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09701071a0 0x7f0970107620 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:27.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.009+0000 7f0977d4f700 1 -- 172.21.15.121:0/2420958055 >> 172.21.15.121:0/2420958055 conn(0x7f09700766b0 msgr2=0x7f0970076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:27.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.009+0000 7f0977d4f700 1 -- 172.21.15.121:0/2420958055 shutdown_connections 2024-06-10T00:28:27.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.009+0000 7f0977d4f700 1 -- 172.21.15.121:0/2420958055 wait complete. 2024-06-10T00:28:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f0977d4f700 1 Processor -- start 2024-06-10T00:28:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f0977d4f700 1 -- start start 2024-06-10T00:28:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f0977d4f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0970105330 0x7f0970198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f0977d4f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09701071a0 0x7f09701991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f0977d4f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0970107b60 0x7f097019c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f0977d4f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0970111730 con 0x7f0970107b60 2024-06-10T00:28:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f0977d4f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f09701115b0 con 0x7f09701071a0 2024-06-10T00:28:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f0977d4f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f09701118b0 con 0x7f0970105330 2024-06-10T00:28:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f09762ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0970107b60 0x7f097019c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f09762ec700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0970107b60 0x7f097019c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46736/0 (socket says 172.21.15.121:46736) 2024-06-10T00:28:27.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f09762ec700 1 -- 172.21.15.121:0/1889138623 learned_addr learned my addr 172.21.15.121:0/1889138623 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f0975aeb700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0970105330 0x7f0970198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f09752ea700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09701071a0 0x7f09701991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f09762ec700 1 -- 172.21.15.121:0/1889138623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0970105330 msgr2=0x7f0970198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f09762ec700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0970105330 0x7f0970198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f09762ec700 1 -- 172.21.15.121:0/1889138623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09701071a0 msgr2=0x7f09701991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f09762ec700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09701071a0 0x7f09701991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.010+0000 7f09762ec700 1 -- 172.21.15.121:0/1889138623 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097019cc00 con 0x7f0970107b60 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.011+0000 7f09762ec700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0970107b60 0x7f097019c4d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f096000d980 tx=0x7f096000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.011+0000 7f096effd700 1 -- 172.21.15.121:0/1889138623 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0960017070 con 0x7f0970107b60 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.011+0000 7f0977d4f700 1 -- 172.21.15.121:0/1889138623 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097019cef0 con 0x7f0970107b60 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.011+0000 7f096effd700 1 -- 172.21.15.121:0/1889138623 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0960005020 con 0x7f0970107b60 2024-06-10T00:28:27.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.011+0000 7f096effd700 1 -- 172.21.15.121:0/1889138623 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096000fbf0 con 0x7f0970107b60 2024-06-10T00:28:27.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.011+0000 7f0977d4f700 1 -- 172.21.15.121:0/1889138623 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09701a5860 con 0x7f0970107b60 2024-06-10T00:28:27.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.012+0000 7f0977d4f700 1 -- 172.21.15.121:0/1889138623 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0970069180 con 0x7f0970107b60 2024-06-10T00:28:27.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.013+0000 7f096effd700 1 -- 172.21.15.121:0/1889138623 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f09600051d0 con 0x7f0970107b60 2024-06-10T00:28:27.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.014+0000 7f096effd700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f09580a2e20 0x7f09580a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:27.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.014+0000 7f0975aeb700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f09580a2e20 0x7f09580a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:27.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.014+0000 7f096effd700 1 -- 172.21.15.121:0/1889138623 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f09600cfa50 con 0x7f0970107b60 2024-06-10T00:28:27.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.014+0000 7f0975aeb700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f09580a2e20 0x7f09580a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f096800f540 tx=0x7f096800b040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:27.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.017+0000 7f096effd700 1 -- 172.21.15.121:0/1889138623 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0960020020 con 0x7f0970107b60 2024-06-10T00:28:27.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.267+0000 7f0977d4f700 1 -- 172.21.15.121:0/1889138623 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09701a5a90 con 0x7f0970107b60 2024-06-10T00:28:27.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.268+0000 7f096effd700 1 -- 172.21.15.121:0/1889138623 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f096009c170 con 0x7f0970107b60 2024-06-10T00:28:27.268 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:27.268 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:27:55.370804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:28:27.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.270+0000 7f0977d4f700 1 -- 172.21.15.121:0/1889138623 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f09580a2e20 msgr2=0x7f09580a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:27.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.270+0000 7f0977d4f700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f09580a2e20 0x7f09580a52e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f096800f540 tx=0x7f096800b040 comp rx=0 tx=0).stop 2024-06-10T00:28:27.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.270+0000 7f0977d4f700 1 -- 172.21.15.121:0/1889138623 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0970107b60 msgr2=0x7f097019c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:27.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.270+0000 7f0977d4f700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0970107b60 0x7f097019c4d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f096000d980 tx=0x7f096000dc90 comp rx=0 tx=0).stop 2024-06-10T00:28:27.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.270+0000 7f0977d4f700 1 -- 172.21.15.121:0/1889138623 shutdown_connections 2024-06-10T00:28:27.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.270+0000 7f0977d4f700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f09580a2e20 0x7f09580a52e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:27.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.270+0000 7f0977d4f700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0970107b60 0x7f097019c4d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:27.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.270+0000 7f0977d4f700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0970105330 0x7f0970198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:27.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.270+0000 7f0977d4f700 1 --2- 172.21.15.121:0/1889138623 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09701071a0 0x7f09701991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:27.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.270+0000 7f0977d4f700 1 -- 172.21.15.121:0/1889138623 >> 172.21.15.121:0/1889138623 conn(0x7f09700766b0 msgr2=0x7f0970100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:27.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.270+0000 7f0977d4f700 1 -- 172.21.15.121:0/1889138623 shutdown_connections 2024-06-10T00:28:27.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:27.271+0000 7f0977d4f700 1 -- 172.21.15.121:0/1889138623 wait complete. 2024-06-10T00:28:27.271 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-06-10T00:28:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:27 smithi193 ceph-mon[36716]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:27 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1889138623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:27.655 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:27 smithi121 ceph-mon[32378]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:27.655 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:27 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1889138623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:27.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:27 smithi135 ceph-mon[36424]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:27.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:27 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1889138623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:27.931 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:28:28.184 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:29.550 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:29 smithi121 ceph-mon[32378]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:29.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:29 smithi193 ceph-mon[36716]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:29.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:29 smithi135 ceph-mon[36424]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.909+0000 7fe3e2545700 1 -- 172.21.15.121:0/1678210138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3dc107570 msgr2=0x7fe3dc107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.909+0000 7fe3e2545700 1 --2- 172.21.15.121:0/1678210138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3dc107570 0x7fe3dc107950 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc009990 tx=0x7fe3cc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:28:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.909+0000 7fe3e2545700 1 -- 172.21.15.121:0/1678210138 shutdown_connections 2024-06-10T00:28:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.909+0000 7fe3e2545700 1 --2- 172.21.15.121:0/1678210138 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3dc101810 0x7fe3dc10f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.909+0000 7fe3e2545700 1 --2- 172.21.15.121:0/1678210138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3dc107570 0x7fe3dc107950 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.909+0000 7fe3e2545700 1 --2- 172.21.15.121:0/1678210138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3dc100e50 0x7fe3dc1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.909+0000 7fe3e2545700 1 -- 172.21.15.121:0/1678210138 >> 172.21.15.121:0/1678210138 conn(0x7fe3dc0766b0 msgr2=0x7fe3dc076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.909+0000 7fe3e2545700 1 -- 172.21.15.121:0/1678210138 shutdown_connections 2024-06-10T00:28:29.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.909+0000 7fe3e2545700 1 -- 172.21.15.121:0/1678210138 wait complete. 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3e2545700 1 Processor -- start 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3e2545700 1 -- start start 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3e2545700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3dc100e50 0x7fe3dc19e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3e2545700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3dc101810 0x7fe3dc19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3e2545700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3dc107570 0x7fe3dc198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3e2545700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe3dc112830 con 0x7fe3dc100e50 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3e2545700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fe3dc1126b0 con 0x7fe3dc107570 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3e2545700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe3dc1129b0 con 0x7fe3dc101810 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3dbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3dc100e50 0x7fe3dc19e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3dbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3dc100e50 0x7fe3dc19e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33206/0 (socket says 172.21.15.121:33206) 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3dbfff700 1 -- 172.21.15.121:0/2698172682 learned_addr learned my addr 172.21.15.121:0/2698172682 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3e0ae2700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3dc107570 0x7fe3dc198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:29.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3db7fe700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3dc101810 0x7fe3dc19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3dbfff700 1 -- 172.21.15.121:0/2698172682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3dc101810 msgr2=0x7fe3dc19ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3dbfff700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3dc101810 0x7fe3dc19ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3dbfff700 1 -- 172.21.15.121:0/2698172682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3dc107570 msgr2=0x7fe3dc198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3dbfff700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3dc107570 0x7fe3dc198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3dbfff700 1 -- 172.21.15.121:0/2698172682 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3dc1991f0 con 0x7fe3dc100e50 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.910+0000 7fe3db7fe700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3dc101810 0x7fe3dc19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.911+0000 7fe3dbfff700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3dc100e50 0x7fe3dc19e8d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc000c00 tx=0x7fe3cc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.911+0000 7fe3d97fa700 1 -- 172.21.15.121:0/2698172682 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc027070 con 0x7fe3dc100e50 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.911+0000 7fe3e2545700 1 -- 172.21.15.121:0/2698172682 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3dc199480 con 0x7fe3dc100e50 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.911+0000 7fe3d97fa700 1 -- 172.21.15.121:0/2698172682 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3cc003780 con 0x7fe3dc100e50 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.911+0000 7fe3d97fa700 1 -- 172.21.15.121:0/2698172682 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3cc022db0 con 0x7fe3dc100e50 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.911+0000 7fe3e2545700 1 -- 172.21.15.121:0/2698172682 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3dc1a57a0 con 0x7fe3dc100e50 2024-06-10T00:28:29.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.912+0000 7fe3e2545700 1 -- 172.21.15.121:0/2698172682 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3dc069180 con 0x7fe3dc100e50 2024-06-10T00:28:29.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.913+0000 7fe3d97fa700 1 -- 172.21.15.121:0/2698172682 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fe3cc02a070 con 0x7fe3dc100e50 2024-06-10T00:28:29.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.913+0000 7fe3d97fa700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe3c40a2e20 0x7fe3c40a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:29.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.914+0000 7fe3db7fe700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe3c40a2e20 0x7fe3c40a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:29.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.914+0000 7fe3d97fa700 1 -- 172.21.15.121:0/2698172682 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fe3cc0d0640 con 0x7fe3dc100e50 2024-06-10T00:28:29.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.914+0000 7fe3db7fe700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe3c40a2e20 0x7fe3c40a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc076c20 tx=0x7fe3d0008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:29.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:29.918+0000 7fe3d97fa700 1 -- 172.21.15.121:0/2698172682 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe3cc013070 con 0x7fe3dc100e50 2024-06-10T00:28:30.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.114+0000 7fe3e2545700 1 -- 172.21.15.121:0/2698172682 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3dc02dca0 con 0x7fe3dc100e50 2024-06-10T00:28:30.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.116+0000 7fe3d97fa700 1 -- 172.21.15.121:0/2698172682 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+25301 (secure 0 0 0) 0x7fe3cc09cdf0 con 0x7fe3dc100e50 2024-06-10T00:28:30.116 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:30.116 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":101,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:27:57.451584+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/4266501776":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/4078460254":"2024-06-11T00:27:57.451553+0000","172.21.15.121:6801/3506007074":"2024-06-11T00:27:57.451553+0000","172.21.15.121:6800/3506007074":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/58990727":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/1002791322":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.118+0000 7fe3e2545700 1 -- 172.21.15.121:0/2698172682 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe3c40a2e20 msgr2=0x7fe3c40a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.118+0000 7fe3e2545700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe3c40a2e20 0x7fe3c40a52e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc076c20 tx=0x7fe3d0008040 comp rx=0 tx=0).stop 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.118+0000 7fe3e2545700 1 -- 172.21.15.121:0/2698172682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3dc100e50 msgr2=0x7fe3dc19e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.118+0000 7fe3e2545700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3dc100e50 0x7fe3dc19e8d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc000c00 tx=0x7fe3cc0047b0 comp rx=0 tx=0).stop 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.118+0000 7fe3e2545700 1 -- 172.21.15.121:0/2698172682 shutdown_connections 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.118+0000 7fe3e2545700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe3c40a2e20 0x7fe3c40a52e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.119+0000 7fe3e2545700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3dc107570 0x7fe3dc198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.119+0000 7fe3e2545700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3dc100e50 0x7fe3dc19e8d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.119+0000 7fe3e2545700 1 --2- 172.21.15.121:0/2698172682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3dc101810 0x7fe3dc19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.119+0000 7fe3e2545700 1 -- 172.21.15.121:0/2698172682 >> 172.21.15.121:0/2698172682 conn(0x7fe3dc0766b0 msgr2=0x7fe3dc0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.119+0000 7fe3e2545700 1 -- 172.21.15.121:0/2698172682 shutdown_connections 2024-06-10T00:28:30.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:30.119+0000 7fe3e2545700 1 -- 172.21.15.121:0/2698172682 wait complete. 2024-06-10T00:28:30.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:30 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2698172682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:28:30.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:30 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2698172682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:28:30.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:30 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2698172682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:28:30.868 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-06-10T00:28:31.111 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:31.516 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:31 smithi121 ceph-mon[32378]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:28:31.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:31 smithi193 ceph-mon[36716]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:28:31.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:31 smithi135 ceph-mon[36424]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:28:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:33 smithi193 ceph-mon[36716]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:33.631 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:33 smithi121 ceph-mon[32378]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:33.658 INFO:teuthology.orchestra.run.smithi121.stdout:{"run_dir":"/var/run/ceph"} 2024-06-10T00:28:33.659 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-06-10T00:28:33.659 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 178 2024-06-10T00:28:33.659 DEBUG:tasks.quiescer:Yielding 2024-06-10T00:28:33.659 INFO:teuthology.run_tasks:Running task workunit... 2024-06-10T00:28:33.674 INFO:tasks.workunit:Pulling workunits from ref 7b7a5d375cf4e4afab96b9d4d229b699ae8304bb 2024-06-10T00:28:33.674 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-06-10T00:28:33.675 INFO:tasks.workunit:timeout=3h 2024-06-10T00:28:33.675 INFO:tasks.workunit:cleanup=True 2024-06-10T00:28:33.675 DEBUG:teuthology.orchestra.run.smithi121:> stat -- /home/ubuntu/cephtest/mnt.0 2024-06-10T00:28:33.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:28:33.692 INFO:teuthology.orchestra.run.smithi121.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-06-10T00:28:33.693 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-06-10T00:28:33.693 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-06-10T00:28:33.693 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-10T00:28:33.693 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-06-10T00:28:33.693 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-06-10 00:25:20.048162469 +0000 2024-06-10T00:28:33.693 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-06-10 00:25:20.048162469 +0000 2024-06-10T00:28:33.693 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-06-10 00:27:54.641646907 +0000 2024-06-10T00:28:33.693 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-06-10 00:25:20.048162469 +0000 2024-06-10T00:28:33.693 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-06-10T00:28:33.693 DEBUG:teuthology.orchestra.run.smithi121:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-06-10T00:28:33.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:33 smithi135 ceph-mon[36424]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:33.766 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7b7a5d375cf4e4afab96b9d4d229b699ae8304bb 2024-06-10T00:28:33.823 INFO:tasks.workunit.client.0.smithi121.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-06-10T00:28:33.926 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:34.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:34 smithi193 ceph-mon[36716]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:28:34.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:34 smithi121 ceph-mon[32378]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:28:34.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:34 smithi135 ceph-mon[36424]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:28:35.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.715+0000 7f60a0b9f700 1 -- 172.21.15.121:0/1905088077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f609c10ee90 msgr2=0x7f609c111280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:35.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.715+0000 7f60a0b9f700 1 --2- 172.21.15.121:0/1905088077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f609c10ee90 0x7f609c111280 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f609400b0a0 tx=0x7f609400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:28:35.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.715+0000 7f60a0b9f700 1 -- 172.21.15.121:0/1905088077 shutdown_connections 2024-06-10T00:28:35.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.715+0000 7f60a0b9f700 1 --2- 172.21.15.121:0/1905088077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f609c10ee90 0x7f609c111280 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:35.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.715+0000 7f60a0b9f700 1 --2- 172.21.15.121:0/1905088077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f609c0746c0 0x7f609c074aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:35.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.715+0000 7f60a0b9f700 1 --2- 172.21.15.121:0/1905088077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f609c074fe0 0x7f609c10e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:35.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.715+0000 7f60a0b9f700 1 -- 172.21.15.121:0/1905088077 >> 172.21.15.121:0/1905088077 conn(0x7f609c0795f0 msgr2=0x7f609c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:35.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.715+0000 7f60a0b9f700 1 -- 172.21.15.121:0/1905088077 shutdown_connections 2024-06-10T00:28:35.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.715+0000 7f60a0b9f700 1 -- 172.21.15.121:0/1905088077 wait complete. 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f60a0b9f700 1 Processor -- start 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f60a0b9f700 1 -- start start 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f60a0b9f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f609c0746c0 0x7f609c1025b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f60a0b9f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f609c074fe0 0x7f609c102af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f60a0b9f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f609c10ee90 0x7f609c103030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f60a0b9f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f609c1149e0 con 0x7f609c0746c0 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f60a0b9f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f609c114860 con 0x7f609c10ee90 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f60a0b9f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f609c114b60 con 0x7f609c074fe0 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f609a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f609c0746c0 0x7f609c1025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f609a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f609c0746c0 0x7f609c1025b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33220/0 (socket says 172.21.15.121:33220) 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f609a59c700 1 -- 172.21.15.121:0/4170980213 learned_addr learned my addr 172.21.15.121:0/4170980213 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f609ad9d700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f609c10ee90 0x7f609c103030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:35.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f6099d9b700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f609c074fe0 0x7f609c102af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f609a59c700 1 -- 172.21.15.121:0/4170980213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f609c074fe0 msgr2=0x7f609c102af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f609a59c700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f609c074fe0 0x7f609c102af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f609a59c700 1 -- 172.21.15.121:0/4170980213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f609c10ee90 msgr2=0x7f609c103030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.716+0000 7f609a59c700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f609c10ee90 0x7f609c103030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.717+0000 7f609a59c700 1 -- 172.21.15.121:0/4170980213 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609c1ab520 con 0x7f609c0746c0 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.717+0000 7f6099d9b700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f609c074fe0 0x7f609c102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.717+0000 7f609a59c700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f609c0746c0 0x7f609c1025b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f608c00d950 tx=0x7f608c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.717+0000 7f608b7fe700 1 -- 172.21.15.121:0/4170980213 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c016070 con 0x7f609c0746c0 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.717+0000 7f60a0b9f700 1 -- 172.21.15.121:0/4170980213 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f609c1ab810 con 0x7f609c0746c0 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.717+0000 7f608b7fe700 1 -- 172.21.15.121:0/4170980213 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f608c009e20 con 0x7f609c0746c0 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.717+0000 7f608b7fe700 1 -- 172.21.15.121:0/4170980213 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c00f620 con 0x7f609c0746c0 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.717+0000 7f60a0b9f700 1 -- 172.21.15.121:0/4170980213 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f609c1abda0 con 0x7f609c0746c0 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.718+0000 7f60a0b9f700 1 -- 172.21.15.121:0/4170980213 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f609c075db0 con 0x7f609c0746c0 2024-06-10T00:28:35.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.719+0000 7f608b7fe700 1 -- 172.21.15.121:0/4170980213 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f608c019070 con 0x7f609c0746c0 2024-06-10T00:28:35.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.720+0000 7f608b7fe700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f60840a2e20 0x7f60840a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:35.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.720+0000 7f6099d9b700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f60840a2e20 0x7f60840a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:35.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.720+0000 7f608b7fe700 1 -- 172.21.15.121:0/4170980213 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f608c0cf550 con 0x7f609c0746c0 2024-06-10T00:28:35.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.720+0000 7f6099d9b700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f60840a2e20 0x7f60840a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f609c103d70 tx=0x7f6090008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:35.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.724+0000 7f608b7fe700 1 -- 172.21.15.121:0/4170980213 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f608c09bc70 con 0x7f609c0746c0 2024-06-10T00:28:35.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.964+0000 7f60a0b9f700 1 -- 172.21.15.121:0/4170980213 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f609c103ae0 con 0x7f609c0746c0 2024-06-10T00:28:35.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.964+0000 7f608b7fe700 1 -- 172.21.15.121:0/4170980213 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f608c0bb070 con 0x7f609c0746c0 2024-06-10T00:28:35.964 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:35.965 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:27:55.370804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:28:35.966 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.966+0000 7f60a0b9f700 1 -- 172.21.15.121:0/4170980213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f60840a2e20 msgr2=0x7f60840a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.966+0000 7f60a0b9f700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f60840a2e20 0x7f60840a52e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f609c103d70 tx=0x7f6090008040 comp rx=0 tx=0).stop 2024-06-10T00:28:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.966+0000 7f60a0b9f700 1 -- 172.21.15.121:0/4170980213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f609c0746c0 msgr2=0x7f609c1025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.966+0000 7f60a0b9f700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f609c0746c0 0x7f609c1025b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f608c00d950 tx=0x7f608c00dc60 comp rx=0 tx=0).stop 2024-06-10T00:28:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.967+0000 7f60a0b9f700 1 -- 172.21.15.121:0/4170980213 shutdown_connections 2024-06-10T00:28:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.967+0000 7f60a0b9f700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f60840a2e20 0x7f60840a52e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.967+0000 7f60a0b9f700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f609c10ee90 0x7f609c103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.967+0000 7f60a0b9f700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f609c0746c0 0x7f609c1025b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.967+0000 7f60a0b9f700 1 --2- 172.21.15.121:0/4170980213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f609c074fe0 0x7f609c102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.967+0000 7f60a0b9f700 1 -- 172.21.15.121:0/4170980213 >> 172.21.15.121:0/4170980213 conn(0x7f609c0795f0 msgr2=0x7f609c0ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.967+0000 7f60a0b9f700 1 -- 172.21.15.121:0/4170980213 shutdown_connections 2024-06-10T00:28:35.967 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:35.967+0000 7f60a0b9f700 1 -- 172.21.15.121:0/4170980213 wait complete. 2024-06-10T00:28:35.968 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-06-10T00:28:36.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:36 smithi121 ceph-mon[32378]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:36.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:36 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/4170980213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:36.747 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-06-10T00:23:59.392036+0000', 'modified': '2024-06-10T00:27:55.370804+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24479, 'mds_1': 24472, 'mds_2': 24455}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.135:6835/284701222', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6834', 'nonce': 284701222}, {'type': 'v1', 'addr': '172.21.15.135:6835', 'nonce': 284701222}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/3023450024', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 3023450024}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 3023450024}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24448': {'gid': 24448, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/2863018656', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 2863018656}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 2863018656}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 25, 'addr': '172.21.15.135:6837/1549198529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6836', 'nonce': 1549198529}, {'type': 'v1', 'addr': '172.21.15.135:6837', 'nonce': 1549198529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24472': {'gid': 24472, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 24, 'addr': '172.21.15.193:6839/2677954983', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 2677954983}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 2677954983}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24479': {'gid': 24479, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.135:6839/1599631648', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.135:6838', 'nonce': 1599631648}, {'type': 'v1', 'addr': '172.21.15.135:6839', 'nonce': 1599631648}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24479, 'qdb_cluster': [24455, 24472, 24479]} 2024-06-10T00:28:36.747 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-06-10T00:28:36.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-06-10T00:28:36.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:36 smithi193 ceph-mon[36716]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:36.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:36 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/4170980213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:37.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:36 smithi135 ceph-mon[36424]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:37.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:36 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/4170980213' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:37.029 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.767+0000 7f688c47b700 1 -- 172.21.15.121:0/4137867542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6884102380 msgr2=0x7f6884105ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.767+0000 7f688c47b700 1 --2- 172.21.15.121:0/4137867542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6884102380 0x7f6884105ff0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f68800099c0 tx=0x7f6880009cd0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.767+0000 7f688c47b700 1 -- 172.21.15.121:0/4137867542 shutdown_connections 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.767+0000 7f688c47b700 1 --2- 172.21.15.121:0/4137867542 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6884106720 0x7f688410e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.767+0000 7f688c47b700 1 --2- 172.21.15.121:0/4137867542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68841019d0 0x7f6884101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.767+0000 7f688c47b700 1 --2- 172.21.15.121:0/4137867542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6884102380 0x7f6884105ff0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.767+0000 7f688c47b700 1 -- 172.21.15.121:0/4137867542 >> 172.21.15.121:0/4137867542 conn(0x7f6884078340 msgr2=0x7f6884078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.767+0000 7f688c47b700 1 -- 172.21.15.121:0/4137867542 shutdown_connections 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.767+0000 7f688c47b700 1 -- 172.21.15.121:0/4137867542 wait complete. 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.767+0000 7f688c47b700 1 Processor -- start 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688c47b700 1 -- start start 2024-06-10T00:28:38.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688c47b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68841019d0 0x7f6884199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688c47b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6884102380 0x7f688419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688c47b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6884106720 0x7f68841a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688c47b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6884111710 con 0x7f68841019d0 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688c47b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6884111590 con 0x7f6884102380 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688c47b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6884111890 con 0x7f6884106720 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688a217700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68841019d0 0x7f6884199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688a217700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68841019d0 0x7f6884199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33230/0 (socket says 172.21.15.121:33230) 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688a217700 1 -- 172.21.15.121:0/4290144363 learned_addr learned my addr 172.21.15.121:0/4290144363 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f6889a16700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6884102380 0x7f688419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688aa18700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6884106720 0x7f68841a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688a217700 1 -- 172.21.15.121:0/4290144363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6884106720 msgr2=0x7f68841a4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688a217700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6884106720 0x7f68841a4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688a217700 1 -- 172.21.15.121:0/4290144363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6884102380 msgr2=0x7f688419a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:38.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688a217700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6884102380 0x7f688419a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688a217700 1 -- 172.21.15.121:0/4290144363 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68841a5110 con 0x7f68841019d0 2024-06-10T00:28:38.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.768+0000 7f688a217700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68841019d0 0x7f6884199b50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f687400b530 tx=0x7f687400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:38.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.769+0000 7f687b7fe700 1 -- 172.21.15.121:0/4290144363 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6874015020 con 0x7f68841019d0 2024-06-10T00:28:38.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.769+0000 7f688c47b700 1 -- 172.21.15.121:0/4290144363 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68841a5340 con 0x7f68841019d0 2024-06-10T00:28:38.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.769+0000 7f687b7fe700 1 -- 172.21.15.121:0/4290144363 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6874011d10 con 0x7f68841019d0 2024-06-10T00:28:38.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.769+0000 7f687b7fe700 1 -- 172.21.15.121:0/4290144363 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687400f2e0 con 0x7f68841019d0 2024-06-10T00:28:38.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.769+0000 7f688c47b700 1 -- 172.21.15.121:0/4290144363 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68841a5800 con 0x7f68841019d0 2024-06-10T00:28:38.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.771+0000 7f687b7fe700 1 -- 172.21.15.121:0/4290144363 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f6874018020 con 0x7f68841019d0 2024-06-10T00:28:38.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.772+0000 7f687b7fe700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68700a2e20 0x7f68700a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.772 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:38 smithi121 ceph-mon[32378]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:38.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.772+0000 7f6889a16700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68700a2e20 0x7f68700a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.772+0000 7f687b7fe700 1 -- 172.21.15.121:0/4290144363 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f68740cf190 con 0x7f68841019d0 2024-06-10T00:28:38.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.772+0000 7f6889a16700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68700a2e20 0x7f68700a52e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6880009990 tx=0x7f6880005fb0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:38.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.772+0000 7f6896cfeb80 1 Processor -- start 2024-06-10T00:28:38.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f6896cfeb80 1 -- start start 2024-06-10T00:28:38.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 0x55e5c8659e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c865a340 0x55e5c8653da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c86542e0 0x55e5c8654700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e5c85a17c0 con 0x55e5c86542e0 2024-06-10T00:28:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55e5c85496d0 con 0x55e5c865a340 2024-06-10T00:28:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e5c859d780 con 0x55e5c8659a20 2024-06-10T00:28:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f688aa18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c86542e0 0x55e5c8654700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f688aa18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c86542e0 0x55e5c8654700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33234/0 (socket says 172.21.15.121:33234) 2024-06-10T00:28:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f688aa18700 1 -- 172.21.15.121:0/2889444840 learned_addr learned my addr 172.21.15.121:0/2889444840 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f6889a16700 1 --2- 172.21.15.121:0/2889444840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c865a340 0x55e5c8653da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f688a217700 1 --2- 172.21.15.121:0/2889444840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 0x55e5c8659e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.773+0000 7f688aa18700 1 -- 172.21.15.121:0/2889444840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 msgr2=0x55e5c8659e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.774+0000 7f688aa18700 1 --2- 172.21.15.121:0/2889444840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 0x55e5c8659e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.774+0000 7f688aa18700 1 -- 172.21.15.121:0/2889444840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c865a340 msgr2=0x55e5c8653da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.774+0000 7f688aa18700 1 --2- 172.21.15.121:0/2889444840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c865a340 0x55e5c8653da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.774+0000 7f688aa18700 1 -- 172.21.15.121:0/2889444840 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5c853ff00 con 0x55e5c86542e0 2024-06-10T00:28:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.774+0000 7f6889a16700 1 --2- 172.21.15.121:0/2889444840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c865a340 0x55e5c8653da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:28:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.774+0000 7f688aa18700 1 --2- 172.21.15.121:0/2889444840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c86542e0 0x55e5c8654700 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f687c009dc0 tx=0x7f687c0077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf5a8ea6a748a45 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.774+0000 7f6878d70700 1 -- 172.21.15.121:0/2889444840 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c014610 con 0x55e5c86542e0 2024-06-10T00:28:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.774+0000 7f6878d70700 1 -- 172.21.15.121:0/2889444840 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f687c019070 con 0x55e5c86542e0 2024-06-10T00:28:38.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.774+0000 7f6878d70700 1 -- 172.21.15.121:0/2889444840 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c013a10 con 0x55e5c86542e0 2024-06-10T00:28:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.775+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2889444840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c86542e0 msgr2=0x55e5c8654700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.775+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/2889444840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c86542e0 0x55e5c8654700 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f687c009dc0 tx=0x7f687c0077e0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.775+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2889444840 shutdown_connections 2024-06-10T00:28:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.775+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/2889444840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c86542e0 0x55e5c8654700 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.775+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/2889444840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 0x55e5c8659e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.775+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/2889444840 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c865a340 0x55e5c8653da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.775+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2889444840 >> 172.21.15.121:0/2889444840 conn(0x55e5c857ac70 msgr2=0x55e5c864a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.775+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2889444840 shutdown_connections 2024-06-10T00:28:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.775+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2889444840 wait complete. 2024-06-10T00:28:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.775+0000 7f6896cfeb80 1 -- start start 2024-06-10T00:28:38.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.775+0000 7f6896cfeb80 1 Processor -- start 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c86542e0 0x55e5c8660030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c8659a20 0x55e5c865d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c865a340 0x55e5c865d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e5c8660d80 con 0x55e5c8659a20 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55e5c8541ba0 con 0x55e5c865a340 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e5c8660f00 con 0x55e5c86542e0 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6896cfeb80 10 client.0 _ll_register_callbacks cb 0x55e5c853b8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6889a16700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c8659a20 0x55e5c865d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6889a16700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c8659a20 0x55e5c865d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33236/0 (socket says 172.21.15.121:33236) 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6889a16700 1 -- 172.21.15.121:0/2059214338 learned_addr learned my addr 172.21.15.121:0/2059214338 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f688a217700 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c86542e0 0x55e5c8660030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f688aa18700 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c865a340 0x55e5c865d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6889a16700 1 -- 172.21.15.121:0/2059214338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c86542e0 msgr2=0x55e5c8660030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:38.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6889a16700 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c86542e0 0x55e5c8660030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6889a16700 1 -- 172.21.15.121:0/2059214338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c865a340 msgr2=0x55e5c865d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:38.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6889a16700 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c865a340 0x55e5c865d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6889a16700 1 -- 172.21.15.121:0/2059214338 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5c84f0880 con 0x55e5c8659a20 2024-06-10T00:28:38.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f6889a16700 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c8659a20 0x55e5c865d330 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6884076580 tx=0x7f688001a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15bf0c740c994d7d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:38.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.776+0000 7f686dffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:38.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f686dffb700 1 -- 172.21.15.121:0/2059214338 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6880024820 con 0x55e5c8659a20 2024-06-10T00:28:38.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f686dffb700 1 -- 172.21.15.121:0/2059214338 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f68800249c0 con 0x55e5c8659a20 2024-06-10T00:28:38.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2059214338 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f687c019070 con 0x55e5c8659a20 2024-06-10T00:28:38.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f686dffb700 1 -- 172.21.15.121:0/2059214338 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f688003e050 con 0x55e5c8659a20 2024-06-10T00:28:38.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f6896cfeb80 10 client.15468 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2059214338 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e5c866b600 con 0x55e5c8659a20 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f686dffb700 1 -- 172.21.15.121:0/2059214338 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f687c019070 con 0x55e5c8659a20 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f686dffb700 1 -- 172.21.15.121:0/2059214338 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f6880030be0 con 0x55e5c8659a20 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f6896cfeb80 10 client.15468 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f6896cfeb80 10 client.15468 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f6896cfeb80 20 client.15468 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55e5c866dc90 0x55e5c866f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f6896cfeb80 4 client.15468 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.777+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2059214338 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5c85aecc0 con 0x55e5c866dc90 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.778+0000 7f688a217700 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55e5c866dc90 0x55e5c866f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.778+0000 7f686dffb700 10 client.15468 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.778+0000 7f688a217700 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55e5c866dc90 0x55e5c866f100 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d7272f293e794bae server_cookie=c3df1e78cb19d1bc in_seq=0 out_seq=0 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f686dffb700 1 -- 172.21.15.121:0/2059214338 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55e5c85aecc0 con 0x55e5c866dc90 2024-06-10T00:28:38.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f686dffb700 10 client.15468 handle_command_reply: tid=0 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 1 client.15468 shutdown 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2059214338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c8659a20 msgr2=0x55e5c865d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c8659a20 0x55e5c865d330 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6884076580 tx=0x7f688001a050 comp rx=0 tx=0).stop 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2059214338 shutdown_connections 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55e5c866dc90 0x55e5c866f100 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c865a340 0x55e5c865d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c86542e0 0x55e5c8660030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/2059214338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c8659a20 0x55e5c865d330 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2059214338 >> 172.21.15.121:0/2059214338 conn(0x55e5c857ac70 msgr2=0x55e5c864e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f686dffb700 0 client.15468 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2059214338 shutdown_connections 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 1 -- 172.21.15.121:0/2059214338 wait complete. 2024-06-10T00:28:38.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.779+0000 7f6896cfeb80 20 client.15468 trim_cache size 0 max 16384 2024-06-10T00:28:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.800+0000 7f6896cfeb80 1 Processor -- start 2024-06-10T00:28:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.800+0000 7f6896cfeb80 1 -- start start 2024-06-10T00:28:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.800+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c86542e0 0x55e5c864a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.800+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 0x55e5c864a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.800+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c865bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.800+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e5c866d440 con 0x55e5c865a340 2024-06-10T00:28:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.800+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55e5c866db10 con 0x55e5c86542e0 2024-06-10T00:28:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.800+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e5c866e1e0 con 0x55e5c8659a20 2024-06-10T00:28:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f688aa18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c865bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f688aa18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c865bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33244/0 (socket says 172.21.15.121:33244) 2024-06-10T00:28:38.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f688aa18700 1 -- 172.21.15.121:0/1765041322 learned_addr learned my addr 172.21.15.121:0/1765041322 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f688aa18700 1 -- 172.21.15.121:0/1765041322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 msgr2=0x55e5c864a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f688aa18700 1 --2- 172.21.15.121:0/1765041322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 0x55e5c864a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f688aa18700 1 -- 172.21.15.121:0/1765041322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c86542e0 msgr2=0x55e5c864a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f688a217700 1 --2- 172.21.15.121:0/1765041322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c86542e0 0x55e5c864a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f688aa18700 1 --2- 172.21.15.121:0/1765041322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c86542e0 0x55e5c864a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f688aa18700 1 -- 172.21.15.121:0/1765041322 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5c8670060 con 0x55e5c865a340 2024-06-10T00:28:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f688aa18700 1 --2- 172.21.15.121:0/1765041322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c865bc40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f687c013620 tx=0x7f687c013650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9be2c14018ee68 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f686ffff700 1 -- 172.21.15.121:0/1765041322 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c004c50 con 0x55e5c865a340 2024-06-10T00:28:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f686ffff700 1 -- 172.21.15.121:0/1765041322 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f687c019070 con 0x55e5c865a340 2024-06-10T00:28:38.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.801+0000 7f686ffff700 1 -- 172.21.15.121:0/1765041322 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c0052a0 con 0x55e5c865a340 2024-06-10T00:28:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.802+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1765041322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 msgr2=0x55e5c865bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.802+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/1765041322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c865bc40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f687c013620 tx=0x7f687c013650 comp rx=0 tx=0).stop 2024-06-10T00:28:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.802+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1765041322 shutdown_connections 2024-06-10T00:28:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.802+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/1765041322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c865bc40 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.802+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/1765041322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c86542e0 0x55e5c864a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.802+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/1765041322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 0x55e5c864a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.802+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1765041322 >> 172.21.15.121:0/1765041322 conn(0x55e5c857ac70 msgr2=0x55e5c864be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.802+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1765041322 shutdown_connections 2024-06-10T00:28:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.802+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1765041322 wait complete. 2024-06-10T00:28:38.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.802+0000 7f6896cfeb80 1 -- start start 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.802+0000 7f6896cfeb80 1 Processor -- start 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c86542e0 0x55e5c866a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 0x55e5c8678580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f6896cfeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c8678cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e5c866ef80 con 0x55e5c865a340 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55e5c866e8b0 con 0x55e5c86542e0 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f6896cfeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e5c866f650 con 0x55e5c8659a20 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f6896cfeb80 10 client.0 _ll_register_callbacks cb 0x55e5c853b8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f688aa18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c8678cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f688aa18700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c8678cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33256/0 (socket says 172.21.15.121:33256) 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f688a217700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c86542e0 0x55e5c866a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f688aa18700 1 -- 172.21.15.121:0/1043030328 learned_addr learned my addr 172.21.15.121:0/1043030328 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f6889a16700 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 0x55e5c8678580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f688aa18700 1 -- 172.21.15.121:0/1043030328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 msgr2=0x55e5c8678580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f688aa18700 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 0x55e5c8678580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f688aa18700 1 -- 172.21.15.121:0/1043030328 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c86542e0 msgr2=0x55e5c866a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f688aa18700 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c86542e0 0x55e5c866a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f688aa18700 1 -- 172.21.15.121:0/1043030328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5c8673630 con 0x55e5c865a340 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f688aa18700 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c8678cc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f687c0096e0 tx=0x7f687c013430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df79efa2fee323d6 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f686dffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f686dffb700 1 -- 172.21.15.121:0/1043030328 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c013c90 con 0x55e5c865a340 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f686dffb700 1 -- 172.21.15.121:0/1043030328 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f687c013e30 con 0x55e5c865a340 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.803+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1043030328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f687c019070 con 0x55e5c865a340 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f686dffb700 1 -- 172.21.15.121:0/1043030328 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f687c0348e0 con 0x55e5c865a340 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f6896cfeb80 10 client.15474 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1043030328 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e5c86d98f0 con 0x55e5c865a340 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f686dffb700 1 -- 172.21.15.121:0/1043030328 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f687c019070 con 0x55e5c865a340 2024-06-10T00:28:38.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f686dffb700 1 -- 172.21.15.121:0/1043030328 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f687c026020 con 0x55e5c865a340 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f6896cfeb80 10 client.15474 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f6896cfeb80 10 client.15474 resolve_mds: resolved name 'h' to daemon mds.h 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f6896cfeb80 20 client.15474 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55e5c86ddd90 0x55e5c86dcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f6896cfeb80 4 client.15474 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1043030328 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "status"}) -- 0x55e5c85a9240 con 0x55e5c86ddd90 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.804+0000 7f688a217700 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55e5c86ddd90 0x55e5c86dcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f686dffb700 10 client.15474 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f688a217700 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55e5c86ddd90 0x55e5c86dcaa0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54fd8d38d0905b41 server_cookie=a05412b95de4c61f in_seq=0 out_seq=0 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f686dffb700 1 -- 172.21.15.121:0/1043030328 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x55e5c85a9240 con 0x55e5c86ddd90 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f686dffb700 10 client.15474 handle_command_reply: tid=0 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 1 client.15474 shutdown 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1043030328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 msgr2=0x55e5c8678cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c8678cc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f687c0096e0 tx=0x7f687c013430 comp rx=0 tx=0).stop 2024-06-10T00:28:38.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1043030328 shutdown_connections 2024-06-10T00:28:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55e5c86ddd90 0x55e5c86dcaa0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e5c865a340 0x55e5c8678cc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e5c86542e0 0x55e5c866a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 1 --2- 172.21.15.121:0/1043030328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5c8659a20 0x55e5c8678580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1043030328 >> 172.21.15.121:0/1043030328 conn(0x55e5c857ac70 msgr2=0x55e5c8657460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f686dffb700 0 client.15474 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:28:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1043030328 shutdown_connections 2024-06-10T00:28:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 1 -- 172.21.15.121:0/1043030328 wait complete. 2024-06-10T00:28:38.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.805+0000 7f6896cfeb80 20 client.15474 trim_cache size 0 max 16384 2024-06-10T00:28:38.810 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:28:38.810 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:28:38.811 INFO:teuthology.orchestra.run.smithi121.stdout: "whoami": 2, 2024-06-10T00:28:38.811 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 24455, 2024-06-10T00:28:38.811 INFO:teuthology.orchestra.run.smithi121.stdout: "want_state": "up:active", 2024-06-10T00:28:38.811 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "up:active", 2024-06-10T00:28:38.811 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs", 2024-06-10T00:28:38.811 INFO:teuthology.orchestra.run.smithi121.stdout: "rank_uptime": 215.55050728099999, 2024-06-10T00:28:38.811 INFO:teuthology.orchestra.run.smithi121.stdout: "mdsmap_epoch": 26, 2024-06-10T00:28:38.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch": 101, 2024-06-10T00:28:38.811 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch_barrier": 100, 2024-06-10T00:28:38.811 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 310.38394603699999 2024-06-10T00:28:38.811 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:28:38.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 -- 172.21.15.121:0/4290144363 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68700a2e20 msgr2=0x7f68700a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:38.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68700a2e20 0x7f68700a52e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6880009990 tx=0x7f6880005fb0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 -- 172.21.15.121:0/4290144363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68841019d0 msgr2=0x7f6884199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:38.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68841019d0 0x7f6884199b50 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f687400b530 tx=0x7f687400b840 comp rx=0 tx=0).stop 2024-06-10T00:28:38.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 -- 172.21.15.121:0/4290144363 shutdown_connections 2024-06-10T00:28:38.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68700a2e20 0x7f68700a52e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6884106720 0x7f68841a4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68841019d0 0x7f6884199b50 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 --2- 172.21.15.121:0/4290144363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6884102380 0x7f688419a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:38.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 -- 172.21.15.121:0/4290144363 >> 172.21.15.121:0/4290144363 conn(0x7f6884078340 msgr2=0x7f68840ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:38.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 -- 172.21.15.121:0/4290144363 shutdown_connections 2024-06-10T00:28:38.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:38.806+0000 7f688c47b700 1 -- 172.21.15.121:0/4290144363 wait complete. 2024-06-10T00:28:38.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:38 smithi193 ceph-mon[36716]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:39.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:38 smithi135 ceph-mon[36424]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:39.498 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-06-10T00:28:39.740 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:40.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:40 smithi193 ceph-mon[36716]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:28:40.886 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:40 smithi121 ceph-mon[32378]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:28:41.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:40 smithi135 ceph-mon[36424]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:28:41.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.511+0000 7fee71fcd700 1 -- 172.21.15.121:0/1679295263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6c1030f0 msgr2=0x7fee6c107110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:41.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.511+0000 7fee71fcd700 1 --2- 172.21.15.121:0/1679295263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6c1030f0 0x7fee6c107110 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fee600099c0 tx=0x7fee60009cd0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.511+0000 7fee71fcd700 1 -- 172.21.15.121:0/1679295263 shutdown_connections 2024-06-10T00:28:41.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.511+0000 7fee71fcd700 1 --2- 172.21.15.121:0/1679295263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee6c107840 0x7fee6c10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.511+0000 7fee71fcd700 1 --2- 172.21.15.121:0/1679295263 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee6c102740 0x7fee6c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.511+0000 7fee71fcd700 1 --2- 172.21.15.121:0/1679295263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6c1030f0 0x7fee6c107110 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.511+0000 7fee71fcd700 1 -- 172.21.15.121:0/1679295263 >> 172.21.15.121:0/1679295263 conn(0x7fee6c0795f0 msgr2=0x7fee6c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.511+0000 7fee71fcd700 1 -- 172.21.15.121:0/1679295263 shutdown_connections 2024-06-10T00:28:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.511+0000 7fee71fcd700 1 -- 172.21.15.121:0/1679295263 wait complete. 2024-06-10T00:28:41.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.512+0000 7fee71fcd700 1 Processor -- start 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.512+0000 7fee71fcd700 1 -- start start 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee71fcd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee6c102740 0x7fee6c19e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee71fcd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee6c1030f0 0x7fee6c19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee71fcd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6c107840 0x7fee6c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee71fcd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c1127d0 con 0x7fee6c107840 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee71fcd700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c112650 con 0x7fee6c102740 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee71fcd700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c112950 con 0x7fee6c1030f0 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6c107840 0x7fee6c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6c107840 0x7fee6c198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52534/0 (socket says 172.21.15.121:52534) 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6bfff700 1 -- 172.21.15.121:0/2084070190 learned_addr learned my addr 172.21.15.121:0/2084070190 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6affd700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee6c1030f0 0x7fee6c19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee6c102740 0x7fee6c19e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6bfff700 1 -- 172.21.15.121:0/2084070190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee6c1030f0 msgr2=0x7fee6c19ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:41.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6bfff700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee6c1030f0 0x7fee6c19ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6bfff700 1 -- 172.21.15.121:0/2084070190 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee6c102740 msgr2=0x7fee6c19e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:41.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6bfff700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee6c102740 0x7fee6c19e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6bfff700 1 -- 172.21.15.121:0/2084070190 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6c1990f0 con 0x7fee6c107840 2024-06-10T00:28:41.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee6c102740 0x7fee6c19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:41.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6affd700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee6c1030f0 0x7fee6c19ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:41.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.513+0000 7fee6bfff700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6c107840 0x7fee6c198990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fee6400b970 tx=0x7fee6400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:41.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.514+0000 7fee68ff9700 1 -- 172.21.15.121:0/2084070190 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64019070 con 0x7fee6c107840 2024-06-10T00:28:41.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.514+0000 7fee71fcd700 1 -- 172.21.15.121:0/2084070190 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee6c1993e0 con 0x7fee6c107840 2024-06-10T00:28:41.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.514+0000 7fee68ff9700 1 -- 172.21.15.121:0/2084070190 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee64014440 con 0x7fee6c107840 2024-06-10T00:28:41.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.514+0000 7fee68ff9700 1 -- 172.21.15.121:0/2084070190 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee640075d0 con 0x7fee6c107840 2024-06-10T00:28:41.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.514+0000 7fee71fcd700 1 -- 172.21.15.121:0/2084070190 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee6c1a56a0 con 0x7fee6c107840 2024-06-10T00:28:41.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.516+0000 7fee68ff9700 1 -- 172.21.15.121:0/2084070190 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7fee6401c070 con 0x7fee6c107840 2024-06-10T00:28:41.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.516+0000 7fee68ff9700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fee540a2e20 0x7fee540a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.517+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fee540a2e20 0x7fee540a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.517+0000 7fee68ff9700 1 -- 172.21.15.121:0/2084070190 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fee640cec80 con 0x7fee6c107840 2024-06-10T00:28:41.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.517+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fee540a2e20 0x7fee540a52e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee6c199ad0 tx=0x7fee5c00a680 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:41.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.517+0000 7fee7c850b80 1 Processor -- start 2024-06-10T00:28:41.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.517+0000 7fee7c850b80 1 -- start start 2024-06-10T00:28:41.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.517+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266eaa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.517+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266eaa830 0x558266ea4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.517+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266ea4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.517+0000 7fee7c850b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x558266d96f10 con 0x558266ea4750 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.517+0000 7fee7c850b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x558266df5800 con 0x558266ea9f10 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.517+0000 7fee7c850b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558266d5d410 con 0x558266eaa830 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266ea4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266ea4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52544/0 (socket says 172.21.15.121:52544) 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee6bfff700 1 -- 172.21.15.121:0/461155983 learned_addr learned my addr 172.21.15.121:0/461155983 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee6affd700 1 --2- 172.21.15.121:0/461155983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266eaa830 0x558266ea4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/461155983 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266eaa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee6bfff700 1 -- 172.21.15.121:0/461155983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266eaa830 msgr2=0x558266ea4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee6bfff700 1 --2- 172.21.15.121:0/461155983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266eaa830 0x558266ea4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee6bfff700 1 -- 172.21.15.121:0/461155983 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 msgr2=0x558266eaa2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee6bfff700 1 --2- 172.21.15.121:0/461155983 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266eaa2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee6bfff700 1 -- 172.21.15.121:0/461155983 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558266d44880 con 0x558266ea4750 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee6bfff700 1 --2- 172.21.15.121:0/461155983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266ea4b70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fee64097780 tx=0x7fee6409aa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12974ec4cd44950 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:41.520 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee51ffb700 1 -- 172.21.15.121:0/461155983 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6409f160 con 0x558266ea4750 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee51ffb700 1 -- 172.21.15.121:0/461155983 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee6409e070 con 0x558266ea4750 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.518+0000 7fee51ffb700 1 -- 172.21.15.121:0/461155983 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6409b270 con 0x558266ea4750 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.519+0000 7fee7c850b80 1 -- 172.21.15.121:0/461155983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 msgr2=0x558266ea4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.519+0000 7fee7c850b80 1 --2- 172.21.15.121:0/461155983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266ea4b70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fee64097780 tx=0x7fee6409aa00 comp rx=0 tx=0).stop 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.519+0000 7fee7c850b80 1 -- 172.21.15.121:0/461155983 shutdown_connections 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.519+0000 7fee7c850b80 1 --2- 172.21.15.121:0/461155983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266ea4b70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.519+0000 7fee7c850b80 1 --2- 172.21.15.121:0/461155983 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266eaa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.519+0000 7fee7c850b80 1 --2- 172.21.15.121:0/461155983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266eaa830 0x558266ea4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.519+0000 7fee7c850b80 1 -- 172.21.15.121:0/461155983 >> 172.21.15.121:0/461155983 conn(0x558266dd1030 msgr2=0x558266e9f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.519+0000 7fee7c850b80 1 -- 172.21.15.121:0/461155983 shutdown_connections 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.519+0000 7fee7c850b80 1 -- 172.21.15.121:0/461155983 wait complete. 2024-06-10T00:28:41.521 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.519+0000 7fee7c850b80 1 -- start start 2024-06-10T00:28:41.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee7c850b80 1 Processor -- start 2024-06-10T00:28:41.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea4750 0x558266eb0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266ead830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266eadd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee7c850b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x558266d95f70 con 0x558266eaa830 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee7c850b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x558266df17c0 con 0x558266ea9f10 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee7c850b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558266eb1210 con 0x558266ea4750 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee7c850b80 10 client.0 _ll_register_callbacks cb 0x558266d8b560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266eadd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee6affd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266ead830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee6affd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266ead830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:60844/0 (socket says 172.21.15.121:60844) 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266eadd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52556/0 (socket says 172.21.15.121:52556) 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee6affd700 1 -- 172.21.15.121:0/1156221856 learned_addr learned my addr 172.21.15.121:0/1156221856 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea4750 0x558266eb0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee6bfff700 1 -- 172.21.15.121:0/1156221856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea4750 msgr2=0x558266eb0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee6bfff700 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea4750 0x558266eb0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee6bfff700 1 -- 172.21.15.121:0/1156221856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 msgr2=0x558266ead830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:41.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee6bfff700 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266ead830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.520+0000 7fee6bfff700 1 -- 172.21.15.121:0/1156221856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558266dbb700 con 0x558266eaa830 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee6bfff700 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266eadd70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fee640a3be0 tx=0x7fee64097050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef7020b5972db3d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee4affd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee4affd700 1 -- 172.21.15.121:0/1156221856 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6409b220 con 0x558266eaa830 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee4affd700 1 -- 172.21.15.121:0/1156221856 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fee64079bd0 con 0x558266eaa830 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee7c850b80 1 -- 172.21.15.121:0/1156221856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee6409e070 con 0x558266eaa830 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee4affd700 1 -- 172.21.15.121:0/1156221856 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fee64084080 con 0x558266eaa830 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee7c850b80 10 client.15486 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee7c850b80 1 -- 172.21.15.121:0/1156221856 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558266ebb910 con 0x558266eaa830 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee4affd700 1 -- 172.21.15.121:0/1156221856 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee6409e070 con 0x558266eaa830 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee4affd700 1 -- 172.21.15.121:0/1156221856 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fee64085040 con 0x558266eaa830 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee7c850b80 10 client.15486 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee7c850b80 10 client.15486 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee7c850b80 20 client.15486 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:41.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee7c850b80 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x558266ebcc60 0x558266ebee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee7c850b80 4 client.15486 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.521+0000 7fee7c850b80 1 -- 172.21.15.121:0/1156221856 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558266e02d00 con 0x558266ebcc60 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.522+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x558266ebcc60 0x558266ebee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.522+0000 7fee4affd700 10 client.15486 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.522+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x558266ebcc60 0x558266ebee00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9bb461ee5043c5b3 server_cookie=71b5f1bdf94dd7ce in_seq=0 out_seq=0 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee4affd700 1 -- 172.21.15.121:0/1156221856 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x558266e02d00 con 0x558266ebcc60 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee4affd700 10 client.15486 handle_command_reply: tid=0 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee7c850b80 1 client.15486 shutdown 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee7c850b80 1 -- 172.21.15.121:0/1156221856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 msgr2=0x558266eadd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee7c850b80 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266eadd70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fee640a3be0 tx=0x7fee64097050 comp rx=0 tx=0).stop 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee7c850b80 1 -- 172.21.15.121:0/1156221856 shutdown_connections 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee7c850b80 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x558266ebcc60 0x558266ebee00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee7c850b80 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266eadd70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee7c850b80 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea4750 0x558266eb0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee7c850b80 1 --2- 172.21.15.121:0/1156221856 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266ead830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee7c850b80 1 -- 172.21.15.121:0/1156221856 >> 172.21.15.121:0/1156221856 conn(0x558266dd1030 msgr2=0x558266e9e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:41.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee4affd700 0 client.15486 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:28:41.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee7c850b80 1 -- 172.21.15.121:0/1156221856 shutdown_connections 2024-06-10T00:28:41.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.523+0000 7fee7c850b80 1 -- 172.21.15.121:0/1156221856 wait complete. 2024-06-10T00:28:41.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.524+0000 7fee7c850b80 20 client.15486 trim_cache size 0 max 16384 2024-06-10T00:28:41.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.544+0000 7fee7c850b80 1 Processor -- start 2024-06-10T00:28:41.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee7c850b80 1 -- start start 2024-06-10T00:28:41.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea4750 0x558266ea4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266eac830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266ec8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee7c850b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x558266ebd140 con 0x558266eaa830 2024-06-10T00:28:41.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee7c850b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x558266ebd810 con 0x558266ea9f10 2024-06-10T00:28:41.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee7c850b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558266ebdee0 con 0x558266ea4750 2024-06-10T00:28:41.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266ec8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266ec8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52564/0 (socket says 172.21.15.121:52564) 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6bfff700 1 -- 172.21.15.121:0/702914933 learned_addr learned my addr 172.21.15.121:0/702914933 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6affd700 1 --2- 172.21.15.121:0/702914933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266eac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/702914933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea4750 0x558266ea4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6bfff700 1 -- 172.21.15.121:0/702914933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea4750 msgr2=0x558266ea4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6bfff700 1 --2- 172.21.15.121:0/702914933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea4750 0x558266ea4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6bfff700 1 -- 172.21.15.121:0/702914933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 msgr2=0x558266eac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6bfff700 1 --2- 172.21.15.121:0/702914933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266eac830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6bfff700 1 -- 172.21.15.121:0/702914933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558266eb0d30 con 0x558266eaa830 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6affd700 1 --2- 172.21.15.121:0/702914933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266eac830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee6bfff700 1 --2- 172.21.15.121:0/702914933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266ec8650 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fee640cc640 tx=0x7fee6409a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc7824cbf5c9399 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee517fa700 1 -- 172.21.15.121:0/702914933 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64099f40 con 0x558266eaa830 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee517fa700 1 -- 172.21.15.121:0/702914933 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee6409e070 con 0x558266eaa830 2024-06-10T00:28:41.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.545+0000 7fee517fa700 1 -- 172.21.15.121:0/702914933 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64079e30 con 0x558266eaa830 2024-06-10T00:28:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.546+0000 7fee7c850b80 1 -- 172.21.15.121:0/702914933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 msgr2=0x558266ec8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.546+0000 7fee7c850b80 1 --2- 172.21.15.121:0/702914933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266ec8650 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fee640cc640 tx=0x7fee6409a7c0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.546+0000 7fee7c850b80 1 -- 172.21.15.121:0/702914933 shutdown_connections 2024-06-10T00:28:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.546+0000 7fee7c850b80 1 --2- 172.21.15.121:0/702914933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266eaa830 0x558266ec8650 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.546+0000 7fee7c850b80 1 --2- 172.21.15.121:0/702914933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea4750 0x558266ea4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.546+0000 7fee7c850b80 1 --2- 172.21.15.121:0/702914933 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266ea9f10 0x558266eac830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.546+0000 7fee7c850b80 1 -- 172.21.15.121:0/702914933 >> 172.21.15.121:0/702914933 conn(0x558266dd1030 msgr2=0x558266eac120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.546+0000 7fee7c850b80 1 -- 172.21.15.121:0/702914933 shutdown_connections 2024-06-10T00:28:41.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.546+0000 7fee7c850b80 1 -- 172.21.15.121:0/702914933 wait complete. 2024-06-10T00:28:41.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.546+0000 7fee7c850b80 1 -- start start 2024-06-10T00:28:41.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee7c850b80 1 Processor -- start 2024-06-10T00:28:41.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266e9eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea9f10 0x558266e9f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee7c850b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266eaa830 0x558266eaf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee7c850b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x558266ebec80 con 0x558266ea4750 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee7c850b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x558266ebe5b0 con 0x558266eaa830 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee7c850b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558266ebf350 con 0x558266ea9f10 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee7c850b80 10 client.0 _ll_register_callbacks cb 0x558266d8b560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee6b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266e9eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee6b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266e9eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52570/0 (socket says 172.21.15.121:52570) 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee6b7fe700 1 -- 172.21.15.121:0/340052390 learned_addr learned my addr 172.21.15.121:0/340052390 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee6affd700 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea9f10 0x558266e9f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee6b7fe700 1 -- 172.21.15.121:0/340052390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea9f10 msgr2=0x558266e9f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee6bfff700 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266eaa830 0x558266eaf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea9f10 0x558266e9f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee6b7fe700 1 -- 172.21.15.121:0/340052390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266eaa830 msgr2=0x558266eaf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266eaa830 0x558266eaf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee6b7fe700 1 -- 172.21.15.121:0/340052390 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558266eaf5a0 con 0x558266ea4750 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.547+0000 7fee6bfff700 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266eaa830 0x558266eaf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee6b7fe700 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266e9eb10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00a5c0 tx=0x7fee5c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a618650efec69ae6 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee4affd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee4affd700 1 -- 172.21.15.121:0/340052390 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c02b070 con 0x558266ea4750 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee4affd700 1 -- 172.21.15.121:0/340052390 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fee5c004220 con 0x558266ea4750 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee7c850b80 1 -- 172.21.15.121:0/340052390 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee6409e070 con 0x558266ea4750 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee4affd700 1 -- 172.21.15.121:0/340052390 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fee5c037050 con 0x558266ea4750 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee7c850b80 10 client.15492 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee7c850b80 1 -- 172.21.15.121:0/340052390 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558266f2daa0 con 0x558266ea4750 2024-06-10T00:28:41.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee4affd700 1 -- 172.21.15.121:0/340052390 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee6409e070 con 0x558266ea4750 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee4affd700 1 -- 172.21.15.121:0/340052390 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fee5c0105f0 con 0x558266ea4750 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee7c850b80 10 client.15492 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee7c850b80 10 client.15492 resolve_mds: resolved name 'i' to daemon mds.i 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee7c850b80 20 client.15492 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee7c850b80 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x558266f326c0 0x558266f313d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee7c850b80 4 client.15492 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.548+0000 7fee7c850b80 1 -- 172.21.15.121:0/340052390 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "status"}) -- 0x558266eab830 con 0x558266f326c0 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.549+0000 7fee6affd700 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x558266f326c0 0x558266f313d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.549+0000 7fee6affd700 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x558266f326c0 0x558266f313d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eff1424550aa80d9 server_cookie=f82e6322fa2761d3 in_seq=0 out_seq=0 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.549+0000 7fee4affd700 10 client.15492 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.549+0000 7fee4affd700 1 -- 172.21.15.121:0/340052390 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x558266eab830 con 0x558266f326c0 2024-06-10T00:28:41.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.549+0000 7fee4affd700 10 client.15492 handle_command_reply: tid=0 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.549+0000 7fee7c850b80 1 client.15492 shutdown 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.549+0000 7fee7c850b80 1 -- 172.21.15.121:0/340052390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 msgr2=0x558266e9eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.549+0000 7fee7c850b80 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266e9eb10 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00a5c0 tx=0x7fee5c0047c0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.549+0000 7fee7c850b80 1 -- 172.21.15.121:0/340052390 shutdown_connections 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.549+0000 7fee7c850b80 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x558266f326c0 0x558266f313d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.549+0000 7fee7c850b80 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558266eaa830 0x558266eaf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee7c850b80 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558266ea4750 0x558266e9eb10 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee7c850b80 1 --2- 172.21.15.121:0/340052390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558266ea9f10 0x558266e9f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee4affd700 0 client.15492 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee7c850b80 1 -- 172.21.15.121:0/340052390 >> 172.21.15.121:0/340052390 conn(0x558266dd1030 msgr2=0x558266eac830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee7c850b80 1 -- 172.21.15.121:0/340052390 shutdown_connections 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee7c850b80 1 -- 172.21.15.121:0/340052390 wait complete. 2024-06-10T00:28:41.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee7c850b80 20 client.15492 trim_cache size 0 max 16384 2024-06-10T00:28:41.556 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:28:41.556 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:28:41.556 INFO:teuthology.orchestra.run.smithi121.stdout: "whoami": 1, 2024-06-10T00:28:41.556 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 24472, 2024-06-10T00:28:41.556 INFO:teuthology.orchestra.run.smithi121.stdout: "want_state": "up:active", 2024-06-10T00:28:41.556 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "up:active", 2024-06-10T00:28:41.556 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs", 2024-06-10T00:28:41.556 INFO:teuthology.orchestra.run.smithi121.stdout: "rank_uptime": 219.30274744299999, 2024-06-10T00:28:41.556 INFO:teuthology.orchestra.run.smithi121.stdout: "mdsmap_epoch": 26, 2024-06-10T00:28:41.556 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch": 101, 2024-06-10T00:28:41.556 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch_barrier": 100, 2024-06-10T00:28:41.557 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 310.23271430699998 2024-06-10T00:28:41.557 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:28:41.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 -- 172.21.15.121:0/2084070190 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fee540a2e20 msgr2=0x7fee540a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:41.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fee540a2e20 0x7fee540a52e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee6c199ad0 tx=0x7fee5c00a680 comp rx=0 tx=0).stop 2024-06-10T00:28:41.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 -- 172.21.15.121:0/2084070190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6c107840 msgr2=0x7fee6c198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:41.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6c107840 0x7fee6c198990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fee6400b970 tx=0x7fee6400bd30 comp rx=0 tx=0).stop 2024-06-10T00:28:41.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 -- 172.21.15.121:0/2084070190 shutdown_connections 2024-06-10T00:28:41.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fee540a2e20 0x7fee540a52e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee6c107840 0x7fee6c198990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee6c102740 0x7fee6c19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 --2- 172.21.15.121:0/2084070190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee6c1030f0 0x7fee6c19ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:41.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 -- 172.21.15.121:0/2084070190 >> 172.21.15.121:0/2084070190 conn(0x7fee6c0795f0 msgr2=0x7fee6c100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:41.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 -- 172.21.15.121:0/2084070190 shutdown_connections 2024-06-10T00:28:41.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:41.550+0000 7fee71fcd700 1 -- 172.21.15.121:0/2084070190 wait complete. 2024-06-10T00:28:42.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-06-10T00:28:42.619 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:42.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:42 smithi121 ceph-mon[32378]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:28:42.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:42 smithi193 ceph-mon[36716]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:28:43.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:42 smithi135 ceph-mon[36424]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:28:44.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.302+0000 7ff39b92c700 1 -- 172.21.15.121:0/4090982206 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3941024e0 msgr2=0x7ff394102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:44.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.302+0000 7ff39b92c700 1 --2- 172.21.15.121:0/4090982206 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3941024e0 0x7ff394102940 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff390009990 tx=0x7ff390009ca0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.302+0000 7ff39b92c700 1 -- 172.21.15.121:0/4090982206 shutdown_connections 2024-06-10T00:28:44.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.302+0000 7ff39b92c700 1 --2- 172.21.15.121:0/4090982206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff394102e80 0x7ff39410f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.302+0000 7ff39b92c700 1 --2- 172.21.15.121:0/4090982206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3941084f0 0x7ff3941088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.302+0000 7ff39b92c700 1 --2- 172.21.15.121:0/4090982206 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3941024e0 0x7ff394102940 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.302+0000 7ff39b92c700 1 -- 172.21.15.121:0/4090982206 >> 172.21.15.121:0/4090982206 conn(0x7ff394079530 msgr2=0x7ff394079940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.302+0000 7ff39b92c700 1 -- 172.21.15.121:0/4090982206 shutdown_connections 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.302+0000 7ff39b92c700 1 -- 172.21.15.121:0/4090982206 wait complete. 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff39b92c700 1 Processor -- start 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff39b92c700 1 -- start start 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff39b92c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3941024e0 0x7ff39419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff39b92c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff394102e80 0x7ff39419ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff39b92c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3941084f0 0x7ff394198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff39b92c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff394112770 con 0x7ff3941084f0 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff39b92c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941125f0 con 0x7ff394102e80 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff39b92c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941128f0 con 0x7ff3941024e0 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff399ec9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3941084f0 0x7ff394198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff399ec9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3941084f0 0x7ff394198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52594/0 (socket says 172.21.15.121:52594) 2024-06-10T00:28:44.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff399ec9700 1 -- 172.21.15.121:0/1174929644 learned_addr learned my addr 172.21.15.121:0/1174929644 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.303+0000 7ff3996c8700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3941024e0 0x7ff39419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff398ec7700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff394102e80 0x7ff39419ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff399ec9700 1 -- 172.21.15.121:0/1174929644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3941024e0 msgr2=0x7ff39419e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff399ec9700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3941024e0 0x7ff39419e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff399ec9700 1 -- 172.21.15.121:0/1174929644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff394102e80 msgr2=0x7ff39419ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff399ec9700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff394102e80 0x7ff39419ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff399ec9700 1 -- 172.21.15.121:0/1174929644 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3941990f0 con 0x7ff3941084f0 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff399ec9700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3941084f0 0x7ff394198990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00ada0 tx=0x7ff38c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff38a7fc700 1 -- 172.21.15.121:0/1174929644 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c017020 con 0x7ff3941084f0 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff38a7fc700 1 -- 172.21.15.121:0/1174929644 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff38c00a0d0 con 0x7ff3941084f0 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff39b92c700 1 -- 172.21.15.121:0/1174929644 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3941993e0 con 0x7ff3941084f0 2024-06-10T00:28:44.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff38a7fc700 1 -- 172.21.15.121:0/1174929644 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c005480 con 0x7ff3941084f0 2024-06-10T00:28:44.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.304+0000 7ff39b92c700 1 -- 172.21.15.121:0/1174929644 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3941a56a0 con 0x7ff3941084f0 2024-06-10T00:28:44.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.306+0000 7ff38a7fc700 1 -- 172.21.15.121:0/1174929644 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7ff38c005620 con 0x7ff3941084f0 2024-06-10T00:28:44.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.307+0000 7ff38a7fc700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3800a2e20 0x7ff3800a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.307+0000 7ff3996c8700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3800a2e20 0x7ff3800a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.307+0000 7ff38a7fc700 1 -- 172.21.15.121:0/1174929644 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff38c0cf0a0 con 0x7ff3941084f0 2024-06-10T00:28:44.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.307+0000 7ff3996c8700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3800a2e20 0x7ff3800a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff394199ad0 tx=0x7ff38400a380 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:44.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.308+0000 7ff3a61afb80 1 Processor -- start 2024-06-10T00:28:44.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.308+0000 7ff3a61afb80 1 -- start start 2024-06-10T00:28:44.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.308+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1d72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.308+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1d1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.308+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 0x5563dc1d1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.308+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5563dc0c3f10 con 0x5563dc1d7830 2024-06-10T00:28:44.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.308+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5563dc122800 con 0x5563dc1d6f10 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.308+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5563dc08a410 con 0x5563dc1d1750 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.308+0000 7ff398ec7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.308+0000 7ff398ec7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1d1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52610/0 (socket says 172.21.15.121:52610) 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.308+0000 7ff398ec7700 1 -- 172.21.15.121:0/1292116445 learned_addr learned my addr 172.21.15.121:0/1292116445 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.309+0000 7ff3996c8700 1 --2- 172.21.15.121:0/1292116445 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1d72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.309+0000 7ff398ec7700 1 -- 172.21.15.121:0/1292116445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 msgr2=0x5563dc1d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.309+0000 7ff398ec7700 1 --2- 172.21.15.121:0/1292116445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 0x5563dc1d1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.309+0000 7ff398ec7700 1 -- 172.21.15.121:0/1292116445 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 msgr2=0x5563dc1d72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.309+0000 7ff398ec7700 1 --2- 172.21.15.121:0/1292116445 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1d72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.309+0000 7ff398ec7700 1 -- 172.21.15.121:0/1292116445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563dc071880 con 0x5563dc1d7830 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.309+0000 7ff3996c8700 1 --2- 172.21.15.121:0/1292116445 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1d72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.309+0000 7ff398ec7700 1 --2- 172.21.15.121:0/1292116445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1d1210 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff390006010 tx=0x7ff390009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0781857b748995 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:44.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.309+0000 7ff37f7fe700 1 -- 172.21.15.121:0/1292116445 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff390027070 con 0x5563dc1d7830 2024-06-10T00:28:44.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.309+0000 7ff37f7fe700 1 -- 172.21.15.121:0/1292116445 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff390005550 con 0x5563dc1d7830 2024-06-10T00:28:44.311 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.309+0000 7ff37f7fe700 1 -- 172.21.15.121:0/1292116445 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39000bc30 con 0x5563dc1d7830 2024-06-10T00:28:44.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.310+0000 7ff3a61afb80 1 -- 172.21.15.121:0/1292116445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 msgr2=0x5563dc1d1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:44.314 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.310+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/1292116445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1d1210 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff390006010 tx=0x7ff390009ee0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.310+0000 7ff3a61afb80 1 -- 172.21.15.121:0/1292116445 shutdown_connections 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.310+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/1292116445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 0x5563dc1d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.310+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/1292116445 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1d72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.310+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/1292116445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1d1210 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.310+0000 7ff3a61afb80 1 -- 172.21.15.121:0/1292116445 >> 172.21.15.121:0/1292116445 conn(0x5563dc0fe030 msgr2=0x5563dc1cc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.310+0000 7ff3a61afb80 1 -- 172.21.15.121:0/1292116445 shutdown_connections 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.310+0000 7ff3a61afb80 1 -- 172.21.15.121:0/1292116445 wait complete. 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.310+0000 7ff3a61afb80 1 -- start start 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff3a61afb80 1 Processor -- start 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 0x5563dc1dd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1da920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1dae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5563dc0c2f70 con 0x5563dc1d7830 2024-06-10T00:28:44.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5563dc11e7c0 con 0x5563dc1d6f10 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5563dc1de1b0 con 0x5563dc1d1750 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff3a61afb80 10 client.0 _ll_register_callbacks cb 0x5563dc0b8560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff399ec9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1dae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff399ec9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1dae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52612/0 (socket says 172.21.15.121:52612) 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff399ec9700 1 -- 172.21.15.121:0/390139172 learned_addr learned my addr 172.21.15.121:0/390139172 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff3996c8700 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 0x5563dc1dd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff398ec7700 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1da920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff399ec9700 1 -- 172.21.15.121:0/390139172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 msgr2=0x5563dc1dd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff399ec9700 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 0x5563dc1dd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff399ec9700 1 -- 172.21.15.121:0/390139172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 msgr2=0x5563dc1da920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff399ec9700 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1da920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.311+0000 7ff399ec9700 1 -- 172.21.15.121:0/390139172 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563dc0e8700 con 0x5563dc1d7830 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.312+0000 7ff398ec7700 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1da920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:44.316 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.312+0000 7ff399ec9700 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1dae60 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00c730 tx=0x7ff38c09ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13875537e5e5733 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.312+0000 7ff37cff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.312+0000 7ff37cff9700 1 -- 172.21.15.121:0/390139172 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c09b150 con 0x5563dc1d7830 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.312+0000 7ff37cff9700 1 -- 172.21.15.121:0/390139172 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff38c09f160 con 0x5563dc1d7830 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.312+0000 7ff3a61afb80 1 -- 172.21.15.121:0/390139172 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff390005550 con 0x5563dc1d7830 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.312+0000 7ff37cff9700 1 -- 172.21.15.121:0/390139172 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff38c09f500 con 0x5563dc1d7830 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.312+0000 7ff3a61afb80 10 client.15504 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.312+0000 7ff3a61afb80 1 -- 172.21.15.121:0/390139172 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5563dc1e8a40 con 0x5563dc1d7830 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.312+0000 7ff37cff9700 1 -- 172.21.15.121:0/390139172 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff390005550 con 0x5563dc1d7830 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.312+0000 7ff37cff9700 1 -- 172.21.15.121:0/390139172 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7ff38c012070 con 0x5563dc1d7830 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.313+0000 7ff3a61afb80 10 client.15504 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.313+0000 7ff3a61afb80 10 client.15504 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.313+0000 7ff3a61afb80 20 client.15504 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.313+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5563dc1eafe0 0x5563dc1ec450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.313+0000 7ff3a61afb80 4 client.15504 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.313+0000 7ff3a61afb80 1 -- 172.21.15.121:0/390139172 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563dc12fd00 con 0x5563dc1eafe0 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.313+0000 7ff3996c8700 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5563dc1eafe0 0x5563dc1ec450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.313+0000 7ff3996c8700 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5563dc1eafe0 0x5563dc1ec450 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a1cc0c802f7d750 server_cookie=9b928dfbf70769f7 in_seq=0 out_seq=0 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.313+0000 7ff37cff9700 10 client.15504 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff37cff9700 1 -- 172.21.15.121:0/390139172 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5563dc12fd00 con 0x5563dc1eafe0 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff37cff9700 10 client.15504 handle_command_reply: tid=0 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff3a61afb80 1 client.15504 shutdown 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff3a61afb80 1 -- 172.21.15.121:0/390139172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 msgr2=0x5563dc1dae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1dae60 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00c730 tx=0x7ff38c09ad40 comp rx=0 tx=0).stop 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff3a61afb80 1 -- 172.21.15.121:0/390139172 shutdown_connections 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5563dc1eafe0 0x5563dc1ec450 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d7830 0x5563dc1dae60 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 0x5563dc1dd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/390139172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1da920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff3a61afb80 1 -- 172.21.15.121:0/390139172 >> 172.21.15.121:0/390139172 conn(0x5563dc0fe030 msgr2=0x5563dc1cb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:44.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff37cff9700 0 client.15504 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:44.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff3a61afb80 1 -- 172.21.15.121:0/390139172 shutdown_connections 2024-06-10T00:28:44.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.314+0000 7ff3a61afb80 1 -- 172.21.15.121:0/390139172 wait complete. 2024-06-10T00:28:44.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.315+0000 7ff3a61afb80 20 client.15504 trim_cache size 0 max 16384 2024-06-10T00:28:44.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff3a61afb80 1 Processor -- start 2024-06-10T00:28:44.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff3a61afb80 1 -- start start 2024-06-10T00:28:44.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 0x5563dc1d8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d6f10 0x5563dc1c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d7830 0x5563dc1d9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5563dc1ea790 con 0x5563dc1d6f10 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5563dc1eae60 con 0x5563dc1d7830 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5563dc1eb530 con 0x5563dc1d1750 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff398ec7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d6f10 0x5563dc1c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff398ec7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d6f10 0x5563dc1c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52614/0 (socket says 172.21.15.121:52614) 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff398ec7700 1 -- 172.21.15.121:0/2681676312 learned_addr learned my addr 172.21.15.121:0/2681676312 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff3996c8700 1 --2- 172.21.15.121:0/2681676312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 0x5563dc1d8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff399ec9700 1 --2- 172.21.15.121:0/2681676312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d7830 0x5563dc1d9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff398ec7700 1 -- 172.21.15.121:0/2681676312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 msgr2=0x5563dc1d8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff398ec7700 1 --2- 172.21.15.121:0/2681676312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 0x5563dc1d8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff398ec7700 1 -- 172.21.15.121:0/2681676312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d7830 msgr2=0x5563dc1d9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:44.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff398ec7700 1 --2- 172.21.15.121:0/2681676312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d7830 0x5563dc1d9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff398ec7700 1 -- 172.21.15.121:0/2681676312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563dc1ea270 con 0x5563dc1d6f10 2024-06-10T00:28:44.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.336+0000 7ff398ec7700 1 --2- 172.21.15.121:0/2681676312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d6f10 0x5563dc1c8320 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff3900098e0 tx=0x7ff390003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6145462390ec1cc8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:44.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff37effd700 1 -- 172.21.15.121:0/2681676312 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff390027050 con 0x5563dc1d6f10 2024-06-10T00:28:44.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff37effd700 1 -- 172.21.15.121:0/2681676312 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff390028040 con 0x5563dc1d6f10 2024-06-10T00:28:44.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff37effd700 1 -- 172.21.15.121:0/2681676312 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39000bc00 con 0x5563dc1d6f10 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2681676312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d6f10 msgr2=0x5563dc1c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/2681676312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d6f10 0x5563dc1c8320 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff3900098e0 tx=0x7ff390003af0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2681676312 shutdown_connections 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/2681676312 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d7830 0x5563dc1d9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/2681676312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d1750 0x5563dc1d8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/2681676312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d6f10 0x5563dc1c8320 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2681676312 >> 172.21.15.121:0/2681676312 conn(0x5563dc0fe030 msgr2=0x5563dc1d9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2681676312 shutdown_connections 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.337+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2681676312 wait complete. 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3a61afb80 1 -- start start 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3a61afb80 1 Processor -- start 2024-06-10T00:28:44.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d1750 0x5563dc1f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1cb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3a61afb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d7830 0x5563dc1cb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5563dc1ec2d0 con 0x5563dc1d1750 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5563dc1ebc00 con 0x5563dc1d6f10 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3a61afb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5563dc1ec9a0 con 0x5563dc1d7830 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3a61afb80 10 client.0 _ll_register_callbacks cb 0x5563dc0b8560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3996c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d1750 0x5563dc1f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3996c8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d1750 0x5563dc1f5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52626/0 (socket says 172.21.15.121:52626) 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.338+0000 7ff3996c8700 1 -- 172.21.15.121:0/2615455817 learned_addr learned my addr 172.21.15.121:0/2615455817 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff398ec7700 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff399ec9700 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d7830 0x5563dc1cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff3996c8700 1 -- 172.21.15.121:0/2615455817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d7830 msgr2=0x5563dc1cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:44.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff3996c8700 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d7830 0x5563dc1cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff3996c8700 1 -- 172.21.15.121:0/2615455817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 msgr2=0x5563dc1cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff3996c8700 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff3996c8700 1 -- 172.21.15.121:0/2615455817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563dc1cc090 con 0x5563dc1d1750 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff3996c8700 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d1750 0x5563dc1f5ff0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x5563dc1eccb0 tx=0x7ff384012070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1646448d656491e7 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff37cff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff37cff9700 1 -- 172.21.15.121:0/2615455817 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff384016070 con 0x5563dc1d1750 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff37cff9700 1 -- 172.21.15.121:0/2615455817 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff384003da0 con 0x5563dc1d1750 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2615455817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff390028040 con 0x5563dc1d1750 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff37cff9700 1 -- 172.21.15.121:0/2615455817 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff384036050 con 0x5563dc1d1750 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff3a61afb80 10 client.15510 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2615455817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5563dc2568d0 con 0x5563dc1d1750 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.339+0000 7ff37cff9700 1 -- 172.21.15.121:0/2615455817 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff390028040 con 0x5563dc1d1750 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.340+0000 7ff37cff9700 1 -- 172.21.15.121:0/2615455817 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7ff384019040 con 0x5563dc1d1750 2024-06-10T00:28:44.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.340+0000 7ff3a61afb80 10 client.15510 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.340+0000 7ff3a61afb80 10 client.15510 resolve_mds: resolved name 'b' to daemon mds.b 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.340+0000 7ff3a61afb80 20 client.15510 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.340+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5563dc25b4e0 0x5563dc25a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.340+0000 7ff3a61afb80 4 client.15510 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.340+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2615455817 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "status"}) -- 0x5563dc12a280 con 0x5563dc25b4e0 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.340+0000 7ff398ec7700 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5563dc25b4e0 0x5563dc25a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.340+0000 7ff37cff9700 10 client.15510 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.340+0000 7ff398ec7700 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5563dc25b4e0 0x5563dc25a1f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e0b8684952d5f1b server_cookie=ddd3afa8f217b5e1 in_seq=0 out_seq=0 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff37cff9700 1 -- 172.21.15.121:0/2615455817 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x5563dc12a280 con 0x5563dc25b4e0 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff37cff9700 10 client.15510 handle_command_reply: tid=0 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 1 client.15510 shutdown 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2615455817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d1750 msgr2=0x5563dc1f5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:44.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d1750 0x5563dc1f5ff0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x5563dc1eccb0 tx=0x7ff384012070 comp rx=0 tx=0).stop 2024-06-10T00:28:44.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2615455817 shutdown_connections 2024-06-10T00:28:44.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5563dc25b4e0 0x5563dc25a1f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563dc1d7830 0x5563dc1cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563dc1d1750 0x5563dc1f5ff0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 1 --2- 172.21.15.121:0/2615455817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5563dc1d6f10 0x5563dc1cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2615455817 >> 172.21.15.121:0/2615455817 conn(0x5563dc0fe030 msgr2=0x5563dc1d5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:44.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff37cff9700 0 client.15510 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:44.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2615455817 shutdown_connections 2024-06-10T00:28:44.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 1 -- 172.21.15.121:0/2615455817 wait complete. 2024-06-10T00:28:44.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.341+0000 7ff3a61afb80 20 client.15510 trim_cache size 0 max 16384 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "cluster_fsid": "6c432244-26be-11ef-bc9b-c7b262605968", 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "whoami": 0, 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 24479, 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "want_state": "up:active", 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "state": "up:active", 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs", 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "rank_uptime": 284.795267728, 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "mdsmap_epoch": 26, 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch": 101, 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_epoch_barrier": 100, 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "uptime": 303.08925976400002 2024-06-10T00:28:44.347 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:28:44.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 -- 172.21.15.121:0/1174929644 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3800a2e20 msgr2=0x7ff3800a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:44.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3800a2e20 0x7ff3800a52e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff394199ad0 tx=0x7ff38400a380 comp rx=0 tx=0).stop 2024-06-10T00:28:44.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 -- 172.21.15.121:0/1174929644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3941084f0 msgr2=0x7ff394198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:44.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3941084f0 0x7ff394198990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00ada0 tx=0x7ff38c00c330 comp rx=0 tx=0).stop 2024-06-10T00:28:44.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 -- 172.21.15.121:0/1174929644 shutdown_connections 2024-06-10T00:28:44.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3800a2e20 0x7ff3800a52e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3941084f0 0x7ff394198990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3941024e0 0x7ff39419e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 --2- 172.21.15.121:0/1174929644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff394102e80 0x7ff39419ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:44.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 -- 172.21.15.121:0/1174929644 >> 172.21.15.121:0/1174929644 conn(0x7ff394079530 msgr2=0x7ff394100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:44.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 -- 172.21.15.121:0/1174929644 shutdown_connections 2024-06-10T00:28:44.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:44.342+0000 7ff39b92c700 1 -- 172.21.15.121:0/1174929644 wait complete. 2024-06-10T00:28:44.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:44 smithi121 ceph-mon[32378]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:28:44.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:44 smithi193 ceph-mon[36716]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:28:44.985 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-06-10T00:28:45.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:44 smithi135 ceph-mon[36424]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:28:46.864 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:28:46.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:46 smithi193 ceph-mon[36716]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:28:46.886 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:46 smithi121 ceph-mon[32378]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:28:47.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:46 smithi135 ceph-mon[36424]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:28:47.115 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:48.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:48 smithi193 ceph-mon[36716]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:28:48.903 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:48 smithi121 ceph-mon[32378]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:28:49.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:48 smithi135 ceph-mon[36424]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:28:49.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.012+0000 7f5ce227b700 1 -- 172.21.15.121:0/1522052940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cdc106750 msgr2=0x7f5cdc10e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:49.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.012+0000 7f5ce227b700 1 --2- 172.21.15.121:0/1522052940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cdc106750 0x7f5cdc10e2d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400b0d0 tx=0x7f5cd400b3e0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.012+0000 7f5ce227b700 1 -- 172.21.15.121:0/1522052940 shutdown_connections 2024-06-10T00:28:49.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.012+0000 7f5ce227b700 1 --2- 172.21.15.121:0/1522052940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cdc106750 0x7f5cdc10e2d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.012+0000 7f5ce227b700 1 --2- 172.21.15.121:0/1522052940 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc101df0 0x7f5cdc1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.012+0000 7f5ce227b700 1 --2- 172.21.15.121:0/1522052940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cdc1027a0 0x7f5cdc106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.012+0000 7f5ce227b700 1 -- 172.21.15.121:0/1522052940 >> 172.21.15.121:0/1522052940 conn(0x7f5cdc077a00 msgr2=0x7f5cdc0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:49.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.012+0000 7f5ce227b700 1 -- 172.21.15.121:0/1522052940 shutdown_connections 2024-06-10T00:28:49.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.012+0000 7f5ce227b700 1 -- 172.21.15.121:0/1522052940 wait complete. 2024-06-10T00:28:49.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5ce227b700 1 Processor -- start 2024-06-10T00:28:49.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5ce227b700 1 -- start start 2024-06-10T00:28:49.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5ce227b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cdc101df0 0x7f5cdc199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5ce227b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cdc1027a0 0x7f5cdc199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5ce227b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc106750 0x7f5cdc1a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5ce227b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1116e0 con 0x7f5cdc1027a0 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5ce227b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc111560 con 0x7f5cdc106750 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5ce227b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc111860 con 0x7f5cdc101df0 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5cd3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cdc1027a0 0x7f5cdc199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5cd3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cdc1027a0 0x7f5cdc199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58344/0 (socket says 172.21.15.121:58344) 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5cd3fff700 1 -- 172.21.15.121:0/3469017031 learned_addr learned my addr 172.21.15.121:0/3469017031 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5ce0818700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc106750 0x7f5cdc1a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cdc101df0 0x7f5cdc199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5cd3fff700 1 -- 172.21.15.121:0/3469017031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cdc101df0 msgr2=0x7f5cdc199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5cd3fff700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cdc101df0 0x7f5cdc199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5cd3fff700 1 -- 172.21.15.121:0/3469017031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc106750 msgr2=0x7f5cdc1a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5cd3fff700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc106750 0x7f5cdc1a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5cd3fff700 1 -- 172.21.15.121:0/3469017031 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cdc1a4fe0 con 0x7f5cdc1027a0 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.013+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cdc101df0 0x7f5cdc199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:49.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.014+0000 7f5cd3fff700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cdc1027a0 0x7f5cdc199f60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400b530 tx=0x7f5cc400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:49.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.014+0000 7f5cd9ffb700 1 -- 172.21.15.121:0/3469017031 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4015020 con 0x7f5cdc1027a0 2024-06-10T00:28:49.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.014+0000 7f5cd9ffb700 1 -- 172.21.15.121:0/3469017031 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cc4011d10 con 0x7f5cdc1027a0 2024-06-10T00:28:49.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.014+0000 7f5ce227b700 1 -- 172.21.15.121:0/3469017031 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cdc1a5210 con 0x7f5cdc1027a0 2024-06-10T00:28:49.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.014+0000 7f5cd9ffb700 1 -- 172.21.15.121:0/3469017031 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc400f2e0 con 0x7f5cdc1027a0 2024-06-10T00:28:49.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.014+0000 7f5ce227b700 1 -- 172.21.15.121:0/3469017031 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cdc1a5720 con 0x7f5cdc1027a0 2024-06-10T00:28:49.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.016+0000 7f5cd9ffb700 1 -- 172.21.15.121:0/3469017031 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f5cc4018020 con 0x7f5cdc1027a0 2024-06-10T00:28:49.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.017+0000 7f5cd9ffb700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5cc80a2e20 0x7f5cc80a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.017+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5cc80a2e20 0x7f5cc80a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.017+0000 7f5cd9ffb700 1 -- 172.21.15.121:0/3469017031 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f5cc40cf0c0 con 0x7f5cdc1027a0 2024-06-10T00:28:49.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.017+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5cc80a2e20 0x7f5cc80a52e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc005fd0 tx=0x7f5ccc005ee0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5cecafeb80 1 Processor -- start 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5cecafeb80 1 -- start start 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11b850 0x55d39e11bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e120010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 0x55d39e120990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d39e009f60 con 0x55d39e120550 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d39e068850 con 0x55d39e11c170 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d39dfd0460 con 0x55d39e11b850 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5ce0818700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 0x55d39e120990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5ce0818700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 0x55d39e120990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58358/0 (socket says 172.21.15.121:58358) 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5ce0818700 1 -- 172.21.15.121:0/3693856479 learned_addr learned my addr 172.21.15.121:0/3693856479 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.018+0000 7f5cd3fff700 1 --2- 172.21.15.121:0/3693856479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e120010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.019+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/3693856479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11b850 0x55d39e11bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.019+0000 7f5ce0818700 1 -- 172.21.15.121:0/3693856479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11b850 msgr2=0x55d39e11bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:49.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.019+0000 7f5ce0818700 1 --2- 172.21.15.121:0/3693856479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11b850 0x55d39e11bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.019+0000 7f5ce0818700 1 -- 172.21.15.121:0/3693856479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 msgr2=0x55d39e120010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:49.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.019+0000 7f5ce0818700 1 --2- 172.21.15.121:0/3693856479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e120010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.019+0000 7f5ce0818700 1 -- 172.21.15.121:0/3693856479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d39dfb78d0 con 0x55d39e120550 2024-06-10T00:28:49.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.019+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/3693856479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11b850 0x55d39e11bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:49.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.019+0000 7f5ce0818700 1 --2- 172.21.15.121:0/3693856479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 0x55d39e120990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009e40 tx=0x7f5cd4006210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e5d9f67efde0a8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:49.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.019+0000 7f5cd27fc700 1 -- 172.21.15.121:0/3693856479 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd40093d0 con 0x55d39e120550 2024-06-10T00:28:49.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.019+0000 7f5cd27fc700 1 -- 172.21.15.121:0/3693856479 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cd40074e0 con 0x55d39e120550 2024-06-10T00:28:49.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.019+0000 7f5cd27fc700 1 -- 172.21.15.121:0/3693856479 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4005240 con 0x55d39e120550 2024-06-10T00:28:49.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.020+0000 7f5cecafeb80 1 -- 172.21.15.121:0/3693856479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 msgr2=0x55d39e120990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.020+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/3693856479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 0x55d39e120990 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009e40 tx=0x7f5cd4006210 comp rx=0 tx=0).stop 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.020+0000 7f5cecafeb80 1 -- 172.21.15.121:0/3693856479 shutdown_connections 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.020+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/3693856479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 0x55d39e120990 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.020+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/3693856479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11b850 0x55d39e11bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.020+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/3693856479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e120010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.020+0000 7f5cecafeb80 1 -- 172.21.15.121:0/3693856479 >> 172.21.15.121:0/3693856479 conn(0x55d39e044080 msgr2=0x55d39e112220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.020+0000 7f5cecafeb80 1 -- 172.21.15.121:0/3693856479 shutdown_connections 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.020+0000 7f5cecafeb80 1 -- 172.21.15.121:0/3693856479 wait complete. 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.020+0000 7f5cecafeb80 1 -- start start 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cecafeb80 1 Processor -- start 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e127aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e124da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e120550 0x55d39e1252e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d39e008fc0 con 0x55d39e11b850 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d39e064810 con 0x55d39e11c170 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d39e128620 con 0x55d39e120550 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cdbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e127aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cecafeb80 10 client.0 _ll_register_callbacks cb 0x55d39dffe5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cdbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e127aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58374/0 (socket says 172.21.15.121:58374) 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cdbfff700 1 -- 172.21.15.121:0/1434022435 learned_addr learned my addr 172.21.15.121:0/1434022435 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cd3fff700 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e124da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cdbfff700 1 -- 172.21.15.121:0/1434022435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e120550 msgr2=0x55d39e1252e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e120550 0x55d39e1252e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cdbfff700 1 -- 172.21.15.121:0/1434022435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 msgr2=0x55d39e124da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e124da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.021+0000 7f5cdbfff700 1 -- 172.21.15.121:0/1434022435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d39e02e750 con 0x55d39e11b850 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e127aa0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc011770 tx=0x7f5ccc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7466ec60532074 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cbb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/1434022435 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc023070 con 0x55d39e11b850 2024-06-10T00:28:49.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/1434022435 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f5ccc01f830 con 0x55d39e11b850 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cecafeb80 1 -- 172.21.15.121:0/1434022435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cd40074e0 con 0x55d39e11b850 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/1434022435 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5ccc01fa30 con 0x55d39e11b850 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cecafeb80 10 client.15522 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cecafeb80 1 -- 172.21.15.121:0/1434022435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d39e132de0 con 0x55d39e11b850 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/1434022435 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cd40074e0 con 0x55d39e11b850 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/1434022435 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f5ccc00f040 con 0x55d39e11b850 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cecafeb80 10 client.15522 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cecafeb80 10 client.15522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cecafeb80 20 client.15522 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d39e1353c0 0x55d39e1377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.022+0000 7f5cecafeb80 4 client.15522 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.023+0000 7f5cecafeb80 1 -- 172.21.15.121:0/1434022435 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d39e075d50 con 0x55d39e1353c0 2024-06-10T00:28:49.026 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.023+0000 7f5cd3fff700 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d39e1353c0 0x55d39e1377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.023+0000 7f5cbb7fe700 10 client.15522 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.023+0000 7f5cd3fff700 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d39e1353c0 0x55d39e1377b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d13ef75fb4ef65d server_cookie=bf77e66db395b763 in_seq=0 out_seq=0 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/1434022435 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d39e075d50 con 0x55d39e1353c0 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cbb7fe700 10 client.15522 handle_command_reply: tid=0 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cecafeb80 1 client.15522 shutdown 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cecafeb80 1 -- 172.21.15.121:0/1434022435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 msgr2=0x55d39e127aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e127aa0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc011770 tx=0x7f5ccc0047c0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cecafeb80 1 -- 172.21.15.121:0/1434022435 shutdown_connections 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d39e1353c0 0x55d39e1377b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e120550 0x55d39e1252e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e127aa0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cbb7fe700 0 client.15522 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/1434022435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e124da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.024+0000 7f5cecafeb80 1 -- 172.21.15.121:0/1434022435 >> 172.21.15.121:0/1434022435 conn(0x55d39e044080 msgr2=0x55d39e121a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:49.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.025+0000 7f5cecafeb80 1 -- 172.21.15.121:0/1434022435 shutdown_connections 2024-06-10T00:28:49.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.025+0000 7f5cecafeb80 1 -- 172.21.15.121:0/1434022435 wait complete. 2024-06-10T00:28:49.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.025+0000 7f5cecafeb80 20 client.15522 trim_cache size 0 max 16384 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.046+0000 7f5cecafeb80 1 Processor -- start 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.046+0000 7f5cecafeb80 1 -- start start 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.046+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e126ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.046+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e13d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.046+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e120550 0x55d39e13d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.046+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1aebd0 con 0x55d39e11b850 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.046+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1ac460 con 0x55d39e11c170 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.046+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1aa2b0 con 0x55d39e120550 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.046+0000 7f5cdbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e126ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cdbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e126ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58380/0 (socket says 172.21.15.121:58380) 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cdbfff700 1 -- 172.21.15.121:0/3268191904 learned_addr learned my addr 172.21.15.121:0/3268191904 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:49.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cd3fff700 1 --2- 172.21.15.121:0/3268191904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e13d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cdbfff700 1 -- 172.21.15.121:0/3268191904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e120550 msgr2=0x55d39e13d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/3268191904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e120550 0x55d39e13d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cdbfff700 1 -- 172.21.15.121:0/3268191904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 msgr2=0x55d39e13d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/3268191904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e13d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cdbfff700 1 -- 172.21.15.121:0/3268191904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d39e125cd0 con 0x55d39e11b850 2024-06-10T00:28:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cd3fff700 1 --2- 172.21.15.121:0/3268191904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e13d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/3268191904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e126ce0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc005410 tx=0x7f5ccc011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac1b9d4de1813ab server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cd1ffb700 1 -- 172.21.15.121:0/3268191904 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc038070 con 0x55d39e11b850 2024-06-10T00:28:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cd1ffb700 1 -- 172.21.15.121:0/3268191904 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ccc017040 con 0x55d39e11b850 2024-06-10T00:28:49.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.047+0000 7f5cd1ffb700 1 -- 172.21.15.121:0/3268191904 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc023040 con 0x55d39e11b850 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.048+0000 7f5cecafeb80 1 -- 172.21.15.121:0/3268191904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 msgr2=0x55d39e126ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.048+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/3268191904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e126ce0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc005410 tx=0x7f5ccc011e10 comp rx=0 tx=0).stop 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.048+0000 7f5cecafeb80 1 -- 172.21.15.121:0/3268191904 shutdown_connections 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.048+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/3268191904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e120550 0x55d39e13d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.048+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/3268191904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e11b850 0x55d39e126ce0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.048+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/3268191904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11c170 0x55d39e13d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.048+0000 7f5cecafeb80 1 -- 172.21.15.121:0/3268191904 >> 172.21.15.121:0/3268191904 conn(0x55d39e044080 msgr2=0x55d39e1265d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.048+0000 7f5cecafeb80 1 -- 172.21.15.121:0/3268191904 shutdown_connections 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.048+0000 7f5cecafeb80 1 -- 172.21.15.121:0/3268191904 wait complete. 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.048+0000 7f5cecafeb80 1 -- start start 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cecafeb80 1 Processor -- start 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11b850 0x55d39e12dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11c170 0x55d39e12e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cecafeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 0x55d39e12a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1aa920 con 0x55d39e120550 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1abd90 con 0x55d39e11b850 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cecafeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1aaff0 con 0x55d39e11c170 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cecafeb80 10 client.0 _ll_register_callbacks cb 0x55d39dffe5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cd3fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11c170 0x55d39e12e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cd3fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11c170 0x55d39e12e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:54176/0 (socket says 172.21.15.121:54176) 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cd3fff700 1 -- 172.21.15.121:0/2374920330 learned_addr learned my addr 172.21.15.121:0/2374920330 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11b850 0x55d39e12dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.049+0000 7f5ce0818700 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 0x55d39e12a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cdbfff700 1 -- 172.21.15.121:0/2374920330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11c170 msgr2=0x55d39e12e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11c170 0x55d39e12e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cdbfff700 1 -- 172.21.15.121:0/2374920330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 msgr2=0x55d39e12a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 0x55d39e12a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cdbfff700 1 -- 172.21.15.121:0/2374920330 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d39e12aa60 con 0x55d39e11b850 2024-06-10T00:28:49.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5ce0818700 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 0x55d39e12a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cdbfff700 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11b850 0x55d39e12dc40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc000c00 tx=0x7f5ccc01fde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da75cd3c6e16b990 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cbb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/2374920330 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc038030 con 0x55d39e11b850 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cecafeb80 1 -- 172.21.15.121:0/2374920330 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ccc017040 con 0x55d39e11b850 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/2374920330 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f5ccc005100 con 0x55d39e11b850 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/2374920330 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5ccc023040 con 0x55d39e11b850 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cecafeb80 10 client.25148 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cecafeb80 1 -- 172.21.15.121:0/2374920330 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d39e1a0c60 con 0x55d39e11b850 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.050+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/2374920330 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ccc017040 con 0x55d39e11b850 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.051+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/2374920330 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f5ccc03ea00 con 0x55d39e11b850 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.051+0000 7f5cecafeb80 10 client.25148 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.051+0000 7f5cecafeb80 10 client.25148 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.051+0000 7f5cecafeb80 20 client.25148 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:49.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.051+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d39e1a10c0 0x55d39e1a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:49.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.051+0000 7f5cecafeb80 4 client.25148 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:28:49.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.051+0000 7f5cecafeb80 1 -- 172.21.15.121:0/2374920330 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d39e0702d0 con 0x55d39e1a10c0 2024-06-10T00:28:49.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.051+0000 7f5cd3fff700 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d39e1a10c0 0x55d39e1a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:49.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.051+0000 7f5cbb7fe700 10 client.25148 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:49.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.051+0000 7f5cd3fff700 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d39e1a10c0 0x55d39e1a34b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=373d785eec5e246c server_cookie=27a19b70a7422bf8 in_seq=0 out_seq=0 2024-06-10T00:28:49.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cbb7fe700 1 -- 172.21.15.121:0/2374920330 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d39e0702d0 con 0x55d39e1a10c0 2024-06-10T00:28:49.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cbb7fe700 10 client.25148 handle_command_reply: tid=0 2024-06-10T00:28:49.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cecafeb80 1 client.25148 shutdown 2024-06-10T00:28:49.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cecafeb80 1 -- 172.21.15.121:0/2374920330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11b850 msgr2=0x55d39e12dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:49.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11b850 0x55d39e12dc40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc000c00 tx=0x7f5ccc01fde0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cecafeb80 1 -- 172.21.15.121:0/2374920330 shutdown_connections 2024-06-10T00:28:49.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d39e1a10c0 0x55d39e1a34b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d39e120550 0x55d39e12a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d39e11b850 0x55d39e12dc40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cecafeb80 1 --2- 172.21.15.121:0/2374920330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d39e11c170 0x55d39e12e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cbb7fe700 0 client.25148 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:28:49.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.052+0000 7f5cecafeb80 1 -- 172.21.15.121:0/2374920330 >> 172.21.15.121:0/2374920330 conn(0x55d39e044080 msgr2=0x55d39e11f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:49.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5cecafeb80 1 -- 172.21.15.121:0/2374920330 shutdown_connections 2024-06-10T00:28:49.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5cecafeb80 1 -- 172.21.15.121:0/2374920330 wait complete. 2024-06-10T00:28:49.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5cecafeb80 20 client.25148 trim_cache size 0 max 16384 2024-06-10T00:28:49.057 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:28:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "no active scrubs running", 2024-06-10T00:28:49.057 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": {} 2024-06-10T00:28:49.057 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:28:49.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5ce227b700 1 -- 172.21.15.121:0/3469017031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5cc80a2e20 msgr2=0x7f5cc80a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:49.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5ce227b700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5cc80a2e20 0x7f5cc80a52e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc005fd0 tx=0x7f5ccc005ee0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5ce227b700 1 -- 172.21.15.121:0/3469017031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cdc1027a0 msgr2=0x7f5cdc199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:49.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5ce227b700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cdc1027a0 0x7f5cdc199f60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400b530 tx=0x7f5cc400b840 comp rx=0 tx=0).stop 2024-06-10T00:28:49.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5ce227b700 1 -- 172.21.15.121:0/3469017031 shutdown_connections 2024-06-10T00:28:49.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5ce227b700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5cc80a2e20 0x7f5cc80a52e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5ce227b700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5cdc106750 0x7f5cdc1a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5ce227b700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5cdc101df0 0x7f5cdc199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:49.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5ce227b700 1 --2- 172.21.15.121:0/3469017031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5cdc1027a0 0x7f5cdc199f60 secure :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400b530 tx=0x7f5cc400b840 comp rx=0 tx=0).stop 2024-06-10T00:28:49.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.053+0000 7f5ce227b700 1 -- 172.21.15.121:0/3469017031 >> 172.21.15.121:0/3469017031 conn(0x7f5cdc077a00 msgr2=0x7f5cdc104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:49.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.054+0000 7f5ce227b700 1 -- 172.21.15.121:0/3469017031 shutdown_connections 2024-06-10T00:28:49.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:49.054+0000 7f5ce227b700 1 -- 172.21.15.121:0/3469017031 wait complete. 2024-06-10T00:28:49.736 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-10T00:28:49.736 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:28:50.031 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:50.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:50 smithi121 ceph-mon[32378]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:28:50.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:50 smithi193 ceph-mon[36716]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:28:51.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:50 smithi135 ceph-mon[36424]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:28:51.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.847+0000 7f4e24ebe700 1 -- 172.21.15.121:0/1193460503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e20101390 msgr2=0x7f4e20101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:51.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.847+0000 7f4e24ebe700 1 --2- 172.21.15.121:0/1193460503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e20101390 0x7f4e20101810 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009990 tx=0x7f4e14009ca0 comp rx=0 tx=0).stop 2024-06-10T00:28:51.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.847+0000 7f4e24ebe700 1 -- 172.21.15.121:0/1193460503 shutdown_connections 2024-06-10T00:28:51.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.847+0000 7f4e24ebe700 1 --2- 172.21.15.121:0/1193460503 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e20101d50 0x7f4e2010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:51.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.847+0000 7f4e24ebe700 1 --2- 172.21.15.121:0/1193460503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e20107400 0x7f4e201077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:51.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.847+0000 7f4e24ebe700 1 --2- 172.21.15.121:0/1193460503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e20101390 0x7f4e20101810 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:51.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.847+0000 7f4e24ebe700 1 -- 172.21.15.121:0/1193460503 >> 172.21.15.121:0/1193460503 conn(0x7f4e20078340 msgr2=0x7f4e20078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:51.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.847+0000 7f4e24ebe700 1 -- 172.21.15.121:0/1193460503 shutdown_connections 2024-06-10T00:28:51.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.847+0000 7f4e24ebe700 1 -- 172.21.15.121:0/1193460503 wait complete. 2024-06-10T00:28:51.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e24ebe700 1 Processor -- start 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e24ebe700 1 -- start start 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e24ebe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e20101390 0x7f4e2019e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e24ebe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e20101d50 0x7f4e2019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e24ebe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e20107400 0x7f4e20198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e24ebe700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4e20111690 con 0x7f4e20101390 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e24ebe700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4e20111510 con 0x7f4e20101d50 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e24ebe700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4e20111810 con 0x7f4e20107400 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e1e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e20101390 0x7f4e2019e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e1e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e20101390 0x7f4e2019e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58396/0 (socket says 172.21.15.121:58396) 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e1e59c700 1 -- 172.21.15.121:0/3995042013 learned_addr learned my addr 172.21.15.121:0/3995042013 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e1dd9b700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e20101d50 0x7f4e2019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e1ed9d700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e20107400 0x7f4e20198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:51.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e1e59c700 1 -- 172.21.15.121:0/3995042013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e20107400 msgr2=0x7f4e20198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:51.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e1e59c700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e20107400 0x7f4e20198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:51.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e1e59c700 1 -- 172.21.15.121:0/3995042013 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e20101d50 msgr2=0x7f4e2019edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:51.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.848+0000 7f4e1e59c700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e20101d50 0x7f4e2019edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:51.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.849+0000 7f4e1e59c700 1 -- 172.21.15.121:0/3995042013 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e201991f0 con 0x7f4e20101390 2024-06-10T00:28:51.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.849+0000 7f4e1e59c700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e20101390 0x7f4e2019e880 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000d980 tx=0x7f4e1000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:51.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.849+0000 7f4e0f7fe700 1 -- 172.21.15.121:0/3995042013 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10017070 con 0x7f4e20101390 2024-06-10T00:28:51.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.849+0000 7f4e24ebe700 1 -- 172.21.15.121:0/3995042013 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e201994e0 con 0x7f4e20101390 2024-06-10T00:28:51.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.849+0000 7f4e0f7fe700 1 -- 172.21.15.121:0/3995042013 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e10005020 con 0x7f4e20101390 2024-06-10T00:28:51.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.849+0000 7f4e0f7fe700 1 -- 172.21.15.121:0/3995042013 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1000fbf0 con 0x7f4e20101390 2024-06-10T00:28:51.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.849+0000 7f4e24ebe700 1 -- 172.21.15.121:0/3995042013 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e201a5750 con 0x7f4e20101390 2024-06-10T00:28:51.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.850+0000 7f4e24ebe700 1 -- 172.21.15.121:0/3995042013 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e20069180 con 0x7f4e20101390 2024-06-10T00:28:51.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.851+0000 7f4e0f7fe700 1 -- 172.21.15.121:0/3995042013 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f4e100051d0 con 0x7f4e20101390 2024-06-10T00:28:51.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.852+0000 7f4e0f7fe700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4e080a2e20 0x7f4e080a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:51.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.852+0000 7f4e1dd9b700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4e080a2e20 0x7f4e080a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:51.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.852+0000 7f4e0f7fe700 1 -- 172.21.15.121:0/3995042013 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f4e100cf770 con 0x7f4e20101390 2024-06-10T00:28:51.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.853+0000 7f4e1dd9b700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4e080a2e20 0x7f4e080a52e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400b5c0 tx=0x7f4e14005960 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:51.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:51.857+0000 7f4e0f7fe700 1 -- 172.21.15.121:0/3995042013 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4e10020020 con 0x7f4e20101390 2024-06-10T00:28:52.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.134+0000 7f4e24ebe700 1 -- 172.21.15.121:0/3995042013 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e201a5920 con 0x7f4e20101390 2024-06-10T00:28:52.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.135+0000 7f4e0f7fe700 1 -- 172.21.15.121:0/3995042013 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f4e1009be90 con 0x7f4e20101390 2024-06-10T00:28:52.136 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:28:52.136 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:27:55.370804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:28:52.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.137+0000 7f4e24ebe700 1 -- 172.21.15.121:0/3995042013 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4e080a2e20 msgr2=0x7f4e080a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:52.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.137+0000 7f4e24ebe700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4e080a2e20 0x7f4e080a52e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400b5c0 tx=0x7f4e14005960 comp rx=0 tx=0).stop 2024-06-10T00:28:52.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.137+0000 7f4e24ebe700 1 -- 172.21.15.121:0/3995042013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e20101390 msgr2=0x7f4e2019e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:52.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.137+0000 7f4e24ebe700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e20101390 0x7f4e2019e880 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000d980 tx=0x7f4e1000dc90 comp rx=0 tx=0).stop 2024-06-10T00:28:52.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.137+0000 7f4e24ebe700 1 -- 172.21.15.121:0/3995042013 shutdown_connections 2024-06-10T00:28:52.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.137+0000 7f4e24ebe700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4e080a2e20 0x7f4e080a52e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:52.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.137+0000 7f4e24ebe700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e20107400 0x7f4e20198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:52.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.137+0000 7f4e24ebe700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e20101390 0x7f4e2019e880 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:52.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.137+0000 7f4e24ebe700 1 --2- 172.21.15.121:0/3995042013 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4e20101d50 0x7f4e2019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:52.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.137+0000 7f4e24ebe700 1 -- 172.21.15.121:0/3995042013 >> 172.21.15.121:0/3995042013 conn(0x7f4e20078340 msgr2=0x7f4e20100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:52.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.138+0000 7f4e24ebe700 1 -- 172.21.15.121:0/3995042013 shutdown_connections 2024-06-10T00:28:52.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:52.138+0000 7f4e24ebe700 1 -- 172.21.15.121:0/3995042013 wait complete. 2024-06-10T00:28:52.143 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-06-10T00:28:52.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:52 smithi193 ceph-mon[36716]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:52.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:52 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3995042013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:52.938 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:52 smithi121 ceph-mon[32378]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:52.938 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:52 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3995042013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:52.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-10T00:28:53.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:52 smithi135 ceph-mon[36424]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:52 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3995042013' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:28:53.214 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:54.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:54 smithi121 ceph-mon[32378]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:28:54.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:54 smithi193 ceph-mon[36716]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:28:55.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:54 smithi135 ceph-mon[36424]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:28:55.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.043+0000 7ff92dda6700 1 -- 172.21.15.121:0/4166303974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff928102380 msgr2=0x7ff928105ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:55.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.043+0000 7ff92dda6700 1 --2- 172.21.15.121:0/4166303974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff928102380 0x7ff928105ff0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0099c0 tx=0x7ff91c009cd0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.043+0000 7ff92dda6700 1 -- 172.21.15.121:0/4166303974 shutdown_connections 2024-06-10T00:28:55.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.043+0000 7ff92dda6700 1 --2- 172.21.15.121:0/4166303974 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff928106720 0x7ff92810e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.043+0000 7ff92dda6700 1 --2- 172.21.15.121:0/4166303974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9281019d0 0x7ff928101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.043+0000 7ff92dda6700 1 --2- 172.21.15.121:0/4166303974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff928102380 0x7ff928105ff0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.043+0000 7ff92dda6700 1 -- 172.21.15.121:0/4166303974 >> 172.21.15.121:0/4166303974 conn(0x7ff928078340 msgr2=0x7ff928078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:55.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.043+0000 7ff92dda6700 1 -- 172.21.15.121:0/4166303974 shutdown_connections 2024-06-10T00:28:55.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.043+0000 7ff92dda6700 1 -- 172.21.15.121:0/4166303974 wait complete. 2024-06-10T00:28:55.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff92dda6700 1 Processor -- start 2024-06-10T00:28:55.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff92dda6700 1 -- start start 2024-06-10T00:28:55.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff92dda6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9281019d0 0x7ff928103090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff92dda6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff928102380 0x7ff9281035d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff92dda6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff928106720 0x7ff928103b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff92dda6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281116b0 con 0x7ff928106720 2024-06-10T00:28:55.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff92dda6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff928111530 con 0x7ff9281019d0 2024-06-10T00:28:55.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff92dda6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff928111830 con 0x7ff928102380 2024-06-10T00:28:55.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff927fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff928106720 0x7ff928103b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff9277fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9281019d0 0x7ff928103090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff927fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff928106720 0x7ff928103b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58414/0 (socket says 172.21.15.121:58414) 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff926ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff928102380 0x7ff9281035d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.044+0000 7ff927fff700 1 -- 172.21.15.121:0/649909097 learned_addr learned my addr 172.21.15.121:0/649909097 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff927fff700 1 -- 172.21.15.121:0/649909097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff928102380 msgr2=0x7ff9281035d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff927fff700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff928102380 0x7ff9281035d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff927fff700 1 -- 172.21.15.121:0/649909097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9281019d0 msgr2=0x7ff928103090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff927fff700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9281019d0 0x7ff928103090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff927fff700 1 -- 172.21.15.121:0/649909097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9281a7200 con 0x7ff928106720 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff9277fe700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9281019d0 0x7ff928103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff927fff700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff928106720 0x7ff928103b10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff920007ee0 tx=0x7ff92000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff924ff9700 1 -- 172.21.15.121:0/649909097 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920004bb0 con 0x7ff928106720 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff92dda6700 1 -- 172.21.15.121:0/649909097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9281a74f0 con 0x7ff928106720 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff924ff9700 1 -- 172.21.15.121:0/649909097 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff920004d50 con 0x7ff928106720 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff924ff9700 1 -- 172.21.15.121:0/649909097 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920005960 con 0x7ff928106720 2024-06-10T00:28:55.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.045+0000 7ff92dda6700 1 -- 172.21.15.121:0/649909097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9281a7a00 con 0x7ff928106720 2024-06-10T00:28:55.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.048+0000 7ff924ff9700 1 -- 172.21.15.121:0/649909097 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7ff92001a020 con 0x7ff928106720 2024-06-10T00:28:55.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff924ff9700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff9100a2e20 0x7ff9100a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff9277fe700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff9100a2e20 0x7ff9100a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff924ff9700 1 -- 172.21.15.121:0/649909097 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff9200cebe0 con 0x7ff928106720 2024-06-10T00:28:55.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff938629b80 1 Processor -- start 2024-06-10T00:28:55.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff938629b80 1 -- start start 2024-06-10T00:28:55.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff9277fe700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff9100a2e20 0x7ff9100a52e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff918011490 tx=0x7ff918009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:55.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff938629b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff938629b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c7f60 0x55922e8c19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff938629b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c1f20 0x55922e8c2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff938629b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55922e813810 con 0x55922e8c7640 2024-06-10T00:28:55.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff938629b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55922e7bb720 con 0x55922e8c7f60 2024-06-10T00:28:55.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.049+0000 7ff938629b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55922e80f7d0 con 0x55922e8c1f20 2024-06-10T00:28:55.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff9277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff9277fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8c7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58424/0 (socket says 172.21.15.121:58424) 2024-06-10T00:28:55.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff9277fe700 1 -- 172.21.15.121:0/2578687385 learned_addr learned my addr 172.21.15.121:0/2578687385 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:55.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff926ffd700 1 --2- 172.21.15.121:0/2578687385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c7f60 0x55922e8c19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff9277fe700 1 -- 172.21.15.121:0/2578687385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c1f20 msgr2=0x55922e8c2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:55.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff9277fe700 1 --2- 172.21.15.121:0/2578687385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c1f20 0x55922e8c2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff9277fe700 1 -- 172.21.15.121:0/2578687385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c7f60 msgr2=0x55922e8c19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:55.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff9277fe700 1 --2- 172.21.15.121:0/2578687385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c7f60 0x55922e8c19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff9277fe700 1 -- 172.21.15.121:0/2578687385 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55922e7b1f50 con 0x55922e8c7640 2024-06-10T00:28:55.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff9277fe700 1 --2- 172.21.15.121:0/2578687385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8c7a20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff9180043b0 tx=0x7ff918014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32bd1e0171062847 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:55.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff90dffb700 1 -- 172.21.15.121:0/2578687385 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918003dc0 con 0x55922e8c7640 2024-06-10T00:28:55.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff90dffb700 1 -- 172.21.15.121:0/2578687385 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff918011e50 con 0x55922e8c7640 2024-06-10T00:28:55.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.050+0000 7ff90dffb700 1 -- 172.21.15.121:0/2578687385 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91800e710 con 0x55922e8c7640 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.051+0000 7ff938629b80 1 -- 172.21.15.121:0/2578687385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 msgr2=0x55922e8c7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.051+0000 7ff938629b80 1 --2- 172.21.15.121:0/2578687385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8c7a20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff9180043b0 tx=0x7ff918014000 comp rx=0 tx=0).stop 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.051+0000 7ff938629b80 1 -- 172.21.15.121:0/2578687385 shutdown_connections 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.051+0000 7ff938629b80 1 --2- 172.21.15.121:0/2578687385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c1f20 0x55922e8c2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.051+0000 7ff938629b80 1 --2- 172.21.15.121:0/2578687385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8c7a20 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.051+0000 7ff938629b80 1 --2- 172.21.15.121:0/2578687385 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c7f60 0x55922e8c19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.051+0000 7ff938629b80 1 -- 172.21.15.121:0/2578687385 >> 172.21.15.121:0/2578687385 conn(0x55922e7eccc0 msgr2=0x55922e8bc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.051+0000 7ff938629b80 1 -- 172.21.15.121:0/2578687385 shutdown_connections 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.051+0000 7ff938629b80 1 -- 172.21.15.121:0/2578687385 wait complete. 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.051+0000 7ff938629b80 1 -- start start 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff938629b80 1 Processor -- start 2024-06-10T00:28:55.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff938629b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c1f20 0x55922e8cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff938629b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff938629b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c7f60 0x55922e8cb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff938629b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55922e8ce9d0 con 0x55922e8c7640 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff938629b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55922e7b3bf0 con 0x55922e8c7f60 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff938629b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55922e8ceb50 con 0x55922e8c1f20 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff938629b80 10 client.0 _ll_register_callbacks cb 0x55922e7ad920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff926ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff926ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58426/0 (socket says 172.21.15.121:58426) 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff926ffd700 1 -- 172.21.15.121:0/268418000 learned_addr learned my addr 172.21.15.121:0/268418000 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff9277fe700 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c1f20 0x55922e8cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.052+0000 7ff927fff700 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c7f60 0x55922e8cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff926ffd700 1 -- 172.21.15.121:0/268418000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c1f20 msgr2=0x55922e8cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff926ffd700 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c1f20 0x55922e8cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff926ffd700 1 -- 172.21.15.121:0/268418000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c7f60 msgr2=0x55922e8cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff926ffd700 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c7f60 0x55922e8cb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff926ffd700 1 -- 172.21.15.121:0/268418000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55922e7628d0 con 0x55922e8c7640 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff926ffd700 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8cb040 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff928076580 tx=0x7ff91c00bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a295aed597c7fb3 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff906ffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff906ffd700 1 -- 172.21.15.121:0/268418000 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c003a60 con 0x55922e8c7640 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff906ffd700 1 -- 172.21.15.121:0/268418000 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff91c003c00 con 0x55922e8c7640 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff938629b80 1 -- 172.21.15.121:0/268418000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff918011e50 con 0x55922e8c7640 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff906ffd700 1 -- 172.21.15.121:0/268418000 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff91c030050 con 0x55922e8c7640 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff938629b80 10 client.15546 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff938629b80 1 -- 172.21.15.121:0/268418000 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55922e8d9380 con 0x55922e8c7640 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.053+0000 7ff906ffd700 1 -- 172.21.15.121:0/268418000 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff918011e50 con 0x55922e8c7640 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.054+0000 7ff906ffd700 1 -- 172.21.15.121:0/268418000 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7ff91c02b030 con 0x55922e8c7640 2024-06-10T00:28:55.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.054+0000 7ff938629b80 10 client.15546 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.054+0000 7ff938629b80 10 client.15546 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.054+0000 7ff938629b80 20 client.15546 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.054+0000 7ff938629b80 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55922e8dba10 0x55922e8dde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.054+0000 7ff938629b80 4 client.15546 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.054+0000 7ff938629b80 1 -- 172.21.15.121:0/268418000 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55922e820d10 con 0x55922e8dba10 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.054+0000 7ff9277fe700 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55922e8dba10 0x55922e8dde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.054+0000 7ff906ffd700 10 client.15546 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.054+0000 7ff9277fe700 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55922e8dba10 0x55922e8dde00 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8f60376e2e2fae68 server_cookie=221afcf62b3fa764 in_seq=0 out_seq=0 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.055+0000 7ff906ffd700 1 -- 172.21.15.121:0/268418000 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55922e820d10 con 0x55922e8dba10 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.055+0000 7ff906ffd700 10 client.15546 handle_command_reply: tid=0 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.055+0000 7ff938629b80 1 client.15546 shutdown 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.055+0000 7ff938629b80 1 -- 172.21.15.121:0/268418000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 msgr2=0x55922e8cb040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.055+0000 7ff938629b80 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8cb040 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ff928076580 tx=0x7ff91c00bee0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.055+0000 7ff938629b80 1 -- 172.21.15.121:0/268418000 shutdown_connections 2024-06-10T00:28:55.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.055+0000 7ff938629b80 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55922e8dba10 0x55922e8dde00 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.055+0000 7ff938629b80 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c7f60 0x55922e8cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.055+0000 7ff938629b80 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c1f20 0x55922e8cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.055+0000 7ff938629b80 1 --2- 172.21.15.121:0/268418000 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8cb040 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.055+0000 7ff938629b80 1 -- 172.21.15.121:0/268418000 >> 172.21.15.121:0/268418000 conn(0x55922e7eccc0 msgr2=0x55922e8c09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:55.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.056+0000 7ff906ffd700 0 client.15546 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:28:55.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.056+0000 7ff938629b80 1 -- 172.21.15.121:0/268418000 shutdown_connections 2024-06-10T00:28:55.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.056+0000 7ff938629b80 1 -- 172.21.15.121:0/268418000 wait complete. 2024-06-10T00:28:55.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.056+0000 7ff938629b80 20 client.15546 trim_cache size 0 max 16384 2024-06-10T00:28:55.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff938629b80 1 Processor -- start 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff938629b80 1 -- start start 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff938629b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 0x55922e8c19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff938629b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8b6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff938629b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7f60 0x55922e8c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff938629b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281b30e0 con 0x55922e8c7640 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff938629b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281b0970 con 0x55922e8c1f20 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff938629b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281ae7c0 con 0x55922e8c7f60 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff926ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff926ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8b6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58430/0 (socket says 172.21.15.121:58430) 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff926ffd700 1 -- 172.21.15.121:0/1706115454 learned_addr learned my addr 172.21.15.121:0/1706115454 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff9277fe700 1 --2- 172.21.15.121:0/1706115454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 0x55922e8c19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff926ffd700 1 -- 172.21.15.121:0/1706115454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7f60 msgr2=0x55922e8c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:55.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.071+0000 7ff927fff700 1 --2- 172.21.15.121:0/1706115454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7f60 0x55922e8c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.072+0000 7ff926ffd700 1 --2- 172.21.15.121:0/1706115454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7f60 0x55922e8c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.072+0000 7ff926ffd700 1 -- 172.21.15.121:0/1706115454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 msgr2=0x55922e8c19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:55.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.072+0000 7ff926ffd700 1 --2- 172.21.15.121:0/1706115454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 0x55922e8c19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.072+0000 7ff926ffd700 1 -- 172.21.15.121:0/1706115454 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55922e8bd520 con 0x55922e8c7640 2024-06-10T00:28:55.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.072+0000 7ff927fff700 1 --2- 172.21.15.121:0/1706115454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7f60 0x55922e8c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:55.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.072+0000 7ff9277fe700 1 --2- 172.21.15.121:0/1706115454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 0x55922e8c19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:55.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.072+0000 7ff926ffd700 1 --2- 172.21.15.121:0/1706115454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8b6690 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0098e0 tx=0x7ff91c009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68907135a3b0cd77 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:55.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.072+0000 7ff90d7fa700 1 -- 172.21.15.121:0/1706115454 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c009e60 con 0x55922e8c7640 2024-06-10T00:28:55.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.072+0000 7ff90d7fa700 1 -- 172.21.15.121:0/1706115454 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff91c003d40 con 0x55922e8c7640 2024-06-10T00:28:55.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.072+0000 7ff90d7fa700 1 -- 172.21.15.121:0/1706115454 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff91c00f460 con 0x55922e8c7640 2024-06-10T00:28:55.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.073+0000 7ff938629b80 1 -- 172.21.15.121:0/1706115454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 msgr2=0x55922e8b6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:55.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.073+0000 7ff938629b80 1 --2- 172.21.15.121:0/1706115454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8b6690 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff91c0098e0 tx=0x7ff91c009290 comp rx=0 tx=0).stop 2024-06-10T00:28:55.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.073+0000 7ff938629b80 1 -- 172.21.15.121:0/1706115454 shutdown_connections 2024-06-10T00:28:55.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.073+0000 7ff938629b80 1 --2- 172.21.15.121:0/1706115454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7f60 0x55922e8c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.073+0000 7ff938629b80 1 --2- 172.21.15.121:0/1706115454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 0x55922e8c19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.073+0000 7ff938629b80 1 --2- 172.21.15.121:0/1706115454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7640 0x55922e8b6690 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.073+0000 7ff938629b80 1 -- 172.21.15.121:0/1706115454 >> 172.21.15.121:0/1706115454 conn(0x55922e7eccc0 msgr2=0x55922e8c8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:55.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.073+0000 7ff938629b80 1 -- 172.21.15.121:0/1706115454 shutdown_connections 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.073+0000 7ff938629b80 1 -- 172.21.15.121:0/1706115454 wait complete. 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.073+0000 7ff938629b80 1 -- start start 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.073+0000 7ff938629b80 1 Processor -- start 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff938629b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 0x55922e8d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff938629b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7640 0x55922e8d4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff938629b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7f60 0x55922e8e2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff938629b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281aee30 con 0x55922e8c7f60 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff938629b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281b02a0 con 0x55922e8c1f20 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff938629b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff9281af500 con 0x55922e8c7640 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff938629b80 10 client.0 _ll_register_callbacks cb 0x55922e7ad920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff927fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7f60 0x55922e8e2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff9277fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 0x55922e8d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff927fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7f60 0x55922e8e2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58434/0 (socket says 172.21.15.121:58434) 2024-06-10T00:28:55.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff927fff700 1 -- 172.21.15.121:0/1410777640 learned_addr learned my addr 172.21.15.121:0/1410777640 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff926ffd700 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7640 0x55922e8d4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff927fff700 1 -- 172.21.15.121:0/1410777640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7640 msgr2=0x55922e8d4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff927fff700 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7640 0x55922e8d4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff927fff700 1 -- 172.21.15.121:0/1410777640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 msgr2=0x55922e8d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff927fff700 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 0x55922e8d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff927fff700 1 -- 172.21.15.121:0/1410777640 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55922e8d4990 con 0x55922e8c7f60 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff926ffd700 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7640 0x55922e8d4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff9277fe700 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 0x55922e8d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff927fff700 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7f60 0x55922e8e2680 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff92009a900 tx=0x7ff92009a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2969de01a5359eb6 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff906ffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.074+0000 7ff906ffd700 1 -- 172.21.15.121:0/1410777640 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920005100 con 0x55922e8c7f60 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff906ffd700 1 -- 172.21.15.121:0/1410777640 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff92009ae60 con 0x55922e8c7f60 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff938629b80 1 -- 172.21.15.121:0/1410777640 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff91c003d40 con 0x55922e8c7f60 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff906ffd700 1 -- 172.21.15.121:0/1410777640 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff9200cc3a0 con 0x55922e8c7f60 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff938629b80 10 client.15552 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:55.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff938629b80 1 -- 172.21.15.121:0/1410777640 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55922e945b70 con 0x55922e8c7f60 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff906ffd700 1 -- 172.21.15.121:0/1410777640 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff91c003d40 con 0x55922e8c7f60 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff906ffd700 1 -- 172.21.15.121:0/1410777640 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7ff9200b7020 con 0x55922e8c7f60 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff938629b80 10 client.15552 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff938629b80 10 client.15552 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff938629b80 20 client.15552 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff938629b80 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55922e9461d0 0x55922e9485c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff938629b80 4 client.15552 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.075+0000 7ff938629b80 1 -- 172.21.15.121:0/1410777640 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55922e81b290 con 0x55922e9461d0 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.076+0000 7ff9277fe700 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55922e9461d0 0x55922e9485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.076+0000 7ff9277fe700 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55922e9461d0 0x55922e9485c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=886fb15b180d2ccc server_cookie=992c8fb7946eda34 in_seq=0 out_seq=0 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.076+0000 7ff906ffd700 10 client.15552 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.076+0000 7ff906ffd700 1 -- 172.21.15.121:0/1410777640 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55922e81b290 con 0x55922e9461d0 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.076+0000 7ff906ffd700 10 client.15552 handle_command_reply: tid=0 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.076+0000 7ff938629b80 1 client.15552 shutdown 2024-06-10T00:28:55.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.076+0000 7ff938629b80 1 -- 172.21.15.121:0/1410777640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7f60 msgr2=0x55922e8e2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:55.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.076+0000 7ff938629b80 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7f60 0x55922e8e2680 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff92009a900 tx=0x7ff92009a840 comp rx=0 tx=0).stop 2024-06-10T00:28:55.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff938629b80 1 -- 172.21.15.121:0/1410777640 shutdown_connections 2024-06-10T00:28:55.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff938629b80 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55922e9461d0 0x55922e9485c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff938629b80 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55922e8c7f60 0x55922e8e2680 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff938629b80 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55922e8c1f20 0x55922e8d3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff938629b80 1 --2- 172.21.15.121:0/1410777640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55922e8c7640 0x55922e8d4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff906ffd700 0 client.15552 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:28:55.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff938629b80 1 -- 172.21.15.121:0/1410777640 >> 172.21.15.121:0/1410777640 conn(0x55922e7eccc0 msgr2=0x55922e8b6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:55.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff938629b80 1 -- 172.21.15.121:0/1410777640 shutdown_connections 2024-06-10T00:28:55.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff938629b80 1 -- 172.21.15.121:0/1410777640 wait complete. 2024-06-10T00:28:55.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff938629b80 20 client.15552 trim_cache size 0 max 16384 2024-06-10T00:28:55.083 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:28:55.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff92dda6700 1 -- 172.21.15.121:0/649909097 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff9100a2e20 msgr2=0x7ff9100a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:55.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff92dda6700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff9100a2e20 0x7ff9100a52e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff918011490 tx=0x7ff918009260 comp rx=0 tx=0).stop 2024-06-10T00:28:55.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff92dda6700 1 -- 172.21.15.121:0/649909097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff928106720 msgr2=0x7ff928103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:55.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff92dda6700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff928106720 0x7ff928103b10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff920007ee0 tx=0x7ff92000d3b0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff92dda6700 1 -- 172.21.15.121:0/649909097 shutdown_connections 2024-06-10T00:28:55.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff92dda6700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff9100a2e20 0x7ff9100a52e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff92dda6700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff928106720 0x7ff928103b10 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff92dda6700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9281019d0 0x7ff928103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff92dda6700 1 --2- 172.21.15.121:0/649909097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff928102380 0x7ff9281035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:55.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.077+0000 7ff92dda6700 1 -- 172.21.15.121:0/649909097 >> 172.21.15.121:0/649909097 conn(0x7ff928078340 msgr2=0x7ff92806af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:55.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.078+0000 7ff92dda6700 1 -- 172.21.15.121:0/649909097 shutdown_connections 2024-06-10T00:28:55.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:55.078+0000 7ff92dda6700 1 -- 172.21.15.121:0/649909097 wait complete. 2024-06-10T00:28:55.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-10T00:28:56.002 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:56.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:56 smithi121 ceph-mon[32378]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:56.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:56 smithi193 ceph-mon[36716]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:57.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:56 smithi135 ceph-mon[36424]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.775+0000 7f9ddb140700 1 -- 172.21.15.121:0/3316144254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd4102510 msgr2=0x7f9dd4102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.775+0000 7f9ddb140700 1 --2- 172.21.15.121:0/3316144254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd4102510 0x7f9dd4102990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009990 tx=0x7f9dc0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.775+0000 7f9ddb140700 1 -- 172.21.15.121:0/3316144254 shutdown_connections 2024-06-10T00:28:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.775+0000 7f9ddb140700 1 --2- 172.21.15.121:0/3316144254 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dd4102ed0 0x7f9dd410f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.775+0000 7f9ddb140700 1 --2- 172.21.15.121:0/3316144254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dd4108510 0x7f9dd41088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.775+0000 7f9ddb140700 1 --2- 172.21.15.121:0/3316144254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd4102510 0x7f9dd4102990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.775+0000 7f9ddb140700 1 -- 172.21.15.121:0/3316144254 >> 172.21.15.121:0/3316144254 conn(0x7f9dd40795b0 msgr2=0x7f9dd40799c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.775+0000 7f9ddb140700 1 -- 172.21.15.121:0/3316144254 shutdown_connections 2024-06-10T00:28:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.775+0000 7f9ddb140700 1 -- 172.21.15.121:0/3316144254 wait complete. 2024-06-10T00:28:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9ddb140700 1 Processor -- start 2024-06-10T00:28:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9ddb140700 1 -- start start 2024-06-10T00:28:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9ddb140700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dd4102510 0x7f9dd419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9ddb140700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dd4102ed0 0x7f9dd419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9ddb140700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd4108510 0x7f9dd4198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9ddb140700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd4112760 con 0x7f9dd4108510 2024-06-10T00:28:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9ddb140700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd41125e0 con 0x7f9dd4102510 2024-06-10T00:28:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9ddb140700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd41128e0 con 0x7f9dd4102ed0 2024-06-10T00:28:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9dd3fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dd4102ed0 0x7f9dd419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9dd3fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dd4102ed0 0x7f9dd419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:54270/0 (socket says 172.21.15.121:54270) 2024-06-10T00:28:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9dd3fff700 1 -- 172.21.15.121:0/3869080266 learned_addr learned my addr 172.21.15.121:0/3869080266 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9dd96dd700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd4108510 0x7f9dd4198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.776+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dd4102510 0x7f9dd419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.777+0000 7f9dd96dd700 1 -- 172.21.15.121:0/3869080266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dd4102ed0 msgr2=0x7f9dd419ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.777+0000 7f9dd96dd700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dd4102ed0 0x7f9dd419ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.777+0000 7f9dd96dd700 1 -- 172.21.15.121:0/3869080266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dd4102510 msgr2=0x7f9dd419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.777+0000 7f9dd96dd700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dd4102510 0x7f9dd419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.777+0000 7f9dd96dd700 1 -- 172.21.15.121:0/3869080266 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd4199240 con 0x7f9dd4108510 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.777+0000 7f9dd96dd700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd4108510 0x7f9dd4198ae0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc007f10 tx=0x7f9dcc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.777+0000 7f9dd1ffb700 1 -- 172.21.15.121:0/3869080266 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc004db0 con 0x7f9dd4108510 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.777+0000 7f9ddb140700 1 -- 172.21.15.121:0/3869080266 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd4199530 con 0x7f9dd4108510 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.777+0000 7f9dd1ffb700 1 -- 172.21.15.121:0/3869080266 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dcc009e20 con 0x7f9dd4108510 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.777+0000 7f9dd1ffb700 1 -- 172.21.15.121:0/3869080266 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc015400 con 0x7f9dd4108510 2024-06-10T00:28:57.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.777+0000 7f9ddb140700 1 -- 172.21.15.121:0/3869080266 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd41a57f0 con 0x7f9dd4108510 2024-06-10T00:28:57.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.781+0000 7f9dd1ffb700 1 -- 172.21.15.121:0/3869080266 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f9dcc00db20 con 0x7f9dd4108510 2024-06-10T00:28:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.781+0000 7f9dd1ffb700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9dc40a2ea0 0x7f9dc40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.781+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9dc40a2ea0 0x7f9dc40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.781+0000 7f9dd1ffb700 1 -- 172.21.15.121:0/3869080266 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f9dcc0cf250 con 0x7f9dd4108510 2024-06-10T00:28:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9de59c3b80 1 Processor -- start 2024-06-10T00:28:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9dc40a2ea0 0x7f9dc40a5360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80045b0 tx=0x7f9dc800b3c0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9de59c3b80 1 -- start start 2024-06-10T00:28:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d01c640 0x55f09d01ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d0169e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d016f20 0x55f09d017340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f09cf68810 con 0x55f09d01c640 2024-06-10T00:28:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55f09cf10720 con 0x55f09d01cf60 2024-06-10T00:28:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f09cf647d0 con 0x55f09d016f20 2024-06-10T00:28:57.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9dd8edc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d01c640 0x55f09d01ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9dd8edc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d01c640 0x55f09d01ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58464/0 (socket says 172.21.15.121:58464) 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9dd8edc700 1 -- 172.21.15.121:0/3757075620 learned_addr learned my addr 172.21.15.121:0/3757075620 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9dd3fff700 1 --2- 172.21.15.121:0/3757075620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d0169e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9dd96dd700 1 --2- 172.21.15.121:0/3757075620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d016f20 0x55f09d017340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9dd8edc700 1 -- 172.21.15.121:0/3757075620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d016f20 msgr2=0x55f09d017340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3757075620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d016f20 0x55f09d017340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9dd8edc700 1 -- 172.21.15.121:0/3757075620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 msgr2=0x55f09d0169e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.782+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3757075620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d0169e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.783+0000 7f9dd8edc700 1 -- 172.21.15.121:0/3757075620 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f09cf06f50 con 0x55f09d01c640 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.783+0000 7f9dd3fff700 1 --2- 172.21.15.121:0/3757075620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d0169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.783+0000 7f9dd96dd700 1 --2- 172.21.15.121:0/3757075620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d016f20 0x55f09d017340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.783+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3757075620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d01c640 0x55f09d01ca20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8003b20 tx=0x7f9dc8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb7e67b281da790 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.783+0000 7f9dbeffd700 1 -- 172.21.15.121:0/3757075620 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc80118d0 con 0x55f09d01c640 2024-06-10T00:28:57.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.783+0000 7f9dbeffd700 1 -- 172.21.15.121:0/3757075620 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dc8004830 con 0x55f09d01c640 2024-06-10T00:28:57.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.783+0000 7f9dbeffd700 1 -- 172.21.15.121:0/3757075620 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8005650 con 0x55f09d01c640 2024-06-10T00:28:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.784+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3757075620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d01c640 msgr2=0x55f09d01ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.784+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3757075620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d01c640 0x55f09d01ca20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8003b20 tx=0x7f9dc8013000 comp rx=0 tx=0).stop 2024-06-10T00:28:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.784+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3757075620 shutdown_connections 2024-06-10T00:28:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.784+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3757075620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d016f20 0x55f09d017340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.784+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3757075620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d01c640 0x55f09d01ca20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.784+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3757075620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d0169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.784+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3757075620 >> 172.21.15.121:0/3757075620 conn(0x55f09cf41cc0 msgr2=0x55f09d011850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.784+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3757075620 shutdown_connections 2024-06-10T00:28:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.784+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3757075620 wait complete. 2024-06-10T00:28:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.784+0000 7f9de59c3b80 1 -- start start 2024-06-10T00:28:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.784+0000 7f9de59c3b80 1 Processor -- start 2024-06-10T00:28:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d022c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01c640 0x55f09d01ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01cf60 0x55f09d0204c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f09d0239d0 con 0x55f09d016f20 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55f09cf08bf0 con 0x55f09d01c640 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f09d023b50 con 0x55f09d01cf60 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd8edc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d022c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd8edc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d022c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58476/0 (socket says 172.21.15.121:58476) 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd8edc700 1 -- 172.21.15.121:0/3196186884 learned_addr learned my addr 172.21.15.121:0/3196186884 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9de59c3b80 10 client.0 _ll_register_callbacks cb 0x55f09cf02920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd3fff700 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01c640 0x55f09d01ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd8edc700 1 -- 172.21.15.121:0/3196186884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01cf60 msgr2=0x55f09d0204c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd96dd700 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01cf60 0x55f09d0204c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01cf60 0x55f09d0204c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd8edc700 1 -- 172.21.15.121:0/3196186884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01c640 msgr2=0x55f09d01ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01c640 0x55f09d01ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd8edc700 1 -- 172.21.15.121:0/3196186884 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f09ceb78d0 con 0x55f09d016f20 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd96dd700 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01cf60 0x55f09d0204c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9db3fff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.785+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d022c80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8011720 tx=0x7f9dc8004c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed5e0f07bf0d78c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9db3fff700 1 -- 172.21.15.121:0/3196186884 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8035060 con 0x55f09d016f20 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9db3fff700 1 -- 172.21.15.121:0/3196186884 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f9dc8003ee0 con 0x55f09d016f20 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3196186884 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dc8004830 con 0x55f09d016f20 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9db3fff700 1 -- 172.21.15.121:0/3196186884 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9dc803c050 con 0x55f09d016f20 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9de59c3b80 10 client.15564 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3196186884 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f09d02e290 con 0x55f09d016f20 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9db3fff700 1 -- 172.21.15.121:0/3196186884 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dc8004830 con 0x55f09d016f20 2024-06-10T00:28:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9db3fff700 1 -- 172.21.15.121:0/3196186884 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f9dc80265d0 con 0x55f09d016f20 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9de59c3b80 10 client.15564 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9de59c3b80 10 client.15564 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9de59c3b80 20 client.15564 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55f09d030870 0x55f09d032c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9de59c3b80 4 client.15564 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.786+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3196186884 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f09cf75d10 con 0x55f09d030870 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.787+0000 7f9dd3fff700 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55f09d030870 0x55f09d032c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.787+0000 7f9dd3fff700 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55f09d030870 0x55f09d032c60 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=369623941b9bc012 server_cookie=d76baf225fc2b4ee in_seq=0 out_seq=0 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.787+0000 7f9db3fff700 10 client.15564 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9db3fff700 1 -- 172.21.15.121:0/3196186884 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55f09cf75d10 con 0x55f09d030870 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9db3fff700 10 client.15564 handle_command_reply: tid=0 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 1 client.15564 shutdown 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3196186884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 msgr2=0x55f09d022c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d022c80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8011720 tx=0x7f9dc8004c10 comp rx=0 tx=0).stop 2024-06-10T00:28:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3196186884 shutdown_connections 2024-06-10T00:28:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55f09d030870 0x55f09d032c60 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9db3fff700 0 client.15564 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:28:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01cf60 0x55f09d0204c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d022c80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3196186884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01c640 0x55f09d01ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3196186884 >> 172.21.15.121:0/3196186884 conn(0x55f09cf41cc0 msgr2=0x55f09d0159d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3196186884 shutdown_connections 2024-06-10T00:28:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3196186884 wait complete. 2024-06-10T00:28:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.788+0000 7f9de59c3b80 20 client.15564 trim_cache size 0 max 16384 2024-06-10T00:28:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9de59c3b80 1 Processor -- start 2024-06-10T00:28:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9de59c3b80 1 -- start start 2024-06-10T00:28:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d0169e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01c640 0x55f09d01dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d00b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd41aec70 con 0x55f09d016f20 2024-06-10T00:28:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd41ac500 con 0x55f09d01cf60 2024-06-10T00:28:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd41aa370 con 0x55f09d01c640 2024-06-10T00:28:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9dd8edc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d0169e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9dd8edc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d0169e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58490/0 (socket says 172.21.15.121:58490) 2024-06-10T00:28:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9dd8edc700 1 -- 172.21.15.121:0/505768292 learned_addr learned my addr 172.21.15.121:0/505768292 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9dd3fff700 1 --2- 172.21.15.121:0/505768292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01c640 0x55f09d01dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9dd8edc700 1 -- 172.21.15.121:0/505768292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01c640 msgr2=0x55f09d01dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:57.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/505768292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01c640 0x55f09d01dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9dd8edc700 1 -- 172.21.15.121:0/505768292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 msgr2=0x55f09d00b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:57.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/505768292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d00b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.805+0000 7f9dd8edc700 1 -- 172.21.15.121:0/505768292 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f09d012720 con 0x55f09d016f20 2024-06-10T00:28:57.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.806+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/505768292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d0169e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8003b20 tx=0x7f9dc8004070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5655a908ca78aa6 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:57.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.806+0000 7f9dbe7fc700 1 -- 172.21.15.121:0/505768292 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc8035040 con 0x55f09d016f20 2024-06-10T00:28:57.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.806+0000 7f9dbe7fc700 1 -- 172.21.15.121:0/505768292 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dc8004d30 con 0x55f09d016f20 2024-06-10T00:28:57.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.806+0000 7f9dbe7fc700 1 -- 172.21.15.121:0/505768292 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc80117d0 con 0x55f09d016f20 2024-06-10T00:28:57.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 -- 172.21.15.121:0/505768292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 msgr2=0x55f09d0169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:57.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/505768292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d0169e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9dc8003b20 tx=0x7f9dc8004070 comp rx=0 tx=0).stop 2024-06-10T00:28:57.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 -- 172.21.15.121:0/505768292 shutdown_connections 2024-06-10T00:28:57.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/505768292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d00b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/505768292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d0169e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/505768292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01c640 0x55f09d01dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 -- 172.21.15.121:0/505768292 >> 172.21.15.121:0/505768292 conn(0x55f09cf41cc0 msgr2=0x55f09d00ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:57.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 -- 172.21.15.121:0/505768292 shutdown_connections 2024-06-10T00:28:57.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 -- 172.21.15.121:0/505768292 wait complete. 2024-06-10T00:28:57.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 -- start start 2024-06-10T00:28:57.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 Processor -- start 2024-06-10T00:28:57.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.807+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d029010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01c640 0x55f09d029550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9de59c3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d024400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd41aa9c0 con 0x55f09d016f20 2024-06-10T00:28:57.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd41abe30 con 0x55f09d01cf60 2024-06-10T00:28:57.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9de59c3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd41ab090 con 0x55f09d01c640 2024-06-10T00:28:57.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9de59c3b80 10 client.0 _ll_register_callbacks cb 0x55f09cf02920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:28:57.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd8edc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d029010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd3fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01c640 0x55f09d029550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd8edc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d029010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:58504/0 (socket says 172.21.15.121:58504) 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd8edc700 1 -- 172.21.15.121:0/3696852311 learned_addr learned my addr 172.21.15.121:0/3696852311 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd96dd700 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d024400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd3fff700 1 -- 172.21.15.121:0/3696852311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 msgr2=0x55f09d024400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd3fff700 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d024400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd3fff700 1 -- 172.21.15.121:0/3696852311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 msgr2=0x55f09d029010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd3fff700 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d029010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd3fff700 1 -- 172.21.15.121:0/3696852311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f09d024c20 con 0x55f09d01c640 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d029010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9dd3fff700 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01c640 0x55f09d029550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55f09d012920 tx=0x7f9dc00259b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3702b4f6c9637020 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.808+0000 7f9db3fff700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9db3fff700 1 -- 172.21.15.121:0/3696852311 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0003b60 con 0x55f09d01c640 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9db3fff700 1 -- 172.21.15.121:0/3696852311 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f9dc0003d00 con 0x55f09d01c640 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3696852311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dc8004d30 con 0x55f09d01c640 2024-06-10T00:28:57.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9db3fff700 1 -- 172.21.15.121:0/3696852311 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9dc003e050 con 0x55f09d01c640 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9de59c3b80 10 client.25132 fetch_fsmap learned FSMap version 26 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3696852311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f09d09adc0 con 0x55f09d01c640 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9db3fff700 1 -- 172.21.15.121:0/3696852311 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dc8004d30 con 0x55f09d01c640 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9db3fff700 1 -- 172.21.15.121:0/3696852311 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f9dc000bc60 con 0x55f09d01c640 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9de59c3b80 10 client.25132 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9de59c3b80 10 client.25132 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9de59c3b80 20 client.25132 populate_metadata read hostname 'smithi121' 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55f09d030870 0x55f09d09d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9de59c3b80 4 client.25132 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.809+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3696852311 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f09cf70290 con 0x55f09d030870 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55f09d030870 0x55f09d09d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9dd8edc700 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55f09d030870 0x55f09d09d510 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6e8cad17f13f0a94 server_cookie=dbf864adb155966c in_seq=0 out_seq=0 2024-06-10T00:28:57.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9db3fff700 10 client.25132 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9db3fff700 1 -- 172.21.15.121:0/3696852311 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f09cf70290 con 0x55f09d030870 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9db3fff700 10 client.25132 handle_command_reply: tid=0 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9de59c3b80 1 client.25132 shutdown 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3696852311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01c640 msgr2=0x55f09d029550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01c640 0x55f09d029550 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55f09d012920 tx=0x7f9dc00259b0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3696852311 shutdown_connections 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55f09d030870 0x55f09d09d510 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f09d01cf60 0x55f09d024400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f09d016f20 0x55f09d029010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9de59c3b80 1 --2- 172.21.15.121:0/3696852311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f09d01c640 0x55f09d029550 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3696852311 >> 172.21.15.121:0/3696852311 conn(0x55f09cf41cc0 msgr2=0x55f09d01b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.810+0000 7f9db3fff700 0 client.25132 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:28:57.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.811+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3696852311 shutdown_connections 2024-06-10T00:28:57.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.811+0000 7f9de59c3b80 1 -- 172.21.15.121:0/3696852311 wait complete. 2024-06-10T00:28:57.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.811+0000 7f9de59c3b80 20 client.25132 trim_cache size 0 max 16384 2024-06-10T00:28:57.816 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.811+0000 7f9ddb140700 1 -- 172.21.15.121:0/3869080266 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9dc40a2ea0 msgr2=0x7f9dc40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.811+0000 7f9ddb140700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9dc40a2ea0 0x7f9dc40a5360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9dc80045b0 tx=0x7f9dc800b3c0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.811+0000 7f9ddb140700 1 -- 172.21.15.121:0/3869080266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd4108510 msgr2=0x7f9dd4198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.811+0000 7f9ddb140700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd4108510 0x7f9dd4198ae0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc007f10 tx=0x7f9dcc00d3b0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.812+0000 7f9ddb140700 1 -- 172.21.15.121:0/3869080266 shutdown_connections 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.812+0000 7f9ddb140700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9dc40a2ea0 0x7f9dc40a5360 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.812+0000 7f9ddb140700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd4108510 0x7f9dd4198ae0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.812+0000 7f9ddb140700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dd4102510 0x7f9dd419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.812+0000 7f9ddb140700 1 --2- 172.21.15.121:0/3869080266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9dd4102ed0 0x7f9dd419ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.812+0000 7f9ddb140700 1 -- 172.21.15.121:0/3869080266 >> 172.21.15.121:0/3869080266 conn(0x7f9dd40795b0 msgr2=0x7f9dd41011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.812+0000 7f9ddb140700 1 -- 172.21.15.121:0/3869080266 shutdown_connections 2024-06-10T00:28:57.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:28:57.812+0000 7f9ddb140700 1 -- 172.21.15.121:0/3869080266 wait complete. 2024-06-10T00:28:58.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-10T00:28:58.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:28:58 smithi193 ceph-mon[36716]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:58.911 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:28:58.933 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:28:58 smithi121 ceph-mon[32378]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:59.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:28:58 smithi135 ceph-mon[36424]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:28:59.986 INFO:tasks.quiescer.fs.[cephfs]:Total duration (60) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-10T00:28:59.986 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-06-10T00:28:59.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:29:00.268 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:00.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:00 smithi121 ceph-mon[32378]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:29:00.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.741+0000 7f6d08759700 1 -- 172.21.15.121:0/865617742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d00100b50 msgr2=0x7f6d00100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:00.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.741+0000 7f6d08759700 1 --2- 172.21.15.121:0/865617742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d00100b50 0x7f6d00100fd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6cf000aaa0 tx=0x7f6cf000adb0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.741+0000 7f6d08759700 1 -- 172.21.15.121:0/865617742 shutdown_connections 2024-06-10T00:29:00.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.741+0000 7f6d08759700 1 --2- 172.21.15.121:0/865617742 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d00101510 0x7f6d0010d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.741+0000 7f6d08759700 1 --2- 172.21.15.121:0/865617742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d00106b90 0x7f6d00106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.741+0000 7f6d08759700 1 --2- 172.21.15.121:0/865617742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d00100b50 0x7f6d00100fd0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.741+0000 7f6d08759700 1 -- 172.21.15.121:0/865617742 >> 172.21.15.121:0/865617742 conn(0x7f6d00077a00 msgr2=0x7f6d000fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:00.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.741+0000 7f6d08759700 1 -- 172.21.15.121:0/865617742 shutdown_connections 2024-06-10T00:29:00.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.741+0000 7f6d08759700 1 -- 172.21.15.121:0/865617742 wait complete. 2024-06-10T00:29:00.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d08759700 1 Processor -- start 2024-06-10T00:29:00.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d08759700 1 -- start start 2024-06-10T00:29:00.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d08759700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d00100b50 0x7f6d00074080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d08759700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d00101510 0x7f6d00072190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d08759700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d00106b90 0x7f6d00072840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d08759700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0006c8c0 con 0x7f6d00106b90 2024-06-10T00:29:00.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d08759700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0006c740 con 0x7f6d00101510 2024-06-10T00:29:00.744 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d08759700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0006ca40 con 0x7f6d00100b50 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d06cf6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d00106b90 0x7f6d00072840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d06cf6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d00106b90 0x7f6d00072840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38678/0 (socket says 172.21.15.121:38678) 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d06cf6700 1 -- 172.21.15.121:0/3285625516 learned_addr learned my addr 172.21.15.121:0/3285625516 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.742+0000 7f6d064f5700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d00100b50 0x7f6d00074080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.743+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d00101510 0x7f6d00072190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.743+0000 7f6d06cf6700 1 -- 172.21.15.121:0/3285625516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d00100b50 msgr2=0x7f6d00074080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.743+0000 7f6d06cf6700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d00100b50 0x7f6d00074080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.743+0000 7f6d06cf6700 1 -- 172.21.15.121:0/3285625516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d00101510 msgr2=0x7f6d00072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.743+0000 7f6d06cf6700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d00101510 0x7f6d00072190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.743+0000 7f6d06cf6700 1 -- 172.21.15.121:0/3285625516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d001af940 con 0x7f6d00106b90 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.743+0000 7f6d06cf6700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d00106b90 0x7f6d00072840 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8007d70 tx=0x7f6cf800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.743+0000 7f6cf77fe700 1 -- 172.21.15.121:0/3285625516 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8016020 con 0x7f6d00106b90 2024-06-10T00:29:00.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.743+0000 7f6d08759700 1 -- 172.21.15.121:0/3285625516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d001afbd0 con 0x7f6d00106b90 2024-06-10T00:29:00.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.743+0000 7f6d08759700 1 -- 172.21.15.121:0/3285625516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d001b00e0 con 0x7f6d00106b90 2024-06-10T00:29:00.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.743+0000 7f6cf77fe700 1 -- 172.21.15.121:0/3285625516 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf800ad40 con 0x7f6d00106b90 2024-06-10T00:29:00.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.744+0000 7f6cf77fe700 1 -- 172.21.15.121:0/3285625516 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf8011600 con 0x7f6d00106b90 2024-06-10T00:29:00.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.747+0000 7f6d12fdcb80 1 Processor -- start 2024-06-10T00:29:00.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d12fdcb80 1 -- start start 2024-06-10T00:29:00.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a867cf60 0x5587a867d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8677260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a86777a0 0x5587a8677bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5587a8569f60 con 0x5587a867d880 2024-06-10T00:29:00.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5587a85c8850 con 0x5587a867cf60 2024-06-10T00:29:00.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5587a8530460 con 0x5587a86777a0 2024-06-10T00:29:00.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d064f5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8677260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d064f5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8677260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38680/0 (socket says 172.21.15.121:38680) 2024-06-10T00:29:00.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d064f5700 1 -- 172.21.15.121:0/4125781670 learned_addr learned my addr 172.21.15.121:0/4125781670 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:00.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/4125781670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a867cf60 0x5587a867d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.748+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/4125781670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a86777a0 0x5587a8677bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.749+0000 7f6d064f5700 1 -- 172.21.15.121:0/4125781670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a86777a0 msgr2=0x5587a8677bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:00.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.749+0000 7f6d064f5700 1 --2- 172.21.15.121:0/4125781670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a86777a0 0x5587a8677bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.749+0000 7f6d064f5700 1 -- 172.21.15.121:0/4125781670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a867cf60 msgr2=0x5587a867d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:00.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.749+0000 7f6d064f5700 1 --2- 172.21.15.121:0/4125781670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a867cf60 0x5587a867d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.749+0000 7f6d064f5700 1 -- 172.21.15.121:0/4125781670 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587a85178d0 con 0x5587a867d880 2024-06-10T00:29:00.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.749+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/4125781670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a867cf60 0x5587a867d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:00.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.749+0000 7f6d064f5700 1 --2- 172.21.15.121:0/4125781670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8677260 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00ac30 tx=0x7f6cfc00af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14f74cee9c3047 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:00.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.749+0000 7f6cf4ff9700 1 -- 172.21.15.121:0/4125781670 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc01d070 con 0x5587a867d880 2024-06-10T00:29:00.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.749+0000 7f6cf4ff9700 1 -- 172.21.15.121:0/4125781670 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cfc00fe90 con 0x5587a867d880 2024-06-10T00:29:00.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.749+0000 7f6cf4ff9700 1 -- 172.21.15.121:0/4125781670 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc010960 con 0x5587a867d880 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.750+0000 7f6cf77fe700 1 -- 172.21.15.121:0/3285625516 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 24) ==== 138010+0+0 (secure 0 0 0) 0x7f6cf8019020 con 0x7f6d00106b90 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.750+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/4125781670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 msgr2=0x5587a8677260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.750+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/4125781670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8677260 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00ac30 tx=0x7f6cfc00af40 comp rx=0 tx=0).stop 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.750+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/4125781670 shutdown_connections 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.750+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/4125781670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a86777a0 0x5587a8677bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.750+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/4125781670 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a867cf60 0x5587a867d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.750+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/4125781670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8677260 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.750+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/4125781670 >> 172.21.15.121:0/4125781670 conn(0x5587a85a4080 msgr2=0x5587a8672220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.750+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/4125781670 shutdown_connections 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.750+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/4125781670 wait complete. 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.751+0000 7f6cf77fe700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6cec0a2db0 0x7f6cec0a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.751+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6cec0a2db0 0x7f6cec0a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.751+0000 7f6d12fdcb80 1 -- start start 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.751+0000 7f6cf77fe700 1 -- 172.21.15.121:0/3285625516 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f6cf80cf6d0 con 0x7f6d00106b90 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.751+0000 7f6d12fdcb80 1 Processor -- start 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.751+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6cec0a2db0 0x7f6cec0a5270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6d0010dd80 tx=0x7f6cf000adb0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a86777a0 0x5587a8683560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867cf60 0x5587a8680860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8680da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5587a8568fc0 con 0x5587a867d880 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5587a85c4810 con 0x5587a86777a0 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5587a8684260 con 0x5587a867cf60 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d12fdcb80 10 client.0 _ll_register_callbacks cb 0x5587a855e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d06cf6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8680da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d06cf6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8680da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38686/0 (socket says 172.21.15.121:38686) 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d06cf6700 1 -- 172.21.15.121:0/3720799798 learned_addr learned my addr 172.21.15.121:0/3720799798 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:00.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d064f5700 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a86777a0 0x5587a8683560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d06cf6700 1 -- 172.21.15.121:0/3720799798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867cf60 msgr2=0x5587a8680860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d06cf6700 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867cf60 0x5587a8680860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d06cf6700 1 -- 172.21.15.121:0/3720799798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a86777a0 msgr2=0x5587a8683560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d06cf6700 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a86777a0 0x5587a8683560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d06cf6700 1 -- 172.21.15.121:0/3720799798 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587a858e750 con 0x5587a867d880 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d064f5700 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a86777a0 0x5587a8683560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6d06cf6700 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8680da0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6cf809b790 tx=0x7f6cf809b3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df829fae737be9b2 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.752+0000 7f6ce9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/3720799798 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf80b4060 con 0x5587a867d880 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/3720799798 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f6cf80860e0 con 0x5587a867d880 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/3720799798 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cfc00fe90 con 0x5587a867d880 2024-06-10T00:29:00.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/3720799798 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6cf809b800 con 0x5587a867d880 2024-06-10T00:29:00.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6d12fdcb80 10 client.15582 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:00.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/3720799798 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5587a868ea10 con 0x5587a867d880 2024-06-10T00:29:00.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/3720799798 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cfc00fe90 con 0x5587a867d880 2024-06-10T00:29:00.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/3720799798 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f6cf809ba60 con 0x5587a867d880 2024-06-10T00:29:00.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6d12fdcb80 10 client.15582 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:00.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6d12fdcb80 10 client.15582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:00.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6d12fdcb80 20 client.15582 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:00.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.753+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5587a8690fa0 0x5587a8693410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.754+0000 7f6d12fdcb80 4 client.15582 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.754+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/3720799798 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587a85d5d50 con 0x5587a8690fa0 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.754+0000 7f6d064f5700 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5587a8690fa0 0x5587a8693410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.754+0000 7f6d064f5700 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5587a8690fa0 0x5587a8693410 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b37618f14efb416 server_cookie=2205d1654fd5dc64 in_seq=0 out_seq=0 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.754+0000 7f6ce9ffb700 10 client.15582 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/3720799798 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5587a85d5d50 con 0x5587a8690fa0 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6ce9ffb700 10 client.15582 handle_command_reply: tid=0 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6d12fdcb80 1 client.15582 shutdown 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/3720799798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 msgr2=0x5587a8680da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8680da0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6cf809b790 tx=0x7f6cf809b3c0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/3720799798 shutdown_connections 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5587a8690fa0 0x5587a8693410 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867d880 0x5587a8680da0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a86777a0 0x5587a8683560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6ce9ffb700 0 client.15582 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:00.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/3720799798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867cf60 0x5587a8680860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.755+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/3720799798 >> 172.21.15.121:0/3720799798 conn(0x5587a85a4080 msgr2=0x5587a8671700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:00.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.756+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/3720799798 shutdown_connections 2024-06-10T00:29:00.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.756+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/3720799798 wait complete. 2024-06-10T00:29:00.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.756+0000 7f6d12fdcb80 20 client.15582 trim_cache size 0 max 16384 2024-06-10T00:29:00.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.784+0000 7f6d12fdcb80 1 Processor -- start 2024-06-10T00:29:00.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.784+0000 7f6d12fdcb80 1 -- start start 2024-06-10T00:29:00.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.784+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a86777a0 0x5587a8671e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.784+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867cf60 0x5587a86723c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.784+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867d880 0x5587a867dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.784+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce4011590 con 0x5587a867cf60 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.784+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce400ee20 con 0x5587a86777a0 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.784+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce400cc70 con 0x5587a867d880 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.784+0000 7f6d05cf4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867cf60 0x5587a86723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.785+0000 7f6d05cf4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867cf60 0x5587a86723c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38692/0 (socket says 172.21.15.121:38692) 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.785+0000 7f6d05cf4700 1 -- 172.21.15.121:0/2461364104 learned_addr learned my addr 172.21.15.121:0/2461364104 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.785+0000 7f6d064f5700 1 --2- 172.21.15.121:0/2461364104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a86777a0 0x5587a8671e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.785+0000 7f6d06cf6700 1 --2- 172.21.15.121:0/2461364104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867d880 0x5587a867dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.785+0000 7f6d05cf4700 1 -- 172.21.15.121:0/2461364104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867d880 msgr2=0x5587a867dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.785+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/2461364104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867d880 0x5587a867dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.785+0000 7f6d05cf4700 1 -- 172.21.15.121:0/2461364104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a86777a0 msgr2=0x5587a8671e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:00.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.785+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/2461364104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a86777a0 0x5587a8671e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.785+0000 7f6d05cf4700 1 -- 172.21.15.121:0/2461364104 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587a8683d60 con 0x5587a867cf60 2024-06-10T00:29:00.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.785+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/2461364104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867cf60 0x5587a86723c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0000c00 tx=0x7f6cf0024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6e0479901a33a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:00.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.785+0000 7f6cebfff700 1 -- 172.21.15.121:0/2461364104 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0034070 con 0x5587a867cf60 2024-06-10T00:29:00.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.786+0000 7f6cebfff700 1 -- 172.21.15.121:0/2461364104 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf0005550 con 0x5587a867cf60 2024-06-10T00:29:00.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.786+0000 7f6cebfff700 1 -- 172.21.15.121:0/2461364104 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0028070 con 0x5587a867cf60 2024-06-10T00:29:00.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.786+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/2461364104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867cf60 msgr2=0x5587a86723c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:00.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.786+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/2461364104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867cf60 0x5587a86723c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0000c00 tx=0x7f6cf0024000 comp rx=0 tx=0).stop 2024-06-10T00:29:00.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.786+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/2461364104 shutdown_connections 2024-06-10T00:29:00.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.786+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/2461364104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867d880 0x5587a867dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.786+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/2461364104 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a86777a0 0x5587a8671e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.786+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/2461364104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a867cf60 0x5587a86723c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.786+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/2461364104 >> 172.21.15.121:0/2461364104 conn(0x5587a85a4080 msgr2=0x5587a867f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.787+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/2461364104 shutdown_connections 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.787+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/2461364104 wait complete. 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.787+0000 7f6d12fdcb80 1 -- start start 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.787+0000 7f6d12fdcb80 1 Processor -- start 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.787+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a86777a0 0x5587a8689e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.787+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867cf60 0x5587a86987e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.787+0000 7f6d12fdcb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a867d880 0x5587a8698ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.787+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce400d2e0 con 0x5587a86777a0 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.787+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce400e750 con 0x5587a867d880 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.787+0000 7f6d12fdcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce400d9b0 con 0x5587a867cf60 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.787+0000 7f6d12fdcb80 10 client.0 _ll_register_callbacks cb 0x5587a855e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d064f5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a86777a0 0x5587a8689e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d064f5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a86777a0 0x5587a8689e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38696/0 (socket says 172.21.15.121:38696) 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d064f5700 1 -- 172.21.15.121:0/1091199673 learned_addr learned my addr 172.21.15.121:0/1091199673 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867cf60 0x5587a86987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d06cf6700 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a867d880 0x5587a8698ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d064f5700 1 -- 172.21.15.121:0/1091199673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867cf60 msgr2=0x5587a86987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d064f5700 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867cf60 0x5587a86987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d064f5700 1 -- 172.21.15.121:0/1091199673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a867d880 msgr2=0x5587a8698ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d064f5700 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a867d880 0x5587a8698ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d064f5700 1 -- 172.21.15.121:0/1091199673 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587a87034c0 con 0x5587a86777a0 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867cf60 0x5587a86987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d064f5700 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a86777a0 0x5587a8689e40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00afd0 tx=0x7f6cfc00fd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20234468d633e3f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6ce9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/1091199673 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc032060 con 0x5587a86777a0 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/1091199673 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f6cfc0046f0 con 0x5587a86777a0 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.788+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/1091199673 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cf0005550 con 0x5587a86777a0 2024-06-10T00:29:00.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.789+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/1091199673 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6cfc039050 con 0x5587a86777a0 2024-06-10T00:29:00.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.789+0000 7f6d12fdcb80 10 client.15588 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:00.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.789+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/1091199673 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5587a87046d0 con 0x5587a86777a0 2024-06-10T00:29:00.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.789+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/1091199673 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cfc01e040 con 0x5587a86777a0 2024-06-10T00:29:00.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.789+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/1091199673 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f6cfc010b50 con 0x5587a86777a0 2024-06-10T00:29:00.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.789+0000 7f6d12fdcb80 10 client.15588 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:00.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.789+0000 7f6d12fdcb80 10 client.15588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:00.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.789+0000 7f6d12fdcb80 20 client.15588 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:00.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.789+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5587a8690fa0 0x5587a8706b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:00.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.789+0000 7f6d12fdcb80 4 client.15588 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:29:00.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.789+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/1091199673 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f6cf0005550 con 0x5587a8690fa0 2024-06-10T00:29:00.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.790+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5587a8690fa0 0x5587a8706b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:00.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.790+0000 7f6d05cf4700 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5587a8690fa0 0x5587a8706b10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68c3b32c6d434412 server_cookie=93e36c50202ef14 in_seq=0 out_seq=0 2024-06-10T00:29:00.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.790+0000 7f6ce9ffb700 10 client.15588 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:00.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.790+0000 7f6ce9ffb700 1 -- 172.21.15.121:0/1091199673 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f6cf0005550 con 0x5587a8690fa0 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.790+0000 7f6ce9ffb700 10 client.15588 handle_command_reply: tid=0 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.790+0000 7f6d12fdcb80 1 client.15588 shutdown 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.790+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/1091199673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a86777a0 msgr2=0x5587a8689e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.790+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a86777a0 0x5587a8689e40 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00afd0 tx=0x7f6cfc00fd20 comp rx=0 tx=0).stop 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.791+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/1091199673 shutdown_connections 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.791+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5587a8690fa0 0x5587a8706b10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.791+0000 7f6ce9ffb700 0 client.15588 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.791+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5587a867d880 0x5587a8698ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.791+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5587a86777a0 0x5587a8689e40 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.791+0000 7f6d12fdcb80 1 --2- 172.21.15.121:0/1091199673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5587a867cf60 0x5587a86987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.791+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/1091199673 >> 172.21.15.121:0/1091199673 conn(0x5587a85a4080 msgr2=0x5587a867dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:00.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.791+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/1091199673 shutdown_connections 2024-06-10T00:29:00.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.791+0000 7f6d12fdcb80 1 -- 172.21.15.121:0/1091199673 wait complete. 2024-06-10T00:29:00.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.791+0000 7f6d12fdcb80 20 client.15588 trim_cache size 0 max 16384 2024-06-10T00:29:00.795 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 -- 172.21.15.121:0/3285625516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6cec0a2db0 msgr2=0x7f6cec0a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6cec0a2db0 0x7f6cec0a5270 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6d0010dd80 tx=0x7f6cf000adb0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 -- 172.21.15.121:0/3285625516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d00106b90 msgr2=0x7f6d00072840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d00106b90 0x7f6d00072840 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8007d70 tx=0x7f6cf800a340 comp rx=0 tx=0).stop 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 -- 172.21.15.121:0/3285625516 shutdown_connections 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6cec0a2db0 0x7f6cec0a5270 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d00106b90 0x7f6d00072840 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d00100b50 0x7f6d00074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 --2- 172.21.15.121:0/3285625516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d00101510 0x7f6d00072190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 -- 172.21.15.121:0/3285625516 >> 172.21.15.121:0/3285625516 conn(0x7f6d00077a00 msgr2=0x7f6d00103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 -- 172.21.15.121:0/3285625516 shutdown_connections 2024-06-10T00:29:00.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:00.792+0000 7f6d08759700 1 -- 172.21.15.121:0/3285625516 wait complete. 2024-06-10T00:29:00.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:00 smithi193 ceph-mon[36716]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:29:00.949 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:29:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:29:00.817+0000 7faf51fc5700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:29:00.949 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:29:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:29:00.817+0000 7faf51fc5700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:29:00.949 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:29:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:29:00.817+0000 7faf51fc5700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:29:00.949 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:29:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:29:00.817+0000 7faf51fc5700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:29:00.949 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:29:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:29:00.817+0000 7faf51fc5700 -1 client.0 error registering admin socket command: (17) File exists 2024-06-10T00:29:01.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:00 smithi135 ceph-mon[36424]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-06-10T00:29:02.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:02 smithi121 ceph-mon[32378]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:29:02.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:02 smithi121 ceph-mon[32378]: mgrmap e25: x(active, since 64s), standbys: z, y 2024-06-10T00:29:02.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:02 smithi193 ceph-mon[36716]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:29:02.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:02 smithi193 ceph-mon[36716]: mgrmap e25: x(active, since 64s), standbys: z, y 2024-06-10T00:29:03.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:02 smithi135 ceph-mon[36424]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:29:03.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:02 smithi135 ceph-mon[36424]: mgrmap e25: x(active, since 64s), standbys: z, y 2024-06-10T00:29:03.443 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-10T00:29:03.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-10T00:29:03.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.462+0000 7f2061d58700 1 -- 172.21.15.121:0/3091346231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c101440 msgr2=0x7f205c10f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:03.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.462+0000 7f2061d58700 1 --2- 172.21.15.121:0/3091346231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c101440 0x7f205c10f330 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f205400b0a0 tx=0x7f205400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:29:03.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.462+0000 7f2061d58700 1 -- 172.21.15.121:0/3091346231 shutdown_connections 2024-06-10T00:29:03.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.462+0000 7f2061d58700 1 --2- 172.21.15.121:0/3091346231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c101440 0x7f205c10f330 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:03.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.462+0000 7f2061d58700 1 --2- 172.21.15.121:0/3091346231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205c108510 0x7f205c1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:03.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.462+0000 7f2061d58700 1 --2- 172.21.15.121:0/3091346231 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f205c100a80 0x7f205c100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:03.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.462+0000 7f2061d58700 1 -- 172.21.15.121:0/3091346231 >> 172.21.15.121:0/3091346231 conn(0x7f205c077a00 msgr2=0x7f205c0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:03.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.462+0000 7f2061d58700 1 -- 172.21.15.121:0/3091346231 shutdown_connections 2024-06-10T00:29:03.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.462+0000 7f2061d58700 1 -- 172.21.15.121:0/3091346231 wait complete. 2024-06-10T00:29:03.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f2061d58700 1 Processor -- start 2024-06-10T00:29:03.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f2061d58700 1 -- start start 2024-06-10T00:29:03.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f2061d58700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f205c100a80 0x7f205c19e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:03.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f2061d58700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c101440 0x7f205c19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:03.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f2061d58700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205c108510 0x7f205c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:03.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f2061d58700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f205c112790 con 0x7f205c101440 2024-06-10T00:29:03.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f2061d58700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f205c112610 con 0x7f205c100a80 2024-06-10T00:29:03.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f2061d58700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f205c112910 con 0x7f205c108510 2024-06-10T00:29:03.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f205affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c101440 0x7f205c19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:03.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f205affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c101440 0x7f205c19ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38736/0 (socket says 172.21.15.121:38736) 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f205affd700 1 -- 172.21.15.121:0/2798548401 learned_addr learned my addr 172.21.15.121:0/2798548401 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f205bfff700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205c108510 0x7f205c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.463+0000 7f205b7fe700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f205c100a80 0x7f205c19e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f205affd700 1 -- 172.21.15.121:0/2798548401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205c108510 msgr2=0x7f205c198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f205affd700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205c108510 0x7f205c198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f205affd700 1 -- 172.21.15.121:0/2798548401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f205c100a80 msgr2=0x7f205c19e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f205affd700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f205c100a80 0x7f205c19e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f205affd700 1 -- 172.21.15.121:0/2798548401 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f205c1990f0 con 0x7f205c101440 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f205b7fe700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f205c100a80 0x7f205c19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f205affd700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c101440 0x7f205c19ed10 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f205000eb30 tx=0x7f205000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f2058ff9700 1 -- 172.21.15.121:0/2798548401 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205000cbd0 con 0x7f205c101440 2024-06-10T00:29:03.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f2058ff9700 1 -- 172.21.15.121:0/2798548401 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2050004500 con 0x7f205c101440 2024-06-10T00:29:03.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f2058ff9700 1 -- 172.21.15.121:0/2798548401 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2050010430 con 0x7f205c101440 2024-06-10T00:29:03.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f2061d58700 1 -- 172.21.15.121:0/2798548401 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f205c1993e0 con 0x7f205c101440 2024-06-10T00:29:03.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.464+0000 7f2061d58700 1 -- 172.21.15.121:0/2798548401 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f205c1a56a0 con 0x7f205c101440 2024-06-10T00:29:03.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.466+0000 7f2061d58700 1 -- 172.21.15.121:0/2798548401 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f205c069180 con 0x7f205c101440 2024-06-10T00:29:03.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.466+0000 7f2058ff9700 1 -- 172.21.15.121:0/2798548401 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f2050003f00 con 0x7f205c101440 2024-06-10T00:29:03.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.467+0000 7f2058ff9700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f20440a2eb0 0x7f20440a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:03.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.467+0000 7f2058ff9700 1 -- 172.21.15.121:0/2798548401 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f20500cfd30 con 0x7f205c101440 2024-06-10T00:29:03.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.467+0000 7f205b7fe700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f20440a2eb0 0x7f20440a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:03.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.468+0000 7f205b7fe700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f20440a2eb0 0x7f20440a5370 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f205c199ad0 tx=0x7f204c009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:03.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.472+0000 7f2058ff9700 1 -- 172.21.15.121:0/2798548401 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f205009c440 con 0x7f205c101440 2024-06-10T00:29:03.767 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:03.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.868+0000 7f2061d58700 1 -- 172.21.15.121:0/2798548401 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f205c1a5930 con 0x7f205c101440 2024-06-10T00:29:03.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.870+0000 7f2058ff9700 1 -- 172.21.15.121:0/2798548401 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f205009bd40 con 0x7f205c101440 2024-06-10T00:29:03.870 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:29:03.870 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:27:55.370804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:29:03.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 -- 172.21.15.121:0/2798548401 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f20440a2eb0 msgr2=0x7f20440a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:03.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f20440a2eb0 0x7f20440a5370 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f205c199ad0 tx=0x7f204c009260 comp rx=0 tx=0).stop 2024-06-10T00:29:03.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 -- 172.21.15.121:0/2798548401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c101440 msgr2=0x7f205c19ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:03.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c101440 0x7f205c19ed10 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f205000eb30 tx=0x7f205000ee40 comp rx=0 tx=0).stop 2024-06-10T00:29:03.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 -- 172.21.15.121:0/2798548401 shutdown_connections 2024-06-10T00:29:03.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f20440a2eb0 0x7f20440a5370 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:03.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f205c108510 0x7f205c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:03.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f205c100a80 0x7f205c19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:03.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 --2- 172.21.15.121:0/2798548401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c101440 0x7f205c19ed10 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:03.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 -- 172.21.15.121:0/2798548401 >> 172.21.15.121:0/2798548401 conn(0x7f205c077a00 msgr2=0x7f205c0fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:03.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 -- 172.21.15.121:0/2798548401 shutdown_connections 2024-06-10T00:29:03.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:03.873+0000 7f2061d58700 1 -- 172.21.15.121:0/2798548401 wait complete. 2024-06-10T00:29:03.874 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-06-10T00:29:04.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:04 smithi121 ceph-mon[32378]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-06-10T00:29:04.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:04 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2798548401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:04.808 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-06-10T00:29:04.808 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-06-10T00:29:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:04 smithi193 ceph-mon[36716]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-06-10T00:29:04.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:04 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2798548401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:04 smithi135 ceph-mon[36424]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-06-10T00:29:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:04 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2798548401' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:05.894 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:06.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:06 smithi193 ceph-mon[36716]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-06-10T00:29:06.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:06 smithi121 ceph-mon[32378]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-06-10T00:29:07.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:06 smithi135 ceph-mon[36424]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-06-10T00:29:07.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.835+0000 7f67639fb700 1 -- 172.21.15.121:0/2884807295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c1073f0 msgr2=0x7f675c1077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:07.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.835+0000 7f67639fb700 1 --2- 172.21.15.121:0/2884807295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c1073f0 0x7f675c1077d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f6758009a10 tx=0x7f6758009d20 comp rx=0 tx=0).stop 2024-06-10T00:29:07.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.835+0000 7f67639fb700 1 -- 172.21.15.121:0/2884807295 shutdown_connections 2024-06-10T00:29:07.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.835+0000 7f67639fb700 1 --2- 172.21.15.121:0/2884807295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f675c101d40 0x7f675c10e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.835+0000 7f67639fb700 1 --2- 172.21.15.121:0/2884807295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c1073f0 0x7f675c1077d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.836+0000 7f67639fb700 1 --2- 172.21.15.121:0/2884807295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c101380 0x7f675c101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.836+0000 7f67639fb700 1 -- 172.21.15.121:0/2884807295 >> 172.21.15.121:0/2884807295 conn(0x7f675c078300 msgr2=0x7f675c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:07.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.836+0000 7f67639fb700 1 -- 172.21.15.121:0/2884807295 shutdown_connections 2024-06-10T00:29:07.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.836+0000 7f67639fb700 1 -- 172.21.15.121:0/2884807295 wait complete. 2024-06-10T00:29:07.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.837+0000 7f67639fb700 1 Processor -- start 2024-06-10T00:29:07.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.837+0000 7f67639fb700 1 -- start start 2024-06-10T00:29:07.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.837+0000 7f67639fb700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c101380 0x7f675c1967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.837+0000 7f67639fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c101d40 0x7f675c196d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.837+0000 7f67639fb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f675c1073f0 0x7f675c197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.837+0000 7f67639fb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f675c110590 con 0x7f675c101d40 2024-06-10T00:29:07.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.837+0000 7f67639fb700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f675c110410 con 0x7f675c1073f0 2024-06-10T00:29:07.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.837+0000 7f67639fb700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f675c110710 con 0x7f675c101380 2024-06-10T00:29:07.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.837+0000 7f6761797700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c101380 0x7f675c1967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.837+0000 7f6761797700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c101380 0x7f675c1967d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:58234/0 (socket says 172.21.15.121:58234) 2024-06-10T00:29:07.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.837+0000 7f6761797700 1 -- 172.21.15.121:0/1213073501 learned_addr learned my addr 172.21.15.121:0/1213073501 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:07.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.838+0000 7f6760f96700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c101d40 0x7f675c196d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.838+0000 7f6761f98700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f675c1073f0 0x7f675c197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.838+0000 7f6761797700 1 -- 172.21.15.121:0/1213073501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f675c1073f0 msgr2=0x7f675c197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:07.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.838+0000 7f6761797700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f675c1073f0 0x7f675c197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.838+0000 7f6761797700 1 -- 172.21.15.121:0/1213073501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c101d40 msgr2=0x7f675c196d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:07.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.838+0000 7f6761797700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c101d40 0x7f675c196d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.838+0000 7f6761797700 1 -- 172.21.15.121:0/1213073501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675c1ab480 con 0x7f675c101380 2024-06-10T00:29:07.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.838+0000 7f6761f98700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f675c1073f0 0x7f675c197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:07.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.838+0000 7f6760f96700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c101d40 0x7f675c196d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:07.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.839+0000 7f6761797700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c101380 0x7f675c1967d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6758018040 tx=0x7f675800b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:07.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.839+0000 7f67527fc700 1 -- 172.21.15.121:0/1213073501 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6758023030 con 0x7f675c101380 2024-06-10T00:29:07.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.839+0000 7f67527fc700 1 -- 172.21.15.121:0/1213073501 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6758003680 con 0x7f675c101380 2024-06-10T00:29:07.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.839+0000 7f67639fb700 1 -- 172.21.15.121:0/1213073501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f675c1ab710 con 0x7f675c101380 2024-06-10T00:29:07.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.839+0000 7f67527fc700 1 -- 172.21.15.121:0/1213073501 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6758004230 con 0x7f675c101380 2024-06-10T00:29:07.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.839+0000 7f67639fb700 1 -- 172.21.15.121:0/1213073501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f675c1abbc0 con 0x7f675c101380 2024-06-10T00:29:07.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.842+0000 7f67527fc700 1 -- 172.21.15.121:0/1213073501 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f6758026070 con 0x7f675c101380 2024-06-10T00:29:07.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.843+0000 7f676e27eb80 1 Processor -- start 2024-06-10T00:29:07.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.843+0000 7f676e27eb80 1 -- start start 2024-06-10T00:29:07.843 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.843+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 0x55c4a3262aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.843+0000 7f67527fc700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f67480a2eb0 0x7f67480a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.843+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a3262fe0 0x55c4a325ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.843+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a325d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.843+0000 7f676e27eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c4a31ae890 con 0x55c4a32626c0 2024-06-10T00:29:07.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.843+0000 7f676e27eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55c4a31567a0 con 0x55c4a325cfa0 2024-06-10T00:29:07.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.843+0000 7f676e27eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c4a31aa850 con 0x55c4a3262fe0 2024-06-10T00:29:07.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f67527fc700 1 -- 172.21.15.121:0/1213073501 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f67580d1370 con 0x7f675c101380 2024-06-10T00:29:07.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6761f98700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f67480a2eb0 0x7f67480a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6761f98700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a325d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6760f96700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a3262fe0 0x55c4a325ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6760f96700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a3262fe0 0x55c4a325ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:58248/0 (socket says 172.21.15.121:58248) 2024-06-10T00:29:07.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6760f96700 1 -- 172.21.15.121:0/160716064 learned_addr learned my addr 172.21.15.121:0/160716064 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:07.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6761797700 1 --2- 172.21.15.121:0/160716064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 0x55c4a3262aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6760f96700 1 -- 172.21.15.121:0/160716064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 msgr2=0x55c4a325d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:07.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6760f96700 1 --2- 172.21.15.121:0/160716064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a325d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6760f96700 1 -- 172.21.15.121:0/160716064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 msgr2=0x55c4a3262aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:07.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6760f96700 1 --2- 172.21.15.121:0/160716064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 0x55c4a3262aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6761f98700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f67480a2eb0 0x7f67480a5370 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6754009790 tx=0x7f6754007a80 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:07.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.844+0000 7f6760f96700 1 -- 172.21.15.121:0/160716064 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4a314cfd0 con 0x55c4a3262fe0 2024-06-10T00:29:07.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.845+0000 7f6760f96700 1 --2- 172.21.15.121:0/160716064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a3262fe0 0x55c4a325ca60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f674c0104d0 tx=0x7f674c0107e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d1323e392e82683 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:07.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.845+0000 7f67477fe700 1 -- 172.21.15.121:0/160716064 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c01c480 con 0x55c4a3262fe0 2024-06-10T00:29:07.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.845+0000 7f67477fe700 1 -- 172.21.15.121:0/160716064 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f674c01cb00 con 0x55c4a3262fe0 2024-06-10T00:29:07.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.845+0000 7f67477fe700 1 -- 172.21.15.121:0/160716064 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c011bf0 con 0x55c4a3262fe0 2024-06-10T00:29:07.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.846+0000 7f676e27eb80 1 -- 172.21.15.121:0/160716064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a3262fe0 msgr2=0x55c4a325ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:07.848 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.846+0000 7f676e27eb80 1 --2- 172.21.15.121:0/160716064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a3262fe0 0x55c4a325ca60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f674c0104d0 tx=0x7f674c0107e0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.846+0000 7f676e27eb80 1 -- 172.21.15.121:0/160716064 shutdown_connections 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.846+0000 7f676e27eb80 1 --2- 172.21.15.121:0/160716064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a325d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.846+0000 7f676e27eb80 1 --2- 172.21.15.121:0/160716064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 0x55c4a3262aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.846+0000 7f676e27eb80 1 --2- 172.21.15.121:0/160716064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a3262fe0 0x55c4a325ca60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.846+0000 7f676e27eb80 1 -- 172.21.15.121:0/160716064 >> 172.21.15.121:0/160716064 conn(0x55c4a3187d40 msgr2=0x55c4a32578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.846+0000 7f676e27eb80 1 -- 172.21.15.121:0/160716064 shutdown_connections 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.846+0000 7f676e27eb80 1 -- 172.21.15.121:0/160716064 wait complete. 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.846+0000 7f676e27eb80 1 -- start start 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.846+0000 7f676e27eb80 1 Processor -- start 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a3268d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a32626c0 0x55c4a3266090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a3262fe0 0x55c4a32665d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f676e27eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3269a50 con 0x55c4a3262fe0 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f676e27eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55c4a314ec70 con 0x55c4a325cfa0 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f676e27eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3269bd0 con 0x55c4a32626c0 2024-06-10T00:29:07.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f6760f96700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a32626c0 0x55c4a3266090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f6760f96700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a32626c0 0x55c4a3266090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:58264/0 (socket says 172.21.15.121:58264) 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f6760f96700 1 -- 172.21.15.121:0/1738231116 learned_addr learned my addr 172.21.15.121:0/1738231116 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f6761797700 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a3268d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f6761f98700 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a3262fe0 0x55c4a32665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f676e27eb80 10 client.0 _ll_register_callbacks cb 0x55c4a31489a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f6760f96700 1 -- 172.21.15.121:0/1738231116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 msgr2=0x55c4a3268d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f6760f96700 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a3268d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f6760f96700 1 -- 172.21.15.121:0/1738231116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a3262fe0 msgr2=0x55c4a32665d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f6760f96700 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a3262fe0 0x55c4a32665d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.847+0000 7f6760f96700 1 -- 172.21.15.121:0/1738231116 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4a30fd950 con 0x55c4a32626c0 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.848+0000 7f6761f98700 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a3262fe0 0x55c4a32665d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.848+0000 7f6760f96700 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a32626c0 0x55c4a3266090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f674c005830 tx=0x7f674c0115c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dd480559d1ae42d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:07.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.848+0000 7f6744ff9700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:29:07.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.848+0000 7f6744ff9700 1 -- 172.21.15.121:0/1738231116 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c004040 con 0x55c4a32626c0 2024-06-10T00:29:07.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.848+0000 7f676e27eb80 1 -- 172.21.15.121:0/1738231116 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f674c01cb00 con 0x55c4a32626c0 2024-06-10T00:29:07.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.848+0000 7f6744ff9700 1 -- 172.21.15.121:0/1738231116 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f674c0041e0 con 0x55c4a32626c0 2024-06-10T00:29:07.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.848+0000 7f6744ff9700 1 -- 172.21.15.121:0/1738231116 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f674c032050 con 0x55c4a32626c0 2024-06-10T00:29:07.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.848+0000 7f676e27eb80 10 client.25162 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:07.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.848+0000 7f676e27eb80 1 -- 172.21.15.121:0/1738231116 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c4a32742e0 con 0x55c4a32626c0 2024-06-10T00:29:07.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.848+0000 7f6744ff9700 1 -- 172.21.15.121:0/1738231116 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f674c01cb00 con 0x55c4a32626c0 2024-06-10T00:29:07.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.849+0000 7f6744ff9700 1 -- 172.21.15.121:0/1738231116 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f674c003980 con 0x55c4a32626c0 2024-06-10T00:29:07.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.849+0000 7f676e27eb80 10 client.25162 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:07.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.849+0000 7f676e27eb80 10 client.25162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:07.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.849+0000 7f676e27eb80 20 client.25162 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:07.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.849+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c4a3276860 0x55c4a3278c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.852 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.849+0000 7f676e27eb80 4 client.25162 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.849+0000 7f676e27eb80 1 -- 172.21.15.121:0/1738231116 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4a31bbd90 con 0x55c4a3276860 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.849+0000 7f6761797700 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c4a3276860 0x55c4a3278c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.850+0000 7f6744ff9700 10 client.25162 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.850+0000 7f6761797700 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c4a3276860 0x55c4a3278c50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7f9c54f9e2e1518 server_cookie=3015de1016f5153f in_seq=0 out_seq=0 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.850+0000 7f6744ff9700 1 -- 172.21.15.121:0/1738231116 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55c4a31bbd90 con 0x55c4a3276860 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.850+0000 7f6744ff9700 10 client.25162 handle_command_reply: tid=0 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.850+0000 7f676e27eb80 1 client.25162 shutdown 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.850+0000 7f676e27eb80 1 -- 172.21.15.121:0/1738231116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a32626c0 msgr2=0x55c4a3266090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.850+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a32626c0 0x55c4a3266090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f674c005830 tx=0x7f674c0115c0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.851+0000 7f676e27eb80 1 -- 172.21.15.121:0/1738231116 shutdown_connections 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.851+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c4a3276860 0x55c4a3278c50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.851+0000 7f6744ff9700 0 client.25162 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.851+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a3262fe0 0x55c4a32665d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.851+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a3268d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.851+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1738231116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a32626c0 0x55c4a3266090 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.853 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.851+0000 7f676e27eb80 1 -- 172.21.15.121:0/1738231116 >> 172.21.15.121:0/1738231116 conn(0x55c4a3187d40 msgr2=0x55c4a325ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:07.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.851+0000 7f676e27eb80 1 -- 172.21.15.121:0/1738231116 shutdown_connections 2024-06-10T00:29:07.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.851+0000 7f676e27eb80 1 -- 172.21.15.121:0/1738231116 wait complete. 2024-06-10T00:29:07.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.851+0000 7f676e27eb80 20 client.25162 trim_cache size 0 max 16384 2024-06-10T00:29:07.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.871+0000 7f676e27eb80 1 Processor -- start 2024-06-10T00:29:07.871 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.871+0000 7f676e27eb80 1 -- start start 2024-06-10T00:29:07.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a325ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 0x55c4a32607b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a3262fe0 0x55c4a3263b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f676e27eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b7370 con 0x55c4a32626c0 2024-06-10T00:29:07.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f676e27eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b4c00 con 0x55c4a325cfa0 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f676e27eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b2a50 con 0x55c4a3262fe0 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f6760f96700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 0x55c4a32607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f6760f96700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 0x55c4a32607b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38790/0 (socket says 172.21.15.121:38790) 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f6760f96700 1 -- 172.21.15.121:0/2502623600 learned_addr learned my addr 172.21.15.121:0/2502623600 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f6761797700 1 --2- 172.21.15.121:0/2502623600 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a325ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f6761797700 1 -- 172.21.15.121:0/2502623600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a3262fe0 msgr2=0x55c4a3263b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f6761797700 1 --2- 172.21.15.121:0/2502623600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a3262fe0 0x55c4a3263b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f6761797700 1 -- 172.21.15.121:0/2502623600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 msgr2=0x55c4a32607b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f6761797700 1 --2- 172.21.15.121:0/2502623600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 0x55c4a32607b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f6761797700 1 -- 172.21.15.121:0/2502623600 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4a3264600 con 0x55c4a325cfa0 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f6760f96700 1 --2- 172.21.15.121:0/2502623600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 0x55c4a32607b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.872+0000 7f6761797700 1 --2- 172.21.15.121:0/2502623600 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a325ca60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f67580ce660 tx=0x7f6758009d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a119e75f52e530e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:07.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.873+0000 7f6746ffd700 1 -- 172.21.15.121:0/2502623600 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67580a17a0 con 0x55c4a325cfa0 2024-06-10T00:29:07.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.873+0000 7f6746ffd700 1 -- 172.21.15.121:0/2502623600 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67580a5fd0 con 0x55c4a325cfa0 2024-06-10T00:29:07.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.873+0000 7f6746ffd700 1 -- 172.21.15.121:0/2502623600 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675809d8b0 con 0x55c4a325cfa0 2024-06-10T00:29:07.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.874+0000 7f676e27eb80 1 -- 172.21.15.121:0/2502623600 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 msgr2=0x55c4a325ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:07.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.874+0000 7f676e27eb80 1 --2- 172.21.15.121:0/2502623600 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a325ca60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f67580ce660 tx=0x7f6758009d20 comp rx=0 tx=0).stop 2024-06-10T00:29:07.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.874+0000 7f676e27eb80 1 -- 172.21.15.121:0/2502623600 shutdown_connections 2024-06-10T00:29:07.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.874+0000 7f676e27eb80 1 --2- 172.21.15.121:0/2502623600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a3262fe0 0x55c4a3263b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.874+0000 7f676e27eb80 1 --2- 172.21.15.121:0/2502623600 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a325cfa0 0x55c4a325ca60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.874+0000 7f676e27eb80 1 --2- 172.21.15.121:0/2502623600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a32626c0 0x55c4a32607b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.874+0000 7f676e27eb80 1 -- 172.21.15.121:0/2502623600 >> 172.21.15.121:0/2502623600 conn(0x55c4a3187d40 msgr2=0x55c4a3263480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.874+0000 7f676e27eb80 1 -- 172.21.15.121:0/2502623600 shutdown_connections 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.874+0000 7f676e27eb80 1 -- 172.21.15.121:0/2502623600 wait complete. 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.874+0000 7f676e27eb80 1 -- start start 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.874+0000 7f676e27eb80 1 Processor -- start 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a325cfa0 0x55c4a326f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a32626c0 0x55c4a326be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f676e27eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a3262fe0 0x55c4a326c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f676e27eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b30c0 con 0x55c4a3262fe0 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f676e27eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b4530 con 0x55c4a32626c0 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f676e27eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f675c1b3790 con 0x55c4a325cfa0 2024-06-10T00:29:07.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f676e27eb80 10 client.0 _ll_register_callbacks cb 0x55c4a31489a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6760f96700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a32626c0 0x55c4a326be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6761797700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a325cfa0 0x55c4a326f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6760f96700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a32626c0 0x55c4a326be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:49148/0 (socket says 172.21.15.121:49148) 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6761797700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a325cfa0 0x55c4a326f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:58294/0 (socket says 172.21.15.121:58294) 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6760f96700 1 -- 172.21.15.121:0/1423758165 learned_addr learned my addr 172.21.15.121:0/1423758165 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6761797700 1 -- 172.21.15.121:0/1423758165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a32626c0 msgr2=0x55c4a326be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6761797700 1 --2- 172.21.15.121:0/1423758165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a32626c0 0x55c4a326be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6761797700 1 -- 172.21.15.121:0/1423758165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a3262fe0 msgr2=0x55c4a326c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6761797700 1 --2- 172.21.15.121:0/1423758165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a3262fe0 0x55c4a326c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6761797700 1 -- 172.21.15.121:0/1423758165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4a326cb90 con 0x55c4a325cfa0 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6761797700 1 --2- 172.21.15.121:0/1423758165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a325cfa0 0x55c4a326f590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55c4a3259060 tx=0x7f675809d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd76c03088af2f0d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6744ff9700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.875+0000 7f6744ff9700 1 -- 172.21.15.121:0/1423758165 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67580a5b40 con 0x55c4a325cfa0 2024-06-10T00:29:07.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f6744ff9700 1 -- 172.21.15.121:0/1423758165 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f67580a5ce0 con 0x55c4a325cfa0 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f676e27eb80 1 -- 172.21.15.121:0/1423758165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67580a5fd0 con 0x55c4a325cfa0 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f6744ff9700 1 -- 172.21.15.121:0/1423758165 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f67580b6200 con 0x55c4a325cfa0 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f676e27eb80 10 client.25165 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f676e27eb80 1 -- 172.21.15.121:0/1423758165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c4a32e22d0 con 0x55c4a325cfa0 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f6744ff9700 1 -- 172.21.15.121:0/1423758165 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67580a5fd0 con 0x55c4a325cfa0 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f6744ff9700 1 -- 172.21.15.121:0/1423758165 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f6758030060 con 0x55c4a325cfa0 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f676e27eb80 10 client.25165 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f676e27eb80 10 client.25165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f676e27eb80 20 client.25165 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1423758165 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c4a32e2790 0x55c4a32e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f676e27eb80 4 client.25165 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-10T00:29:07.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.876+0000 7f676e27eb80 1 -- 172.21.15.121:0/1423758165 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c4a31b6310 con 0x55c4a32e2790 2024-06-10T00:29:07.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.877+0000 7f6760f96700 1 --2- 172.21.15.121:0/1423758165 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c4a32e2790 0x55c4a32e4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:07.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.877+0000 7f6760f96700 1 --2- 172.21.15.121:0/1423758165 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c4a32e2790 0x55c4a32e4bb0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ef2624d9a5fccf3 server_cookie=f6390f20df40113d in_seq=0 out_seq=0 2024-06-10T00:29:07.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.877+0000 7f6744ff9700 10 client.25165 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:07.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.878+0000 7f6744ff9700 1 -- 172.21.15.121:0/1423758165 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c4a31b6310 con 0x55c4a32e2790 2024-06-10T00:29:07.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.878+0000 7f6744ff9700 10 client.25165 handle_command_reply: tid=0 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.878+0000 7f676e27eb80 1 client.25165 shutdown 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.878+0000 7f676e27eb80 1 -- 172.21.15.121:0/1423758165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a325cfa0 msgr2=0x55c4a326f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.878+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1423758165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a325cfa0 0x55c4a326f590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55c4a3259060 tx=0x7f675809d520 comp rx=0 tx=0).stop 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.879+0000 7f676e27eb80 1 -- 172.21.15.121:0/1423758165 shutdown_connections 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.879+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1423758165 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c4a32e2790 0x55c4a32e4bb0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.879+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1423758165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c4a3262fe0 0x55c4a326c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.879+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1423758165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4a325cfa0 0x55c4a326f590 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.879+0000 7f676e27eb80 1 --2- 172.21.15.121:0/1423758165 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c4a32626c0 0x55c4a326be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.879+0000 7f676e27eb80 1 -- 172.21.15.121:0/1423758165 >> 172.21.15.121:0/1423758165 conn(0x55c4a3187d40 msgr2=0x55c4a3250e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.879+0000 7f6744ff9700 0 client.25165 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.879+0000 7f676e27eb80 1 -- 172.21.15.121:0/1423758165 shutdown_connections 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.879+0000 7f676e27eb80 1 -- 172.21.15.121:0/1423758165 wait complete. 2024-06-10T00:29:07.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.879+0000 7f676e27eb80 20 client.25165 trim_cache size 0 max 16384 2024-06-10T00:29:07.882 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:29:07.882 INFO:teuthology.orchestra.run.smithi121.stdout: "return_code": 0, 2024-06-10T00:29:07.882 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_tag": "2fd515e1-8021-4bac-a5f5-05e68d4de548", 2024-06-10T00:29:07.882 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": "asynchronous" 2024-06-10T00:29:07.882 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:29:07.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.880+0000 7f67639fb700 1 -- 172.21.15.121:0/1213073501 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f67480a2eb0 msgr2=0x7f67480a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:07.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.880+0000 7f67639fb700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f67480a2eb0 0x7f67480a5370 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6754009790 tx=0x7f6754007a80 comp rx=0 tx=0).stop 2024-06-10T00:29:07.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.880+0000 7f67639fb700 1 -- 172.21.15.121:0/1213073501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c101380 msgr2=0x7f675c1967d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:07.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.880+0000 7f67639fb700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c101380 0x7f675c1967d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6758018040 tx=0x7f675800b980 comp rx=0 tx=0).stop 2024-06-10T00:29:07.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.880+0000 7f67639fb700 1 -- 172.21.15.121:0/1213073501 shutdown_connections 2024-06-10T00:29:07.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.880+0000 7f67639fb700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f67480a2eb0 0x7f67480a5370 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.880+0000 7f67639fb700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f675c1073f0 0x7f675c197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.880+0000 7f67639fb700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f675c101380 0x7f675c1967d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.880+0000 7f67639fb700 1 --2- 172.21.15.121:0/1213073501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c101d40 0x7f675c196d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:07.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.880+0000 7f67639fb700 1 -- 172.21.15.121:0/1213073501 >> 172.21.15.121:0/1213073501 conn(0x7f675c078300 msgr2=0x7f675c100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:07.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.881+0000 7f67639fb700 1 -- 172.21.15.121:0/1213073501 shutdown_connections 2024-06-10T00:29:07.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:07.881+0000 7f67639fb700 1 -- 172.21.15.121:0/1213073501 wait complete. 2024-06-10T00:29:08.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.041+0000 7effd52d7700 1 -- 172.21.15.121:0/3471774609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd01033a0 msgr2=0x7effd0103780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:08.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.041+0000 7effd52d7700 1 --2- 172.21.15.121:0/3471774609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd01033a0 0x7effd0103780 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7effc0009990 tx=0x7effc0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.041+0000 7effd52d7700 1 -- 172.21.15.121:0/3471774609 shutdown_connections 2024-06-10T00:29:08.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.041+0000 7effd52d7700 1 --2- 172.21.15.121:0/3471774609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effd0101520 0x7effd0111210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.041+0000 7effd52d7700 1 --2- 172.21.15.121:0/3471774609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd01033a0 0x7effd0103780 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.041+0000 7effd52d7700 1 --2- 172.21.15.121:0/3471774609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effd0100a80 0x7effd0100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.041+0000 7effd52d7700 1 -- 172.21.15.121:0/3471774609 >> 172.21.15.121:0/3471774609 conn(0x7effd0077a30 msgr2=0x7effd00fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.042+0000 7effd52d7700 1 -- 172.21.15.121:0/3471774609 shutdown_connections 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.042+0000 7effd52d7700 1 -- 172.21.15.121:0/3471774609 wait complete. 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.042+0000 7effd52d7700 1 Processor -- start 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effd52d7700 1 -- start start 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effd52d7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd0100a80 0x7effd019ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effd52d7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effd0101520 0x7effd019b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effd52d7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effd01033a0 0x7effd019f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effd52d7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7effd01149b0 con 0x7effd0100a80 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effd52d7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7effd0114830 con 0x7effd0101520 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effd52d7700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7effd0114b30 con 0x7effd01033a0 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effceffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd0100a80 0x7effd019ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effceffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd0100a80 0x7effd019ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38830/0 (socket says 172.21.15.121:38830) 2024-06-10T00:29:08.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effceffd700 1 -- 172.21.15.121:0/3148054880 learned_addr learned my addr 172.21.15.121:0/3148054880 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effceffd700 1 -- 172.21.15.121:0/3148054880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effd01033a0 msgr2=0x7effd019f670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effceffd700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effd01033a0 0x7effd019f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effceffd700 1 -- 172.21.15.121:0/3148054880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effd0101520 msgr2=0x7effd019b2e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effceffd700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effd0101520 0x7effd019b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.043+0000 7effceffd700 1 -- 172.21.15.121:0/3148054880 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd019fdb0 con 0x7effd0100a80 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.044+0000 7effceffd700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd0100a80 0x7effd019ada0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7effc0000c00 tx=0x7effc000bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.044+0000 7effb7fff700 1 -- 172.21.15.121:0/3148054880 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0027070 con 0x7effd0100a80 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.044+0000 7effd52d7700 1 -- 172.21.15.121:0/3148054880 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd01a0040 con 0x7effd0100a80 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.044+0000 7effb7fff700 1 -- 172.21.15.121:0/3148054880 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effc0003d70 con 0x7effd0100a80 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.044+0000 7effb7fff700 1 -- 172.21.15.121:0/3148054880 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc000f460 con 0x7effd0100a80 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.044+0000 7effd52d7700 1 -- 172.21.15.121:0/3148054880 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd01a7920 con 0x7effd0100a80 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.046+0000 7effb7fff700 1 -- 172.21.15.121:0/3148054880 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7effc0009d10 con 0x7effd0100a80 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.047+0000 7effb7fff700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7effb80a2fd0 0x7effb80a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.047+0000 7effb7fff700 1 -- 172.21.15.121:0/3148054880 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7effc00d0620 con 0x7effd0100a80 2024-06-10T00:29:08.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.047+0000 7effce7fc700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7effb80a2fd0 0x7effb80a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.048+0000 7effce7fc700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7effb80a2fd0 0x7effb80a5490 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7effd00fffc0 tx=0x7effc4009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effdfb5ab80 1 Processor -- start 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effdfb5ab80 1 -- start start 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f4629af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f4629b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564f461ec9a0 con 0x564f462a15f0 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x564f461948b0 con 0x564f462a0cd0 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564f461e8960 con 0x564f4629b460 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effce7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f4629af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effce7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f4629af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38836/0 (socket says 172.21.15.121:38836) 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effce7fc700 1 -- 172.21.15.121:0/3612783233 learned_addr learned my addr 172.21.15.121:0/3612783233 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effceffd700 1 --2- 172.21.15.121:0/3612783233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effcf7fe700 1 --2- 172.21.15.121:0/3612783233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f4629b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.050+0000 7effce7fc700 1 -- 172.21.15.121:0/3612783233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 msgr2=0x564f4629b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.051+0000 7effce7fc700 1 --2- 172.21.15.121:0/3612783233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f4629b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.051+0000 7effce7fc700 1 -- 172.21.15.121:0/3612783233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 msgr2=0x564f462a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.051+0000 7effce7fc700 1 --2- 172.21.15.121:0/3612783233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.051+0000 7effce7fc700 1 -- 172.21.15.121:0/3612783233 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f4618b0e0 con 0x564f462a15f0 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.051+0000 7effceffd700 1 --2- 172.21.15.121:0/3612783233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.051+0000 7effce7fc700 1 --2- 172.21.15.121:0/3612783233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f4629af20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7effc400eb40 tx=0x7effc4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=606c41bfc4f1ab01 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.051+0000 7effb57fa700 1 -- 172.21.15.121:0/3612783233 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc4020070 con 0x564f462a15f0 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.051+0000 7effb57fa700 1 -- 172.21.15.121:0/3612783233 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effc400ee50 con 0x564f462a15f0 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.051+0000 7effb57fa700 1 -- 172.21.15.121:0/3612783233 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc400f3c0 con 0x564f462a15f0 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.052+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3612783233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 msgr2=0x564f4629af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.052+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/3612783233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f4629af20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7effc400eb40 tx=0x7effc4011000 comp rx=0 tx=0).stop 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.052+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3612783233 shutdown_connections 2024-06-10T00:29:08.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.052+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/3612783233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f4629b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.052+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/3612783233 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.052+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/3612783233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f4629af20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.052+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3612783233 >> 172.21.15.121:0/3612783233 conn(0x564f461c5e50 msgr2=0x564f462961b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.052+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3612783233 shutdown_connections 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.052+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3612783233 wait complete. 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effdfb5ab80 1 -- start start 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effdfb5ab80 1 Processor -- start 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f462a72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f462a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564f462a8060 con 0x564f462a15f0 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x564f4618cd80 con 0x564f462a0cd0 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564f462a81e0 con 0x564f4629b460 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effcf7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f462a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effdfb5ab80 10 client.0 _ll_register_callbacks cb 0x564f46186ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effceffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f462a72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effceffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f462a72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:58348/0 (socket says 172.21.15.121:58348) 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effceffd700 1 -- 172.21.15.121:0/2749999186 learned_addr learned my addr 172.21.15.121:0/2749999186 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effce7fc700 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.053+0000 7effcf7fe700 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f462a4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38838/0 (socket says 172.21.15.121:38838) 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effceffd700 1 -- 172.21.15.121:0/2749999186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 msgr2=0x564f462a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effceffd700 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effceffd700 1 -- 172.21.15.121:0/2749999186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 msgr2=0x564f462a4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effceffd700 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f462a4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effceffd700 1 -- 172.21.15.121:0/2749999186 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f4613ba60 con 0x564f4629b460 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effceffd700 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f462a72f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7effc00cdfe0 tx=0x7effc009cb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fcfe1d0d2a5fe2b server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effb2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effb2ff5700 1 -- 172.21.15.121:0/2749999186 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc00cdd40 con 0x564f4629b460 2024-06-10T00:29:08.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effb2ff5700 1 -- 172.21.15.121:0/2749999186 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7effc00a1b30 con 0x564f4629b460 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2749999186 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effc400ee50 con 0x564f4629b460 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effb2ff5700 1 -- 172.21.15.121:0/2749999186 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7effc00d0450 con 0x564f4629b460 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effdfb5ab80 10 client.25174 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.054+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2749999186 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564f462a80c0 con 0x564f4629b460 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.055+0000 7effb2ff5700 1 -- 172.21.15.121:0/2749999186 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effc400ee50 con 0x564f4629b460 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.055+0000 7effb2ff5700 1 -- 172.21.15.121:0/2749999186 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7effc0085bc0 con 0x564f4629b460 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.055+0000 7effdfb5ab80 10 client.25174 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.055+0000 7effdfb5ab80 10 client.25174 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.055+0000 7effdfb5ab80 20 client.25174 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.055+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x564f462b4cd0 0x564f462b7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.055+0000 7effdfb5ab80 4 client.25174 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.055+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2749999186 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f461f9ea0 con 0x564f462b4cd0 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.055+0000 7effce7fc700 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x564f462b4cd0 0x564f462b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.055+0000 7effce7fc700 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x564f462b4cd0 0x564f462b7100 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc3b6bcd915b05f1 server_cookie=cb1d530c37fc864d in_seq=0 out_seq=0 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.055+0000 7effb2ff5700 10 client.25174 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.056+0000 7effb2ff5700 1 -- 172.21.15.121:0/2749999186 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x564f461f9ea0 con 0x564f462b4cd0 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.056+0000 7effb2ff5700 10 client.25174 handle_command_reply: tid=0 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.056+0000 7effdfb5ab80 1 client.25174 shutdown 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.056+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2749999186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 msgr2=0x564f462a72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.056+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f462a72f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7effc00cdfe0 tx=0x7effc009cb30 comp rx=0 tx=0).stop 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.056+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2749999186 shutdown_connections 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.056+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x564f462b4cd0 0x564f462b7100 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.056+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f462a4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.057+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f462a72f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.057+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/2749999186 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.057+0000 7effb2ff5700 0 client.25174 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.057+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2749999186 >> 172.21.15.121:0/2749999186 conn(0x564f461c5e50 msgr2=0x564f46299f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.057+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2749999186 shutdown_connections 2024-06-10T00:29:08.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.057+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2749999186 wait complete. 2024-06-10T00:29:08.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.057+0000 7effdfb5ab80 20 client.25174 trim_cache size 0 max 16384 2024-06-10T00:29:08.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.095+0000 7effdfb5ab80 1 Processor -- start 2024-06-10T00:29:08.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.095+0000 7effdfb5ab80 1 -- start start 2024-06-10T00:29:08.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.095+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f4628fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.095+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a0cd0 0x564f462a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a15f0 0x564f462a1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7effd01b2ee0 con 0x564f462a0cd0 2024-06-10T00:29:08.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7effd01b0770 con 0x564f462a15f0 2024-06-10T00:29:08.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7effd01ae5c0 con 0x564f4629b460 2024-06-10T00:29:08.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effce7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a0cd0 0x564f462a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effce7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a0cd0 0x564f462a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38848/0 (socket says 172.21.15.121:38848) 2024-06-10T00:29:08.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effce7fc700 1 -- 172.21.15.121:0/2618696425 learned_addr learned my addr 172.21.15.121:0/2618696425 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:08.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effceffd700 1 --2- 172.21.15.121:0/2618696425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f4628fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effcf7fe700 1 --2- 172.21.15.121:0/2618696425 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a15f0 0x564f462a1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effceffd700 1 -- 172.21.15.121:0/2618696425 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a15f0 msgr2=0x564f462a1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:08.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effceffd700 1 --2- 172.21.15.121:0/2618696425 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a15f0 0x564f462a1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effceffd700 1 -- 172.21.15.121:0/2618696425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a0cd0 msgr2=0x564f462a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:08.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effceffd700 1 --2- 172.21.15.121:0/2618696425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a0cd0 0x564f462a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.096+0000 7effceffd700 1 -- 172.21.15.121:0/2618696425 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f462a4e80 con 0x564f4629b460 2024-06-10T00:29:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.097+0000 7effceffd700 1 --2- 172.21.15.121:0/2618696425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f4628fa50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7effc009a8a0 tx=0x7effc000bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b19a2441371e8127 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.097+0000 7effb4ff9700 1 -- 172.21.15.121:0/2618696425 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc00a51d0 con 0x564f4629b460 2024-06-10T00:29:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.097+0000 7effb4ff9700 1 -- 172.21.15.121:0/2618696425 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effc00b5070 con 0x564f4629b460 2024-06-10T00:29:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.097+0000 7effb4ff9700 1 -- 172.21.15.121:0/2618696425 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc00cdda0 con 0x564f4629b460 2024-06-10T00:29:08.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.098+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2618696425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 msgr2=0x564f4628fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:08.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.098+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/2618696425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f4628fa50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7effc009a8a0 tx=0x7effc000bfd0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.098+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2618696425 shutdown_connections 2024-06-10T00:29:08.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.098+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/2618696425 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a15f0 0x564f462a1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.098+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/2618696425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f4628fa50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.098+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/2618696425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a0cd0 0x564f462a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.098+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2618696425 >> 172.21.15.121:0/2618696425 conn(0x564f461c5e50 msgr2=0x564f462a2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.098+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2618696425 shutdown_connections 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.098+0000 7effdfb5ab80 1 -- 172.21.15.121:0/2618696425 wait complete. 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.099+0000 7effdfb5ab80 1 -- start start 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.099+0000 7effdfb5ab80 1 Processor -- start 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.099+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f462bc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.099+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462bcc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.099+0000 7effdfb5ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f462bd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7effd01aec30 con 0x564f462a15f0 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7effd01b00a0 con 0x564f462a0cd0 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effdfb5ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7effd01af300 con 0x564f4629b460 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effce7fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462bcc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effce7fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462bcc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:49194/0 (socket says 172.21.15.121:49194) 2024-06-10T00:29:08.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effce7fc700 1 -- 172.21.15.121:0/3455358825 learned_addr learned my addr 172.21.15.121:0/3455358825 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:08.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effdfb5ab80 10 client.0 _ll_register_callbacks cb 0x564f46186ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:08.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effceffd700 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f462bc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effce7fc700 1 -- 172.21.15.121:0/3455358825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 msgr2=0x564f462bc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:08.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effce7fc700 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f462bc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effce7fc700 1 -- 172.21.15.121:0/3455358825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 msgr2=0x564f462bd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:08.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effce7fc700 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f462bd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effce7fc700 1 -- 172.21.15.121:0/3455358825 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f46327280 con 0x564f462a0cd0 2024-06-10T00:29:08.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.100+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3455358825 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effc00b5070 con 0x564f462a0cd0 2024-06-10T00:29:08.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.101+0000 7effce7fc700 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462bcc20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effc400e630 tx=0x7effc401cd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86e4ed4da39d96d0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:08.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.101+0000 7effb2ff5700 10 client.25244 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:29:08.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.101+0000 7effb2ff5700 1 -- 172.21.15.121:0/3455358825 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc4014040 con 0x564f462a0cd0 2024-06-10T00:29:08.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.101+0000 7effb2ff5700 1 -- 172.21.15.121:0/3455358825 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7effc4010b90 con 0x564f462a0cd0 2024-06-10T00:29:08.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.101+0000 7effb2ff5700 1 -- 172.21.15.121:0/3455358825 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7effc4020040 con 0x564f462a0cd0 2024-06-10T00:29:08.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.101+0000 7effdfb5ab80 10 client.25244 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:08.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.101+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3455358825 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({fsmap=26}) -- 0x564f463283f0 con 0x564f462a0cd0 2024-06-10T00:29:08.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.102+0000 7effb2ff5700 1 -- 172.21.15.121:0/3455358825 <== mon.2 v2:172.21.15.135:3300/0 4 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7effc401c420 con 0x564f462a0cd0 2024-06-10T00:29:08.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.102+0000 7effdfb5ab80 10 client.25244 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:08.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.102+0000 7effdfb5ab80 10 client.25244 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:29:08.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.102+0000 7effdfb5ab80 20 client.25244 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:08.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.102+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x564f463288a0 0x564f4632acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:08.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.102+0000 7effdfb5ab80 4 client.25244 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-06-10T00:29:08.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.102+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3455358825 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x564f461f4420 con 0x564f463288a0 2024-06-10T00:29:08.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.102+0000 7effceffd700 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x564f463288a0 0x564f4632acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:08.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.103+0000 7effb2ff5700 10 client.25244 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:08.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.103+0000 7effceffd700 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x564f463288a0 0x564f4632acc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f8c8fe2ac4439a3 server_cookie=b1bcce2a7e161b4a in_seq=0 out_seq=0 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.112+0000 7effb2ff5700 1 -- 172.21.15.121:0/3455358825 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x564f461f4420 con 0x564f463288a0 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.112+0000 7effb2ff5700 10 client.25244 handle_command_reply: tid=0 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.112+0000 7effdfb5ab80 1 client.25244 shutdown 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.112+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3455358825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 msgr2=0x564f462bcc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.112+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462bcc20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7effc400e630 tx=0x7effc401cd20 comp rx=0 tx=0).stop 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.113+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3455358825 shutdown_connections 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.113+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x564f463288a0 0x564f4632acc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.113+0000 7effb2ff5700 0 client.25244 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.113+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f462a15f0 0x564f462bd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.113+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f4629b460 0x564f462bc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.113+0000 7effdfb5ab80 1 --2- 172.21.15.121:0/3455358825 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564f462a0cd0 0x564f462bcc20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.113+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3455358825 >> 172.21.15.121:0/3455358825 conn(0x564f461c5e50 msgr2=0x564f462958c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:08.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.113+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3455358825 shutdown_connections 2024-06-10T00:29:08.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.113+0000 7effdfb5ab80 1 -- 172.21.15.121:0/3455358825 wait complete. 2024-06-10T00:29:08.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.113+0000 7effdfb5ab80 20 client.25244 trim_cache size 0 max 16384 2024-06-10T00:29:08.114 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:29:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 26, 2024-06-10T00:29:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:29:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 2, 2024-06-10T00:29:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:29:08.114 INFO:teuthology.orchestra.run.smithi121.stdout: "25c0a77": { 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 2, 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 150.0, 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:29:08.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:08.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:08.116 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:29:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.114+0000 7effd52d7700 1 -- 172.21.15.121:0/3148054880 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7effb80a2fd0 msgr2=0x7effb80a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:08.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.114+0000 7effd52d7700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7effb80a2fd0 0x7effb80a5490 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7effd00fffc0 tx=0x7effc4009260 comp rx=0 tx=0).stop 2024-06-10T00:29:08.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.114+0000 7effd52d7700 1 -- 172.21.15.121:0/3148054880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd0100a80 msgr2=0x7effd019ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:08.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.114+0000 7effd52d7700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd0100a80 0x7effd019ada0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7effc0000c00 tx=0x7effc000bb60 comp rx=0 tx=0).stop 2024-06-10T00:29:08.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.115+0000 7effd52d7700 1 -- 172.21.15.121:0/3148054880 shutdown_connections 2024-06-10T00:29:08.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.115+0000 7effd52d7700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7effb80a2fd0 0x7effb80a5490 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.115+0000 7effd52d7700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7effd01033a0 0x7effd019f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.115+0000 7effd52d7700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd0100a80 0x7effd019ada0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.115+0000 7effd52d7700 1 --2- 172.21.15.121:0/3148054880 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7effd0101520 0x7effd019b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:08.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.115+0000 7effd52d7700 1 -- 172.21.15.121:0/3148054880 >> 172.21.15.121:0/3148054880 conn(0x7effd0077a30 msgr2=0x7effd00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:08.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.115+0000 7effd52d7700 1 -- 172.21.15.121:0/3148054880 shutdown_connections 2024-06-10T00:29:08.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:08.115+0000 7effd52d7700 1 -- 172.21.15.121:0/3148054880 wait complete. 2024-06-10T00:29:08.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:08 smithi121 ceph-mon[32378]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-06-10T00:29:08.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:08 smithi121 ceph-mon[32378]: scrub queued for path: / 2024-06-10T00:29:08.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:08 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:08.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:08 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:29:08.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:08 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:08.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:08 smithi193 ceph-mon[36716]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-06-10T00:29:08.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:08 smithi193 ceph-mon[36716]: scrub queued for path: / 2024-06-10T00:29:08.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:08 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:08.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:08 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:29:08.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:08 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:08.909 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:29:09.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:08 smithi135 ceph-mon[36424]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-06-10T00:29:09.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:08 smithi135 ceph-mon[36424]: scrub queued for path: / 2024-06-10T00:29:09.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:08 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:09.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:08 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:29:09.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:08 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:09.777 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '25c0a77', quiesce took 9.8 seconds. Will release after: 20.2 2024-06-10T00:29:09.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:09 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:29:09.924 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:09.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:09 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:29:10.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:09 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:29:10.613 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:10 smithi121 ceph-mon[32378]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-06-10T00:29:10.613 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:10 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:29:10.613 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:10 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:29:10.613 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:10 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:29:10.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:10 smithi193 ceph-mon[36716]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-06-10T00:29:10.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:10 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:29:10.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:10 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:29:10.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:10 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:29:11.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:10 smithi135 ceph-mon[36424]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-06-10T00:29:11.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:10 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:29:11.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:10 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:29:11.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:10 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:29:11.723 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:11 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:29:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.776+0000 7fba79f73700 1 -- 172.21.15.121:0/1531506328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba741073d0 msgr2=0x7fba741077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.776+0000 7fba79f73700 1 --2- 172.21.15.121:0/1531506328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba741073d0 0x7fba741077b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fba64009990 tx=0x7fba64009ca0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.777+0000 7fba79f73700 1 -- 172.21.15.121:0/1531506328 shutdown_connections 2024-06-10T00:29:11.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.777+0000 7fba79f73700 1 --2- 172.21.15.121:0/1531506328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba74101d30 0x7fba7410e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.777+0000 7fba79f73700 1 --2- 172.21.15.121:0/1531506328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba741073d0 0x7fba741077b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.777+0000 7fba79f73700 1 --2- 172.21.15.121:0/1531506328 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba74101390 0x7fba741017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.777+0000 7fba79f73700 1 -- 172.21.15.121:0/1531506328 >> 172.21.15.121:0/1531506328 conn(0x7fba740782c0 msgr2=0x7fba740786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.777+0000 7fba79f73700 1 -- 172.21.15.121:0/1531506328 shutdown_connections 2024-06-10T00:29:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.777+0000 7fba79f73700 1 -- 172.21.15.121:0/1531506328 wait complete. 2024-06-10T00:29:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.777+0000 7fba79f73700 1 Processor -- start 2024-06-10T00:29:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.778+0000 7fba79f73700 1 -- start start 2024-06-10T00:29:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.778+0000 7fba79f73700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba74101390 0x7fba7419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.778+0000 7fba79f73700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba74101d30 0x7fba7419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.778+0000 7fba79f73700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba741073d0 0x7fba74198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.779 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.778+0000 7fba79f73700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fba74111660 con 0x7fba74101d30 2024-06-10T00:29:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.778+0000 7fba79f73700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fba741114e0 con 0x7fba74101390 2024-06-10T00:29:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.778+0000 7fba79f73700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba741117e0 con 0x7fba741073d0 2024-06-10T00:29:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.778+0000 7fba737fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba74101390 0x7fba7419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.778+0000 7fba737fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba74101390 0x7fba7419e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:60564/0 (socket says 172.21.15.121:60564) 2024-06-10T00:29:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.778+0000 7fba737fe700 1 -- 172.21.15.121:0/108768439 learned_addr learned my addr 172.21.15.121:0/108768439 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:11.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.779+0000 7fba73fff700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba741073d0 0x7fba74198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.779+0000 7fba72ffd700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba74101d30 0x7fba7419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.779+0000 7fba737fe700 1 -- 172.21.15.121:0/108768439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba741073d0 msgr2=0x7fba74198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:11.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.779+0000 7fba737fe700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba741073d0 0x7fba74198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.779+0000 7fba737fe700 1 -- 172.21.15.121:0/108768439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba74101d30 msgr2=0x7fba7419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.779+0000 7fba737fe700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba74101d30 0x7fba7419ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.779+0000 7fba737fe700 1 -- 172.21.15.121:0/108768439 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba74199070 con 0x7fba74101390 2024-06-10T00:29:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.779+0000 7fba73fff700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba741073d0 0x7fba74198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.779+0000 7fba72ffd700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba74101d30 0x7fba7419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.780+0000 7fba737fe700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba74101390 0x7fba7419e6c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba6400b5c0 tx=0x7fba640057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.780+0000 7fba70ff9700 1 -- 172.21.15.121:0/108768439 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba64026070 con 0x7fba74101390 2024-06-10T00:29:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.780+0000 7fba79f73700 1 -- 172.21.15.121:0/108768439 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba74199300 con 0x7fba74101390 2024-06-10T00:29:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.780+0000 7fba79f73700 1 -- 172.21.15.121:0/108768439 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba741a5520 con 0x7fba74101390 2024-06-10T00:29:11.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.780+0000 7fba70ff9700 1 -- 172.21.15.121:0/108768439 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba64004190 con 0x7fba74101390 2024-06-10T00:29:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.780+0000 7fba70ff9700 1 -- 172.21.15.121:0/108768439 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba64021410 con 0x7fba74101390 2024-06-10T00:29:11.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.784+0000 7fba70ff9700 1 -- 172.21.15.121:0/108768439 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7fba64003cb0 con 0x7fba74101390 2024-06-10T00:29:11.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.785+0000 7fba70ff9700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba5c0a2f30 0x7fba5c0a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.785+0000 7fba72ffd700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba5c0a2f30 0x7fba5c0a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.785+0000 7fba70ff9700 1 -- 172.21.15.121:0/108768439 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fba640d0910 con 0x7fba74101390 2024-06-10T00:29:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.785+0000 7fba847f6b80 1 Processor -- start 2024-06-10T00:29:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.786+0000 7fba847f6b80 1 -- start start 2024-06-10T00:29:11.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.786+0000 7fba72ffd700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba5c0a2f30 0x7fba5c0a53f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fba68005fd0 tx=0x7fba68005e30 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.786+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 0x55fa67576340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.786+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa67576880 0x55fa67570260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.786+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa675707a0 0x55fa67570bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.786+0000 7fba847f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fa67462f60 con 0x55fa67576880 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.786+0000 7fba72ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa67576880 0x55fa67570260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.786+0000 7fba72ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa67576880 0x55fa67570260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44448/0 (socket says 172.21.15.121:44448) 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.786+0000 7fba72ffd700 1 -- 172.21.15.121:0/2181284029 learned_addr learned my addr 172.21.15.121:0/2181284029 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.786+0000 7fba737fe700 1 --2- 172.21.15.121:0/2181284029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 0x55fa67576340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.787+0000 7fba847f6b80 1 -- 172.21.15.121:0/2181284029 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55fa674c1850 con 0x55fa675707a0 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.787+0000 7fba847f6b80 1 -- 172.21.15.121:0/2181284029 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fa67429460 con 0x55fa67575f60 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.787+0000 7fba73fff700 1 --2- 172.21.15.121:0/2181284029 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa675707a0 0x55fa67570bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.787+0000 7fba72ffd700 1 -- 172.21.15.121:0/2181284029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 msgr2=0x55fa67576340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.787+0000 7fba72ffd700 1 --2- 172.21.15.121:0/2181284029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 0x55fa67576340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.787+0000 7fba72ffd700 1 -- 172.21.15.121:0/2181284029 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa675707a0 msgr2=0x55fa67570bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.787+0000 7fba72ffd700 1 --2- 172.21.15.121:0/2181284029 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa675707a0 0x55fa67570bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.787+0000 7fba72ffd700 1 -- 172.21.15.121:0/2181284029 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa674108d0 con 0x55fa67576880 2024-06-10T00:29:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.787+0000 7fba73fff700 1 --2- 172.21.15.121:0/2181284029 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa675707a0 0x55fa67570bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.787+0000 7fba72ffd700 1 --2- 172.21.15.121:0/2181284029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa67576880 0x55fa67570260 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fba6800eef0 tx=0x7fba68011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1584128fcdad6559 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:11.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.788+0000 7fba59ffb700 1 -- 172.21.15.121:0/2181284029 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68021070 con 0x55fa67576880 2024-06-10T00:29:11.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.788+0000 7fba59ffb700 1 -- 172.21.15.121:0/2181284029 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba68010da0 con 0x55fa67576880 2024-06-10T00:29:11.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.788+0000 7fba59ffb700 1 -- 172.21.15.121:0/2181284029 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6800f790 con 0x55fa67576880 2024-06-10T00:29:11.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.789+0000 7fba847f6b80 1 -- 172.21.15.121:0/2181284029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa67576880 msgr2=0x55fa67570260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:11.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.789+0000 7fba847f6b80 1 --2- 172.21.15.121:0/2181284029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa67576880 0x55fa67570260 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fba6800eef0 tx=0x7fba68011000 comp rx=0 tx=0).stop 2024-06-10T00:29:11.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.789+0000 7fba847f6b80 1 -- 172.21.15.121:0/2181284029 shutdown_connections 2024-06-10T00:29:11.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.789+0000 7fba847f6b80 1 --2- 172.21.15.121:0/2181284029 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa675707a0 0x55fa67570bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.789+0000 7fba847f6b80 1 --2- 172.21.15.121:0/2181284029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 0x55fa67576340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.789+0000 7fba847f6b80 1 --2- 172.21.15.121:0/2181284029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa67576880 0x55fa67570260 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.789+0000 7fba847f6b80 1 -- 172.21.15.121:0/2181284029 >> 172.21.15.121:0/2181284029 conn(0x55fa6749d080 msgr2=0x55fa6756b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.790+0000 7fba847f6b80 1 -- 172.21.15.121:0/2181284029 shutdown_connections 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.790+0000 7fba847f6b80 1 -- 172.21.15.121:0/2181284029 wait complete. 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.790+0000 7fba847f6b80 1 -- start start 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.790+0000 7fba847f6b80 1 Processor -- start 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.790+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa675707a0 0x55fa6757c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.790+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67575f60 0x55fa675798e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.790+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa67576880 0x55fa67579e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.790+0000 7fba847f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fa67461fc0 con 0x55fa67576880 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.790+0000 7fba847f6b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55fa674bd810 con 0x55fa67575f60 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.790+0000 7fba847f6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fa6757d260 con 0x55fa675707a0 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba847f6b80 10 client.0 _ll_register_callbacks cb 0x55fa674575b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:11.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba737fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa675707a0 0x55fa6757c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba737fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa675707a0 0x55fa6757c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:34468/0 (socket says 172.21.15.121:34468) 2024-06-10T00:29:11.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba737fe700 1 -- 172.21.15.121:0/568244018 learned_addr learned my addr 172.21.15.121:0/568244018 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:11.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba72ffd700 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67575f60 0x55fa675798e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba737fe700 1 -- 172.21.15.121:0/568244018 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67575f60 msgr2=0x55fa675798e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:11.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba737fe700 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67575f60 0x55fa675798e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba737fe700 1 -- 172.21.15.121:0/568244018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa67576880 msgr2=0x55fa67579e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-10T00:29:11.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba737fe700 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa67576880 0x55fa67579e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba737fe700 1 -- 172.21.15.121:0/568244018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa67487750 con 0x55fa675707a0 2024-06-10T00:29:11.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba737fe700 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa675707a0 0x55fa6757c5e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fba64006010 tx=0x7fba6409c310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60577bd89dadd1b server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:11.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba52ffd700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:29:11.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba52ffd700 1 -- 172.21.15.121:0/568244018 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba640872c0 con 0x55fa675707a0 2024-06-10T00:29:11.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.791+0000 7fba847f6b80 1 -- 172.21.15.121:0/568244018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba68010da0 con 0x55fa675707a0 2024-06-10T00:29:11.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba52ffd700 1 -- 172.21.15.121:0/568244018 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fba640a5030 con 0x55fa675707a0 2024-06-10T00:29:11.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba52ffd700 1 -- 172.21.15.121:0/568244018 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fba640a0c90 con 0x55fa675707a0 2024-06-10T00:29:11.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba847f6b80 10 client.25192 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:11.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba847f6b80 1 -- 172.21.15.121:0/568244018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fa67587a40 con 0x55fa675707a0 2024-06-10T00:29:11.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba52ffd700 1 -- 172.21.15.121:0/568244018 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba640a54c0 con 0x55fa675707a0 2024-06-10T00:29:11.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba52ffd700 1 -- 172.21.15.121:0/568244018 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fba6409bd30 con 0x55fa675707a0 2024-06-10T00:29:11.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba847f6b80 10 client.25192 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:11.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba847f6b80 10 client.25192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:11.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba847f6b80 20 client.25192 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:11.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba847f6b80 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fa6758a000 0x55fa6758c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba847f6b80 4 client.25192 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:11.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.792+0000 7fba847f6b80 1 -- 172.21.15.121:0/568244018 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fba68010da0 con 0x55fa6758a000 2024-06-10T00:29:11.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.793+0000 7fba72ffd700 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fa6758a000 0x55fa6758c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.793+0000 7fba52ffd700 10 client.25192 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:11.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.793+0000 7fba72ffd700 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fa6758a000 0x55fa6758c420 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f28f3c47e9440fc1 server_cookie=a8dbc7b8c6bddf3a in_seq=0 out_seq=0 2024-06-10T00:29:11.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba52ffd700 1 -- 172.21.15.121:0/568244018 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x7fba68010da0 con 0x55fa6758a000 2024-06-10T00:29:11.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba52ffd700 10 client.25192 handle_command_reply: tid=0 2024-06-10T00:29:11.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba847f6b80 1 client.25192 shutdown 2024-06-10T00:29:11.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba847f6b80 1 -- 172.21.15.121:0/568244018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa675707a0 msgr2=0x55fa6757c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:11.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba847f6b80 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa675707a0 0x55fa6757c5e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fba64006010 tx=0x7fba6409c310 comp rx=0 tx=0).stop 2024-06-10T00:29:11.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba847f6b80 1 -- 172.21.15.121:0/568244018 shutdown_connections 2024-06-10T00:29:11.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba847f6b80 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fa6758a000 0x55fa6758c420 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba847f6b80 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa67576880 0x55fa67579e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba52ffd700 0 client.25192 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:11.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba847f6b80 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa675707a0 0x55fa6757c5e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba847f6b80 1 --2- 172.21.15.121:0/568244018 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67575f60 0x55fa675798e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba847f6b80 1 -- 172.21.15.121:0/568244018 >> 172.21.15.121:0/568244018 conn(0x55fa6749d080 msgr2=0x55fa6756a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:11.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba847f6b80 1 -- 172.21.15.121:0/568244018 shutdown_connections 2024-06-10T00:29:11.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.794+0000 7fba847f6b80 1 -- 172.21.15.121:0/568244018 wait complete. 2024-06-10T00:29:11.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.795+0000 7fba847f6b80 20 client.25192 trim_cache size 0 max 16384 2024-06-10T00:29:11.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.815+0000 7fba847f6b80 1 Processor -- start 2024-06-10T00:29:11.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.815+0000 7fba847f6b80 1 -- start start 2024-06-10T00:29:11.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.815+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa675707a0 0x55fa6756ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.815+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67575f60 0x55fa6756b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.815+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67576880 0x55fa67591d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.815+0000 7fba847f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fba60011580 con 0x55fa675707a0 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.815+0000 7fba847f6b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fba6000ee10 con 0x55fa67575f60 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.815+0000 7fba847f6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba6000cc40 con 0x55fa67576880 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba73fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67576880 0x55fa67591d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba73fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67576880 0x55fa67591d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:34476/0 (socket says 172.21.15.121:34476) 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba73fff700 1 -- 172.21.15.121:0/3185622498 learned_addr learned my addr 172.21.15.121:0/3185622498 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba73fff700 1 -- 172.21.15.121:0/3185622498 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67575f60 msgr2=0x55fa6756b3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba73fff700 1 --2- 172.21.15.121:0/3185622498 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67575f60 0x55fa6756b3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba73fff700 1 -- 172.21.15.121:0/3185622498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa675707a0 msgr2=0x55fa6756ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba73fff700 1 --2- 172.21.15.121:0/3185622498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa675707a0 0x55fa6756ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba73fff700 1 -- 172.21.15.121:0/3185622498 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa67575a00 con 0x55fa67576880 2024-06-10T00:29:11.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba73fff700 1 --2- 172.21.15.121:0/3185622498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67576880 0x55fa67591d10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba6c007690 tx=0x7fba6c0076c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=961b5681e12ded18 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba597fa700 1 -- 172.21.15.121:0/3185622498 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c01a030 con 0x55fa67576880 2024-06-10T00:29:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba597fa700 1 -- 172.21.15.121:0/3185622498 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba6c016910 con 0x55fa67576880 2024-06-10T00:29:11.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.816+0000 7fba597fa700 1 -- 172.21.15.121:0/3185622498 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c01eb00 con 0x55fa67576880 2024-06-10T00:29:11.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.817+0000 7fba847f6b80 1 -- 172.21.15.121:0/3185622498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67576880 msgr2=0x55fa67591d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:11.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.817+0000 7fba847f6b80 1 --2- 172.21.15.121:0/3185622498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67576880 0x55fa67591d10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fba6c007690 tx=0x7fba6c0076c0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.817+0000 7fba847f6b80 1 -- 172.21.15.121:0/3185622498 shutdown_connections 2024-06-10T00:29:11.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.817+0000 7fba847f6b80 1 --2- 172.21.15.121:0/3185622498 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67576880 0x55fa67591d10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.817+0000 7fba847f6b80 1 --2- 172.21.15.121:0/3185622498 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa675707a0 0x55fa6756ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.817+0000 7fba847f6b80 1 --2- 172.21.15.121:0/3185622498 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67575f60 0x55fa6756b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.817+0000 7fba847f6b80 1 -- 172.21.15.121:0/3185622498 >> 172.21.15.121:0/3185622498 conn(0x55fa6749d080 msgr2=0x55fa6757b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:11.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.818+0000 7fba847f6b80 1 -- 172.21.15.121:0/3185622498 shutdown_connections 2024-06-10T00:29:11.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.818+0000 7fba847f6b80 1 -- 172.21.15.121:0/3185622498 wait complete. 2024-06-10T00:29:11.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.818+0000 7fba847f6b80 1 -- start start 2024-06-10T00:29:11.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.818+0000 7fba847f6b80 1 Processor -- start 2024-06-10T00:29:11.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa675707a0 0x55fa67582bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 0x55fa6757f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba847f6b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67576880 0x55fa6757fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba847f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fba6000d2d0 con 0x55fa675707a0 2024-06-10T00:29:11.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba847f6b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fba6000e740 con 0x55fa67576880 2024-06-10T00:29:11.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba847f6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba6000d9a0 con 0x55fa67575f60 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba847f6b80 10 client.0 _ll_register_callbacks cb 0x55fa674575b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba72ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 0x55fa6757f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba737fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa675707a0 0x55fa67582bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba72ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 0x55fa6757f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:34478/0 (socket says 172.21.15.121:34478) 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba72ffd700 1 -- 172.21.15.121:0/2084380503 learned_addr learned my addr 172.21.15.121:0/2084380503 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba72ffd700 1 -- 172.21.15.121:0/2084380503 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67576880 msgr2=0x55fa6757fa90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba72ffd700 1 --2- 172.21.15.121:0/2084380503 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67576880 0x55fa6757fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba72ffd700 1 -- 172.21.15.121:0/2084380503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa675707a0 msgr2=0x55fa67582bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba72ffd700 1 --2- 172.21.15.121:0/2084380503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa675707a0 0x55fa67582bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.819+0000 7fba72ffd700 1 -- 172.21.15.121:0/2084380503 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa675801d0 con 0x55fa67575f60 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.820+0000 7fba72ffd700 1 --2- 172.21.15.121:0/2084380503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 0x55fa6757f550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba680109f0 tx=0x7fba680047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d22e6e0483bb82e2 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.820+0000 7fba52ffd700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.820+0000 7fba52ffd700 1 -- 172.21.15.121:0/2084380503 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68021040 con 0x55fa67575f60 2024-06-10T00:29:11.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.820+0000 7fba847f6b80 1 -- 172.21.15.121:0/2084380503 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba6c016910 con 0x55fa67575f60 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.820+0000 7fba52ffd700 1 -- 172.21.15.121:0/2084380503 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fba68004c60 con 0x55fa67575f60 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.820+0000 7fba52ffd700 1 -- 172.21.15.121:0/2084380503 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fba68041050 con 0x55fa67575f60 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.820+0000 7fba847f6b80 10 client.25198 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.820+0000 7fba847f6b80 1 -- 172.21.15.121:0/2084380503 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fa675f5c80 con 0x55fa67575f60 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.820+0000 7fba52ffd700 1 -- 172.21.15.121:0/2084380503 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba6c016910 con 0x55fa67575f60 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.821+0000 7fba52ffd700 1 -- 172.21.15.121:0/2084380503 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fba6801d6b0 con 0x55fa67575f60 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.821+0000 7fba847f6b80 10 client.25198 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.821+0000 7fba847f6b80 10 client.25198 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.821+0000 7fba847f6b80 20 client.25198 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.821+0000 7fba847f6b80 1 --2- 172.21.15.121:0/2084380503 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fa6758a000 0x55fa675f8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.821+0000 7fba847f6b80 4 client.25198 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.821+0000 7fba847f6b80 1 -- 172.21.15.121:0/2084380503 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fa674ced50 con 0x55fa6758a000 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.821+0000 7fba737fe700 1 --2- 172.21.15.121:0/2084380503 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fa6758a000 0x55fa675f8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:11.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.821+0000 7fba52ffd700 10 client.25198 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:11.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.821+0000 7fba737fe700 1 --2- 172.21.15.121:0/2084380503 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fa6758a000 0x55fa675f8120 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a474e5ca5b638045 server_cookie=43d598d1e8f2ad3c in_seq=0 out_seq=0 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba52ffd700 1 -- 172.21.15.121:0/2084380503 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fa674ced50 con 0x55fa6758a000 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba52ffd700 10 client.25198 handle_command_reply: tid=0 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 1 client.25198 shutdown 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 1 -- 172.21.15.121:0/2084380503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 msgr2=0x55fa6757f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 1 --2- 172.21.15.121:0/2084380503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 0x55fa6757f550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba680109f0 tx=0x7fba680047c0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 1 -- 172.21.15.121:0/2084380503 shutdown_connections 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 1 --2- 172.21.15.121:0/2084380503 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fa6758a000 0x55fa675f8120 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 1 --2- 172.21.15.121:0/2084380503 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fa67576880 0x55fa6757fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 1 --2- 172.21.15.121:0/2084380503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fa675707a0 0x55fa67582bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 1 --2- 172.21.15.121:0/2084380503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa67575f60 0x55fa6757f550 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 1 -- 172.21.15.121:0/2084380503 >> 172.21.15.121:0/2084380503 conn(0x55fa6749d080 msgr2=0x55fa675749f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:11.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba52ffd700 0 client.25198 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:11.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 1 -- 172.21.15.121:0/2084380503 shutdown_connections 2024-06-10T00:29:11.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 1 -- 172.21.15.121:0/2084380503 wait complete. 2024-06-10T00:29:11.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.822+0000 7fba847f6b80 20 client.25198 trim_cache size 0 max 16384 2024-06-10T00:29:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:29:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "no active scrubs running", 2024-06-10T00:29:11.827 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": {} 2024-06-10T00:29:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:29:11.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.823+0000 7fba79f73700 1 -- 172.21.15.121:0/108768439 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba5c0a2f30 msgr2=0x7fba5c0a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:11.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.823+0000 7fba79f73700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba5c0a2f30 0x7fba5c0a53f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fba68005fd0 tx=0x7fba68005e30 comp rx=0 tx=0).stop 2024-06-10T00:29:11.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.823+0000 7fba79f73700 1 -- 172.21.15.121:0/108768439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba74101390 msgr2=0x7fba7419e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:11.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.823+0000 7fba79f73700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba74101390 0x7fba7419e6c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fba6400b5c0 tx=0x7fba640057c0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.823+0000 7fba79f73700 1 -- 172.21.15.121:0/108768439 shutdown_connections 2024-06-10T00:29:11.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.823+0000 7fba79f73700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba5c0a2f30 0x7fba5c0a53f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.823+0000 7fba79f73700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba741073d0 0x7fba74198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.823+0000 7fba79f73700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba74101390 0x7fba7419e6c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.823+0000 7fba79f73700 1 --2- 172.21.15.121:0/108768439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba74101d30 0x7fba7419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:11.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.823+0000 7fba79f73700 1 -- 172.21.15.121:0/108768439 >> 172.21.15.121:0/108768439 conn(0x7fba740782c0 msgr2=0x7fba74100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:11.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.824+0000 7fba79f73700 1 -- 172.21.15.121:0/108768439 shutdown_connections 2024-06-10T00:29:11.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:11.824+0000 7fba79f73700 1 -- 172.21.15.121:0/108768439 wait complete. 2024-06-10T00:29:11.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:11 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:29:12.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:11 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:29:12.699 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-10T00:29:12.699 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:29:12.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:12 smithi193 ceph-mon[36716]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-06-10T00:29:12.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:12 smithi121 ceph-mon[32378]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-06-10T00:29:12.963 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:13.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:12 smithi135 ceph-mon[36424]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-06-10T00:29:14.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.818+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/4187489401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44107400 msgr2=0x7f4b441077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:14.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.818+0000 7f4b4a6b5700 1 --2- 172.21.15.121:0/4187489401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44107400 0x7f4b441077e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4b34009990 tx=0x7f4b34009ca0 comp rx=0 tx=0).stop 2024-06-10T00:29:14.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.818+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/4187489401 shutdown_connections 2024-06-10T00:29:14.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.818+0000 7f4b4a6b5700 1 --2- 172.21.15.121:0/4187489401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b44101d50 0x7f4b4410e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:14.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.818+0000 7f4b4a6b5700 1 --2- 172.21.15.121:0/4187489401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44107400 0x7f4b441077e0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:14.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.818+0000 7f4b4a6b5700 1 --2- 172.21.15.121:0/4187489401 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b44101390 0x7f4b44101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:14.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.818+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/4187489401 >> 172.21.15.121:0/4187489401 conn(0x7f4b44078340 msgr2=0x7f4b44078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:14.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.818+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/4187489401 shutdown_connections 2024-06-10T00:29:14.819 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:14 smithi121 ceph-mon[32378]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-06-10T00:29:14.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.819+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/4187489401 wait complete. 2024-06-10T00:29:14.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.820+0000 7f4b4a6b5700 1 Processor -- start 2024-06-10T00:29:14.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.820+0000 7f4b4a6b5700 1 -- start start 2024-06-10T00:29:14.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.821+0000 7f4b4a6b5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44101390 0x7f4b4419e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:14.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.821+0000 7f4b4a6b5700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b44101d50 0x7f4b4419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:14.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.821+0000 7f4b4a6b5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b44107400 0x7f4b44198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:14.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.821+0000 7f4b4a6b5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4b441116f0 con 0x7f4b44101390 2024-06-10T00:29:14.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.821+0000 7f4b4a6b5700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44111570 con 0x7f4b44101d50 2024-06-10T00:29:14.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.821+0000 7f4b4a6b5700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b44111870 con 0x7f4b44107400 2024-06-10T00:29:14.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.821+0000 7f4b43fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44101390 0x7f4b4419e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:14.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.821+0000 7f4b43fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44101390 0x7f4b4419e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44484/0 (socket says 172.21.15.121:44484) 2024-06-10T00:29:14.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.821+0000 7f4b43fff700 1 -- 172.21.15.121:0/3692025085 learned_addr learned my addr 172.21.15.121:0/3692025085 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:14.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.821+0000 7f4b437fe700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b44101d50 0x7f4b4419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:14.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.822+0000 7f4b48c52700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b44107400 0x7f4b44198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:14.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.822+0000 7f4b43fff700 1 -- 172.21.15.121:0/3692025085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b44107400 msgr2=0x7f4b44198dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:14.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.822+0000 7f4b43fff700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b44107400 0x7f4b44198dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:14.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.822+0000 7f4b43fff700 1 -- 172.21.15.121:0/3692025085 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b44101d50 msgr2=0x7f4b4419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:14.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.822+0000 7f4b43fff700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b44101d50 0x7f4b4419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:14.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.822+0000 7f4b43fff700 1 -- 172.21.15.121:0/3692025085 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b44199610 con 0x7f4b44101390 2024-06-10T00:29:14.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.822+0000 7f4b43fff700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44101390 0x7f4b4419e9f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4b34005910 tx=0x7f4b340047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:14.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.822+0000 7f4b417fa700 1 -- 172.21.15.121:0/3692025085 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34027070 con 0x7f4b44101390 2024-06-10T00:29:14.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.823+0000 7f4b417fa700 1 -- 172.21.15.121:0/3692025085 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b34003780 con 0x7f4b44101390 2024-06-10T00:29:14.823 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.823+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/3692025085 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b441998a0 con 0x7f4b44101390 2024-06-10T00:29:14.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.824+0000 7f4b417fa700 1 -- 172.21.15.121:0/3692025085 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b34022da0 con 0x7f4b44101390 2024-06-10T00:29:14.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.824+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/3692025085 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b441a5810 con 0x7f4b44101390 2024-06-10T00:29:14.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.825+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/3692025085 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b44102ab0 con 0x7f4b44101390 2024-06-10T00:29:14.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.826+0000 7f4b417fa700 1 -- 172.21.15.121:0/3692025085 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f4b3402a070 con 0x7f4b44101390 2024-06-10T00:29:14.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.827+0000 7f4b417fa700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4b2c0a2e10 0x7f4b2c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:14.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.828+0000 7f4b437fe700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4b2c0a2e10 0x7f4b2c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:14.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.828+0000 7f4b417fa700 1 -- 172.21.15.121:0/3692025085 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f4b340d4060 con 0x7f4b44101390 2024-06-10T00:29:14.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.828+0000 7f4b437fe700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4b2c0a2e10 0x7f4b2c0a52d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4b4410e750 tx=0x7f4b3800b410 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:14.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:14.837+0000 7f4b417fa700 1 -- 172.21.15.121:0/3692025085 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4b34013070 con 0x7f4b44101390 2024-06-10T00:29:14.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:14 smithi193 ceph-mon[36716]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-06-10T00:29:15.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:14 smithi135 ceph-mon[36424]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-06-10T00:29:15.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.225+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/3692025085 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b440691b0 con 0x7f4b44101390 2024-06-10T00:29:15.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.226+0000 7f4b417fa700 1 -- 172.21.15.121:0/3692025085 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f4b440691b0 con 0x7f4b44101390 2024-06-10T00:29:15.226 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:29:15.226 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:27:55.370804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:29:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.228+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/3692025085 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4b2c0a2e10 msgr2=0x7f4b2c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.228+0000 7f4b4a6b5700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4b2c0a2e10 0x7f4b2c0a52d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4b4410e750 tx=0x7f4b3800b410 comp rx=0 tx=0).stop 2024-06-10T00:29:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.228+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/3692025085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44101390 msgr2=0x7f4b4419e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.228+0000 7f4b4a6b5700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44101390 0x7f4b4419e9f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4b34005910 tx=0x7f4b340047b0 comp rx=0 tx=0).stop 2024-06-10T00:29:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.229+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/3692025085 shutdown_connections 2024-06-10T00:29:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.229+0000 7f4b4a6b5700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4b2c0a2e10 0x7f4b2c0a52d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.229+0000 7f4b4a6b5700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b44107400 0x7f4b44198dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.229+0000 7f4b4a6b5700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b44101390 0x7f4b4419e9f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:15.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.229+0000 7f4b4a6b5700 1 --2- 172.21.15.121:0/3692025085 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4b44101d50 0x7f4b4419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.229+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/3692025085 >> 172.21.15.121:0/3692025085 conn(0x7f4b44078340 msgr2=0x7f4b440fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.229+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/3692025085 shutdown_connections 2024-06-10T00:29:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:15.229+0000 7f4b4a6b5700 1 -- 172.21.15.121:0/3692025085 wait complete. 2024-06-10T00:29:15.230 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-06-10T00:29:15.880 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:15 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3692025085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:15.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:15 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3692025085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:16.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:15 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3692025085' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:16.024 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-10T00:29:16.375 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:16.759 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:16 smithi121 ceph-mon[32378]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:29:16.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:16 smithi193 ceph-mon[36716]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:29:17.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:16 smithi135 ceph-mon[36424]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:29:18.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.220+0000 7f5e85c76700 1 -- 172.21.15.121:0/3320940886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e80101510 msgr2=0x7f5e8010d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:18.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.220+0000 7f5e85c76700 1 --2- 172.21.15.121:0/3320940886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e80101510 0x7f5e8010d9d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800b0a0 tx=0x7f5e7800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.220+0000 7f5e85c76700 1 -- 172.21.15.121:0/3320940886 shutdown_connections 2024-06-10T00:29:18.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.220+0000 7f5e85c76700 1 --2- 172.21.15.121:0/3320940886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e80101510 0x7f5e8010d9d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.220+0000 7f5e85c76700 1 --2- 172.21.15.121:0/3320940886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e80106b90 0x7f5e80106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.220+0000 7f5e85c76700 1 --2- 172.21.15.121:0/3320940886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e80100b50 0x7f5e80100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.220+0000 7f5e85c76700 1 -- 172.21.15.121:0/3320940886 >> 172.21.15.121:0/3320940886 conn(0x7f5e80077a00 msgr2=0x7f5e800fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:18.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.221+0000 7f5e85c76700 1 -- 172.21.15.121:0/3320940886 shutdown_connections 2024-06-10T00:29:18.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.221+0000 7f5e85c76700 1 -- 172.21.15.121:0/3320940886 wait complete. 2024-06-10T00:29:18.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.222+0000 7f5e85c76700 1 Processor -- start 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.222+0000 7f5e85c76700 1 -- start start 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.222+0000 7f5e85c76700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e80100b50 0x7f5e801a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.222+0000 7f5e85c76700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e80101510 0x7f5e801a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.222+0000 7f5e85c76700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e80106b90 0x7f5e8019a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.222+0000 7f5e85c76700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5e80110620 con 0x7f5e80100b50 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.222+0000 7f5e85c76700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5e801104a0 con 0x7f5e80101510 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.222+0000 7f5e85c76700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e801107a0 con 0x7f5e80106b90 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e80100b50 0x7f5e801a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7effd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e80101510 0x7f5e801a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7effd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e80101510 0x7f5e801a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:60626/0 (socket says 172.21.15.121:60626) 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e80100b50 0x7f5e801a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44502/0 (socket says 172.21.15.121:44502) 2024-06-10T00:29:18.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7effd700 1 -- 172.21.15.121:0/3533638108 learned_addr learned my addr 172.21.15.121:0/3533638108 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:18.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7ffff700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e80106b90 0x7f5e8019a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7effd700 1 -- 172.21.15.121:0/3533638108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e80106b90 msgr2=0x7f5e8019a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:18.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7effd700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e80106b90 0x7f5e8019a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7effd700 1 -- 172.21.15.121:0/3533638108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e80100b50 msgr2=0x7f5e801a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7effd700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e80100b50 0x7f5e801a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7effd700 1 -- 172.21.15.121:0/3533638108 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8019b080 con 0x7f5e80101510 2024-06-10T00:29:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7ffff700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e80106b90 0x7f5e8019a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.223+0000 7f5e7effd700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e80101510 0x7f5e801a0c50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400c9b0 tx=0x7f5e7400ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.224+0000 7f5e7cff9700 1 -- 172.21.15.121:0/3533638108 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74007b00 con 0x7f5e80101510 2024-06-10T00:29:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.224+0000 7f5e85c76700 1 -- 172.21.15.121:0/3533638108 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e8019b370 con 0x7f5e80101510 2024-06-10T00:29:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.224+0000 7f5e7cff9700 1 -- 172.21.15.121:0/3533638108 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e7400ce50 con 0x7f5e80101510 2024-06-10T00:29:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.224+0000 7f5e7cff9700 1 -- 172.21.15.121:0/3533638108 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74018660 con 0x7f5e80101510 2024-06-10T00:29:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.224+0000 7f5e85c76700 1 -- 172.21.15.121:0/3533638108 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e801a75e0 con 0x7f5e80101510 2024-06-10T00:29:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.226+0000 7f5e7cff9700 1 -- 172.21.15.121:0/3533638108 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f5e7400f450 con 0x7f5e80101510 2024-06-10T00:29:18.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.227+0000 7f5e7cff9700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e680a2f10 0x7f5e680a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.227+0000 7f5e7cff9700 1 -- 172.21.15.121:0/3533638108 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f5e740cf9c0 con 0x7f5e80101510 2024-06-10T00:29:18.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.227+0000 7f5e7f7fe700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e680a2f10 0x7f5e680a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.228+0000 7f5e7f7fe700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e680a2f10 0x7f5e680a53d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5e80107230 tx=0x7f5e7000a380 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:18.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e904f9b80 1 Processor -- start 2024-06-10T00:29:18.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e904f9b80 1 -- start start 2024-06-10T00:29:18.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e904f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e3c640 0x560996e3ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e904f9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e369e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e7effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e369e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e7effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e369e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44510/0 (socket says 172.21.15.121:44510) 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e7effd700 1 -- 172.21.15.121:0/483748712 learned_addr learned my addr 172.21.15.121:0/483748712 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/483748712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e36f20 0x560996e37340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e904f9b80 1 -- 172.21.15.121:0/483748712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x560996d88810 con 0x560996e3cf60 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e904f9b80 1 -- 172.21.15.121:0/483748712 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x560996d30720 con 0x560996e36f20 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e7f7fe700 1 --2- 172.21.15.121:0/483748712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e3c640 0x560996e3ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.229+0000 7f5e904f9b80 1 -- 172.21.15.121:0/483748712 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560996d847d0 con 0x560996e3c640 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.230+0000 7f5e7ffff700 1 --2- 172.21.15.121:0/483748712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e36f20 0x560996e37340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.230+0000 7f5e7effd700 1 -- 172.21.15.121:0/483748712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e3c640 msgr2=0x560996e3ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.230+0000 7f5e7effd700 1 --2- 172.21.15.121:0/483748712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e3c640 0x560996e3ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.230+0000 7f5e7effd700 1 -- 172.21.15.121:0/483748712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e36f20 msgr2=0x560996e37340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:18.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.230+0000 7f5e7effd700 1 --2- 172.21.15.121:0/483748712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e36f20 0x560996e37340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.230+0000 7f5e7effd700 1 -- 172.21.15.121:0/483748712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560996d26f50 con 0x560996e3cf60 2024-06-10T00:29:18.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.230+0000 7f5e7ffff700 1 --2- 172.21.15.121:0/483748712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e36f20 0x560996e37340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:29:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.230+0000 7f5e7effd700 1 --2- 172.21.15.121:0/483748712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e369e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5e74097d00 tx=0x7f5e7409bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=543b9e4a5b3d3e09 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.230+0000 7f5e65ffb700 1 -- 172.21.15.121:0/483748712 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e740a4510 con 0x560996e3cf60 2024-06-10T00:29:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.230+0000 7f5e65ffb700 1 -- 172.21.15.121:0/483748712 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e74007ca0 con 0x560996e3cf60 2024-06-10T00:29:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.230+0000 7f5e65ffb700 1 -- 172.21.15.121:0/483748712 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e740a4080 con 0x560996e3cf60 2024-06-10T00:29:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.231+0000 7f5e904f9b80 1 -- 172.21.15.121:0/483748712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 msgr2=0x560996e369e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.231+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/483748712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e369e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5e74097d00 tx=0x7f5e7409bff0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.231+0000 7f5e904f9b80 1 -- 172.21.15.121:0/483748712 shutdown_connections 2024-06-10T00:29:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.231+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/483748712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e36f20 0x560996e37340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.231+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/483748712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e3c640 0x560996e3ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.231+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/483748712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e369e0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.232+0000 7f5e904f9b80 1 -- 172.21.15.121:0/483748712 >> 172.21.15.121:0/483748712 conn(0x560996d61cc0 msgr2=0x560996e31850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:18.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.232+0000 7f5e904f9b80 1 -- 172.21.15.121:0/483748712 shutdown_connections 2024-06-10T00:29:18.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.232+0000 7f5e904f9b80 1 -- 172.21.15.121:0/483748712 wait complete. 2024-06-10T00:29:18.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.232+0000 7f5e904f9b80 1 -- start start 2024-06-10T00:29:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.233+0000 7f5e904f9b80 1 Processor -- start 2024-06-10T00:29:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.233+0000 7f5e904f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e42d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.233+0000 7f5e904f9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3c640 0x560996e40010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.233+0000 7f5e904f9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3cf60 0x560996e40550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.233+0000 7f5e7effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3c640 0x560996e40010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.233+0000 7f5e7effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3c640 0x560996e40010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44522/0 (socket says 172.21.15.121:44522) 2024-06-10T00:29:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.233+0000 7f5e7effd700 1 -- 172.21.15.121:0/3673043182 learned_addr learned my addr 172.21.15.121:0/3673043182 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3673043182 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x560996e439d0 con 0x560996e3c640 2024-06-10T00:29:18.236 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e7f7fe700 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e42d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3673043182 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x560996d28bf0 con 0x560996e3cf60 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3673043182 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560996e43b50 con 0x560996e36f20 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e7ffff700 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3cf60 0x560996e40550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e7effd700 1 -- 172.21.15.121:0/3673043182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 msgr2=0x560996e42d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e7effd700 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e42d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e7effd700 1 -- 172.21.15.121:0/3673043182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3cf60 msgr2=0x560996e40550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e7effd700 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3cf60 0x560996e40550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e7effd700 1 -- 172.21.15.121:0/3673043182 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560996cd78d0 con 0x560996e3c640 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e7effd700 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3c640 0x560996e40010 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5e74000c00 tx=0x7f5e7400a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325507fe0fac0059 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.234+0000 7f5e5effd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.235+0000 7f5e5effd700 1 -- 172.21.15.121:0/3673043182 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7409cbd0 con 0x560996e3c640 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.235+0000 7f5e5effd700 1 -- 172.21.15.121:0/3673043182 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f5e7409cd70 con 0x560996e3c640 2024-06-10T00:29:18.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.235+0000 7f5e904f9b80 10 client.0 _ll_register_callbacks cb 0x560996d22920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.235+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3673043182 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e74007ca0 con 0x560996e3c640 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.235+0000 7f5e5effd700 1 -- 172.21.15.121:0/3673043182 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5e740a4080 con 0x560996e3c640 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.235+0000 7f5e904f9b80 10 client.15657 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.235+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3673043182 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({fsmap=26}) -- 0x560996e4de70 con 0x560996e3c640 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.235+0000 7f5e5effd700 1 -- 172.21.15.121:0/3673043182 <== mon.0 v2:172.21.15.121:3300/0 4 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f5e7400f740 con 0x560996e3c640 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.235+0000 7f5e904f9b80 10 client.15657 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.236+0000 7f5e904f9b80 10 client.15657 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.236+0000 7f5e904f9b80 20 client.15657 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.236+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x560996e50340 0x560996e52730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.236+0000 7f5e904f9b80 4 client.15657 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.236+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3673043182 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560996d95d10 con 0x560996e50340 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.236+0000 7f5e7f7fe700 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x560996e50340 0x560996e52730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.236+0000 7f5e7f7fe700 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x560996e50340 0x560996e52730 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c66cf39e9c2be357 server_cookie=491c35541172b7df in_seq=0 out_seq=0 2024-06-10T00:29:18.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.236+0000 7f5e5effd700 10 client.15657 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:29:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e5effd700 1 -- 172.21.15.121:0/3673043182 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x560996d95d10 con 0x560996e50340 2024-06-10T00:29:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e5effd700 10 client.15657 handle_command_reply: tid=0 2024-06-10T00:29:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e904f9b80 1 client.15657 shutdown 2024-06-10T00:29:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3673043182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3c640 msgr2=0x560996e40010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3c640 0x560996e40010 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5e74000c00 tx=0x7f5e7400a5f0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3673043182 shutdown_connections 2024-06-10T00:29:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x560996e50340 0x560996e52730 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3cf60 0x560996e40550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e42d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/3673043182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3c640 0x560996e40010 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3673043182 >> 172.21.15.121:0/3673043182 conn(0x560996d61cc0 msgr2=0x560996e359d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:18.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.237+0000 7f5e5effd700 0 client.15657 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:29:18.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.238+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3673043182 shutdown_connections 2024-06-10T00:29:18.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.238+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3673043182 wait complete. 2024-06-10T00:29:18.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.238+0000 7f5e904f9b80 20 client.15657 trim_cache size 0 max 16384 2024-06-10T00:29:18.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.267+0000 7f5e904f9b80 1 Processor -- start 2024-06-10T00:29:18.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.267+0000 7f5e904f9b80 1 -- start start 2024-06-10T00:29:18.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.267+0000 7f5e904f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e369e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.267+0000 7f5e904f9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3c640 0x560996e3dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.267+0000 7f5e904f9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e57d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.267+0000 7f5e904f9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5e801b2de0 con 0x560996e3cf60 2024-06-10T00:29:18.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.267+0000 7f5e904f9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5e801b0670 con 0x560996e3c640 2024-06-10T00:29:18.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.267+0000 7f5e904f9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e801ae4c0 con 0x560996e36f20 2024-06-10T00:29:18.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.267+0000 7f5e7ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e57d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.268+0000 7f5e7ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e57d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44528/0 (socket says 172.21.15.121:44528) 2024-06-10T00:29:18.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.268+0000 7f5e7ffff700 1 -- 172.21.15.121:0/3135689968 learned_addr learned my addr 172.21.15.121:0/3135689968 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:18.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.268+0000 7f5e7f7fe700 1 --2- 172.21.15.121:0/3135689968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e369e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.268+0000 7f5e7effd700 1 --2- 172.21.15.121:0/3135689968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3c640 0x560996e3dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.268+0000 7f5e7ffff700 1 -- 172.21.15.121:0/3135689968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 msgr2=0x560996e369e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:18.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.268+0000 7f5e7ffff700 1 --2- 172.21.15.121:0/3135689968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e369e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.268+0000 7f5e7ffff700 1 -- 172.21.15.121:0/3135689968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3c640 msgr2=0x560996e3dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:18.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.268+0000 7f5e7ffff700 1 --2- 172.21.15.121:0/3135689968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3c640 0x560996e3dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.268+0000 7f5e7ffff700 1 -- 172.21.15.121:0/3135689968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560996e40f90 con 0x560996e3cf60 2024-06-10T00:29:18.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.268+0000 7f5e7ffff700 1 --2- 172.21.15.121:0/3135689968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e57d30 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800b0a0 tx=0x7f5e78003c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd9f524ff758a30 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:18.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.269+0000 7f5e657fa700 1 -- 172.21.15.121:0/3135689968 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7802a070 con 0x560996e3cf60 2024-06-10T00:29:18.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.269+0000 7f5e657fa700 1 -- 172.21.15.121:0/3135689968 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e780079c0 con 0x560996e3cf60 2024-06-10T00:29:18.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.269+0000 7f5e657fa700 1 -- 172.21.15.121:0/3135689968 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e780056e0 con 0x560996e3cf60 2024-06-10T00:29:18.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.269+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3135689968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 msgr2=0x560996e57d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:18.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.269+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/3135689968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e57d30 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800b0a0 tx=0x7f5e78003c40 comp rx=0 tx=0).stop 2024-06-10T00:29:18.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.270+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3135689968 shutdown_connections 2024-06-10T00:29:18.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.270+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/3135689968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e57d30 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.270+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/3135689968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e369e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.270+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/3135689968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3c640 0x560996e3dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.270+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3135689968 >> 172.21.15.121:0/3135689968 conn(0x560996d61cc0 msgr2=0x560996e316f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:18.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.270+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3135689968 shutdown_connections 2024-06-10T00:29:18.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.270+0000 7f5e904f9b80 1 -- 172.21.15.121:0/3135689968 wait complete. 2024-06-10T00:29:18.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.270+0000 7f5e904f9b80 1 -- start start 2024-06-10T00:29:18.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.270+0000 7f5e904f9b80 1 Processor -- start 2024-06-10T00:29:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e904f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e48eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e904f9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3c640 0x560996e493f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e904f9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e463c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e904f9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5e801aeb30 con 0x560996e3cf60 2024-06-10T00:29:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e7f7fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e48eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e7f7fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e48eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:34552/0 (socket says 172.21.15.121:34552) 2024-06-10T00:29:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e7f7fe700 1 -- 172.21.15.121:0/87491708 learned_addr learned my addr 172.21.15.121:0/87491708 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e904f9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5e801affa0 con 0x560996e3c640 2024-06-10T00:29:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e904f9b80 1 -- 172.21.15.121:0/87491708 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e801af200 con 0x560996e36f20 2024-06-10T00:29:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e7ffff700 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e463c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e7effd700 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3c640 0x560996e493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.271+0000 7f5e904f9b80 10 client.0 _ll_register_callbacks cb 0x560996d22920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e7f7fe700 1 -- 172.21.15.121:0/87491708 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3c640 msgr2=0x560996e493f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e7f7fe700 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3c640 0x560996e493f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e7f7fe700 1 -- 172.21.15.121:0/87491708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 msgr2=0x560996e463c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e7f7fe700 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e463c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e7f7fe700 1 -- 172.21.15.121:0/87491708 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560996e46bc0 con 0x560996e36f20 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e7f7fe700 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e48eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000f6a0 tx=0x7f5e700047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fafd0c338a67b4f9 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e5effd700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e5effd700 1 -- 172.21.15.121:0/87491708 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e70016070 con 0x560996e36f20 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e5effd700 1 -- 172.21.15.121:0/87491708 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f5e70029410 con 0x560996e36f20 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e904f9b80 1 -- 172.21.15.121:0/87491708 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e780079c0 con 0x560996e36f20 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e5effd700 1 -- 172.21.15.121:0/87491708 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5e7000fe70 con 0x560996e36f20 2024-06-10T00:29:18.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e904f9b80 10 client.25219 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:18.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.272+0000 7f5e904f9b80 1 -- 172.21.15.121:0/87491708 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560996ec2fe0 con 0x560996e36f20 2024-06-10T00:29:18.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.273+0000 7f5e5effd700 1 -- 172.21.15.121:0/87491708 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e780079c0 con 0x560996e36f20 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.274+0000 7f5e5effd700 1 -- 172.21.15.121:0/87491708 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f5e70030030 con 0x560996e36f20 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.274+0000 7f5e904f9b80 10 client.25219 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.274+0000 7f5e904f9b80 10 client.25219 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.274+0000 7f5e904f9b80 20 client.25219 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.274+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x560996ec3640 0x560996ec5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.274+0000 7f5e904f9b80 4 client.25219 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.274+0000 7f5e904f9b80 1 -- 172.21.15.121:0/87491708 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560996d90290 con 0x560996ec3640 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.274+0000 7f5e7effd700 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x560996ec3640 0x560996ec5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.274+0000 7f5e5effd700 10 client.25219 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.274+0000 7f5e7effd700 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x560996ec3640 0x560996ec5a50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7173971271b9df77 server_cookie=bedcf1993b8233d7 in_seq=0 out_seq=0 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e5effd700 1 -- 172.21.15.121:0/87491708 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560996d90290 con 0x560996ec3640 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e5effd700 10 client.25219 handle_command_reply: tid=0 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 1 client.25219 shutdown 2024-06-10T00:29:18.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 1 -- 172.21.15.121:0/87491708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 msgr2=0x560996e48eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e48eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000f6a0 tx=0x7f5e700047c0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 1 -- 172.21.15.121:0/87491708 shutdown_connections 2024-06-10T00:29:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x560996ec3640 0x560996ec5a50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560996e3cf60 0x560996e463c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560996e36f20 0x560996e48eb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e5effd700 0 client.25219 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:29:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 1 --2- 172.21.15.121:0/87491708 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560996e3c640 0x560996e493f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 1 -- 172.21.15.121:0/87491708 >> 172.21.15.121:0/87491708 conn(0x560996d61cc0 msgr2=0x560996e3d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 1 -- 172.21.15.121:0/87491708 shutdown_connections 2024-06-10T00:29:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 1 -- 172.21.15.121:0/87491708 wait complete. 2024-06-10T00:29:18.278 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.275+0000 7f5e904f9b80 20 client.25219 trim_cache size 0 max 16384 2024-06-10T00:29:18.279 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:29:18.279 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 -- 172.21.15.121:0/3533638108 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e680a2f10 msgr2=0x7f5e680a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e680a2f10 0x7f5e680a53d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5e80107230 tx=0x7f5e7000a380 comp rx=0 tx=0).stop 2024-06-10T00:29:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 -- 172.21.15.121:0/3533638108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e80101510 msgr2=0x7f5e801a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e80101510 0x7f5e801a0c50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5e7400c9b0 tx=0x7f5e7400ccc0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 -- 172.21.15.121:0/3533638108 shutdown_connections 2024-06-10T00:29:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e680a2f10 0x7f5e680a53d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e80106b90 0x7f5e8019a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e80100b50 0x7f5e801a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 --2- 172.21.15.121:0/3533638108 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e80101510 0x7f5e801a0c50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 -- 172.21.15.121:0/3533638108 >> 172.21.15.121:0/3533638108 conn(0x7f5e80077a00 msgr2=0x7f5e8010cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 -- 172.21.15.121:0/3533638108 shutdown_connections 2024-06-10T00:29:18.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:18.276+0000 7f5e85c76700 1 -- 172.21.15.121:0/3533638108 wait complete. 2024-06-10T00:29:18.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:18 smithi121 ceph-mon[32378]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:29:19.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:18 smithi135 ceph-mon[36424]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:29:19.080 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-10T00:29:19.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:18 smithi193 ceph-mon[36716]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 5 op/s 2024-06-10T00:29:19.355 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:20.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:20 smithi121 ceph-mon[32378]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2024-06-10T00:29:21.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:20 smithi135 ceph-mon[36424]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2024-06-10T00:29:21.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:20 smithi193 ceph-mon[36716]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2024-06-10T00:29:21.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.224+0000 7f70db9bc700 1 -- 172.21.15.121:0/147769366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d4101690 msgr2=0x7f70d4101b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:21.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.224+0000 7f70db9bc700 1 --2- 172.21.15.121:0/147769366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d4101690 0x7f70d4101b10 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f70c40099c0 tx=0x7f70c4009cd0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.224+0000 7f70db9bc700 1 -- 172.21.15.121:0/147769366 shutdown_connections 2024-06-10T00:29:21.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.224+0000 7f70db9bc700 1 --2- 172.21.15.121:0/147769366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d4102050 0x7f70d410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.224+0000 7f70db9bc700 1 --2- 172.21.15.121:0/147769366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f70d4107590 0x7f70d4107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.224+0000 7f70db9bc700 1 --2- 172.21.15.121:0/147769366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d4101690 0x7f70d4101b10 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.224+0000 7f70db9bc700 1 -- 172.21.15.121:0/147769366 >> 172.21.15.121:0/147769366 conn(0x7f70d4078150 msgr2=0x7f70d4078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:21.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.225+0000 7f70db9bc700 1 -- 172.21.15.121:0/147769366 shutdown_connections 2024-06-10T00:29:21.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.225+0000 7f70db9bc700 1 -- 172.21.15.121:0/147769366 wait complete. 2024-06-10T00:29:21.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.226+0000 7f70db9bc700 1 Processor -- start 2024-06-10T00:29:21.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.226+0000 7f70db9bc700 1 -- start start 2024-06-10T00:29:21.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.226+0000 7f70db9bc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d4101690 0x7f70d419d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.226+0000 7f70db9bc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d4102050 0x7f70d419dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.226+0000 7f70db9bc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f70d4107590 0x7f70d4197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.226+0000 7f70db9bc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f70d4111710 con 0x7f70d4102050 2024-06-10T00:29:21.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.226+0000 7f70db9bc700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f70d4111590 con 0x7f70d4107590 2024-06-10T00:29:21.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.226+0000 7f70db9bc700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f70d4111890 con 0x7f70d4101690 2024-06-10T00:29:21.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d9758700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d4101690 0x7f70d419d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d8f57700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d4102050 0x7f70d419dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d9758700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d4101690 0x7f70d419d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:42876/0 (socket says 172.21.15.121:42876) 2024-06-10T00:29:21.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d8f57700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d4102050 0x7f70d419dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60920/0 (socket says 172.21.15.121:60920) 2024-06-10T00:29:21.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d9758700 1 -- 172.21.15.121:0/1779214423 learned_addr learned my addr 172.21.15.121:0/1779214423 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:21.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d9f59700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f70d4107590 0x7f70d4197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d9758700 1 -- 172.21.15.121:0/1779214423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f70d4107590 msgr2=0x7f70d4197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:21.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d9758700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f70d4107590 0x7f70d4197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d9758700 1 -- 172.21.15.121:0/1779214423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d4102050 msgr2=0x7f70d419dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:21.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d9758700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d4102050 0x7f70d419dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d9758700 1 -- 172.21.15.121:0/1779214423 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d41980b0 con 0x7f70d4101690 2024-06-10T00:29:21.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d9f59700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f70d4107590 0x7f70d4197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:21.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.227+0000 7f70d9758700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d4101690 0x7f70d419d790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009ef0 tx=0x7f70d000ff50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:21.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.228+0000 7f70ca7fc700 1 -- 172.21.15.121:0/1779214423 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d001e070 con 0x7f70d4101690 2024-06-10T00:29:21.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.228+0000 7f70db9bc700 1 -- 172.21.15.121:0/1779214423 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d41983a0 con 0x7f70d4101690 2024-06-10T00:29:21.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.228+0000 7f70db9bc700 1 -- 172.21.15.121:0/1779214423 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d406ab60 con 0x7f70d4101690 2024-06-10T00:29:21.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.228+0000 7f70ca7fc700 1 -- 172.21.15.121:0/1779214423 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70d0007c50 con 0x7f70d4101690 2024-06-10T00:29:21.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.228+0000 7f70ca7fc700 1 -- 172.21.15.121:0/1779214423 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d000eb90 con 0x7f70d4101690 2024-06-10T00:29:21.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.231+0000 7f70ca7fc700 1 -- 172.21.15.121:0/1779214423 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f70d0021070 con 0x7f70d4101690 2024-06-10T00:29:21.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.232+0000 7f70ca7fc700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f70c00a2f30 0x7f70c00a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.232+0000 7f70ca7fc700 1 -- 172.21.15.121:0/1779214423 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f70d00d2d20 con 0x7f70d4101690 2024-06-10T00:29:21.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.232+0000 7f70d8f57700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f70c00a2f30 0x7f70c00a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.232+0000 7f70e623fb80 1 Processor -- start 2024-06-10T00:29:21.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.232+0000 7f70e623fb80 1 -- start start 2024-06-10T00:29:21.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70d8f57700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f70c00a2f30 0x7f70c00a53f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f70d41061d0 tx=0x7f70c4005fb0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:21.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad5262bf60 0x55ad5262c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262c880 0x55ad52626260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad526267a0 0x55ad52626bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70e623fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ad52518f60 con 0x55ad5262bf60 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70e623fb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55ad52577850 con 0x55ad526267a0 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70e623fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ad524df460 con 0x55ad5262c880 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70d9f59700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad526267a0 0x55ad52626bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70d9f59700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad526267a0 0x55ad52626bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:60450/0 (socket says 172.21.15.121:60450) 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70d9f59700 1 -- 172.21.15.121:0/4116380770 learned_addr learned my addr 172.21.15.121:0/4116380770 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70d9f59700 1 -- 172.21.15.121:0/4116380770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262c880 msgr2=0x55ad52626260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70d9f59700 1 --2- 172.21.15.121:0/4116380770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262c880 0x55ad52626260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70d9f59700 1 -- 172.21.15.121:0/4116380770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad5262bf60 msgr2=0x55ad5262c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70d9f59700 1 --2- 172.21.15.121:0/4116380770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad5262bf60 0x55ad5262c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70d9f59700 1 -- 172.21.15.121:0/4116380770 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad524c68d0 con 0x55ad526267a0 2024-06-10T00:29:21.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.233+0000 7f70d9f59700 1 --2- 172.21.15.121:0/4116380770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad526267a0 0x55ad52626bc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00a760 tx=0x7f70cc0078d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f48b0c5157831109 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:21.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.234+0000 7f70bf7fe700 1 -- 172.21.15.121:0/4116380770 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc014600 con 0x55ad526267a0 2024-06-10T00:29:21.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.234+0000 7f70bf7fe700 1 -- 172.21.15.121:0/4116380770 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70cc019070 con 0x55ad526267a0 2024-06-10T00:29:21.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.234+0000 7f70bf7fe700 1 -- 172.21.15.121:0/4116380770 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc008af0 con 0x55ad526267a0 2024-06-10T00:29:21.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.234+0000 7f70e623fb80 1 -- 172.21.15.121:0/4116380770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad526267a0 msgr2=0x55ad52626bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:21.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.234+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4116380770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad526267a0 0x55ad52626bc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00a760 tx=0x7f70cc0078d0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.235+0000 7f70e623fb80 1 -- 172.21.15.121:0/4116380770 shutdown_connections 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.235+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4116380770 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad526267a0 0x55ad52626bc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.235+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4116380770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad5262bf60 0x55ad5262c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.235+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4116380770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262c880 0x55ad52626260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.235+0000 7f70e623fb80 1 -- 172.21.15.121:0/4116380770 >> 172.21.15.121:0/4116380770 conn(0x55ad52553080 msgr2=0x55ad52621220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.235+0000 7f70e623fb80 1 -- 172.21.15.121:0/4116380770 shutdown_connections 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.235+0000 7f70e623fb80 1 -- 172.21.15.121:0/4116380770 wait complete. 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.235+0000 7f70e623fb80 1 -- start start 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.235+0000 7f70e623fb80 1 Processor -- start 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.236+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad526267a0 0x55ad52632690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.236+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262bf60 0x55ad5262fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.236+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad5262c880 0x55ad5262ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.236+0000 7f70e623fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ad52517fc0 con 0x55ad5262c880 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.236+0000 7f70d9758700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad526267a0 0x55ad52632690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.236+0000 7f70d9758700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad526267a0 0x55ad52632690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:42904/0 (socket says 172.21.15.121:42904) 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.236+0000 7f70d9758700 1 -- 172.21.15.121:0/4136107420 learned_addr learned my addr 172.21.15.121:0/4136107420 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.236+0000 7f70d8f57700 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262bf60 0x55ad5262fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.236+0000 7f70e623fb80 1 -- 172.21.15.121:0/4136107420 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55ad52573810 con 0x55ad5262bf60 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.236+0000 7f70e623fb80 1 -- 172.21.15.121:0/4136107420 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ad52633200 con 0x55ad526267a0 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.237+0000 7f70d9758700 1 -- 172.21.15.121:0/4136107420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262bf60 msgr2=0x55ad5262fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.237+0000 7f70d9758700 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262bf60 0x55ad5262fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.237+0000 7f70d9758700 1 -- 172.21.15.121:0/4136107420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad5262c880 msgr2=0x55ad5262ff40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.237+0000 7f70d9758700 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad5262c880 0x55ad5262ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.237+0000 7f70d9758700 1 -- 172.21.15.121:0/4136107420 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad5253d750 con 0x55ad526267a0 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.237+0000 7f70e623fb80 10 client.0 _ll_register_callbacks cb 0x55ad5250d5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.237+0000 7f70e623fb80 1 -- 172.21.15.121:0/4136107420 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70cc019070 con 0x55ad526267a0 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.237+0000 7f70d9758700 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad526267a0 0x55ad52632690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f70d0000c00 tx=0x7f70d0007e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b996a84dbdb26ff server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.237+0000 7f70bcff9700 10 client.25225 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.237+0000 7f70bcff9700 1 -- 172.21.15.121:0/4136107420 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d00a2fb0 con 0x55ad526267a0 2024-06-10T00:29:21.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70bcff9700 1 -- 172.21.15.121:0/4136107420 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f70d00a75f0 con 0x55ad526267a0 2024-06-10T00:29:21.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70bcff9700 1 -- 172.21.15.121:0/4136107420 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f70d00a7870 con 0x55ad526267a0 2024-06-10T00:29:21.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70e623fb80 10 client.25225 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:21.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70e623fb80 1 -- 172.21.15.121:0/4136107420 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=26}) -- 0x55ad5263d980 con 0x55ad526267a0 2024-06-10T00:29:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70bcff9700 1 -- 172.21.15.121:0/4136107420 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f70d000f260 con 0x55ad526267a0 2024-06-10T00:29:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70e623fb80 10 client.25225 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70e623fb80 10 client.25225 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:29:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70e623fb80 20 client.25225 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ad5263feb0 0x55ad526422f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70e623fb80 4 client.25225 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70e623fb80 1 -- 172.21.15.121:0/4136107420 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad52584d50 con 0x55ad5263feb0 2024-06-10T00:29:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.238+0000 7f70d8f57700 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ad5263feb0 0x55ad526422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.239+0000 7f70bcff9700 10 client.25225 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.239+0000 7f70d8f57700 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ad5263feb0 0x55ad526422f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5110d7c60d4a0afc server_cookie=1068367992003e1a in_seq=0 out_seq=0 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70bcff9700 1 -- 172.21.15.121:0/4136107420 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55ad52584d50 con 0x55ad5263feb0 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70bcff9700 10 client.25225 handle_command_reply: tid=0 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 1 client.25225 shutdown 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 1 -- 172.21.15.121:0/4136107420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad526267a0 msgr2=0x55ad52632690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad526267a0 0x55ad52632690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f70d0000c00 tx=0x7f70d0007e70 comp rx=0 tx=0).stop 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 1 -- 172.21.15.121:0/4136107420 shutdown_connections 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ad5263feb0 0x55ad526422f0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70bcff9700 0 client.25225 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad5262c880 0x55ad5262ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad526267a0 0x55ad52632690 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4136107420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262bf60 0x55ad5262fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 1 -- 172.21.15.121:0/4136107420 >> 172.21.15.121:0/4136107420 conn(0x55ad52553080 msgr2=0x55ad52620700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:21.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 1 -- 172.21.15.121:0/4136107420 shutdown_connections 2024-06-10T00:29:21.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 1 -- 172.21.15.121:0/4136107420 wait complete. 2024-06-10T00:29:21.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.240+0000 7f70e623fb80 20 client.25225 trim_cache size 0 max 16384 2024-06-10T00:29:21.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.256+0000 7f70e623fb80 1 Processor -- start 2024-06-10T00:29:21.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.256+0000 7f70e623fb80 1 -- start start 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.256+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad5263c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.256+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262bf60 0x55ad5262d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.256+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262c880 0x55ad52632240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.256+0000 7f70e623fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f70b8011560 con 0x55ad526267a0 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.256+0000 7f70e623fb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800edf0 con 0x55ad5262bf60 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.256+0000 7f70e623fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800cc40 con 0x55ad5262c880 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.256+0000 7f70d9758700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad5263c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.256+0000 7f70d9758700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad5263c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60950/0 (socket says 172.21.15.121:60950) 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.256+0000 7f70d9758700 1 -- 172.21.15.121:0/833841674 learned_addr learned my addr 172.21.15.121:0/833841674 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70d8f57700 1 --2- 172.21.15.121:0/833841674 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262bf60 0x55ad5262d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70d9f59700 1 --2- 172.21.15.121:0/833841674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262c880 0x55ad52632240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70d9758700 1 -- 172.21.15.121:0/833841674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262c880 msgr2=0x55ad52632240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:21.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70d9758700 1 --2- 172.21.15.121:0/833841674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262c880 0x55ad52632240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70d9758700 1 -- 172.21.15.121:0/833841674 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262bf60 msgr2=0x55ad5262d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70d9758700 1 --2- 172.21.15.121:0/833841674 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262bf60 0x55ad5262d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70d9758700 1 -- 172.21.15.121:0/833841674 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad52627380 con 0x55ad526267a0 2024-06-10T00:29:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70d8f57700 1 --2- 172.21.15.121:0/833841674 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262bf60 0x55ad5262d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70d9f59700 1 --2- 172.21.15.121:0/833841674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262c880 0x55ad52632240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:29:21.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70d9758700 1 --2- 172.21.15.121:0/833841674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad5263c3b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f70d00a7340 tx=0x7f70d00d2000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03c1596f06de8bc server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70beffd700 1 -- 172.21.15.121:0/833841674 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d009aea0 con 0x55ad526267a0 2024-06-10T00:29:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70beffd700 1 -- 172.21.15.121:0/833841674 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70cc019070 con 0x55ad526267a0 2024-06-10T00:29:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.257+0000 7f70beffd700 1 -- 172.21.15.121:0/833841674 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d009e050 con 0x55ad526267a0 2024-06-10T00:29:21.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.258+0000 7f70e623fb80 1 -- 172.21.15.121:0/833841674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 msgr2=0x55ad5263c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.258+0000 7f70e623fb80 1 --2- 172.21.15.121:0/833841674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad5263c3b0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f70d00a7340 tx=0x7f70d00d2000 comp rx=0 tx=0).stop 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.258+0000 7f70e623fb80 1 -- 172.21.15.121:0/833841674 shutdown_connections 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.258+0000 7f70e623fb80 1 --2- 172.21.15.121:0/833841674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262c880 0x55ad52632240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.258+0000 7f70e623fb80 1 --2- 172.21.15.121:0/833841674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad5263c3b0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.258+0000 7f70e623fb80 1 --2- 172.21.15.121:0/833841674 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262bf60 0x55ad5262d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.258+0000 7f70e623fb80 1 -- 172.21.15.121:0/833841674 >> 172.21.15.121:0/833841674 conn(0x55ad52553080 msgr2=0x55ad52629e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.258+0000 7f70e623fb80 1 -- 172.21.15.121:0/833841674 shutdown_connections 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.258+0000 7f70e623fb80 1 -- 172.21.15.121:0/833841674 wait complete. 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.258+0000 7f70e623fb80 1 -- start start 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.258+0000 7f70e623fb80 1 Processor -- start 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad52638700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262bf60 0x55ad52638c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70e623fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262c880 0x55ad526354f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70e623fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800d2b0 con 0x55ad526267a0 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70e623fb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800e720 con 0x55ad5262c880 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70e623fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f70b800d980 con 0x55ad5262bf60 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70e623fb80 10 client.0 _ll_register_callbacks cb 0x55ad5250d5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70d9758700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad52638700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70d9758700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad52638700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60964/0 (socket says 172.21.15.121:60964) 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70d9758700 1 -- 172.21.15.121:0/4218493805 learned_addr learned my addr 172.21.15.121:0/4218493805 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70d8f57700 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262bf60 0x55ad52638c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70d9f59700 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262c880 0x55ad526354f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70d9758700 1 -- 172.21.15.121:0/4218493805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262bf60 msgr2=0x55ad52638c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70d9758700 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262bf60 0x55ad52638c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70d9758700 1 -- 172.21.15.121:0/4218493805 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262c880 msgr2=0x55ad526354f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.259+0000 7f70d9758700 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262c880 0x55ad526354f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.260+0000 7f70d9758700 1 -- 172.21.15.121:0/4218493805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad52635d30 con 0x55ad526267a0 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.260+0000 7f70d9758700 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad52638700 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f70d000e740 tx=0x7f70d009ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8248e709774ef66e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.260+0000 7f70bcff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.260+0000 7f70bcff9700 1 -- 172.21.15.121:0/4218493805 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d009aea0 con 0x55ad526267a0 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.260+0000 7f70bcff9700 1 -- 172.21.15.121:0/4218493805 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f70d000e430 con 0x55ad526267a0 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.260+0000 7f70e623fb80 1 -- 172.21.15.121:0/4218493805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70cc019070 con 0x55ad526267a0 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.260+0000 7f70bcff9700 1 -- 172.21.15.121:0/4218493805 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f70d00ae050 con 0x55ad526267a0 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.260+0000 7f70e623fb80 10 client.15675 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.260+0000 7f70e623fb80 1 -- 172.21.15.121:0/4218493805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ad52635d30 con 0x55ad526267a0 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.260+0000 7f70bcff9700 1 -- 172.21.15.121:0/4218493805 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70cc019070 con 0x55ad526267a0 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.261+0000 7f70bcff9700 1 -- 172.21.15.121:0/4218493805 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f70d009dac0 con 0x55ad526267a0 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.261+0000 7f70e623fb80 10 client.15675 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.261+0000 7f70e623fb80 10 client.15675 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.261+0000 7f70e623fb80 20 client.15675 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.261+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ad526b4120 0x55ad526b6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.261+0000 7f70e623fb80 4 client.15675 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:29:21.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.261+0000 7f70e623fb80 1 -- 172.21.15.121:0/4218493805 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ad5257f2d0 con 0x55ad526b4120 2024-06-10T00:29:21.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.261+0000 7f70d8f57700 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ad526b4120 0x55ad526b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:21.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.261+0000 7f70d8f57700 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ad526b4120 0x55ad526b6510 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=457a3de8a3052c92 server_cookie=31f49c4e1cb1cf16 in_seq=0 out_seq=0 2024-06-10T00:29:21.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.261+0000 7f70bcff9700 10 client.15675 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:29:21.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70bcff9700 1 -- 172.21.15.121:0/4218493805 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ad5257f2d0 con 0x55ad526b4120 2024-06-10T00:29:21.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70bcff9700 10 client.15675 handle_command_reply: tid=0 2024-06-10T00:29:21.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 1 client.15675 shutdown 2024-06-10T00:29:21.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 1 -- 172.21.15.121:0/4218493805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 msgr2=0x55ad52638700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:21.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad52638700 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f70d000e740 tx=0x7f70d009ed40 comp rx=0 tx=0).stop 2024-06-10T00:29:21.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 1 -- 172.21.15.121:0/4218493805 shutdown_connections 2024-06-10T00:29:21.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ad526b4120 0x55ad526b6510 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ad5262c880 0x55ad526354f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ad526267a0 0x55ad52638700 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 1 --2- 172.21.15.121:0/4218493805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ad5262bf60 0x55ad52638c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 1 -- 172.21.15.121:0/4218493805 >> 172.21.15.121:0/4218493805 conn(0x55ad52553080 msgr2=0x55ad5262aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:21.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70bcff9700 0 client.15675 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:29:21.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 1 -- 172.21.15.121:0/4218493805 shutdown_connections 2024-06-10T00:29:21.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 1 -- 172.21.15.121:0/4218493805 wait complete. 2024-06-10T00:29:21.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.262+0000 7f70e623fb80 20 client.15675 trim_cache size 0 max 16384 2024-06-10T00:29:21.267 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:29:21.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.263+0000 7f70db9bc700 1 -- 172.21.15.121:0/1779214423 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f70c00a2f30 msgr2=0x7f70c00a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:21.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.263+0000 7f70db9bc700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f70c00a2f30 0x7f70c00a53f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f70d41061d0 tx=0x7f70c4005fb0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.263+0000 7f70db9bc700 1 -- 172.21.15.121:0/1779214423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d4101690 msgr2=0x7f70d419d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:21.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.263+0000 7f70db9bc700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d4101690 0x7f70d419d790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009ef0 tx=0x7f70d000ff50 comp rx=0 tx=0).stop 2024-06-10T00:29:21.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.263+0000 7f70db9bc700 1 -- 172.21.15.121:0/1779214423 shutdown_connections 2024-06-10T00:29:21.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.263+0000 7f70db9bc700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f70c00a2f30 0x7f70c00a53f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.263+0000 7f70db9bc700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f70d4107590 0x7f70d4197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.263+0000 7f70db9bc700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d4101690 0x7f70d419d790 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.263+0000 7f70db9bc700 1 --2- 172.21.15.121:0/1779214423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70d4102050 0x7f70d419dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:21.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.263+0000 7f70db9bc700 1 -- 172.21.15.121:0/1779214423 >> 172.21.15.121:0/1779214423 conn(0x7f70d4078150 msgr2=0x7f70d4100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:21.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.264+0000 7f70db9bc700 1 -- 172.21.15.121:0/1779214423 shutdown_connections 2024-06-10T00:29:21.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:21.264+0000 7f70db9bc700 1 -- 172.21.15.121:0/1779214423 wait complete. 2024-06-10T00:29:22.933 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:22 smithi193 ceph-mon[36716]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:29:22.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:22 smithi121 ceph-mon[32378]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:29:23.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:22 smithi135 ceph-mon[36424]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:29:24.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:24 smithi121 ceph-mon[32378]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-06-10T00:29:25.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:24 smithi135 ceph-mon[36424]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-06-10T00:29:25.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:24 smithi193 ceph-mon[36716]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-06-10T00:29:26.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-10T00:29:26.549 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:26.892 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:26 smithi121 ceph-mon[32378]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:29:27.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:26 smithi135 ceph-mon[36424]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:29:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:26 smithi193 ceph-mon[36716]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:29:28.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.653+0000 7f8cddd83700 1 -- 172.21.15.121:0/3540459904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cd81024f0 msgr2=0x7f8cd810e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:28.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.653+0000 7f8cddd83700 1 --2- 172.21.15.121:0/3540459904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cd81024f0 0x7f8cd810e200 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000b0a0 tx=0x7f8cd000b3b0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.653+0000 7f8cddd83700 1 -- 172.21.15.121:0/3540459904 shutdown_connections 2024-06-10T00:29:28.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.653+0000 7f8cddd83700 1 --2- 172.21.15.121:0/3540459904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cd81024f0 0x7f8cd810e200 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.653+0000 7f8cddd83700 1 --2- 172.21.15.121:0/3540459904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8106380 0x7f8cd8106760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.653+0000 7f8cddd83700 1 --2- 172.21.15.121:0/3540459904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cd8101b30 0x7f8cd8101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.653+0000 7f8cddd83700 1 -- 172.21.15.121:0/3540459904 >> 172.21.15.121:0/3540459904 conn(0x7f8cd8077990 msgr2=0x7f8cd80fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:28.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.653+0000 7f8cddd83700 1 -- 172.21.15.121:0/3540459904 shutdown_connections 2024-06-10T00:29:28.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.654+0000 7f8cddd83700 1 -- 172.21.15.121:0/3540459904 wait complete. 2024-06-10T00:29:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.654+0000 7f8cddd83700 1 Processor -- start 2024-06-10T00:29:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cddd83700 1 -- start start 2024-06-10T00:29:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cddd83700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8101b30 0x7f8cd819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cddd83700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cd81024f0 0x7f8cd819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cddd83700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cd8106380 0x7f8cd8198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cddd83700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd81117a0 con 0x7f8cd8106380 2024-06-10T00:29:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cddd83700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd8111620 con 0x7f8cd8101b30 2024-06-10T00:29:28.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cddd83700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd8111920 con 0x7f8cd81024f0 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cd7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cd8106380 0x7f8cd8198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cd7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cd8106380 0x7f8cd8198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60978/0 (socket says 172.21.15.121:60978) 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cd7fff700 1 -- 172.21.15.121:0/3083086090 learned_addr learned my addr 172.21.15.121:0/3083086090 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cd77fe700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8101b30 0x7f8cd819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.655+0000 7f8cd6ffd700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cd81024f0 0x7f8cd819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.656+0000 7f8cd7fff700 1 -- 172.21.15.121:0/3083086090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cd81024f0 msgr2=0x7f8cd819edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.656+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cd81024f0 0x7f8cd819edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.656+0000 7f8cd7fff700 1 -- 172.21.15.121:0/3083086090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8101b30 msgr2=0x7f8cd819e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.656+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8101b30 0x7f8cd819e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.656+0000 7f8cd7fff700 1 -- 172.21.15.121:0/3083086090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd81991b0 con 0x7f8cd8106380 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.656+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cd8106380 0x7f8cd8198a50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00062a0 tx=0x7f8cd0004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.656+0000 7f8cd4ff9700 1 -- 172.21.15.121:0/3083086090 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0021070 con 0x7f8cd8106380 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.656+0000 7f8cd4ff9700 1 -- 172.21.15.121:0/3083086090 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cd0003c00 con 0x7f8cd8106380 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.656+0000 7f8cddd83700 1 -- 172.21.15.121:0/3083086090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cd8199440 con 0x7f8cd8106380 2024-06-10T00:29:28.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.656+0000 7f8cddd83700 1 -- 172.21.15.121:0/3083086090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cd81a5760 con 0x7f8cd8106380 2024-06-10T00:29:28.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.657+0000 7f8cd4ff9700 1 -- 172.21.15.121:0/3083086090 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0004880 con 0x7f8cd8106380 2024-06-10T00:29:28.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.659+0000 7f8cd4ff9700 1 -- 172.21.15.121:0/3083086090 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 25) ==== 138067+0+0 (secure 0 0 0) 0x7f8cd001d470 con 0x7f8cd8106380 2024-06-10T00:29:28.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.659+0000 7f8cd4ff9700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cc00a2f30 0x7f8cc00a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.659+0000 7f8cd4ff9700 1 -- 172.21.15.121:0/3083086090 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8cd000f020 con 0x7f8cd8106380 2024-06-10T00:29:28.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.660+0000 7f8cd77fe700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cc00a2f30 0x7f8cc00a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.660+0000 7f8cd77fe700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cc00a2f30 0x7f8cc00a53f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8199b90 tx=0x7f8cc800b410 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:28.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.660+0000 7f8ce8606b80 1 Processor -- start 2024-06-10T00:29:28.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.660+0000 7f8ce8606b80 1 -- start start 2024-06-10T00:29:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.660+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 0x5624f6bfaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.660+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bf49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.660+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6bf5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.660+0000 7f8ce8606b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5624f6b46810 con 0x5624f6bf4f20 2024-06-10T00:29:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.660+0000 7f8ce8606b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5624f6aee720 con 0x5624f6bfa640 2024-06-10T00:29:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.660+0000 7f8ce8606b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5624f6b427d0 con 0x5624f6bfaf60 2024-06-10T00:29:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cd7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6bf5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cd7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6bf5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60988/0 (socket says 172.21.15.121:60988) 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cd7fff700 1 -- 172.21.15.121:0/1650447793 learned_addr learned my addr 172.21.15.121:0/1650447793 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cd77fe700 1 --2- 172.21.15.121:0/1650447793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 0x5624f6bfaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cd6ffd700 1 --2- 172.21.15.121:0/1650447793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bf49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cd7fff700 1 -- 172.21.15.121:0/1650447793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 msgr2=0x5624f6bf49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/1650447793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bf49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cd7fff700 1 -- 172.21.15.121:0/1650447793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 msgr2=0x5624f6bfaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/1650447793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 0x5624f6bfaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cd7fff700 1 -- 172.21.15.121:0/1650447793 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624f6ae4f50 con 0x5624f6bf4f20 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/1650447793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6bf5340 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0009fd0 tx=0x7f8cd009cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b564e60d10f0b34 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cbdffb700 1 -- 172.21.15.121:0/1650447793 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd00a1250 con 0x5624f6bf4f20 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cbdffb700 1 -- 172.21.15.121:0/1650447793 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cd00a5a80 con 0x5624f6bf4f20 2024-06-10T00:29:28.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.661+0000 7f8cbdffb700 1 -- 172.21.15.121:0/1650447793 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd009d360 con 0x5624f6bf4f20 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.662+0000 7f8ce8606b80 1 -- 172.21.15.121:0/1650447793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 msgr2=0x5624f6bf5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.662+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/1650447793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6bf5340 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0009fd0 tx=0x7f8cd009cab0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.662+0000 7f8ce8606b80 1 -- 172.21.15.121:0/1650447793 shutdown_connections 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.662+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/1650447793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6bf5340 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.662+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/1650447793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 0x5624f6bfaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.662+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/1650447793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bf49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.662+0000 7f8ce8606b80 1 -- 172.21.15.121:0/1650447793 >> 172.21.15.121:0/1650447793 conn(0x5624f6b1fcc0 msgr2=0x5624f6bef850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.662+0000 7f8ce8606b80 1 -- 172.21.15.121:0/1650447793 shutdown_connections 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.662+0000 7f8ce8606b80 1 -- 172.21.15.121:0/1650447793 wait complete. 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.663+0000 7f8ce8606b80 1 -- start start 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.663+0000 7f8ce8606b80 1 Processor -- start 2024-06-10T00:29:28.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.663+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c00d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.663+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 0x5624f6bfe010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.663+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bfe550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.663+0000 7f8ce8606b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5624f6c016f0 con 0x5624f6bf4f20 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.663+0000 7f8cd77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c00d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.663+0000 7f8cd77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c00d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32768/0 (socket says 172.21.15.121:32768) 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.663+0000 7f8cd77fe700 1 -- 172.21.15.121:0/3999075349 learned_addr learned my addr 172.21.15.121:0/3999075349 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.663+0000 7f8ce8606b80 1 -- 172.21.15.121:0/3999075349 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5624f6ae6bf0 con 0x5624f6bfa640 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.663+0000 7f8ce8606b80 1 -- 172.21.15.121:0/3999075349 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5624f6c01870 con 0x5624f6bfaf60 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bfe550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8ce8606b80 10 client.0 _ll_register_callbacks cb 0x5624f6ae0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8cd77fe700 1 -- 172.21.15.121:0/3999075349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 msgr2=0x5624f6bfe550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8cd77fe700 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bfe550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8cd77fe700 1 -- 172.21.15.121:0/3999075349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 msgr2=0x5624f6bfe010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:29:28.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8cd77fe700 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 0x5624f6bfe010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8cd77fe700 1 -- 172.21.15.121:0/3999075349 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624f6a958d0 con 0x5624f6bf4f20 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8cd77fe700 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c00d10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8cc80116f0 tx=0x7f8cc80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b3247151727bf server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8cb6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/3999075349 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc800be60 con 0x5624f6bf4f20 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/3999075349 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8cc801f840 con 0x5624f6bf4f20 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.664+0000 7f8ce8606b80 1 -- 172.21.15.121:0/3999075349 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cd00a5a80 con 0x5624f6bf4f20 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/3999075349 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8cc8031050 con 0x5624f6bf4f20 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8ce8606b80 10 client.15687 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8ce8606b80 1 -- 172.21.15.121:0/3999075349 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5624f6c0c0b0 con 0x5624f6bf4f20 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/3999075349 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cd00a5a80 con 0x5624f6bf4f20 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/3999075349 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f8cc8028bd0 con 0x5624f6bf4f20 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8ce8606b80 10 client.15687 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8ce8606b80 10 client.15687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8ce8606b80 20 client.15687 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:28.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5624f6c0eb20 0x5624f6c10fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8ce8606b80 4 client.15687 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8ce8606b80 1 -- 172.21.15.121:0/3999075349 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624f6b53d10 con 0x5624f6c0eb20 2024-06-10T00:29:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.665+0000 7f8cd6ffd700 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5624f6c0eb20 0x5624f6c10fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.666+0000 7f8cb6ffd700 10 client.15687 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:28.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.666+0000 7f8cd6ffd700 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5624f6c0eb20 0x5624f6c10fc0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=283a70ce889ef3c1 server_cookie=52eaee0e2fbb55c in_seq=0 out_seq=0 2024-06-10T00:29:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.666+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/3999075349 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5624f6b53d10 con 0x5624f6c0eb20 2024-06-10T00:29:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8cb6ffd700 10 client.15687 handle_command_reply: tid=0 2024-06-10T00:29:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 1 client.15687 shutdown 2024-06-10T00:29:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 1 -- 172.21.15.121:0/3999075349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 msgr2=0x5624f6c00d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c00d10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8cc80116f0 tx=0x7f8cc80047c0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 1 -- 172.21.15.121:0/3999075349 shutdown_connections 2024-06-10T00:29:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5624f6c0eb20 0x5624f6c10fc0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bfe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c00d10 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/3999075349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 0x5624f6bfe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 1 -- 172.21.15.121:0/3999075349 >> 172.21.15.121:0/3999075349 conn(0x5624f6b1fcc0 msgr2=0x5624f6bf39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8cb6ffd700 0 client.15687 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 1 -- 172.21.15.121:0/3999075349 shutdown_connections 2024-06-10T00:29:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 1 -- 172.21.15.121:0/3999075349 wait complete. 2024-06-10T00:29:28.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.667+0000 7f8ce8606b80 20 client.15687 trim_cache size 0 max 16384 2024-06-10T00:29:28.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.683+0000 7f8ce8606b80 1 Processor -- start 2024-06-10T00:29:28.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.683+0000 7f8ce8606b80 1 -- start start 2024-06-10T00:29:28.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bf4f20 0x5624f6bf49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bfa640 0x5624f6be9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bfcf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8ce8606b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd81b0d40 con 0x5624f6bfa640 2024-06-10T00:29:28.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8ce8606b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd81ae5d0 con 0x5624f6bf4f20 2024-06-10T00:29:28.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8ce8606b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd81ac420 con 0x5624f6bfaf60 2024-06-10T00:29:28.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bfa640 0x5624f6be9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bfa640 0x5624f6be9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32778/0 (socket says 172.21.15.121:32778) 2024-06-10T00:29:28.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd77fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bf4f20 0x5624f6bf49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd6ffd700 1 -- 172.21.15.121:0/1113507582 learned_addr learned my addr 172.21.15.121:0/1113507582 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:28.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/1113507582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bfcf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd7fff700 1 -- 172.21.15.121:0/1113507582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bf4f20 msgr2=0x5624f6bf49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:28.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/1113507582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bf4f20 0x5624f6bf49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd7fff700 1 -- 172.21.15.121:0/1113507582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bfa640 msgr2=0x5624f6be9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:28.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/1113507582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bfa640 0x5624f6be9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd7fff700 1 -- 172.21.15.121:0/1113507582 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624f6bfad90 con 0x5624f6bfaf60 2024-06-10T00:29:28.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd6ffd700 1 --2- 172.21.15.121:0/1113507582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bfa640 0x5624f6be9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.684+0000 7f8cd77fe700 1 --2- 172.21.15.121:0/1113507582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bf4f20 0x5624f6bf49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.685+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/1113507582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bfcf70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x5624f6c01500 tx=0x7f8cd000b9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cef63dbd54c9360f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.685+0000 7f8cbd7fa700 1 -- 172.21.15.121:0/1113507582 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd00ce050 con 0x5624f6bfaf60 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.685+0000 7f8cbd7fa700 1 -- 172.21.15.121:0/1113507582 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cd00a1250 con 0x5624f6bfaf60 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.685+0000 7f8cbd7fa700 1 -- 172.21.15.121:0/1113507582 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd009d100 con 0x5624f6bfaf60 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.686+0000 7f8ce8606b80 1 -- 172.21.15.121:0/1113507582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 msgr2=0x5624f6bfcf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.686+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/1113507582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bfcf70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x5624f6c01500 tx=0x7f8cd000b9e0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.686+0000 7f8ce8606b80 1 -- 172.21.15.121:0/1113507582 shutdown_connections 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.686+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/1113507582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6bfcf70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.686+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/1113507582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bf4f20 0x5624f6bf49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.686+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/1113507582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bfa640 0x5624f6be9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.686+0000 7f8ce8606b80 1 -- 172.21.15.121:0/1113507582 >> 172.21.15.121:0/1113507582 conn(0x5624f6b1fcc0 msgr2=0x5624f6bf7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.686+0000 7f8ce8606b80 1 -- 172.21.15.121:0/1113507582 shutdown_connections 2024-06-10T00:29:28.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.686+0000 7f8ce8606b80 1 -- 172.21.15.121:0/1113507582 wait complete. 2024-06-10T00:29:28.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.686+0000 7f8ce8606b80 1 -- start start 2024-06-10T00:29:28.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8ce8606b80 1 Processor -- start 2024-06-10T00:29:28.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c16ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 0x5624f6c06b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8ce8606b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6c07070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8ce8606b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd81aca90 con 0x5624f6bf4f20 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8ce8606b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd81adf00 con 0x5624f6bfa640 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8ce8606b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd81ad160 con 0x5624f6bfaf60 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8ce8606b80 10 client.0 _ll_register_callbacks cb 0x5624f6ae0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8cd77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c16ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8cd77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c16ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:32792/0 (socket says 172.21.15.121:32792) 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8cd77fe700 1 -- 172.21.15.121:0/2368762657 learned_addr learned my addr 172.21.15.121:0/2368762657 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8cd6ffd700 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 0x5624f6c06b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8cd7fff700 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6c07070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8cd77fe700 1 -- 172.21.15.121:0/2368762657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 msgr2=0x5624f6c07070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8cd77fe700 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6c07070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8cd77fe700 1 -- 172.21.15.121:0/2368762657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 msgr2=0x5624f6c06b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8cd77fe700 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 0x5624f6c06b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.687+0000 7f8cd77fe700 1 -- 172.21.15.121:0/2368762657 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624f6c076c0 con 0x5624f6bf4f20 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.688+0000 7f8cd77fe700 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c16ed0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8cc80116f0 tx=0x7f8cc801e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a4ef348a15b9ec server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:28.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.688+0000 7f8cb6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.688+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/2368762657 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc800be60 con 0x5624f6bf4f20 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.688+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/2368762657 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8cc801f640 con 0x5624f6bf4f20 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.688+0000 7f8ce8606b80 1 -- 172.21.15.121:0/2368762657 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cd00a1250 con 0x5624f6bf4f20 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.688+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/2368762657 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8cc8047050 con 0x5624f6bf4f20 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.688+0000 7f8ce8606b80 10 client.15693 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.688+0000 7f8ce8606b80 1 -- 172.21.15.121:0/2368762657 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5624f6c158e0 con 0x5624f6bf4f20 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.688+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/2368762657 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cc8039070 con 0x5624f6bf4f20 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.688+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/2368762657 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f8cc803e890 con 0x5624f6bf4f20 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.688+0000 7f8ce8606b80 10 client.15693 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.689+0000 7f8ce8606b80 10 client.15693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.689+0000 7f8ce8606b80 20 client.15693 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.689+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5624f6c16000 0x5624f6c7fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.689+0000 7f8ce8606b80 4 client.15693 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:29:28.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.689+0000 7f8ce8606b80 1 -- 172.21.15.121:0/2368762657 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f8cd00a1250 con 0x5624f6c16000 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.689+0000 7f8cd6ffd700 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5624f6c16000 0x5624f6c7fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.689+0000 7f8cd6ffd700 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5624f6c16000 0x5624f6c7fc80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e70a85f52bbe7c2 server_cookie=c08b7570056b985e in_seq=0 out_seq=0 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.689+0000 7f8cb6ffd700 10 client.15693 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.689+0000 7f8cb6ffd700 1 -- 172.21.15.121:0/2368762657 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f8cd00a1250 con 0x5624f6c16000 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.689+0000 7f8cb6ffd700 10 client.15693 handle_command_reply: tid=0 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.689+0000 7f8ce8606b80 1 client.15693 shutdown 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8ce8606b80 1 -- 172.21.15.121:0/2368762657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 msgr2=0x5624f6c16ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c16ed0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8cc80116f0 tx=0x7f8cc801e8f0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8ce8606b80 1 -- 172.21.15.121:0/2368762657 shutdown_connections 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5624f6c16000 0x5624f6c7fc80 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5624f6bfaf60 0x5624f6c07070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5624f6bf4f20 0x5624f6c16ed0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8ce8606b80 1 --2- 172.21.15.121:0/2368762657 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5624f6bfa640 0x5624f6c06b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8ce8606b80 1 -- 172.21.15.121:0/2368762657 >> 172.21.15.121:0/2368762657 conn(0x5624f6b1fcc0 msgr2=0x5624f6bfb420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:28.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8cb6ffd700 0 client.15693 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:28.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8ce8606b80 1 -- 172.21.15.121:0/2368762657 shutdown_connections 2024-06-10T00:29:28.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8ce8606b80 1 -- 172.21.15.121:0/2368762657 wait complete. 2024-06-10T00:29:28.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.690+0000 7f8ce8606b80 20 client.15693 trim_cache size 0 max 16384 2024-06-10T00:29:28.694 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:29:28.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 -- 172.21.15.121:0/3083086090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cc00a2f30 msgr2=0x7f8cc00a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:28.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cc00a2f30 0x7f8cc00a53f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8199b90 tx=0x7f8cc800b410 comp rx=0 tx=0).stop 2024-06-10T00:29:28.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 -- 172.21.15.121:0/3083086090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cd8106380 msgr2=0x7f8cd8198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:28.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cd8106380 0x7f8cd8198a50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00062a0 tx=0x7f8cd0004d70 comp rx=0 tx=0).stop 2024-06-10T00:29:28.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 -- 172.21.15.121:0/3083086090 shutdown_connections 2024-06-10T00:29:28.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cc00a2f30 0x7f8cc00a53f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cd8106380 0x7f8cd8198a50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8cd8101b30 0x7f8cd819e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 --2- 172.21.15.121:0/3083086090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8cd81024f0 0x7f8cd819edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:28.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 -- 172.21.15.121:0/3083086090 >> 172.21.15.121:0/3083086090 conn(0x7f8cd8077990 msgr2=0x7f8cd8100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:28.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 -- 172.21.15.121:0/3083086090 shutdown_connections 2024-06-10T00:29:28.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:28.691+0000 7f8cddd83700 1 -- 172.21.15.121:0/3083086090 wait complete. 2024-06-10T00:29:28.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:28 smithi121 ceph-mon[32378]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:29:29.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:28 smithi135 ceph-mon[36424]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:29:29.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:28 smithi193 ceph-mon[36716]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:29:29.309 INFO:tasks.workunit.client.0.smithi121.stderr:Updating files: 93% (11779/12576) Updating files: 94% (11822/12576) Updating files: 95% (11948/12576) Updating files: 96% (12073/12576) Updating files: 97% (12199/12576) Updating files: 98% (12325/12576) Updating files: 99% (12451/12576) Updating files: 100% (12576/12576) Updating files: 100% (12576/12576), done. 2024-06-10T00:29:29.793 INFO:tasks.workunit.client.0.smithi121.stderr:Note: switching to '7b7a5d375cf4e4afab96b9d4d229b699ae8304bb'. 2024-06-10T00:29:29.793 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr:changes and commit them, and you can discard any commits you make in this 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr:state without impacting any branches by switching back to a branch. 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr:If you want to create a new branch to retain commits you create, you may 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr:do so (now or later) by using -c with the switch command. Example: 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr: git switch -c 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr:Or undo this operation with: 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr: git switch - 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-06-10T00:29:29.794 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-06-10T00:29:29.795 INFO:tasks.workunit.client.0.smithi121.stderr:HEAD is now at 7b7a5d375cf Merge pull request #57941 from zdover23/wip-doc-2024-06-09-backport-57939-to-squid 2024-06-10T00:29:29.798 DEBUG:teuthology.orchestra.run.smithi121:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-06-10T00:29:29.828 INFO:tasks.workunit.client.0.smithi121.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-06-10T00:29:29.830 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-06-10T00:29:29.830 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-06-10T00:29:29.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:29:30.221 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:30.548 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-10T00:29:30.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-10T00:29:30.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:30 smithi121 ceph-mon[32378]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 853 B/s wr, 8 op/s 2024-06-10T00:29:30.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:30 smithi121 ceph-mon[32378]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-06-10T00:29:31.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:30 smithi135 ceph-mon[36424]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 853 B/s wr, 8 op/s 2024-06-10T00:29:31.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:30 smithi135 ceph-mon[36424]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-06-10T00:29:31.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:30 smithi193 ceph-mon[36716]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 853 B/s wr, 8 op/s 2024-06-10T00:29:31.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:30 smithi193 ceph-mon[36716]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-06-10T00:29:31.289 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-06-10T00:29:31.422 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-06-10T00:29:31.476 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-06-10T00:29:31.478 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-06-10T00:29:31.478 INFO:tasks.workunit.client.0.smithi121.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-06-10T00:29:31.518 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-06-10T00:29:31.521 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:29:31.521 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-06-10T00:29:31.537 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-06-10T00:29:31.538 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-06-10T00:29:31.538 DEBUG:teuthology.orchestra.run.smithi121:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7b7a5d375cf4e4afab96b9d4d229b699ae8304bb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-06-10T00:29:32.570 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:32.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.586+0000 7f2cb80e6700 1 -- 172.21.15.121:0/353416451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0107870 msgr2=0x7f2cb010f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:32.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.586+0000 7f2cb80e6700 1 --2- 172.21.15.121:0/353416451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0107870 0x7f2cb010f3f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800b0d0 tx=0x7f2ca800b3e0 comp rx=0 tx=0).stop 2024-06-10T00:29:32.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.586+0000 7f2cb80e6700 1 -- 172.21.15.121:0/353416451 shutdown_connections 2024-06-10T00:29:32.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.586+0000 7f2cb80e6700 1 --2- 172.21.15.121:0/353416451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0107870 0x7f2cb010f3f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:32.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.586+0000 7f2cb80e6700 1 --2- 172.21.15.121:0/353416451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cb01027a0 0x7f2cb0102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:32.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.586+0000 7f2cb80e6700 1 --2- 172.21.15.121:0/353416451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb0103150 0x7f2cb0107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:32.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.586+0000 7f2cb80e6700 1 -- 172.21.15.121:0/353416451 >> 172.21.15.121:0/353416451 conn(0x7f2cb0079520 msgr2=0x7f2cb0079930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:32.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.586+0000 7f2cb80e6700 1 -- 172.21.15.121:0/353416451 shutdown_connections 2024-06-10T00:29:32.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.586+0000 7f2cb80e6700 1 -- 172.21.15.121:0/353416451 wait complete. 2024-06-10T00:29:32.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.587+0000 7f2cb80e6700 1 Processor -- start 2024-06-10T00:29:32.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.587+0000 7f2cb80e6700 1 -- start start 2024-06-10T00:29:32.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.587+0000 7f2cb80e6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cb01027a0 0x7f2cb0074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:32.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.587+0000 7f2cb80e6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb0103150 0x7f2cb0074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:32.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.587+0000 7f2cb80e6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0107870 0x7f2cb0075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:32.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.587+0000 7f2cb80e6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb01127b0 con 0x7f2cb0107870 2024-06-10T00:29:32.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.587+0000 7f2cb80e6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb0112630 con 0x7f2cb0103150 2024-06-10T00:29:32.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.587+0000 7f2cb80e6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb0112930 con 0x7f2cb01027a0 2024-06-10T00:29:32.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb6683700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0107870 0x7f2cb0075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb6683700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0107870 0x7f2cb0075210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60794/0 (socket says 172.21.15.121:60794) 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb6683700 1 -- 172.21.15.121:0/3899120628 learned_addr learned my addr 172.21.15.121:0/3899120628 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb5e82700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cb01027a0 0x7f2cb0074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb5681700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb0103150 0x7f2cb0074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb6683700 1 -- 172.21.15.121:0/3899120628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cb01027a0 msgr2=0x7f2cb0074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb6683700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cb01027a0 0x7f2cb0074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb6683700 1 -- 172.21.15.121:0/3899120628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb0103150 msgr2=0x7f2cb0074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb6683700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb0103150 0x7f2cb0074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb6683700 1 -- 172.21.15.121:0/3899120628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb01a7040 con 0x7f2cb0107870 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb6683700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0107870 0x7f2cb0075210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8006070 tx=0x7f2ca8004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2ca6ffd700 1 -- 172.21.15.121:0/3899120628 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8021070 con 0x7f2cb0107870 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2ca6ffd700 1 -- 172.21.15.121:0/3899120628 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ca8007b80 con 0x7f2cb0107870 2024-06-10T00:29:32.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb80e6700 1 -- 172.21.15.121:0/3899120628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb01a72d0 con 0x7f2cb0107870 2024-06-10T00:29:32.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2ca6ffd700 1 -- 172.21.15.121:0/3899120628 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca80052a0 con 0x7f2cb0107870 2024-06-10T00:29:32.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.588+0000 7f2cb80e6700 1 -- 172.21.15.121:0/3899120628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb01a7780 con 0x7f2cb0107870 2024-06-10T00:29:32.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.590+0000 7f2cb80e6700 1 -- 172.21.15.121:0/3899120628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cb010aa40 con 0x7f2cb0107870 2024-06-10T00:29:32.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.591+0000 7f2ca6ffd700 1 -- 172.21.15.121:0/3899120628 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2ca8024070 con 0x7f2cb0107870 2024-06-10T00:29:32.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.592+0000 7f2ca6ffd700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2c9c0a2d90 0x7f2c9c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:32.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.592+0000 7f2cb5e82700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2c9c0a2d90 0x7f2c9c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:32.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.592+0000 7f2ca6ffd700 1 -- 172.21.15.121:0/3899120628 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f2ca800f020 con 0x7f2cb0107870 2024-06-10T00:29:32.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.593+0000 7f2cb5e82700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2c9c0a2d90 0x7f2c9c0a5250 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0004630 tx=0x7f2ca000b3c0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:32.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.596+0000 7f2ca6ffd700 1 -- 172.21.15.121:0/3899120628 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2ca802b020 con 0x7f2cb0107870 2024-06-10T00:29:32.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.846+0000 7f2cb80e6700 1 -- 172.21.15.121:0/3899120628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cb0069180 con 0x7f2cb0107870 2024-06-10T00:29:32.847 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:32 smithi121 ceph-mon[32378]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 767 B/s wr, 5 op/s 2024-06-10T00:29:32.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.847+0000 7f2ca6ffd700 1 -- 172.21.15.121:0/3899120628 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f2ca809db10 con 0x7f2cb0107870 2024-06-10T00:29:32.848 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:29:32.848 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:27:55.370804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:29:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.849+0000 7f2cb80e6700 1 -- 172.21.15.121:0/3899120628 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2c9c0a2d90 msgr2=0x7f2c9c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.849+0000 7f2cb80e6700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2c9c0a2d90 0x7f2c9c0a5250 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0004630 tx=0x7f2ca000b3c0 comp rx=0 tx=0).stop 2024-06-10T00:29:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.849+0000 7f2cb80e6700 1 -- 172.21.15.121:0/3899120628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0107870 msgr2=0x7f2cb0075210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.849+0000 7f2cb80e6700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0107870 0x7f2cb0075210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8006070 tx=0x7f2ca8004d00 comp rx=0 tx=0).stop 2024-06-10T00:29:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.849+0000 7f2cb80e6700 1 -- 172.21.15.121:0/3899120628 shutdown_connections 2024-06-10T00:29:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.849+0000 7f2cb80e6700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2c9c0a2d90 0x7f2c9c0a5250 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.849+0000 7f2cb80e6700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0107870 0x7f2cb0075210 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.850+0000 7f2cb80e6700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2cb01027a0 0x7f2cb0074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:32.850 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.850+0000 7f2cb80e6700 1 --2- 172.21.15.121:0/3899120628 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2cb0103150 0x7f2cb0074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.850+0000 7f2cb80e6700 1 -- 172.21.15.121:0/3899120628 >> 172.21.15.121:0/3899120628 conn(0x7f2cb0079520 msgr2=0x7f2cb0100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.850+0000 7f2cb80e6700 1 -- 172.21.15.121:0/3899120628 shutdown_connections 2024-06-10T00:29:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:32.850+0000 7f2cb80e6700 1 -- 172.21.15.121:0/3899120628 wait complete. 2024-06-10T00:29:32.851 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-06-10T00:29:33.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:32 smithi135 ceph-mon[36424]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 767 B/s wr, 5 op/s 2024-06-10T00:29:33.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:32 smithi193 ceph-mon[36716]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 767 B/s wr, 5 op/s 2024-06-10T00:29:33.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:33 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3899120628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:34.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:33 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3899120628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:34.061 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db --set-id 25c0a77 --release --await' 2024-06-10T00:29:34.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db --set-id 25c0a77 --release --await 2024-06-10T00:29:34.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:33 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3899120628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:34.739 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:34 smithi121 ceph-mon[32378]: pgmap v51: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:29:35.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:34 smithi135 ceph-mon[36424]: pgmap v51: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:29:35.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:34 smithi193 ceph-mon[36716]: pgmap v51: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-06-10T00:29:35.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.133+0000 7f4622bc4700 1 -- 172.21.15.121:0/1945569798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c108510 msgr2=0x7f461c1088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:35.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.133+0000 7f4622bc4700 1 --2- 172.21.15.121:0/1945569798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c108510 0x7f461c1088f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4610009990 tx=0x7f4610009ca0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.133+0000 7f4622bc4700 1 -- 172.21.15.121:0/1945569798 shutdown_connections 2024-06-10T00:29:35.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.133+0000 7f4622bc4700 1 --2- 172.21.15.121:0/1945569798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101440 0x7f461c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.133+0000 7f4622bc4700 1 --2- 172.21.15.121:0/1945569798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c108510 0x7f461c1088f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.133+0000 7f4622bc4700 1 --2- 172.21.15.121:0/1945569798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c100a80 0x7f461c100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.133+0000 7f4622bc4700 1 -- 172.21.15.121:0/1945569798 >> 172.21.15.121:0/1945569798 conn(0x7f461c077a00 msgr2=0x7f461c0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:35.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.134+0000 7f4622bc4700 1 -- 172.21.15.121:0/1945569798 shutdown_connections 2024-06-10T00:29:35.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.134+0000 7f4622bc4700 1 -- 172.21.15.121:0/1945569798 wait complete. 2024-06-10T00:29:35.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.134+0000 7f4622bc4700 1 Processor -- start 2024-06-10T00:29:35.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.134+0000 7f4622bc4700 1 -- start start 2024-06-10T00:29:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.134+0000 7f4622bc4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c100a80 0x7f461c101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.134+0000 7f4622bc4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101440 0x7f461c1021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4622bc4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c108510 0x7f461c1026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4622bc4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f461c112790 con 0x7f461c100a80 2024-06-10T00:29:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4622bc4700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f461c112610 con 0x7f461c108510 2024-06-10T00:29:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4622bc4700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f461c112910 con 0x7f461c101440 2024-06-10T00:29:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4620960700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c100a80 0x7f461c101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4620960700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c100a80 0x7f461c101c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60808/0 (socket says 172.21.15.121:60808) 2024-06-10T00:29:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4620960700 1 -- 172.21.15.121:0/1678212213 learned_addr learned my addr 172.21.15.121:0/1678212213 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:35.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4621161700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c108510 0x7f461c1026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f461bfff700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101440 0x7f461c1021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4620960700 1 -- 172.21.15.121:0/1678212213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101440 msgr2=0x7f461c1021a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4620960700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101440 0x7f461c1021a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4620960700 1 -- 172.21.15.121:0/1678212213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c108510 msgr2=0x7f461c1026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4620960700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c108510 0x7f461c1026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4620960700 1 -- 172.21.15.121:0/1678212213 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c1a7170 con 0x7f461c100a80 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4621161700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c108510 0x7f461c1026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4620960700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c100a80 0x7f461c101c60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4610000c00 tx=0x7f4610003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4619ffb700 1 -- 172.21.15.121:0/1678212213 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610027070 con 0x7f461c100a80 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4622bc4700 1 -- 172.21.15.121:0/1678212213 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461c1a7400 con 0x7f461c100a80 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4619ffb700 1 -- 172.21.15.121:0/1678212213 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4610004190 con 0x7f461c100a80 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.135+0000 7f4619ffb700 1 -- 172.21.15.121:0/1678212213 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461000f460 con 0x7f461c100a80 2024-06-10T00:29:35.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.136+0000 7f4622bc4700 1 -- 172.21.15.121:0/1678212213 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461c1a78b0 con 0x7f461c100a80 2024-06-10T00:29:35.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.138+0000 7f4619ffb700 1 -- 172.21.15.121:0/1678212213 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f461002a070 con 0x7f461c100a80 2024-06-10T00:29:35.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.139+0000 7f4619ffb700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f460c0a2d90 0x7f460c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.139+0000 7f461bfff700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f460c0a2d90 0x7f460c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.139+0000 7f4619ffb700 1 -- 172.21.15.121:0/1678212213 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f46100d0720 con 0x7f461c100a80 2024-06-10T00:29:35.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.139+0000 7f462d447b80 1 Processor -- start 2024-06-10T00:29:35.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.139+0000 7f462d447b80 1 -- start start 2024-06-10T00:29:35.139 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.139+0000 7f461bfff700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f460c0a2d90 0x7f460c0a5250 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4608005fd0 tx=0x7f460800a560 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:35.140 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f462d447b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a76c0 0x55c6061a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f462d447b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f462d447b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f462d447b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c6060f3890 con 0x55c6061a7fe0 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f462d447b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55c60609b7a0 con 0x55c6061a1fa0 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f462d447b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c6060ef850 con 0x55c6061a76c0 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f461bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f461bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60812/0 (socket says 172.21.15.121:60812) 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f461bfff700 1 -- 172.21.15.121:0/2902556152 learned_addr learned my addr 172.21.15.121:0/2902556152 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f4621161700 1 --2- 172.21.15.121:0/2902556152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f4620960700 1 --2- 172.21.15.121:0/2902556152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a76c0 0x55c6061a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f461bfff700 1 -- 172.21.15.121:0/2902556152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a76c0 msgr2=0x55c6061a7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f461bfff700 1 --2- 172.21.15.121:0/2902556152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a76c0 0x55c6061a7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f461bfff700 1 -- 172.21.15.121:0/2902556152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 msgr2=0x55c6061a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:35.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f461bfff700 1 --2- 172.21.15.121:0/2902556152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f461bfff700 1 -- 172.21.15.121:0/2902556152 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c606091fd0 con 0x55c6061a7fe0 2024-06-10T00:29:35.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f4621161700 1 --2- 172.21.15.121:0/2902556152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:35.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.140+0000 7f461bfff700 1 --2- 172.21.15.121:0/2902556152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061a1a60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f460800ec00 tx=0x7f4608011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6ee36a2c6fa744 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:35.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.141+0000 7f4606ffd700 1 -- 172.21.15.121:0/2902556152 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608021070 con 0x55c6061a7fe0 2024-06-10T00:29:35.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.141+0000 7f4606ffd700 1 -- 172.21.15.121:0/2902556152 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4608010c50 con 0x55c6061a7fe0 2024-06-10T00:29:35.142 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.141+0000 7f4606ffd700 1 -- 172.21.15.121:0/2902556152 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460800f6b0 con 0x55c6061a7fe0 2024-06-10T00:29:35.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.141+0000 7f462d447b80 1 -- 172.21.15.121:0/2902556152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 msgr2=0x55c6061a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:35.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.141+0000 7f462d447b80 1 --2- 172.21.15.121:0/2902556152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061a1a60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f460800ec00 tx=0x7f4608011000 comp rx=0 tx=0).stop 2024-06-10T00:29:35.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.141+0000 7f462d447b80 1 -- 172.21.15.121:0/2902556152 shutdown_connections 2024-06-10T00:29:35.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.141+0000 7f462d447b80 1 --2- 172.21.15.121:0/2902556152 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.141+0000 7f462d447b80 1 --2- 172.21.15.121:0/2902556152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a76c0 0x55c6061a7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.141+0000 7f462d447b80 1 --2- 172.21.15.121:0/2902556152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061a1a60 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.141+0000 7f462d447b80 1 -- 172.21.15.121:0/2902556152 >> 172.21.15.121:0/2902556152 conn(0x55c6060ccd40 msgr2=0x55c60619c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:35.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.142+0000 7f462d447b80 1 -- 172.21.15.121:0/2902556152 shutdown_connections 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.142+0000 7f462d447b80 1 -- 172.21.15.121:0/2902556152 wait complete. 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.142+0000 7f462d447b80 1 -- start start 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.142+0000 7f462d447b80 1 Processor -- start 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.142+0000 7f462d447b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061add50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.142+0000 7f462d447b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a76c0 0x55c6061ab050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.142+0000 7f462d447b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061ab590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.142+0000 7f462d447b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c6061aea50 con 0x55c6061a7fe0 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.142+0000 7f462d447b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55c606093c70 con 0x55c6061a1fa0 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.142+0000 7f462d447b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c6061aebd0 con 0x55c6061a76c0 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f462d447b80 10 client.0 _ll_register_callbacks cb 0x55c60608d9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f4621161700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061ab590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f4621161700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061ab590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60826/0 (socket says 172.21.15.121:60826) 2024-06-10T00:29:35.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f4621161700 1 -- 172.21.15.121:0/235100710 learned_addr learned my addr 172.21.15.121:0/235100710 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f4620960700 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f461bfff700 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a76c0 0x55c6061ab050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f4621161700 1 -- 172.21.15.121:0/235100710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a76c0 msgr2=0x55c6061ab050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f4621161700 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a76c0 0x55c6061ab050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f4621161700 1 -- 172.21.15.121:0/235100710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 msgr2=0x55c6061add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f4621161700 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061add50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f4621161700 1 -- 172.21.15.121:0/235100710 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c606042950 con 0x55c6061a7fe0 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f461bfff700 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a76c0 0x55c6061ab050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f4621161700 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061ab590 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4614008f50 tx=0x7f4614010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c2dac338475603 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f45fbfff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f45fbfff700 1 -- 172.21.15.121:0/235100710 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614010dd0 con 0x55c6061a7fe0 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f45fbfff700 1 -- 172.21.15.121:0/235100710 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f4614016470 con 0x55c6061a7fe0 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.143+0000 7f462d447b80 1 -- 172.21.15.121:0/235100710 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4608010c50 con 0x55c6061a7fe0 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f45fbfff700 1 -- 172.21.15.121:0/235100710 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4614024050 con 0x55c6061a7fe0 2024-06-10T00:29:35.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f462d447b80 10 client.15711 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f462d447b80 1 -- 172.21.15.121:0/235100710 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c6061b9400 con 0x55c6061a7fe0 2024-06-10T00:29:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f45fbfff700 1 -- 172.21.15.121:0/235100710 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4608010c50 con 0x55c6061a7fe0 2024-06-10T00:29:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f45fbfff700 1 -- 172.21.15.121:0/235100710 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f46140157d0 con 0x55c6061a7fe0 2024-06-10T00:29:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f462d447b80 10 client.15711 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f462d447b80 10 client.15711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f462d447b80 20 client.15711 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f462d447b80 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c6061bba40 0x55c6061bde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f462d447b80 4 client.15711 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f462d447b80 1 -- 172.21.15.121:0/235100710 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c606100d90 con 0x55c6061bba40 2024-06-10T00:29:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.144+0000 7f4620960700 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c6061bba40 0x55c6061bde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.145+0000 7f45fbfff700 10 client.15711 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:35.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.145+0000 7f4620960700 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c6061bba40 0x55c6061bde90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=accb8e6a1cba2713 server_cookie=2dc2283d94b7cc83 in_seq=0 out_seq=0 2024-06-10T00:29:35.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f45fbfff700 1 -- 172.21.15.121:0/235100710 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55c606100d90 con 0x55c6061bba40 2024-06-10T00:29:35.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f45fbfff700 10 client.15711 handle_command_reply: tid=0 2024-06-10T00:29:35.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 1 client.15711 shutdown 2024-06-10T00:29:35.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 1 -- 172.21.15.121:0/235100710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 msgr2=0x55c6061ab590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:35.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061ab590 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4614008f50 tx=0x7f4614010430 comp rx=0 tx=0).stop 2024-06-10T00:29:35.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 1 -- 172.21.15.121:0/235100710 shutdown_connections 2024-06-10T00:29:35.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c6061bba40 0x55c6061bde90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f45fbfff700 0 client.15711 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:35.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a7fe0 0x55c6061ab590 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 1 --2- 172.21.15.121:0/235100710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a76c0 0x55c6061ab050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 1 -- 172.21.15.121:0/235100710 >> 172.21.15.121:0/235100710 conn(0x55c6060ccd40 msgr2=0x55c6061a0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:35.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 1 -- 172.21.15.121:0/235100710 shutdown_connections 2024-06-10T00:29:35.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 1 -- 172.21.15.121:0/235100710 wait complete. 2024-06-10T00:29:35.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.146+0000 7f462d447b80 20 client.15711 trim_cache size 0 max 16384 2024-06-10T00:29:35.170 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:35.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.172+0000 7f462d447b80 1 Processor -- start 2024-06-10T00:29:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.172+0000 7f462d447b80 1 -- start start 2024-06-10T00:29:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f462d447b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f462d447b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c606196710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f462d447b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a7fe0 0x55c6061aa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f462d447b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1b2e70 con 0x55c6061a76c0 2024-06-10T00:29:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f462d447b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1b0700 con 0x55c6061a1fa0 2024-06-10T00:29:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f462d447b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1ae550 con 0x55c6061a7fe0 2024-06-10T00:29:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f461bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c606196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f461bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c606196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60840/0 (socket says 172.21.15.121:60840) 2024-06-10T00:29:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f461bfff700 1 -- 172.21.15.121:0/1943390857 learned_addr learned my addr 172.21.15.121:0/1943390857 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:35.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f4620960700 1 --2- 172.21.15.121:0/1943390857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f4621161700 1 --2- 172.21.15.121:0/1943390857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a7fe0 0x55c6061aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f461bfff700 1 -- 172.21.15.121:0/1943390857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a7fe0 msgr2=0x55c6061aa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:35.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f461bfff700 1 --2- 172.21.15.121:0/1943390857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a7fe0 0x55c6061aa840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f461bfff700 1 -- 172.21.15.121:0/1943390857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 msgr2=0x55c6061a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:35.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f461bfff700 1 --2- 172.21.15.121:0/1943390857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f461bfff700 1 -- 172.21.15.121:0/1943390857 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6061abf50 con 0x55c6061a76c0 2024-06-10T00:29:35.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f4620960700 1 --2- 172.21.15.121:0/1943390857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:35.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.173+0000 7f461bfff700 1 --2- 172.21.15.121:0/1943390857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c606196710 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f460800eba0 tx=0x7f4608010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae9ee8dfcb1fc4e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:35.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.174+0000 7f46067fc700 1 -- 172.21.15.121:0/1943390857 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4608021040 con 0x55c6061a76c0 2024-06-10T00:29:35.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.174+0000 7f46067fc700 1 -- 172.21.15.121:0/1943390857 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4608005720 con 0x55c6061a76c0 2024-06-10T00:29:35.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.174+0000 7f46067fc700 1 -- 172.21.15.121:0/1943390857 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460801d6b0 con 0x55c6061a76c0 2024-06-10T00:29:35.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.174+0000 7f462d447b80 1 -- 172.21.15.121:0/1943390857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 msgr2=0x55c606196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:35.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.174+0000 7f462d447b80 1 --2- 172.21.15.121:0/1943390857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c606196710 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f460800eba0 tx=0x7f4608010cb0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 -- 172.21.15.121:0/1943390857 shutdown_connections 2024-06-10T00:29:35.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 --2- 172.21.15.121:0/1943390857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a7fe0 0x55c6061aa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 --2- 172.21.15.121:0/1943390857 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a1fa0 0x55c6061a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 --2- 172.21.15.121:0/1943390857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c606196710 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 -- 172.21.15.121:0/1943390857 >> 172.21.15.121:0/1943390857 conn(0x55c6060ccd40 msgr2=0x55c6061ac880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 -- 172.21.15.121:0/1943390857 shutdown_connections 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 -- 172.21.15.121:0/1943390857 wait complete. 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 -- start start 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 Processor -- start 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a1fa0 0x55c6061b4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c6061b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.175+0000 7f462d447b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a7fe0 0x55c6061b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f462d447b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1aebc0 con 0x55c6061a76c0 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f462d447b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1b0030 con 0x55c6061a7fe0 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f462d447b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1af290 con 0x55c6061a1fa0 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f462d447b80 10 client.0 _ll_register_callbacks cb 0x55c60608d9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f461bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c6061b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f461bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c6061b0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60842/0 (socket says 172.21.15.121:60842) 2024-06-10T00:29:35.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f461bfff700 1 -- 172.21.15.121:0/2835294563 learned_addr learned my addr 172.21.15.121:0/2835294563 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f4621161700 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a7fe0 0x55c6061b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f4620960700 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a1fa0 0x55c6061b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f461bfff700 1 -- 172.21.15.121:0/2835294563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a1fa0 msgr2=0x55c6061b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f461bfff700 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a1fa0 0x55c6061b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f461bfff700 1 -- 172.21.15.121:0/2835294563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a7fe0 msgr2=0x55c6061b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f461bfff700 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a7fe0 0x55c6061b13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f461bfff700 1 -- 172.21.15.121:0/2835294563 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6061b19e0 con 0x55c6061a76c0 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f4620960700 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a1fa0 0x55c6061b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f461bfff700 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c6061b0e60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4608010710 tx=0x7f4608004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa08f3413b134f64 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f45fbfff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f45fbfff700 1 -- 172.21.15.121:0/2835294563 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460803f060 con 0x55c6061a76c0 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f462d447b80 1 -- 172.21.15.121:0/2835294563 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c6060fb310 con 0x55c6061a76c0 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.176+0000 7f45fbfff700 1 -- 172.21.15.121:0/2835294563 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f4608005450 con 0x55c6061a76c0 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f45fbfff700 1 -- 172.21.15.121:0/2835294563 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4608014070 con 0x55c6061a76c0 2024-06-10T00:29:35.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f462d447b80 10 client.15717 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f462d447b80 1 -- 172.21.15.121:0/2835294563 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c6062275f0 con 0x55c6061a76c0 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f45fbfff700 1 -- 172.21.15.121:0/2835294563 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c6060fb310 con 0x55c6061a76c0 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f45fbfff700 1 -- 172.21.15.121:0/2835294563 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f4608026570 con 0x55c6061a76c0 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f462d447b80 10 client.15717 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f462d447b80 10 client.15717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f462d447b80 20 client.15717 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f462d447b80 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c606227a60 0x55c606229e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f462d447b80 4 client.15717 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f462d447b80 1 -- 172.21.15.121:0/2835294563 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c605f752d0 con 0x55c606227a60 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.177+0000 7f4620960700 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c606227a60 0x55c606229e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.178+0000 7f4620960700 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c606227a60 0x55c606229e80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f70426ae8714c03 server_cookie=93dee284cc59ecab in_seq=0 out_seq=0 2024-06-10T00:29:35.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.178+0000 7f45fbfff700 10 client.15717 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:35.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f45fbfff700 1 -- 172.21.15.121:0/2835294563 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c605f752d0 con 0x55c606227a60 2024-06-10T00:29:35.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f45fbfff700 10 client.15717 handle_command_reply: tid=0 2024-06-10T00:29:35.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f462d447b80 1 client.15717 shutdown 2024-06-10T00:29:35.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f462d447b80 1 -- 172.21.15.121:0/2835294563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 msgr2=0x55c6061b0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:35.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f462d447b80 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c6061b0e60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4608010710 tx=0x7f4608004920 comp rx=0 tx=0).stop 2024-06-10T00:29:35.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f462d447b80 1 -- 172.21.15.121:0/2835294563 shutdown_connections 2024-06-10T00:29:35.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f462d447b80 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55c606227a60 0x55c606229e80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f462d447b80 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c6061a7fe0 0x55c6061b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f45fbfff700 0 client.15717 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:35.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f462d447b80 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6061a1fa0 0x55c6061b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f462d447b80 1 --2- 172.21.15.121:0/2835294563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6061a76c0 0x55c6061b0e60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.179+0000 7f462d447b80 1 -- 172.21.15.121:0/2835294563 >> 172.21.15.121:0/2835294563 conn(0x55c6060ccd40 msgr2=0x55c606196590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:35.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.180+0000 7f462d447b80 1 -- 172.21.15.121:0/2835294563 shutdown_connections 2024-06-10T00:29:35.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.180+0000 7f462d447b80 1 -- 172.21.15.121:0/2835294563 wait complete. 2024-06-10T00:29:35.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.180+0000 7f462d447b80 20 client.15717 trim_cache size 0 max 16384 2024-06-10T00:29:35.183 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:29:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "return_code": 0, 2024-06-10T00:29:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_tag": "1e08fe48-11fb-4fee-b701-91f68c0a4d54", 2024-06-10T00:29:35.184 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": "asynchronous" 2024-06-10T00:29:35.184 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:29:35.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.180+0000 7f4622bc4700 1 -- 172.21.15.121:0/1678212213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f460c0a2d90 msgr2=0x7f460c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:35.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.180+0000 7f4622bc4700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f460c0a2d90 0x7f460c0a5250 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4608005fd0 tx=0x7f460800a560 comp rx=0 tx=0).stop 2024-06-10T00:29:35.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.181+0000 7f4622bc4700 1 -- 172.21.15.121:0/1678212213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c100a80 msgr2=0x7f461c101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:35.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.181+0000 7f4622bc4700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c100a80 0x7f461c101c60 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4610000c00 tx=0x7f4610003730 comp rx=0 tx=0).stop 2024-06-10T00:29:35.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.181+0000 7f4622bc4700 1 -- 172.21.15.121:0/1678212213 shutdown_connections 2024-06-10T00:29:35.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.181+0000 7f4622bc4700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f460c0a2d90 0x7f460c0a5250 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.181+0000 7f4622bc4700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c108510 0x7f461c1026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.181+0000 7f4622bc4700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c100a80 0x7f461c101c60 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.181+0000 7f4622bc4700 1 --2- 172.21.15.121:0/1678212213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c101440 0x7f461c1021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:35.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.181+0000 7f4622bc4700 1 -- 172.21.15.121:0/1678212213 >> 172.21.15.121:0/1678212213 conn(0x7f461c077a00 msgr2=0x7f461c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:35.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.181+0000 7f4622bc4700 1 -- 172.21.15.121:0/1678212213 shutdown_connections 2024-06-10T00:29:35.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:35.181+0000 7f4622bc4700 1 -- 172.21.15.121:0/1678212213 wait complete. 2024-06-10T00:29:35.892 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:35 smithi121 ceph-mon[32378]: scrub queued for path: / 2024-06-10T00:29:35.892 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:35 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:35.893 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:35 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:29:35.893 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:35 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:36.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:35 smithi135 ceph-mon[36424]: scrub queued for path: / 2024-06-10T00:29:36.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:35 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:36.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:35 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:29:36.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:35 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:36.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:35 smithi193 ceph-mon[36716]: scrub queued for path: / 2024-06-10T00:29:36.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:35 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:36.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:35 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:29:36.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:35 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:29:36.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:29:36.793 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:36 smithi121 ceph-mon[32378]: pgmap v52: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-06-10T00:29:36.793 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:36 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:29:37.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:36 smithi135 ceph-mon[36424]: pgmap v52: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-06-10T00:29:37.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:36 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:29:37.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:36 smithi193 ceph-mon[36716]: pgmap v52: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-06-10T00:29:37.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:36 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:29:37.411 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:37.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.632+0000 7f2e31ac2700 1 -- 172.21.15.121:0/2147676904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e2c107400 msgr2=0x7f2e2c1077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:37.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.632+0000 7f2e31ac2700 1 --2- 172.21.15.121:0/2147676904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e2c107400 0x7f2e2c1077e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c009990 tx=0x7f2e1c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.632+0000 7f2e31ac2700 1 -- 172.21.15.121:0/2147676904 shutdown_connections 2024-06-10T00:29:37.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.632+0000 7f2e31ac2700 1 --2- 172.21.15.121:0/2147676904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e2c101d50 0x7f2e2c10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.632+0000 7f2e31ac2700 1 --2- 172.21.15.121:0/2147676904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e2c107400 0x7f2e2c1077e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.632+0000 7f2e31ac2700 1 --2- 172.21.15.121:0/2147676904 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e2c101390 0x7f2e2c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.632+0000 7f2e31ac2700 1 -- 172.21.15.121:0/2147676904 >> 172.21.15.121:0/2147676904 conn(0x7f2e2c078340 msgr2=0x7f2e2c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.632+0000 7f2e31ac2700 1 -- 172.21.15.121:0/2147676904 shutdown_connections 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.632+0000 7f2e31ac2700 1 -- 172.21.15.121:0/2147676904 wait complete. 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e31ac2700 1 Processor -- start 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e31ac2700 1 -- start start 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e31ac2700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e2c101390 0x7f2e2c19d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e31ac2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e2c101d50 0x7f2e2c19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e31ac2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e2c107400 0x7f2e2c197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e31ac2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1116a0 con 0x7f2e2c101d50 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e31ac2700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c111520 con 0x7f2e2c107400 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e31ac2700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c111820 con 0x7f2e2c101390 2024-06-10T00:29:37.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e23fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e2c101d50 0x7f2e2c19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e23fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e2c101d50 0x7f2e2c19dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60854/0 (socket says 172.21.15.121:60854) 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e23fff700 1 -- 172.21.15.121:0/3615657269 learned_addr learned my addr 172.21.15.121:0/3615657269 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e2bfff700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e2c107400 0x7f2e2c197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e2c101390 0x7f2e2c19d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.633+0000 7f2e23fff700 1 -- 172.21.15.121:0/3615657269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e2c101390 msgr2=0x7f2e2c19d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e23fff700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e2c101390 0x7f2e2c19d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e23fff700 1 -- 172.21.15.121:0/3615657269 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e2c107400 msgr2=0x7f2e2c197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e23fff700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e2c107400 0x7f2e2c197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e23fff700 1 -- 172.21.15.121:0/3615657269 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2c1980e0 con 0x7f2e2c101d50 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e2bfff700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e2c107400 0x7f2e2c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e2c101390 0x7f2e2c19d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e23fff700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e2c101d50 0x7f2e2c19dd00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400d950 tx=0x7f2e1400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:37.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e297fa700 1 -- 172.21.15.121:0/3615657269 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14016070 con 0x7f2e2c101d50 2024-06-10T00:29:37.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e31ac2700 1 -- 172.21.15.121:0/3615657269 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e2c1983d0 con 0x7f2e2c101d50 2024-06-10T00:29:37.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e297fa700 1 -- 172.21.15.121:0/3615657269 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e14009b40 con 0x7f2e2c101d50 2024-06-10T00:29:37.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e297fa700 1 -- 172.21.15.121:0/3615657269 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1400f620 con 0x7f2e2c101d50 2024-06-10T00:29:37.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.634+0000 7f2e31ac2700 1 -- 172.21.15.121:0/3615657269 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e2c06a170 con 0x7f2e2c101d50 2024-06-10T00:29:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.636+0000 7f2e297fa700 1 -- 172.21.15.121:0/3615657269 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2e14019070 con 0x7f2e2c101d50 2024-06-10T00:29:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.637+0000 7f2e297fa700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2e180a4fa0 0x7f2e180a7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.637+0000 7f2e297fa700 1 -- 172.21.15.121:0/3615657269 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f2e140cf820 con 0x7f2e2c101d50 2024-06-10T00:29:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.637+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2e180a4fa0 0x7f2e180a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.637+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2e180a4fa0 0x7f2e180a7460 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c000c00 tx=0x7f2e1c005fb0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e3c345b80 1 Processor -- start 2024-06-10T00:29:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e3c345b80 1 -- start start 2024-06-10T00:29:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586c560 0x55fbb5865e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb58663d0 0x55fbb5866810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fbb57b7910 con 0x55fbb586bc40 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55fbb575f820 con 0x55fbb58663d0 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fbb57b38d0 con 0x55fbb586c560 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60856/0 (socket says 172.21.15.121:60856) 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/1713495340 learned_addr learned my addr 172.21.15.121:0/1713495340 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e23fff700 1 --2- 172.21.15.121:0/1713495340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586c560 0x55fbb5865e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e2bfff700 1 --2- 172.21.15.121:0/1713495340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb58663d0 0x55fbb5866810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/1713495340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586c560 msgr2=0x55fbb5865e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/1713495340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586c560 0x55fbb5865e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/1713495340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb58663d0 msgr2=0x55fbb5866810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/1713495340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb58663d0 0x55fbb5866810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.638+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/1713495340 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbb5756050 con 0x55fbb586bc40 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.639+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/1713495340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586c020 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c009750 tx=0x7f2e1c005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884ef6a7cf0d5a47 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.639+0000 7f2e21ffb700 1 -- 172.21.15.121:0/1713495340 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c003d40 con 0x55fbb586bc40 2024-06-10T00:29:37.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.639+0000 7f2e21ffb700 1 -- 172.21.15.121:0/1713495340 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e1c00bcb0 con 0x55fbb586bc40 2024-06-10T00:29:37.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.639+0000 7f2e21ffb700 1 -- 172.21.15.121:0/1713495340 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c02e9a0 con 0x55fbb586bc40 2024-06-10T00:29:37.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.640+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1713495340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 msgr2=0x55fbb586c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:37.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.640+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/1713495340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586c020 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c009750 tx=0x7f2e1c005000 comp rx=0 tx=0).stop 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.640+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1713495340 shutdown_connections 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.640+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/1713495340 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb58663d0 0x55fbb5866810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.640+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/1713495340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586c020 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.640+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/1713495340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586c560 0x55fbb5865e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.640+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1713495340 >> 172.21.15.121:0/1713495340 conn(0x55fbb5790dc0 msgr2=0x55fbb5861120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.640+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1713495340 shutdown_connections 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.640+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1713495340 wait complete. 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.640+0000 7f2e3c345b80 1 -- start start 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.640+0000 7f2e3c345b80 1 Processor -- start 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb58663d0 0x55fbb5872290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586c560 0x55fbb586fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fbb5872f70 con 0x55fbb586bc40 2024-06-10T00:29:37.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55fbb5757cf0 con 0x55fbb586c560 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fbb58730f0 con 0x55fbb58663d0 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e3c345b80 10 client.0 _ll_register_callbacks cb 0x55fbb5751a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e23fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e23fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60858/0 (socket says 172.21.15.121:60858) 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e23fff700 1 -- 172.21.15.121:0/1610582701 learned_addr learned my addr 172.21.15.121:0/1610582701 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb58663d0 0x55fbb5872290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e2bfff700 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586c560 0x55fbb586fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/1610582701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586c560 msgr2=0x55fbb586fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586c560 0x55fbb586fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/1610582701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 msgr2=0x55fbb586f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/1610582701 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbb57069d0 con 0x55fbb58663d0 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e23fff700 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb58663d0 0x55fbb5872290 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c009350 tx=0x7f2e1c00bbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5598e639fc03992a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:37.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.641+0000 7f2e0affd700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e0affd700 1 -- 172.21.15.121:0/1610582701 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c03b060 con 0x55fbb58663d0 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e0affd700 1 -- 172.21.15.121:0/1610582701 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f2e1c02e7b0 con 0x55fbb58663d0 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1610582701 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e1c00bcb0 con 0x55fbb58663d0 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e0affd700 1 -- 172.21.15.121:0/1610582701 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e1c005480 con 0x55fbb58663d0 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e3c345b80 10 client.25273 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1610582701 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fbb5872f70 con 0x55fbb58663d0 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e0affd700 1 -- 172.21.15.121:0/1610582701 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e1c00bcb0 con 0x55fbb58663d0 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e0affd700 1 -- 172.21.15.121:0/1610582701 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f2e1c020070 con 0x55fbb58663d0 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e3c345b80 10 client.25273 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e3c345b80 10 client.25273 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e3c345b80 20 client.25273 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.642+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fbb587fbb0 0x55fbb5881fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.643+0000 7f2e3c345b80 4 client.25273 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:37.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.643+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1610582701 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbb57c4e10 con 0x55fbb587fbb0 2024-06-10T00:29:37.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.643+0000 7f2e23fff700 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fbb587fbb0 0x55fbb5881fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.643+0000 7f2e23fff700 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fbb587fbb0 0x55fbb5881fa0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99c79f4f9a3c6669 server_cookie=e77095ea834614d2 in_seq=0 out_seq=0 2024-06-10T00:29:37.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.643+0000 7f2e0affd700 10 client.25273 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e0affd700 1 -- 172.21.15.121:0/1610582701 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55fbb57c4e10 con 0x55fbb587fbb0 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e0affd700 10 client.25273 handle_command_reply: tid=0 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e3c345b80 1 client.25273 shutdown 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1610582701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb58663d0 msgr2=0x55fbb5872290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb58663d0 0x55fbb5872290 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c009350 tx=0x7f2e1c00bbf0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1610582701 shutdown_connections 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fbb587fbb0 0x55fbb5881fa0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586c560 0x55fbb586fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb58663d0 0x55fbb5872290 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e0affd700 0 client.25273 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/1610582701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb586bc40 0x55fbb586f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1610582701 >> 172.21.15.121:0/1610582701 conn(0x55fbb5790dc0 msgr2=0x55fbb5864e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.644+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1610582701 shutdown_connections 2024-06-10T00:29:37.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.645+0000 7f2e3c345b80 1 -- 172.21.15.121:0/1610582701 wait complete. 2024-06-10T00:29:37.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.645+0000 7f2e3c345b80 20 client.25273 trim_cache size 0 max 16384 2024-06-10T00:29:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e3c345b80 1 Processor -- start 2024-06-10T00:29:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e3c345b80 1 -- start start 2024-06-10T00:29:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5865e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586bc40 0x55fbb586d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586c560 0x55fbb586e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1aebe0 con 0x55fbb58663d0 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1ac470 con 0x55fbb586c560 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1aa2c0 con 0x55fbb586bc40 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5865e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5865e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60864/0 (socket says 172.21.15.121:60864) 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/809111781 learned_addr learned my addr 172.21.15.121:0/809111781 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.671+0000 7f2e23fff700 1 --2- 172.21.15.121:0/809111781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586bc40 0x55fbb586d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.672+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/809111781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586bc40 msgr2=0x55fbb586d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.672+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/809111781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586bc40 0x55fbb586d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.672+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/809111781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586c560 msgr2=0x55fbb586e710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.672+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/809111781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586c560 0x55fbb586e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.672+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/809111781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbb586db00 con 0x55fbb58663d0 2024-06-10T00:29:37.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.672+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/809111781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5865e40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c003a30 tx=0x7f2e1c004fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950804daaecfdcb8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:37.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.673+0000 7f2e217fa700 1 -- 172.21.15.121:0/809111781 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c03b040 con 0x55fbb58663d0 2024-06-10T00:29:37.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.673+0000 7f2e217fa700 1 -- 172.21.15.121:0/809111781 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e1c022590 con 0x55fbb58663d0 2024-06-10T00:29:37.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.673+0000 7f2e217fa700 1 -- 172.21.15.121:0/809111781 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c02ea10 con 0x55fbb58663d0 2024-06-10T00:29:37.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.674+0000 7f2e3c345b80 1 -- 172.21.15.121:0/809111781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 msgr2=0x55fbb5865e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:37.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.674+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/809111781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5865e40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c003a30 tx=0x7f2e1c004fe0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.674+0000 7f2e3c345b80 1 -- 172.21.15.121:0/809111781 shutdown_connections 2024-06-10T00:29:37.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.674+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/809111781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586c560 0x55fbb586e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.674+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/809111781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5865e40 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.674+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/809111781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586bc40 0x55fbb586d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.674+0000 7f2e3c345b80 1 -- 172.21.15.121:0/809111781 >> 172.21.15.121:0/809111781 conn(0x55fbb5790dc0 msgr2=0x55fbb5860830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:37.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.674+0000 7f2e3c345b80 1 -- 172.21.15.121:0/809111781 shutdown_connections 2024-06-10T00:29:37.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.674+0000 7f2e3c345b80 1 -- 172.21.15.121:0/809111781 wait complete. 2024-06-10T00:29:37.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e3c345b80 1 -- start start 2024-06-10T00:29:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e3c345b80 1 Processor -- start 2024-06-10T00:29:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5874820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586bc40 0x55fbb5874d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e3c345b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586c560 0x55fbb58e9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1aa930 con 0x55fbb58663d0 2024-06-10T00:29:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1abda0 con 0x55fbb586bc40 2024-06-10T00:29:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e3c345b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e2c1ab000 con 0x55fbb586c560 2024-06-10T00:29:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e3c345b80 10 client.0 _ll_register_callbacks cb 0x55fbb5751a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5874820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e2b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5874820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60874/0 (socket says 172.21.15.121:60874) 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/3093815769 learned_addr learned my addr 172.21.15.121:0/3093815769 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e23fff700 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586bc40 0x55fbb5874d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.675+0000 7f2e2bfff700 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586c560 0x55fbb58e9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/3093815769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586c560 msgr2=0x55fbb58e9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586c560 0x55fbb58e9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/3093815769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586bc40 msgr2=0x55fbb5874d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586bc40 0x55fbb5874d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e2b7fe700 1 -- 172.21.15.121:0/3093815769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbb58e9ab0 con 0x55fbb58663d0 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e2b7fe700 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5874820 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c031040 tx=0x7f2e1c02ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164202feb69db332 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e0affd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e0affd700 1 -- 172.21.15.121:0/3093815769 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c03b040 con 0x55fbb58663d0 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e3c345b80 1 -- 172.21.15.121:0/3093815769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e1c022590 con 0x55fbb58663d0 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e0affd700 1 -- 172.21.15.121:0/3093815769 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f2e1c022ac0 con 0x55fbb58663d0 2024-06-10T00:29:37.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e0affd700 1 -- 172.21.15.121:0/3093815769 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e1c056050 con 0x55fbb58663d0 2024-06-10T00:29:37.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e3c345b80 10 client.15735 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:37.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.676+0000 7f2e3c345b80 1 -- 172.21.15.121:0/3093815769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55fbb58ead70 con 0x55fbb58663d0 2024-06-10T00:29:37.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.677+0000 7f2e0affd700 1 -- 172.21.15.121:0/3093815769 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e1c022590 con 0x55fbb58663d0 2024-06-10T00:29:37.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.677+0000 7f2e0affd700 1 -- 172.21.15.121:0/3093815769 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f2e1c020040 con 0x55fbb58663d0 2024-06-10T00:29:37.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.677+0000 7f2e3c345b80 10 client.15735 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:37.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.677+0000 7f2e3c345b80 10 client.15735 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:29:37.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.677+0000 7f2e3c345b80 20 client.15735 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:37.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.677+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fbb58eb180 0x55fbb58ed570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:37.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.677+0000 7f2e3c345b80 4 client.15735 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "25c0a77", "release": true, "await": true}] 2024-06-10T00:29:37.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.677+0000 7f2e3c345b80 1 -- 172.21.15.121:0/3093815769 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "set_id": "25c0a77", "release": true, "await": true}) -- 0x55fbb57bf390 con 0x55fbb58eb180 2024-06-10T00:29:37.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.677+0000 7f2e23fff700 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fbb58eb180 0x55fbb58ed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:37.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.677+0000 7f2e0affd700 10 client.15735 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:37.680 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.677+0000 7f2e23fff700 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fbb58eb180 0x55fbb58ed570 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cc6be31c4c858b0 server_cookie=709ce45fd638df82 in_seq=0 out_seq=0 2024-06-10T00:29:37.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e0affd700 1 -- 172.21.15.121:0/3093815769 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+588 (crc 0 0 0) 0x55fbb57bf390 con 0x55fbb58eb180 2024-06-10T00:29:37.681 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e0affd700 10 client.15735 handle_command_reply: tid=0 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 1 client.15735 shutdown 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 1 -- 172.21.15.121:0/3093815769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 msgr2=0x55fbb5874820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5874820 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c031040 tx=0x7f2e1c02ea60 comp rx=0 tx=0).stop 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 1 -- 172.21.15.121:0/3093815769 shutdown_connections 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fbb58eb180 0x55fbb58ed570 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fbb586c560 0x55fbb58e9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fbb58663d0 0x55fbb5874820 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e0affd700 0 client.15735 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 1 --2- 172.21.15.121:0/3093815769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fbb586bc40 0x55fbb5874d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 1 -- 172.21.15.121:0/3093815769 >> 172.21.15.121:0/3093815769 conn(0x55fbb5790dc0 msgr2=0x55fbb58610c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 1 -- 172.21.15.121:0/3093815769 shutdown_connections 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 1 -- 172.21.15.121:0/3093815769 wait complete. 2024-06-10T00:29:37.682 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.681+0000 7f2e3c345b80 20 client.15735 trim_cache size 0 max 16384 2024-06-10T00:29:37.683 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:29:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 26, 2024-06-10T00:29:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:29:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 4, 2024-06-10T00:29:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:29:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "25c0a77": { 2024-06-10T00:29:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 4, 2024-06-10T00:29:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:29:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:29:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:29:37.683 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 150.0, 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:37.684 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:37.685 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:29:37.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 -- 172.21.15.121:0/3615657269 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2e180a4fa0 msgr2=0x7f2e180a7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:37.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2e180a4fa0 0x7f2e180a7460 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c000c00 tx=0x7f2e1c005fb0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 -- 172.21.15.121:0/3615657269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e2c101d50 msgr2=0x7f2e2c19dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:37.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e2c101d50 0x7f2e2c19dd00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400d950 tx=0x7f2e1400dd10 comp rx=0 tx=0).stop 2024-06-10T00:29:37.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 -- 172.21.15.121:0/3615657269 shutdown_connections 2024-06-10T00:29:37.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2e180a4fa0 0x7f2e180a7460 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e2c107400 0x7f2e2c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e2c101390 0x7f2e2c19d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 --2- 172.21.15.121:0/3615657269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e2c101d50 0x7f2e2c19dd00 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:37.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 -- 172.21.15.121:0/3615657269 >> 172.21.15.121:0/3615657269 conn(0x7f2e2c078340 msgr2=0x7f2e2c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:37.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 -- 172.21.15.121:0/3615657269 shutdown_connections 2024-06-10T00:29:37.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:37.682+0000 7f2e31ac2700 1 -- 172.21.15.121:0/3615657269 wait complete. 2024-06-10T00:29:37.686 INFO:tasks.workunit.client.0.smithi121.stdout:getting blogbench 2024-06-10T00:29:37.686 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'getting blogbench' 2024-06-10T00:29:37.687 INFO:tasks.workunit.client.0.smithi121.stderr:+ wget http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-06-10T00:29:37.767 INFO:tasks.workunit.client.0.smithi121.stderr:--2024-06-10 00:29:37-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-06-10T00:29:37.767 INFO:tasks.workunit.client.0.smithi121.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-06-10T00:29:37.789 INFO:tasks.workunit.client.0.smithi121.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-06-10T00:29:37.810 INFO:tasks.workunit.client.0.smithi121.stderr:HTTP request sent, awaiting response... 200 OK 2024-06-10T00:29:37.810 INFO:tasks.workunit.client.0.smithi121.stderr:Length: 103355 (101K) [application/octet-stream] 2024-06-10T00:29:37.811 INFO:tasks.workunit.client.0.smithi121.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-06-10T00:29:37.814 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-06-10T00:29:37.852 INFO:tasks.workunit.client.0.smithi121.stderr: 0K .......... .......... .......... .......... .......... 49% 1.22M 0s 2024-06-10T00:29:37.872 INFO:tasks.workunit.client.0.smithi121.stderr: 50K .......... .......... .......... .......... .......... 99% 2.36M 0s 2024-06-10T00:29:37.872 INFO:tasks.workunit.client.0.smithi121.stderr: 100K 100% 1.74T=0.06s 2024-06-10T00:29:37.872 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-06-10T00:29:37.872 INFO:tasks.workunit.client.0.smithi121.stderr:2024-06-10 00:29:37 (1.63 MB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-06-10T00:29:37.872 INFO:tasks.workunit.client.0.smithi121.stderr: 2024-06-10T00:29:37.872 INFO:tasks.workunit.client.0.smithi121.stderr:+ tar -xvf blogbench-1.0.tar.bz2 2024-06-10T00:29:37.952 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/ 2024-06-10T00:29:37.953 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/m4/ 2024-06-10T00:29:37.954 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-06-10T00:29:37.955 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/man/ 2024-06-10T00:29:37.956 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/man/blogbench.8 2024-06-10T00:29:37.957 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/man/Makefile.am 2024-06-10T00:29:37.958 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/man/Makefile.in 2024-06-10T00:29:37.959 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/ 2024-06-10T00:29:37.960 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/mysnprintf.c 2024-06-10T00:29:37.961 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/mysnprintf.h 2024-06-10T00:29:37.962 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/blogbench.c 2024-06-10T00:29:37.963 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/blogbench.h 2024-06-10T00:29:37.964 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/fakesnprintf.c 2024-06-10T00:29:37.965 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/fakesnprintf.h 2024-06-10T00:29:37.966 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/blogbench_p.h 2024-06-10T00:29:37.967 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-06-10T00:29:37.968 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-06-10T00:29:37.969 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/Makefile.am 2024-06-10T00:29:37.970 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/Makefile.in 2024-06-10T00:29:37.971 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/process.c 2024-06-10T00:29:37.972 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/globals.h 2024-06-10T00:29:37.973 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/commenter.c 2024-06-10T00:29:37.974 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/rewriter.c 2024-06-10T00:29:37.975 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/reader.c 2024-06-10T00:29:37.976 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/writer.c 2024-06-10T00:29:37.977 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/src/helpers.c 2024-06-10T00:29:37.978 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/NEWS 2024-06-10T00:29:37.979 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/compile 2024-06-10T00:29:37.980 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/depcomp 2024-06-10T00:29:37.981 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/aclocal.m4 2024-06-10T00:29:37.982 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/README 2024-06-10T00:29:37.983 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/configure 2024-06-10T00:29:37.987 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/configure.ac 2024-06-10T00:29:37.988 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/config.guess 2024-06-10T00:29:37.989 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/install-sh 2024-06-10T00:29:37.990 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/config.sub 2024-06-10T00:29:37.991 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/missing 2024-06-10T00:29:37.992 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/mkinstalldirs 2024-06-10T00:29:37.993 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/Makefile.am 2024-06-10T00:29:37.994 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/Makefile.in 2024-06-10T00:29:37.995 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/config.h.in 2024-06-10T00:29:37.996 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/AUTHORS 2024-06-10T00:29:37.997 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/INSTALL 2024-06-10T00:29:37.998 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/ChangeLog 2024-06-10T00:29:37.998 INFO:tasks.workunit.client.0.smithi121.stdout:blogbench-1.0/COPYING 2024-06-10T00:29:38.000 INFO:tasks.workunit.client.0.smithi121.stderr:+ cd blogbench-1.0/ 2024-06-10T00:29:38.000 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'making blogbench' 2024-06-10T00:29:38.000 INFO:tasks.workunit.client.0.smithi121.stdout:making blogbench 2024-06-10T00:29:38.001 INFO:tasks.workunit.client.0.smithi121.stderr:+ ./configure 2024-06-10T00:29:38.158 INFO:tasks.workunit.client.0.smithi121.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-06-10T00:29:38.688 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '25c0a77', total seconds elapsed: 38.7 2024-06-10T00:29:38.688 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.0 2024-06-10T00:29:38.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:29:38.779 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:38 smithi121 ceph-mon[32378]: pgmap v53: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-06-10T00:29:39.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:38 smithi135 ceph-mon[36424]: pgmap v53: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-06-10T00:29:39.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:38 smithi193 ceph-mon[36716]: pgmap v53: 129 pgs: 129 active+clean; 1.3 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-06-10T00:29:39.166 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether build environment is sane... yes 2024-06-10T00:29:39.174 INFO:tasks.workunit.client.0.smithi121.stdout:checking for gawk... gawk 2024-06-10T00:29:39.189 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether make sets $(MAKE)... yes 2024-06-10T00:29:39.198 INFO:tasks.workunit.client.0.smithi121.stdout:checking for gcc... gcc 2024-06-10T00:29:39.244 INFO:tasks.workunit.client.0.smithi121.stdout:checking for C compiler default output file name... a.out 2024-06-10T00:29:39.245 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether the C compiler works... yes 2024-06-10T00:29:39.248 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether we are cross compiling... no 2024-06-10T00:29:39.283 INFO:tasks.workunit.client.0.smithi121.stdout:checking for suffix of executables... 2024-06-10T00:29:39.318 INFO:tasks.workunit.client.0.smithi121.stdout:checking for suffix of object files... o 2024-06-10T00:29:39.359 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether we are using the GNU C compiler... yes 2024-06-10T00:29:39.392 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether gcc accepts -g... yes 2024-06-10T00:29:39.532 INFO:tasks.workunit.client.0.smithi121.stdout:checking for gcc option to accept ANSI C... none needed 2024-06-10T00:29:39.601 INFO:tasks.workunit.client.0.smithi121.stdout:checking for style of include used by make... GNU 2024-06-10T00:29:39.703 INFO:tasks.workunit.client.0.smithi121.stdout:checking dependency style of gcc... gcc3 2024-06-10T00:29:39.703 INFO:tasks.workunit.client.0.smithi121.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-06-10T00:29:39.799 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:39.840 INFO:tasks.workunit.client.0.smithi121.stdout:checking how to run the C preprocessor... gcc -E 2024-06-10T00:29:39.883 INFO:tasks.workunit.client.0.smithi121.stdout:checking for egrep... grep -E 2024-06-10T00:29:39.892 INFO:tasks.workunit.client.0.smithi121.stdout:checking for AIX... no 2024-06-10T00:29:39.958 INFO:tasks.workunit.client.0.smithi121.stdout:checking for library containing strerror... none required 2024-06-10T00:29:40.151 INFO:tasks.workunit.client.0.smithi121.stdout:checking for ANSI C header files... yes 2024-06-10T00:29:40.205 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-06-10T00:29:40.254 INFO:tasks.workunit.client.0.smithi121.stdout:checking for sys/types.h... yes 2024-06-10T00:29:40.312 INFO:tasks.workunit.client.0.smithi121.stdout:checking for sys/stat.h... yes 2024-06-10T00:29:40.362 INFO:tasks.workunit.client.0.smithi121.stdout:checking for stdlib.h... yes 2024-06-10T00:29:40.453 INFO:tasks.workunit.client.0.smithi121.stdout:checking for string.h... yes 2024-06-10T00:29:40.516 INFO:tasks.workunit.client.0.smithi121.stdout:checking for memory.h... yes 2024-06-10T00:29:40.566 INFO:tasks.workunit.client.0.smithi121.stdout:checking for strings.h... yes 2024-06-10T00:29:40.616 INFO:tasks.workunit.client.0.smithi121.stdout:checking for inttypes.h... yes 2024-06-10T00:29:40.666 INFO:tasks.workunit.client.0.smithi121.stdout:checking for stdint.h... yes 2024-06-10T00:29:40.717 INFO:tasks.workunit.client.0.smithi121.stdout:checking for unistd.h... yes 2024-06-10T00:29:40.729 INFO:tasks.workunit.client.0.smithi121.stdout:checking for memory.h... (cached) yes 2024-06-10T00:29:40.734 INFO:tasks.workunit.client.0.smithi121.stdout:checking for strings.h... (cached) yes 2024-06-10T00:29:40.739 INFO:tasks.workunit.client.0.smithi121.stdout:checking for unistd.h... (cached) yes 2024-06-10T00:29:40.797 INFO:tasks.workunit.client.0.smithi121.stdout:checking locale.h usability... yes 2024-06-10T00:29:40.817 INFO:tasks.workunit.client.0.smithi121.stdout:checking locale.h presence... yes 2024-06-10T00:29:40.818 INFO:tasks.workunit.client.0.smithi121.stdout:checking for locale.h... yes 2024-06-10T00:29:40.834 INFO:tasks.workunit.client.0.smithi121.stdout:checking for stdlib.h... (cached) yes 2024-06-10T00:29:40.839 INFO:tasks.workunit.client.0.smithi121.stdout:checking for string.h... (cached) yes 2024-06-10T00:29:40.919 INFO:tasks.workunit.client.0.smithi121.stdout:checking sys/socket.h usability... yes 2024-06-10T00:29:40.941 INFO:tasks.workunit.client.0.smithi121.stdout:checking sys/socket.h presence... yes 2024-06-10T00:29:40.942 INFO:tasks.workunit.client.0.smithi121.stdout:checking for sys/socket.h... yes 2024-06-10T00:29:40.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:40 smithi121 ceph-mon[32378]: pgmap v54: 129 pgs: 129 active+clean; 1.5 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-10T00:29:40.996 INFO:tasks.workunit.client.0.smithi121.stdout:checking sys/time.h usability... yes 2024-06-10T00:29:41.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:40 smithi135 ceph-mon[36424]: pgmap v54: 129 pgs: 129 active+clean; 1.5 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-10T00:29:41.017 INFO:tasks.workunit.client.0.smithi121.stdout:checking sys/time.h presence... yes 2024-06-10T00:29:41.018 INFO:tasks.workunit.client.0.smithi121.stdout:checking for sys/time.h... yes 2024-06-10T00:29:41.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.763+0000 7f4bf2946700 1 -- 172.21.15.121:0/99158987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec1013c0 msgr2=0x7f4bec101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:41.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.763+0000 7f4bf2946700 1 --2- 172.21.15.121:0/99158987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec1013c0 0x7f4bec101840 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009990 tx=0x7f4be0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.763+0000 7f4bf2946700 1 -- 172.21.15.121:0/99158987 shutdown_connections 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.763+0000 7f4bf2946700 1 --2- 172.21.15.121:0/99158987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bec101d80 0x7f4bec10e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.763+0000 7f4bf2946700 1 --2- 172.21.15.121:0/99158987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bec107430 0x7f4bec107810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.763+0000 7f4bf2946700 1 --2- 172.21.15.121:0/99158987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec1013c0 0x7f4bec101840 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.763+0000 7f4bf2946700 1 -- 172.21.15.121:0/99158987 >> 172.21.15.121:0/99158987 conn(0x7f4bec0783d0 msgr2=0x7f4bec0787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.763+0000 7f4bf2946700 1 -- 172.21.15.121:0/99158987 shutdown_connections 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.763+0000 7f4bf2946700 1 -- 172.21.15.121:0/99158987 wait complete. 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.764+0000 7f4bf2946700 1 Processor -- start 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.764+0000 7f4bf2946700 1 -- start start 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.764+0000 7f4bf2946700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec1013c0 0x7f4bec199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.764+0000 7f4bf2946700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bec101d80 0x7f4bec19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.764+0000 7f4bf2946700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bec107430 0x7f4bec1a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.764+0000 7f4bf2946700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1116a0 con 0x7f4bec1013c0 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.764+0000 7f4bf2946700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec111520 con 0x7f4bec101d80 2024-06-10T00:29:41.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.764+0000 7f4bf2946700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec111820 con 0x7f4bec107430 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.764+0000 7f4bebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec1013c0 0x7f4bec199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.764+0000 7f4bebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec1013c0 0x7f4bec199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39404/0 (socket says 172.21.15.121:39404) 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.764+0000 7f4bebfff700 1 -- 172.21.15.121:0/3885780324 learned_addr learned my addr 172.21.15.121:0/3885780324 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4beb7fe700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bec101d80 0x7f4bec19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4bf0ee3700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bec107430 0x7f4bec1a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4bebfff700 1 -- 172.21.15.121:0/3885780324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bec107430 msgr2=0x7f4bec1a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4bebfff700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bec107430 0x7f4bec1a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4bebfff700 1 -- 172.21.15.121:0/3885780324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bec101d80 msgr2=0x7f4bec19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4bebfff700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bec101d80 0x7f4bec19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4bebfff700 1 -- 172.21.15.121:0/3885780324 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bec1a50c0 con 0x7f4bec1013c0 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4bebfff700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec1013c0 0x7f4bec199b00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00e9a0 tx=0x7f4bdc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4be97fa700 1 -- 172.21.15.121:0/3885780324 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc00ca10 con 0x7f4bec1013c0 2024-06-10T00:29:41.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4bf2946700 1 -- 172.21.15.121:0/3885780324 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bec1a5290 con 0x7f4bec1013c0 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4be97fa700 1 -- 172.21.15.121:0/3885780324 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bdc00cbb0 con 0x7f4bec1013c0 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4be97fa700 1 -- 172.21.15.121:0/3885780324 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc018680 con 0x7f4bec1013c0 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.765+0000 7f4bf2946700 1 -- 172.21.15.121:0/3885780324 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bec1a57f0 con 0x7f4bec1013c0 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.767+0000 7f4be97fa700 1 -- 172.21.15.121:0/3885780324 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4bdc010430 con 0x7f4bec1013c0 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.768+0000 7f4be97fa700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4bd40a2d90 0x7f4bd40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.768+0000 7f4beb7fe700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4bd40a2d90 0x7f4bd40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.768+0000 7f4be97fa700 1 -- 172.21.15.121:0/3885780324 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f4bdc0cf9e0 con 0x7f4bec1013c0 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.768+0000 7f4beb7fe700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4bd40a2d90 0x7f4bd40a5250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4be000b2e0 tx=0x7f4be0005960 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.769+0000 7f4bfd1c9b80 1 Processor -- start 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.769+0000 7f4bfd1c9b80 1 -- start start 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.769+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c29a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.769+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 0x55e7f6c239e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.769+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c24340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.769+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e7f6b75810 con 0x55e7f6c29f60 2024-06-10T00:29:41.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.769+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55e7f6b1d720 con 0x55e7f6c23f20 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.769+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e7f6b717d0 con 0x55e7f6c29640 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.769+0000 7f4beb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 0x55e7f6c239e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.769+0000 7f4beb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 0x55e7f6c239e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39416/0 (socket says 172.21.15.121:39416) 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.769+0000 7f4beb7fe700 1 -- 172.21.15.121:0/2611942051 learned_addr learned my addr 172.21.15.121:0/2611942051 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4bf0ee3700 1 --2- 172.21.15.121:0/2611942051 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c24340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4bebfff700 1 --2- 172.21.15.121:0/2611942051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c29a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4bf0ee3700 1 -- 172.21.15.121:0/2611942051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 msgr2=0x55e7f6c29a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4bf0ee3700 1 --2- 172.21.15.121:0/2611942051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c29a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4bf0ee3700 1 -- 172.21.15.121:0/2611942051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 msgr2=0x55e7f6c239e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4bf0ee3700 1 --2- 172.21.15.121:0/2611942051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 0x55e7f6c239e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4bf0ee3700 1 -- 172.21.15.121:0/2611942051 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7f6b13f50 con 0x55e7f6c23f20 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4beb7fe700 1 --2- 172.21.15.121:0/2611942051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 0x55e7f6c239e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4bf0ee3700 1 --2- 172.21.15.121:0/2611942051 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c24340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4be4003f20 tx=0x7f4be4009ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc2782b8db53e6ca server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:41.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4bd27fc700 1 -- 172.21.15.121:0/2611942051 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4004bb0 con 0x55e7f6c23f20 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4bd27fc700 1 -- 172.21.15.121:0/2611942051 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4be4004d50 con 0x55e7f6c23f20 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.770+0000 7f4bd27fc700 1 -- 172.21.15.121:0/2611942051 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4007d20 con 0x55e7f6c23f20 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.771+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/2611942051 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 msgr2=0x55e7f6c24340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.771+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/2611942051 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c24340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4be4003f20 tx=0x7f4be4009ba0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.771+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/2611942051 shutdown_connections 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.771+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/2611942051 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c24340 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.771+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/2611942051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c29a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.771+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/2611942051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 0x55e7f6c239e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.771+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/2611942051 >> 172.21.15.121:0/2611942051 conn(0x55e7f6b4ecc0 msgr2=0x55e7f6c1e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.771+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/2611942051 shutdown_connections 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.771+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/2611942051 wait complete. 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bfd1c9b80 1 -- start start 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bfd1c9b80 1 Processor -- start 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c2fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.045 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29640 0x55e7f6c2cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29f60 0x55e7f6c2d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e7f6c309d0 con 0x55e7f6c29640 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55e7f6b15bf0 con 0x55e7f6c23f20 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e7f6c30b50 con 0x55e7f6c29f60 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bfd1c9b80 10 client.0 _ll_register_callbacks cb 0x55e7f6b0f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bebfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c2fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4beb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29640 0x55e7f6c2cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bebfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c2fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:57520/0 (socket says 172.21.15.121:57520) 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.772+0000 7f4bebfff700 1 -- 172.21.15.121:0/3626997589 learned_addr learned my addr 172.21.15.121:0/3626997589 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bebfff700 1 -- 172.21.15.121:0/3626997589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29f60 msgr2=0x55e7f6c2d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bebfff700 1 --2- 172.21.15.121:0/3626997589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29f60 0x55e7f6c2d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bebfff700 1 -- 172.21.15.121:0/3626997589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29640 msgr2=0x55e7f6c2cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bebfff700 1 --2- 172.21.15.121:0/3626997589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29640 0x55e7f6c2cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bebfff700 1 -- 172.21.15.121:0/3626997589 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7f6ac48d0 con 0x55e7f6c23f20 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bebfff700 1 --2- 172.21.15.121:0/3626997589 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c2fcd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc097f70 tx=0x7f4bdc09c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e9b5fed3b0158bc server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bcb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/3626997589 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc0a42f0 con 0x55e7f6c23f20 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/3626997589 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f4bdc0a4490 con 0x55e7f6c23f20 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/3626997589 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4be4004d50 con 0x55e7f6c23f20 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/3626997589 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4bdc010720 con 0x55e7f6c23f20 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bfd1c9b80 10 client.25352 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/3626997589 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e7f6c3b290 con 0x55e7f6c23f20 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/3626997589 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bdc065020 con 0x55e7f6c23f20 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.773+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/3626997589 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f4bdc09a9d0 con 0x55e7f6c23f20 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.774+0000 7f4bfd1c9b80 10 client.25352 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.774+0000 7f4bfd1c9b80 10 client.25352 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.774+0000 7f4bfd1c9b80 20 client.25352 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.774+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/3626997589 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e7f6c3d820 0x55e7f6c3fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.774+0000 7f4bfd1c9b80 4 client.25352 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.774+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/3626997589 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7f6b82d10 con 0x55e7f6c3d820 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.774+0000 7f4beb7fe700 1 --2- 172.21.15.121:0/3626997589 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e7f6c3d820 0x55e7f6c3fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.774+0000 7f4bcb7fe700 10 client.25352 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.774+0000 7f4beb7fe700 1 --2- 172.21.15.121:0/3626997589 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e7f6c3d820 0x55e7f6c3fc10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c37a23480160edc server_cookie=e5f089220db7c7d3 in_seq=0 out_seq=0 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/3626997589 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55e7f6b82d10 con 0x55e7f6c3d820 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bcb7fe700 10 client.25352 handle_command_reply: tid=0 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 1 client.25352 shutdown 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/3626997589 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 msgr2=0x55e7f6c2fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/3626997589 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c2fcd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc097f70 tx=0x7f4bdc09c350 comp rx=0 tx=0).stop 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/3626997589 shutdown_connections 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/3626997589 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e7f6c3d820 0x55e7f6c3fc10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/3626997589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29f60 0x55e7f6c2d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bcb7fe700 0 client.25352 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:41.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/3626997589 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c2fcd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/3626997589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29640 0x55e7f6c2cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/3626997589 >> 172.21.15.121:0/3626997589 conn(0x55e7f6b4ecc0 msgr2=0x55e7f6c229d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/3626997589 shutdown_connections 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/3626997589 wait complete. 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.775+0000 7f4bfd1c9b80 20 client.25352 trim_cache size 0 max 16384 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bfd1c9b80 1 Processor -- start 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bfd1c9b80 1 -- start start 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c239e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c2d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 0x55e7f6c2bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1b0fa0 con 0x55e7f6c29f60 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1ae830 con 0x55e7f6c23f20 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1ac680 con 0x55e7f6c29640 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bf0ee3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 0x55e7f6c2bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bebfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c239e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bebfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c239e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:57528/0 (socket says 172.21.15.121:57528) 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bebfff700 1 -- 172.21.15.121:0/1749195050 learned_addr learned my addr 172.21.15.121:0/1749195050 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4beb7fe700 1 --2- 172.21.15.121:0/1749195050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c2d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bebfff700 1 -- 172.21.15.121:0/1749195050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 msgr2=0x55e7f6c2d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bebfff700 1 --2- 172.21.15.121:0/1749195050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c2d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bebfff700 1 -- 172.21.15.121:0/1749195050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 msgr2=0x55e7f6c2bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bebfff700 1 --2- 172.21.15.121:0/1749195050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 0x55e7f6c2bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.797+0000 7f4bebfff700 1 -- 172.21.15.121:0/1749195050 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7f6c2b540 con 0x55e7f6c23f20 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.798+0000 7f4bebfff700 1 --2- 172.21.15.121:0/1749195050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c239e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc000c00 tx=0x7f4bdc010ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9d0c3c36a8962fc server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.798+0000 7f4bd1ffb700 1 -- 172.21.15.121:0/1749195050 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc09a9d0 con 0x55e7f6c23f20 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.798+0000 7f4bd1ffb700 1 -- 172.21.15.121:0/1749195050 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bdc065030 con 0x55e7f6c23f20 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.798+0000 7f4bd1ffb700 1 -- 172.21.15.121:0/1749195050 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc0a42f0 con 0x55e7f6c23f20 2024-06-10T00:29:41.050 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.799+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/1749195050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 msgr2=0x55e7f6c239e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.799+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/1749195050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c239e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc000c00 tx=0x7f4bdc010ad0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.799+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/1749195050 shutdown_connections 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.799+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/1749195050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c29f60 0x55e7f6c2bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.799+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/1749195050 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c23f20 0x55e7f6c239e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.799+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/1749195050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c2d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.799+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/1749195050 >> 172.21.15.121:0/1749195050 conn(0x55e7f6b4ecc0 msgr2=0x55e7f6c2a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.799+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/1749195050 shutdown_connections 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.799+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/1749195050 wait complete. 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.799+0000 7f4bfd1c9b80 1 -- start start 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.799+0000 7f4bfd1c9b80 1 Processor -- start 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c23f20 0x55e7f6c36090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c365d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bfd1c9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c29f60 0x55e7f6c32e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1accf0 con 0x55e7f6c23f20 2024-06-10T00:29:41.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1ae160 con 0x55e7f6c29f60 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bfd1c9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec1ad3c0 con 0x55e7f6c29640 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bfd1c9b80 10 client.0 _ll_register_callbacks cb 0x55e7f6b0f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c23f20 0x55e7f6c36090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c23f20 0x55e7f6c36090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39454/0 (socket says 172.21.15.121:39454) 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bebfff700 1 -- 172.21.15.121:0/54865450 learned_addr learned my addr 172.21.15.121:0/54865450 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bf0ee3700 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c29f60 0x55e7f6c32e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4beb7fe700 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bebfff700 1 -- 172.21.15.121:0/54865450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 msgr2=0x55e7f6c365d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bebfff700 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c365d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bebfff700 1 -- 172.21.15.121:0/54865450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c29f60 msgr2=0x55e7f6c32e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bebfff700 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c29f60 0x55e7f6c32e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bebfff700 1 -- 172.21.15.121:0/54865450 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7f6c33650 con 0x55e7f6c23f20 2024-06-10T00:29:41.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bf0ee3700 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c29f60 0x55e7f6c32e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4beb7fe700 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bebfff700 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c23f20 0x55e7f6c36090 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc09aef0 tx=0x7f4bdc09c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ba4f46f26e0294 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bcb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/54865450 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc09b650 con 0x55e7f6c23f20 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/54865450 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f4bdc0a42f0 con 0x55e7f6c23f20 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.800+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/54865450 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bdc065030 con 0x55e7f6c23f20 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/54865450 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4bdc0a44f0 con 0x55e7f6c23f20 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4bfd1c9b80 10 client.15753 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/54865450 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e7f6ca8ff0 con 0x55e7f6c23f20 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/54865450 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bdc065030 con 0x55e7f6c23f20 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/54865450 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7f4bdc0ba1b0 con 0x55e7f6c23f20 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4bfd1c9b80 10 client.15753 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4bfd1c9b80 10 client.15753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4bfd1c9b80 20 client.15753 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e7f6ca9310 0x55e7f6cab700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:41.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4bfd1c9b80 4 client.15753 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/54865450 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e7f6b7d290 con 0x55e7f6ca9310 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.801+0000 7f4beb7fe700 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e7f6ca9310 0x55e7f6cab700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bcb7fe700 10 client.15753 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4beb7fe700 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e7f6ca9310 0x55e7f6cab700 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19425c9d4293b997 server_cookie=6f417b25aaf383b4 in_seq=0 out_seq=0 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bcb7fe700 1 -- 172.21.15.121:0/54865450 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e7f6b7d290 con 0x55e7f6ca9310 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bcb7fe700 10 client.15753 handle_command_reply: tid=0 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bfd1c9b80 1 client.15753 shutdown 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/54865450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c23f20 msgr2=0x55e7f6c36090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c23f20 0x55e7f6c36090 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc09aef0 tx=0x7f4bdc09c350 comp rx=0 tx=0).stop 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/54865450 shutdown_connections 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e7f6ca9310 0x55e7f6cab700 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e7f6c29f60 0x55e7f6c32e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bcb7fe700 0 client.15753 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e7f6c23f20 0x55e7f6c36090 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bfd1c9b80 1 --2- 172.21.15.121:0/54865450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7f6c29640 0x55e7f6c365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/54865450 >> 172.21.15.121:0/54865450 conn(0x55e7f6b4ecc0 msgr2=0x55e7f6c28140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/54865450 shutdown_connections 2024-06-10T00:29:41.055 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:29:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "no active scrubs running", 2024-06-10T00:29:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": {} 2024-06-10T00:29:41.055 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:29:41.055 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.802+0000 7f4bfd1c9b80 1 -- 172.21.15.121:0/54865450 wait complete. 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bfd1c9b80 20 client.15753 trim_cache size 0 max 16384 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 -- 172.21.15.121:0/3885780324 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4bd40a2d90 msgr2=0x7f4bd40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4bd40a2d90 0x7f4bd40a5250 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4be000b2e0 tx=0x7f4be0005960 comp rx=0 tx=0).stop 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 -- 172.21.15.121:0/3885780324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec1013c0 msgr2=0x7f4bec199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec1013c0 0x7f4bec199b00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00e9a0 tx=0x7f4bdc00ecb0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 -- 172.21.15.121:0/3885780324 shutdown_connections 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4bd40a2d90 0x7f4bd40a5250 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bec107430 0x7f4bec1a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec1013c0 0x7f4bec199b00 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 --2- 172.21.15.121:0/3885780324 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4bec101d80 0x7f4bec19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 -- 172.21.15.121:0/3885780324 >> 172.21.15.121:0/3885780324 conn(0x7f4bec0783d0 msgr2=0x7f4bec1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 -- 172.21.15.121:0/3885780324 shutdown_connections 2024-06-10T00:29:41.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:40.803+0000 7f4bf2946700 1 -- 172.21.15.121:0/3885780324 wait complete. 2024-06-10T00:29:41.079 INFO:tasks.workunit.client.0.smithi121.stdout:checking fcntl.h usability... yes 2024-06-10T00:29:41.099 INFO:tasks.workunit.client.0.smithi121.stdout:checking fcntl.h presence... yes 2024-06-10T00:29:41.100 INFO:tasks.workunit.client.0.smithi121.stdout:checking for fcntl.h... yes 2024-06-10T00:29:41.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:40 smithi193 ceph-mon[36716]: pgmap v54: 129 pgs: 129 active+clean; 1.5 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-06-10T00:29:41.180 INFO:tasks.workunit.client.0.smithi121.stdout:checking sys/fcntl.h usability... yes 2024-06-10T00:29:41.201 INFO:tasks.workunit.client.0.smithi121.stdout:checking sys/fcntl.h presence... yes 2024-06-10T00:29:41.202 INFO:tasks.workunit.client.0.smithi121.stdout:checking for sys/fcntl.h... yes 2024-06-10T00:29:41.365 INFO:tasks.workunit.client.0.smithi121.stdout:checking sys/param.h usability... yes 2024-06-10T00:29:41.388 INFO:tasks.workunit.client.0.smithi121.stdout:checking sys/param.h presence... yes 2024-06-10T00:29:41.389 INFO:tasks.workunit.client.0.smithi121.stdout:checking for sys/param.h... yes 2024-06-10T00:29:41.439 INFO:tasks.workunit.client.0.smithi121.stdout:checking ioctl.h usability... no 2024-06-10T00:29:41.460 INFO:tasks.workunit.client.0.smithi121.stdout:checking ioctl.h presence... no 2024-06-10T00:29:41.460 INFO:tasks.workunit.client.0.smithi121.stdout:checking for ioctl.h... no 2024-06-10T00:29:41.512 INFO:tasks.workunit.client.0.smithi121.stdout:checking sys/ioctl.h usability... yes 2024-06-10T00:29:41.532 INFO:tasks.workunit.client.0.smithi121.stdout:checking sys/ioctl.h presence... yes 2024-06-10T00:29:41.533 INFO:tasks.workunit.client.0.smithi121.stdout:checking for sys/ioctl.h... yes 2024-06-10T00:29:41.585 INFO:tasks.workunit.client.0.smithi121.stdout:checking return type of signal handlers... void 2024-06-10T00:29:41.631 INFO:tasks.workunit.client.0.smithi121.stdout:checking for size_t... yes 2024-06-10T00:29:41.673 INFO:tasks.workunit.client.0.smithi121.stdout:checking for dev_t... yes 2024-06-10T00:29:41.717 INFO:tasks.workunit.client.0.smithi121.stdout:checking for short... yes 2024-06-10T00:29:41.800 INFO:tasks.workunit.client.0.smithi121.stdout:checking size of short... 2 2024-06-10T00:29:41.847 INFO:tasks.workunit.client.0.smithi121.stdout:checking for int... yes 2024-06-10T00:29:41.904 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-10T00:29:41.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:29:41.920 INFO:tasks.workunit.client.0.smithi121.stdout:checking size of int... 4 2024-06-10T00:29:41.968 INFO:tasks.workunit.client.0.smithi121.stdout:checking for long... yes 2024-06-10T00:29:42.045 INFO:tasks.workunit.client.0.smithi121.stdout:checking size of long... 8 2024-06-10T00:29:42.098 INFO:tasks.workunit.client.0.smithi121.stdout:checking for long long... yes 2024-06-10T00:29:42.169 INFO:tasks.workunit.client.0.smithi121.stdout:checking size of long long... 8 2024-06-10T00:29:42.212 INFO:tasks.workunit.client.0.smithi121.stdout:checking for ssize_t... yes 2024-06-10T00:29:42.252 INFO:tasks.workunit.client.0.smithi121.stdout:checking for sig_atomic_t... yes 2024-06-10T00:29:42.389 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether gcc needs -traditional... no 2024-06-10T00:29:42.423 INFO:tasks.workunit.client.0.smithi121.stdout:checking for an ANSI C-conforming const... yes 2024-06-10T00:29:42.450 INFO:tasks.workunit.client.0.smithi121.stdout:checking for inline... inline 2024-06-10T00:29:42.509 INFO:tasks.workunit.client.0.smithi121.stdout:checking for working alloca.h... yes 2024-06-10T00:29:42.555 INFO:tasks.workunit.client.0.smithi121.stdout:checking for alloca... yes 2024-06-10T00:29:42.624 INFO:tasks.workunit.client.0.smithi121.stdout:checking for working memcmp... yes 2024-06-10T00:29:42.708 INFO:tasks.workunit.client.0.smithi121.stdout:checking for strftime... yes 2024-06-10T00:29:42.792 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-06-10T00:29:42.904 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether stat accepts an empty string... no 2024-06-10T00:29:42.936 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:42 smithi121 ceph-mon[32378]: pgmap v55: 129 pgs: 129 active+clean; 1.5 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 11 KiB/s wr, 5 op/s 2024-06-10T00:29:42.971 INFO:tasks.workunit.client.0.smithi121.stdout:checking for vprintf... yes 2024-06-10T00:29:43.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:42 smithi135 ceph-mon[36424]: pgmap v55: 129 pgs: 129 active+clean; 1.5 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 11 KiB/s wr, 5 op/s 2024-06-10T00:29:43.054 INFO:tasks.workunit.client.0.smithi121.stdout:checking for _doprnt... no 2024-06-10T00:29:43.076 INFO:tasks.workunit.client.0.smithi121.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-06-10T00:29:43.088 INFO:tasks.workunit.client.0.smithi121.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-06-10T00:29:43.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:42 smithi193 ceph-mon[36716]: pgmap v55: 129 pgs: 129 active+clean; 1.5 MiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 11 KiB/s wr, 5 op/s 2024-06-10T00:29:43.174 INFO:tasks.workunit.client.0.smithi121.stdout:checking for the pthreads library -lpthreads... no 2024-06-10T00:29:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.186+0000 7fa944a45700 1 -- 172.21.15.121:0/2080915332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa940101380 msgr2=0x7fa940101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.186+0000 7fa944a45700 1 --2- 172.21.15.121:0/2080915332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa940101380 0x7fa940101800 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa934009990 tx=0x7fa934009ca0 comp rx=0 tx=0).stop 2024-06-10T00:29:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.186+0000 7fa944a45700 1 -- 172.21.15.121:0/2080915332 shutdown_connections 2024-06-10T00:29:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.186+0000 7fa944a45700 1 --2- 172.21.15.121:0/2080915332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa940101d40 0x7fa94010e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.186+0000 7fa944a45700 1 --2- 172.21.15.121:0/2080915332 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9401073f0 0x7fa9401077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:43.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.186+0000 7fa944a45700 1 --2- 172.21.15.121:0/2080915332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa940101380 0x7fa940101800 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:43.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.186+0000 7fa944a45700 1 -- 172.21.15.121:0/2080915332 >> 172.21.15.121:0/2080915332 conn(0x7fa940078300 msgr2=0x7fa940078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:43.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.186+0000 7fa944a45700 1 -- 172.21.15.121:0/2080915332 shutdown_connections 2024-06-10T00:29:43.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.186+0000 7fa944a45700 1 -- 172.21.15.121:0/2080915332 wait complete. 2024-06-10T00:29:43.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.187+0000 7fa944a45700 1 Processor -- start 2024-06-10T00:29:43.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.187+0000 7fa944a45700 1 -- start start 2024-06-10T00:29:43.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.187+0000 7fa944a45700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa940101380 0x7fa94019d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:43.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.187+0000 7fa944a45700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa940101d40 0x7fa94019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:43.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa944a45700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9401073f0 0x7fa9401979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:43.188 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa944a45700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa940111690 con 0x7fa940101d40 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa944a45700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fa940111510 con 0x7fa9401073f0 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa944a45700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa940111810 con 0x7fa940101380 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93dd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa940101d40 0x7fa94019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93dd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa940101d40 0x7fa94019dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39470/0 (socket says 172.21.15.121:39470) 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93dd9b700 1 -- 172.21.15.121:0/3451451365 learned_addr learned my addr 172.21.15.121:0/3451451365 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93e59c700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa940101380 0x7fa94019d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93ed9d700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9401073f0 0x7fa9401979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93dd9b700 1 -- 172.21.15.121:0/3451451365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa940101380 msgr2=0x7fa94019d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93dd9b700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa940101380 0x7fa94019d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93dd9b700 1 -- 172.21.15.121:0/3451451365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9401073f0 msgr2=0x7fa9401979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93dd9b700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9401073f0 0x7fa9401979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93dd9b700 1 -- 172.21.15.121:0/3451451365 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa940198130 con 0x7fa940101d40 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93e59c700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa940101380 0x7fa94019d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:43.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.188+0000 7fa93dd9b700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa940101d40 0x7fa94019dd50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa93400b5c0 tx=0x7fa9340047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.189+0000 7fa92f7fe700 1 -- 172.21.15.121:0/3451451365 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa934027070 con 0x7fa940101d40 2024-06-10T00:29:43.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.189+0000 7fa92f7fe700 1 -- 172.21.15.121:0/3451451365 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9340040b0 con 0x7fa940101d40 2024-06-10T00:29:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.189+0000 7fa944a45700 1 -- 172.21.15.121:0/3451451365 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9401983c0 con 0x7fa940101d40 2024-06-10T00:29:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.189+0000 7fa944a45700 1 -- 172.21.15.121:0/3451451365 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9401987e0 con 0x7fa940101d40 2024-06-10T00:29:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.190+0000 7fa92f7fe700 1 -- 172.21.15.121:0/3451451365 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93400f460 con 0x7fa940101d40 2024-06-10T00:29:43.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.190+0000 7fa944a45700 1 -- 172.21.15.121:0/3451451365 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa920005330 con 0x7fa940101d40 2024-06-10T00:29:43.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.192+0000 7fa92f7fe700 1 -- 172.21.15.121:0/3451451365 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa93402a070 con 0x7fa940101d40 2024-06-10T00:29:43.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.193+0000 7fa92f7fe700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9280a2d90 0x7fa9280a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:43.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.193+0000 7fa93e59c700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9280a2d90 0x7fa9280a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:43.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.193+0000 7fa92f7fe700 1 -- 172.21.15.121:0/3451451365 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fa9340d0be0 con 0x7fa940101d40 2024-06-10T00:29:43.194 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.194+0000 7fa93e59c700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9280a2d90 0x7fa9280a5250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa930005e10 tx=0x7fa930005d60 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:43.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.196+0000 7fa92f7fe700 1 -- 172.21.15.121:0/3451451365 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa934031070 con 0x7fa940101d40 2024-06-10T00:29:43.233 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether pthreads work without any flags... yes 2024-06-10T00:29:43.233 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:43.280 INFO:tasks.workunit.client.0.smithi121.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-06-10T00:29:43.281 INFO:tasks.workunit.client.0.smithi121.stdout:checking if more special flags are required for pthreads... no 2024-06-10T00:29:43.281 INFO:tasks.workunit.client.0.smithi121.stdout:checking for cc_r... gcc 2024-06-10T00:29:43.360 INFO:tasks.workunit.client.0.smithi121.stdout:checking for setrlimit... yes 2024-06-10T00:29:43.425 INFO:tasks.workunit.client.0.smithi121.stdout:checking for setproctitle... no 2024-06-10T00:29:43.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.466+0000 7fa944a45700 1 -- 172.21.15.121:0/3451451365 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa920005620 con 0x7fa940101d40 2024-06-10T00:29:43.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.467+0000 7fa92f7fe700 1 -- 172.21.15.121:0/3451451365 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7fa934013070 con 0x7fa940101d40 2024-06-10T00:29:43.467 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:29:43.467 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:27:55.370804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:29:43.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.470+0000 7fa944a45700 1 -- 172.21.15.121:0/3451451365 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9280a2d90 msgr2=0x7fa9280a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:43.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.470+0000 7fa944a45700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9280a2d90 0x7fa9280a5250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa930005e10 tx=0x7fa930005d60 comp rx=0 tx=0).stop 2024-06-10T00:29:43.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.470+0000 7fa944a45700 1 -- 172.21.15.121:0/3451451365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa940101d40 msgr2=0x7fa94019dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:43.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.470+0000 7fa944a45700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa940101d40 0x7fa94019dd50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa93400b5c0 tx=0x7fa9340047b0 comp rx=0 tx=0).stop 2024-06-10T00:29:43.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.470+0000 7fa944a45700 1 -- 172.21.15.121:0/3451451365 shutdown_connections 2024-06-10T00:29:43.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.471+0000 7fa944a45700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9280a2d90 0x7fa9280a5250 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:43.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.471+0000 7fa944a45700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa9401073f0 0x7fa9401979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:43.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.471+0000 7fa944a45700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa940101380 0x7fa94019d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:43.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.471+0000 7fa944a45700 1 --2- 172.21.15.121:0/3451451365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa940101d40 0x7fa94019dd50 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:43.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.471+0000 7fa944a45700 1 -- 172.21.15.121:0/3451451365 >> 172.21.15.121:0/3451451365 conn(0x7fa940078300 msgr2=0x7fa9400fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:43.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.471+0000 7fa944a45700 1 -- 172.21.15.121:0/3451451365 shutdown_connections 2024-06-10T00:29:43.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:43.471+0000 7fa944a45700 1 -- 172.21.15.121:0/3451451365 wait complete. 2024-06-10T00:29:43.474 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-06-10T00:29:43.494 INFO:tasks.workunit.client.0.smithi121.stdout:checking for getopt_long... yes 2024-06-10T00:29:43.556 INFO:tasks.workunit.client.0.smithi121.stdout:checking for strtoull... yes 2024-06-10T00:29:43.622 INFO:tasks.workunit.client.0.smithi121.stdout:checking for strtoq... yes 2024-06-10T00:29:43.683 INFO:tasks.workunit.client.0.smithi121.stdout:checking for snprintf... yes 2024-06-10T00:29:43.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:43 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3451451365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:43.767 INFO:tasks.workunit.client.0.smithi121.stdout:checking for vsnprintf... yes 2024-06-10T00:29:43.851 INFO:tasks.workunit.client.0.smithi121.stdout:checking for vfprintf... yes 2024-06-10T00:29:43.913 INFO:tasks.workunit.client.0.smithi121.stdout:checking for gethostname... yes 2024-06-10T00:29:43.975 INFO:tasks.workunit.client.0.smithi121.stdout:checking for setlocale... yes 2024-06-10T00:29:44.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:43 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3451451365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:44.035 INFO:tasks.workunit.client.0.smithi121.stdout:checking for sync... yes 2024-06-10T00:29:44.100 INFO:tasks.workunit.client.0.smithi121.stdout:checking whether snprintf is C99 conformant... done 2024-06-10T00:29:44.111 INFO:tasks.workunit.client.0.smithi121.stdout:configure: creating ./config.status 2024-06-10T00:29:44.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:43 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3451451365' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:44.262 INFO:tasks.workunit.client.0.smithi121.stdout:config.status: creating Makefile 2024-06-10T00:29:44.278 INFO:tasks.workunit.client.0.smithi121.stdout:config.status: creating src/Makefile 2024-06-10T00:29:44.297 INFO:tasks.workunit.client.0.smithi121.stdout:config.status: creating man/Makefile 2024-06-10T00:29:44.309 INFO:tasks.workunit.client.0.smithi121.stdout:config.status: creating config.h 2024-06-10T00:29:44.359 INFO:tasks.workunit.client.0.smithi121.stdout:config.status: executing depfiles commands 2024-06-10T00:29:44.430 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0' 2024-06-10T00:29:44.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db / --timeout 90 --expiration 150.0 --await-for 45.0 2024-06-10T00:29:44.492 INFO:tasks.workunit.client.0.smithi121.stderr:+ make 2024-06-10T00:29:44.505 INFO:tasks.workunit.client.0.smithi121.stdout:make all-recursive 2024-06-10T00:29:44.507 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-06-10T00:29:44.509 INFO:tasks.workunit.client.0.smithi121.stdout:Making all in src 2024-06-10T00:29:44.515 INFO:tasks.workunit.client.0.smithi121.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-06-10T00:29:44.516 INFO:tasks.workunit.client.0.smithi121.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-06-10T00:29:44.516 INFO:tasks.workunit.client.0.smithi121.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-06-10T00:29:44.638 INFO:tasks.workunit.client.0.smithi121.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-06-10T00:29:44.639 INFO:tasks.workunit.client.0.smithi121.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-06-10T00:29:44.708 INFO:tasks.workunit.client.0.smithi121.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-06-10T00:29:44.709 INFO:tasks.workunit.client.0.smithi121.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-06-10T00:29:44.761 INFO:tasks.workunit.client.0.smithi121.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-06-10T00:29:44.761 INFO:tasks.workunit.client.0.smithi121.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-06-10T00:29:44.813 INFO:tasks.workunit.client.0.smithi121.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-06-10T00:29:44.813 INFO:tasks.workunit.client.0.smithi121.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-06-10T00:29:44.864 INFO:tasks.workunit.client.0.smithi121.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-06-10T00:29:44.864 INFO:tasks.workunit.client.0.smithi121.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-06-10T00:29:44.928 INFO:tasks.workunit.client.0.smithi121.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-06-10T00:29:44.928 INFO:tasks.workunit.client.0.smithi121.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-06-10T00:29:44.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:44 smithi121 ceph-mon[32378]: pgmap v56: 129 pgs: 129 active+clean; 4.6 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 287 KiB/s wr, 29 op/s 2024-06-10T00:29:45.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:44 smithi135 ceph-mon[36424]: pgmap v56: 129 pgs: 129 active+clean; 4.6 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 287 KiB/s wr, 29 op/s 2024-06-10T00:29:45.030 INFO:tasks.workunit.client.0.smithi121.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-06-10T00:29:45.030 INFO:tasks.workunit.client.0.smithi121.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-06-10T00:29:45.066 INFO:tasks.workunit.client.0.smithi121.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-06-10T00:29:45.066 INFO:tasks.workunit.client.0.smithi121.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-06-10T00:29:45.112 INFO:tasks.workunit.client.0.smithi121.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-06-10T00:29:45.112 INFO:tasks.workunit.client.0.smithi121.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-06-10T00:29:45.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:44 smithi193 ceph-mon[36716]: pgmap v56: 129 pgs: 129 active+clean; 4.6 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 287 KiB/s wr, 29 op/s 2024-06-10T00:29:45.155 INFO:tasks.workunit.client.0.smithi121.stdout:gcc -g -O2 -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-06-10T00:29:45.174 INFO:tasks.workunit.client.0.smithi121.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-06-10T00:29:45.174 INFO:tasks.workunit.client.0.smithi121.stdout:Making all in man 2024-06-10T00:29:45.201 INFO:tasks.workunit.client.0.smithi121.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-06-10T00:29:45.201 INFO:tasks.workunit.client.0.smithi121.stdout:make[2]: Nothing to be done for 'all'. 2024-06-10T00:29:45.201 INFO:tasks.workunit.client.0.smithi121.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-06-10T00:29:45.202 INFO:tasks.workunit.client.0.smithi121.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-06-10T00:29:45.203 INFO:tasks.workunit.client.0.smithi121.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-06-10T00:29:45.203 INFO:tasks.workunit.client.0.smithi121.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-06-10T00:29:45.204 INFO:tasks.workunit.client.0.smithi121.stderr:+ cd src 2024-06-10T00:29:45.204 INFO:tasks.workunit.client.0.smithi121.stderr:+ mkdir blogtest_in 2024-06-10T00:29:45.206 INFO:tasks.workunit.client.0.smithi121.stdout:running blogbench 2024-06-10T00:29:45.206 INFO:tasks.workunit.client.0.smithi121.stderr:+ echo 'running blogbench' 2024-06-10T00:29:45.206 INFO:tasks.workunit.client.0.smithi121.stderr:+ ./blogbench -d blogtest_in 2024-06-10T00:29:45.207 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-06-10T00:29:45.207 INFO:tasks.workunit.client.0.smithi121.stdout:Frequency = 10 secs 2024-06-10T00:29:45.207 INFO:tasks.workunit.client.0.smithi121.stdout:Scratch dir = [blogtest_in] 2024-06-10T00:29:46.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:46 smithi121 ceph-mon[32378]: pgmap v57: 129 pgs: 129 active+clean; 4.6 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 287 KiB/s wr, 26 op/s 2024-06-10T00:29:47.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:46 smithi135 ceph-mon[36424]: pgmap v57: 129 pgs: 129 active+clean; 4.6 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 287 KiB/s wr, 26 op/s 2024-06-10T00:29:47.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:46 smithi193 ceph-mon[36716]: pgmap v57: 129 pgs: 129 active+clean; 4.6 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 287 KiB/s wr, 26 op/s 2024-06-10T00:29:47.289 INFO:tasks.workunit.client.0.smithi121.stdout:Spawning 3 writers... 2024-06-10T00:29:47.290 INFO:tasks.workunit.client.0.smithi121.stdout:Spawning 1 rewriters... 2024-06-10T00:29:47.290 INFO:tasks.workunit.client.0.smithi121.stdout:Spawning 5 commenters... 2024-06-10T00:29:47.290 INFO:tasks.workunit.client.0.smithi121.stdout:Spawning 100 readers... 2024-06-10T00:29:47.291 INFO:tasks.workunit.client.0.smithi121.stdout:Benchmarking for 30 iterations. 2024-06-10T00:29:47.292 INFO:tasks.workunit.client.0.smithi121.stdout:The test will run during 5 minutes. 2024-06-10T00:29:47.292 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-06-10T00:29:47.292 INFO:tasks.workunit.client.0.smithi121.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-06-10T00:29:47.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.894+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2787412703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c48108500 msgr2=0x7f8c481088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:47.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.894+0000 7f8c4fc53700 1 --2- 172.21.15.121:0/2787412703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c48108500 0x7f8c481088e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8c44009990 tx=0x7f8c44009ca0 comp rx=0 tx=0).stop 2024-06-10T00:29:47.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.894+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2787412703 shutdown_connections 2024-06-10T00:29:47.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.894+0000 7f8c4fc53700 1 --2- 172.21.15.121:0/2787412703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c4806ab30 0x7f8c4810f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:47.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.894+0000 7f8c4fc53700 1 --2- 172.21.15.121:0/2787412703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c48108500 0x7f8c481088e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:47.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.894+0000 7f8c4fc53700 1 --2- 172.21.15.121:0/2787412703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c4806a170 0x7f8c4806a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:47.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.894+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2787412703 >> 172.21.15.121:0/2787412703 conn(0x7f8c48078340 msgr2=0x7f8c48078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.894+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2787412703 shutdown_connections 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.894+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2787412703 wait complete. 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.894+0000 7f8c4fc53700 1 Processor -- start 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.894+0000 7f8c4fc53700 1 -- start start 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4fc53700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c4806a170 0x7f8c4819e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4fc53700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4806ab30 0x7f8c4819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4fc53700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c48108500 0x7f8c48198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4fc53700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8c48112780 con 0x7f8c4806ab30 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4fc53700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8c48112600 con 0x7f8c48108500 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4fc53700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8c48112900 con 0x7f8c4806a170 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4d1ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4806ab30 0x7f8c4819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4d9ef700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c4806a170 0x7f8c4819e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:47.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4d1ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4806ab30 0x7f8c4819ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39480/0 (socket says 172.21.15.121:39480) 2024-06-10T00:29:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4e1f0700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c48108500 0x7f8c48198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4d1ee700 1 -- 172.21.15.121:0/2591505160 learned_addr learned my addr 172.21.15.121:0/2591505160 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4d1ee700 1 -- 172.21.15.121:0/2591505160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c4806a170 msgr2=0x7f8c4819e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4d1ee700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c4806a170 0x7f8c4819e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4d1ee700 1 -- 172.21.15.121:0/2591505160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c48108500 msgr2=0x7f8c48198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4d1ee700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c48108500 0x7f8c48198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4d1ee700 1 -- 172.21.15.121:0/2591505160 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c48199200 con 0x7f8c4806ab30 2024-06-10T00:29:47.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c4d1ee700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4806ab30 0x7f8c4819ee20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800e9a0 tx=0x7f8c3800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:47.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.895+0000 7f8c3effd700 1 -- 172.21.15.121:0/2591505160 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3800ca10 con 0x7f8c4806ab30 2024-06-10T00:29:47.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.896+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2591505160 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c481994f0 con 0x7f8c4806ab30 2024-06-10T00:29:47.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.896+0000 7f8c3effd700 1 -- 172.21.15.121:0/2591505160 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c3800cbb0 con 0x7f8c4806ab30 2024-06-10T00:29:47.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.896+0000 7f8c3effd700 1 -- 172.21.15.121:0/2591505160 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c38018680 con 0x7f8c4806ab30 2024-06-10T00:29:47.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.896+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2591505160 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c481a57b0 con 0x7f8c4806ab30 2024-06-10T00:29:47.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.896+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2591505160 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c4810a9e0 con 0x7f8c4806ab30 2024-06-10T00:29:47.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.898+0000 7f8c3effd700 1 -- 172.21.15.121:0/2591505160 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8c38010430 con 0x7f8c4806ab30 2024-06-10T00:29:47.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.898+0000 7f8c3effd700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c340a2e10 0x7f8c340a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:47.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.899+0000 7f8c4d9ef700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c340a2e10 0x7f8c340a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:47.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.899+0000 7f8c3effd700 1 -- 172.21.15.121:0/2591505160 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8c380cfe00 con 0x7f8c4806ab30 2024-06-10T00:29:47.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.899+0000 7f8c4d9ef700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c340a2e10 0x7f8c340a52d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8c4400b2e0 tx=0x7f8c4401a040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:47.902 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:47.902+0000 7f8c3effd700 1 -- 172.21.15.121:0/2591505160 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8c3809c690 con 0x7f8c4806ab30 2024-06-10T00:29:47.903 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:48.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.164+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2591505160 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c48199fd0 con 0x7f8c4806ab30 2024-06-10T00:29:48.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.165+0000 7f8c3effd700 1 -- 172.21.15.121:0/2591505160 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11885 (secure 0 0 0) 0x7f8c3809bf90 con 0x7f8c4806ab30 2024-06-10T00:29:48.165 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:29:48.165 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:27:55.370804+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:29:48.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2591505160 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c340a2e10 msgr2=0x7f8c340a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:48.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c340a2e10 0x7f8c340a52d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8c4400b2e0 tx=0x7f8c4401a040 comp rx=0 tx=0).stop 2024-06-10T00:29:48.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2591505160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4806ab30 msgr2=0x7f8c4819ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:48.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4806ab30 0x7f8c4819ee20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8c3800e9a0 tx=0x7f8c3800ecb0 comp rx=0 tx=0).stop 2024-06-10T00:29:48.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2591505160 shutdown_connections 2024-06-10T00:29:48.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8c340a2e10 0x7f8c340a52d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:48.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8c48108500 0x7f8c48198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:48.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c4806a170 0x7f8c4819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:48.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 --2- 172.21.15.121:0/2591505160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4806ab30 0x7f8c4819ee20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:48.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2591505160 >> 172.21.15.121:0/2591505160 conn(0x7f8c48078340 msgr2=0x7f8c48100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:48.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2591505160 shutdown_connections 2024-06-10T00:29:48.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:48.167+0000 7f8c4fc53700 1 -- 172.21.15.121:0/2591505160 wait complete. 2024-06-10T00:29:48.168 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 26 2024-06-10T00:29:49.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:48 smithi135 ceph-mon[36424]: pgmap v58: 129 pgs: 129 active+clean; 4.6 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 287 KiB/s wr, 26 op/s 2024-06-10T00:29:49.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:48 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2591505160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:49.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:48 smithi193 ceph-mon[36716]: pgmap v58: 129 pgs: 129 active+clean; 4.6 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 287 KiB/s wr, 26 op/s 2024-06-10T00:29:49.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:48 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2591505160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:49.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:48 smithi121 ceph-mon[32378]: pgmap v58: 129 pgs: 129 active+clean; 4.6 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 287 KiB/s wr, 26 op/s 2024-06-10T00:29:49.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:48 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2591505160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:29:49.251 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-10T00:29:50.212 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:50.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.216+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/1126761738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbc806ab60 msgr2=0x7fcbc806afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:50.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.216+0000 7fcbcf3cb700 1 --2- 172.21.15.121:0/1126761738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbc806ab60 0x7fcbc806afe0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fcbb80099c0 tx=0x7fcbb8009cd0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.216+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/1126761738 shutdown_connections 2024-06-10T00:29:50.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.216+0000 7fcbcf3cb700 1 --2- 172.21.15.121:0/1126761738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc806b520 0x7fcbc810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.216+0000 7fcbcf3cb700 1 --2- 172.21.15.121:0/1126761738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbc8108560 0x7fcbc8108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.216+0000 7fcbcf3cb700 1 --2- 172.21.15.121:0/1126761738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbc806ab60 0x7fcbc806afe0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.216+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/1126761738 >> 172.21.15.121:0/1126761738 conn(0x7fcbc80780f0 msgr2=0x7fcbc8078500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:50.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.216+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/1126761738 shutdown_connections 2024-06-10T00:29:50.217 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.216+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/1126761738 wait complete. 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.217+0000 7fcbcf3cb700 1 Processor -- start 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.217+0000 7fcbcf3cb700 1 -- start start 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.217+0000 7fcbcf3cb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc806ab60 0x7fcbc819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.217+0000 7fcbcf3cb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbc806b520 0x7fcbc819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.217+0000 7fcbcf3cb700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbc8108560 0x7fcbc8198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.217+0000 7fcbcf3cb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc81127f0 con 0x7fcbc806b520 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.217+0000 7fcbcf3cb700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc8112670 con 0x7fcbc806ab60 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.217+0000 7fcbcf3cb700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc8112970 con 0x7fcbc8108560 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcd167700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc806ab60 0x7fcbc819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcc966700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbc806b520 0x7fcbc819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcd167700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc806ab60 0x7fcbc819e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:33590/0 (socket says 172.21.15.121:33590) 2024-06-10T00:29:50.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcd167700 1 -- 172.21.15.121:0/597167349 learned_addr learned my addr 172.21.15.121:0/597167349 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcd968700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbc8108560 0x7fcbc8198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcd167700 1 -- 172.21.15.121:0/597167349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbc8108560 msgr2=0x7fcbc8198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcd167700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbc8108560 0x7fcbc8198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcd167700 1 -- 172.21.15.121:0/597167349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbc806b520 msgr2=0x7fcbc819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcd167700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbc806b520 0x7fcbc819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcd167700 1 -- 172.21.15.121:0/597167349 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc81990d0 con 0x7fcbc806ab60 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcd167700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc806ab60 0x7fcbc819e7b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400d980 tx=0x7fcbc400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbbe7fc700 1 -- 172.21.15.121:0/597167349 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4017070 con 0x7fcbc806ab60 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbbe7fc700 1 -- 172.21.15.121:0/597167349 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbc400fd70 con 0x7fcbc806ab60 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/597167349 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc81993c0 con 0x7fcbc806ab60 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.218+0000 7fcbbe7fc700 1 -- 172.21.15.121:0/597167349 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc400e3b0 con 0x7fcbc806ab60 2024-06-10T00:29:50.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.219+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/597167349 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc81a5680 con 0x7fcbc806ab60 2024-06-10T00:29:50.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.221+0000 7fcbbe7fc700 1 -- 172.21.15.121:0/597167349 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fcbc400f890 con 0x7fcbc806ab60 2024-06-10T00:29:50.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.222+0000 7fcbbe7fc700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcbb40a2e30 0x7fcbb40a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.222+0000 7fcbcc966700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcbb40a2e30 0x7fcbb40a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.222+0000 7fcbbe7fc700 1 -- 172.21.15.121:0/597167349 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcbc40cf340 con 0x7fcbc806ab60 2024-06-10T00:29:50.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.222+0000 7fcbcc966700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcbb40a2e30 0x7fcbb40a52f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8078680 tx=0x7fcbb8005fb0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:50.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbd9c4eb80 1 Processor -- start 2024-06-10T00:29:50.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbd9c4eb80 1 -- start start 2024-06-10T00:29:50.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298bcd0 0x55d64298c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d642985f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d6429868a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d6428d79a0 con 0x55d64298c5f0 2024-06-10T00:29:50.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d64287f8b0 con 0x55d642986460 2024-06-10T00:29:50.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d6428d3960 con 0x55d64298bcd0 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbcc966700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d642985f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbcc966700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d642985f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42622/0 (socket says 172.21.15.121:42622) 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbcc966700 1 -- 172.21.15.121:0/3548537318 learned_addr learned my addr 172.21.15.121:0/3548537318 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbcd968700 1 --2- 172.21.15.121:0/3548537318 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d6429868a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.224+0000 7fcbcd167700 1 --2- 172.21.15.121:0/3548537318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298bcd0 0x55d64298c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.225+0000 7fcbcc966700 1 -- 172.21.15.121:0/3548537318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298bcd0 msgr2=0x55d64298c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.225+0000 7fcbcc966700 1 --2- 172.21.15.121:0/3548537318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298bcd0 0x55d64298c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.225+0000 7fcbcc966700 1 -- 172.21.15.121:0/3548537318 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 msgr2=0x55d6429868a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.225+0000 7fcbcc966700 1 --2- 172.21.15.121:0/3548537318 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d6429868a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.225+0000 7fcbcc966700 1 -- 172.21.15.121:0/3548537318 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6428760e0 con 0x55d64298c5f0 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.225+0000 7fcbcd167700 1 --2- 172.21.15.121:0/3548537318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298bcd0 0x55d64298c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.225+0000 7fcbcc966700 1 --2- 172.21.15.121:0/3548537318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d642985f20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8015000 tx=0x7fcbb801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dac1f57a7cb7787 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:50.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.225+0000 7fcbb37fe700 1 -- 172.21.15.121:0/3548537318 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8034070 con 0x55d64298c5f0 2024-06-10T00:29:50.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.225+0000 7fcbb37fe700 1 -- 172.21.15.121:0/3548537318 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb800be30 con 0x55d64298c5f0 2024-06-10T00:29:50.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.225+0000 7fcbb37fe700 1 -- 172.21.15.121:0/3548537318 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb802f910 con 0x55d64298c5f0 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.226+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/3548537318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 msgr2=0x55d642985f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.226+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/3548537318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d642985f20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8015000 tx=0x7fcbb801f000 comp rx=0 tx=0).stop 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.226+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/3548537318 shutdown_connections 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.226+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/3548537318 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d6429868a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.226+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/3548537318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298bcd0 0x55d64298c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.226+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/3548537318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d642985f20 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.226+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/3548537318 >> 172.21.15.121:0/3548537318 conn(0x55d6428b0e50 msgr2=0x55d6429811b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.226+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/3548537318 shutdown_connections 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.226+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/3548537318 wait complete. 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbd9c4eb80 1 -- start start 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbd9c4eb80 1 Processor -- start 2024-06-10T00:29:50.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d642992370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298bcd0 0x55d64298f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d64298fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d642993000 con 0x55d64298c5f0 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d642877d80 con 0x55d642986460 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d642993180 con 0x55d64298bcd0 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbd9c4eb80 10 client.0 _ll_register_callbacks cb 0x55d642871ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbcd968700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d64298fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbcd968700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d64298fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42632/0 (socket says 172.21.15.121:42632) 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbcd968700 1 -- 172.21.15.121:0/331591336 learned_addr learned my addr 172.21.15.121:0/331591336 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbcd167700 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d642992370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.227+0000 7fcbcc966700 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298bcd0 0x55d64298f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbcd968700 1 -- 172.21.15.121:0/331591336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298bcd0 msgr2=0x55d64298f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbcd968700 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298bcd0 0x55d64298f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbcd968700 1 -- 172.21.15.121:0/331591336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 msgr2=0x55d642992370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbcd968700 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d642992370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbcd968700 1 -- 172.21.15.121:0/331591336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d642826a60 con 0x55d64298c5f0 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbcd968700 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d64298fc00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0011f60 tx=0x7fcbc0014600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5190beaca56b82a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbb0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/331591336 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00090d0 con 0x55d64298c5f0 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/331591336 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcbc0009270 con 0x55d64298c5f0 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/331591336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbb800be30 con 0x55d64298c5f0 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/331591336 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcbc0027050 con 0x55d64298c5f0 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbd9c4eb80 10 client.15777 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/331591336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d64299d970 con 0x55d64298c5f0 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.228+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/331591336 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb800be30 con 0x55d64298c5f0 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.229+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/331591336 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fcbc0007500 con 0x55d64298c5f0 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.229+0000 7fcbd9c4eb80 10 client.15777 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.229+0000 7fcbd9c4eb80 10 client.15777 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:29:50.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.229+0000 7fcbd9c4eb80 20 client.15777 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:50.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.229+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d64299ffb0 0x55d6429a2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.229+0000 7fcbd9c4eb80 4 client.15777 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:50.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.229+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/331591336 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6428e4ea0 con 0x55d64299ffb0 2024-06-10T00:29:50.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.229+0000 7fcbcd167700 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d64299ffb0 0x55d6429a2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.229+0000 7fcbb0ff9700 10 client.15777 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:50.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.229+0000 7fcbcd167700 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d64299ffb0 0x55d6429a2400 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e4caad87d968cd0 server_cookie=6ae9071e26021e94 in_seq=0 out_seq=0 2024-06-10T00:29:50.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.233+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/331591336 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d6428e4ea0 con 0x55d64299ffb0 2024-06-10T00:29:50.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.233+0000 7fcbb0ff9700 10 client.15777 handle_command_reply: tid=0 2024-06-10T00:29:50.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.233+0000 7fcbd9c4eb80 1 client.15777 shutdown 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/331591336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 msgr2=0x55d64298fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d64298fc00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0011f60 tx=0x7fcbc0014600 comp rx=0 tx=0).stop 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/331591336 shutdown_connections 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d64299ffb0 0x55d6429a2400 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbb0ff9700 0 client.15777 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298c5f0 0x55d64298fc00 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d642992370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/331591336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298bcd0 0x55d64298f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/331591336 >> 172.21.15.121:0/331591336 conn(0x55d6428b0e50 msgr2=0x55d642984f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/331591336 shutdown_connections 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/331591336 wait complete. 2024-06-10T00:29:50.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.234+0000 7fcbd9c4eb80 20 client.15777 trim_cache size 0 max 16384 2024-06-10T00:29:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.261+0000 7fcbd9c4eb80 1 Processor -- start 2024-06-10T00:29:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.261+0000 7fcbd9c4eb80 1 -- start start 2024-06-10T00:29:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.261+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d64298fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.261+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d64298d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.261+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298c5f0 0x55d6429a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.261+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc81b0d70 con 0x55d64298bcd0 2024-06-10T00:29:50.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.261+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc81ae600 con 0x55d642986460 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.261+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc81ac450 con 0x55d64298c5f0 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.261+0000 7fcbcc966700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d64298d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbcc966700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d64298d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42636/0 (socket says 172.21.15.121:42636) 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbcc966700 1 -- 172.21.15.121:0/1116167946 learned_addr learned my addr 172.21.15.121:0/1116167946 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbcd167700 1 --2- 172.21.15.121:0/1116167946 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d64298fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbcc966700 1 -- 172.21.15.121:0/1116167946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298c5f0 msgr2=0x55d6429a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbcc966700 1 --2- 172.21.15.121:0/1116167946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298c5f0 0x55d6429a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbcc966700 1 -- 172.21.15.121:0/1116167946 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 msgr2=0x55d64298fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbcc966700 1 --2- 172.21.15.121:0/1116167946 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d64298fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbcc966700 1 -- 172.21.15.121:0/1116167946 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d64298dc30 con 0x55d64298bcd0 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbcd167700 1 --2- 172.21.15.121:0/1116167946 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d64298fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbcc966700 1 --2- 172.21.15.121:0/1116167946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d64298d1a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fcbb80098e0 tx=0x7fcbb8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bc17c1208a6ae2 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbb2ffd700 1 -- 172.21.15.121:0/1116167946 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8034040 con 0x55d64298bcd0 2024-06-10T00:29:50.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbb2ffd700 1 -- 172.21.15.121:0/1116167946 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb8024960 con 0x55d64298bcd0 2024-06-10T00:29:50.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.262+0000 7fcbb2ffd700 1 -- 172.21.15.121:0/1116167946 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb802f950 con 0x55d64298bcd0 2024-06-10T00:29:50.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.263+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1116167946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 msgr2=0x55d64298d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:50.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.263+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/1116167946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d64298d1a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fcbb80098e0 tx=0x7fcbb8009310 comp rx=0 tx=0).stop 2024-06-10T00:29:50.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.263+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1116167946 shutdown_connections 2024-06-10T00:29:50.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.263+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/1116167946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298c5f0 0x55d6429a7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.263+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/1116167946 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d64298fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.263+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/1116167946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d64298d1a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.263+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1116167946 >> 172.21.15.121:0/1116167946 conn(0x55d6428b0e50 msgr2=0x55d64298ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.263+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1116167946 shutdown_connections 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.263+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1116167946 wait complete. 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.263+0000 7fcbd9c4eb80 1 -- start start 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbd9c4eb80 1 Processor -- start 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d642998930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d642995410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbd9c4eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298c5f0 0x55d642995950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc81acac0 con 0x55d64298bcd0 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc81adf30 con 0x55d642986460 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbd9c4eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc81ad190 con 0x55d64298c5f0 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbd9c4eb80 10 client.0 _ll_register_callbacks cb 0x55d642871ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:50.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbcc966700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d642995410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbcc966700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d642995410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42638/0 (socket says 172.21.15.121:42638) 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbcc966700 1 -- 172.21.15.121:0/1943569834 learned_addr learned my addr 172.21.15.121:0/1943569834 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbcd167700 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d642998930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbcd968700 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298c5f0 0x55d642995950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbcc966700 1 -- 172.21.15.121:0/1943569834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298c5f0 msgr2=0x55d642995950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.264+0000 7fcbcc966700 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298c5f0 0x55d642995950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbcc966700 1 -- 172.21.15.121:0/1943569834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 msgr2=0x55d642998930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbcc966700 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d642998930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbcc966700 1 -- 172.21.15.121:0/1943569834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d642995e90 con 0x55d64298bcd0 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbcd167700 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d642998930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbcc966700 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d642995410 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8000c00 tx=0x7fcbb800bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88edfe815e3cf46 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbb0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/1943569834 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8034040 con 0x55d64298bcd0 2024-06-10T00:29:50.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/1943569834 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcbb8024690 con 0x55d64298bcd0 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1943569834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbb8024960 con 0x55d64298bcd0 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/1943569834 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcbb8053050 con 0x55d64298bcd0 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbd9c4eb80 10 client.15783 fetch_fsmap learned FSMap version 26 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1943569834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d642a0b9a0 con 0x55d64298bcd0 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/1943569834 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb8024960 con 0x55d64298bcd0 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.265+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/1943569834 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 26) ==== 8346+0+0 (secure 0 0 0) 0x7fcbb8037030 con 0x55d64298bcd0 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.266+0000 7fcbd9c4eb80 10 client.15783 fetch_fsmap finished waiting for FSMap version 26 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.266+0000 7fcbd9c4eb80 10 client.15783 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.266+0000 7fcbd9c4eb80 20 client.15783 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.266+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d64299ffb0 0x55d642a0dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.266+0000 7fcbd9c4eb80 4 client.15783 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}] 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.266+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1943569834 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.0, "await_for": 45.0}) -- 0x55d6428df420 con 0x55d64299ffb0 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.266+0000 7fcbcd167700 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d64299ffb0 0x55d642a0dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.266+0000 7fcbb0ff9700 10 client.15783 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:50.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:50.266+0000 7fcbcd167700 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d64299ffb0 0x55d642a0dc20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67fea14e5c273d76 server_cookie=569bac4232273229 in_seq=0 out_seq=0 2024-06-10T00:29:50.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:50 smithi121 ceph-mon[32378]: pgmap v59: 129 pgs: 129 active+clean; 7.5 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 618 KiB/s wr, 61 op/s 2024-06-10T00:29:51.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:50 smithi135 ceph-mon[36424]: pgmap v59: 129 pgs: 129 active+clean; 7.5 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 618 KiB/s wr, 61 op/s 2024-06-10T00:29:51.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:50 smithi193 ceph-mon[36716]: pgmap v59: 129 pgs: 129 active+clean; 7.5 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 618 KiB/s wr, 61 op/s 2024-06-10T00:29:52.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.027+0000 7fcf39a44700 1 -- 172.21.15.121:0/4166313301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf341084e0 msgr2=0x7fcf341088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.027+0000 7fcf39a44700 1 --2- 172.21.15.121:0/4166313301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf341084e0 0x7fcf341088c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fcf240099c0 tx=0x7fcf24009cd0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.027+0000 7fcf39a44700 1 -- 172.21.15.121:0/4166313301 shutdown_connections 2024-06-10T00:29:52.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.027+0000 7fcf39a44700 1 --2- 172.21.15.121:0/4166313301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf3406ab10 0x7fcf3410f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.027+0000 7fcf39a44700 1 --2- 172.21.15.121:0/4166313301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf341084e0 0x7fcf341088c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.027+0000 7fcf39a44700 1 --2- 172.21.15.121:0/4166313301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf3406a170 0x7fcf3406a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.027+0000 7fcf39a44700 1 -- 172.21.15.121:0/4166313301 >> 172.21.15.121:0/4166313301 conn(0x7fcf340782c0 msgr2=0x7fcf340786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:52.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.027+0000 7fcf39a44700 1 -- 172.21.15.121:0/4166313301 shutdown_connections 2024-06-10T00:29:52.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.027+0000 7fcf39a44700 1 -- 172.21.15.121:0/4166313301 wait complete. 2024-06-10T00:29:52.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.028+0000 7fcf39a44700 1 Processor -- start 2024-06-10T00:29:52.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.028+0000 7fcf39a44700 1 -- start start 2024-06-10T00:29:52.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.028+0000 7fcf39a44700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf3406a170 0x7fcf34101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.028+0000 7fcf39a44700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf3406ab10 0x7fcf341019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.028+0000 7fcf39a44700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf341084e0 0x7fcf34101ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.028+0000 7fcf39a44700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcf34112740 con 0x7fcf3406ab10 2024-06-10T00:29:52.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.028+0000 7fcf39a44700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcf341125c0 con 0x7fcf341084e0 2024-06-10T00:29:52.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.028+0000 7fcf39a44700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf341128c0 con 0x7fcf3406a170 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf3406ab10 0x7fcf341019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf337fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf341084e0 0x7fcf34101ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf3406ab10 0x7fcf341019a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42662/0 (socket says 172.21.15.121:42662) 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf327fc700 1 -- 172.21.15.121:0/215378842 learned_addr learned my addr 172.21.15.121:0/215378842 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf32ffd700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf3406a170 0x7fcf34101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf327fc700 1 -- 172.21.15.121:0/215378842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf3406a170 msgr2=0x7fcf34101460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf327fc700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf3406a170 0x7fcf34101460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf327fc700 1 -- 172.21.15.121:0/215378842 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf341084e0 msgr2=0x7fcf34101ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf327fc700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf341084e0 0x7fcf34101ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf327fc700 1 -- 172.21.15.121:0/215378842 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf341a7110 con 0x7fcf3406ab10 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf337fe700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf341084e0 0x7fcf34101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf327fc700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf3406ab10 0x7fcf341019a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800e9a0 tx=0x7fcf2800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf38a42700 1 -- 172.21.15.121:0/215378842 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2800ca10 con 0x7fcf3406ab10 2024-06-10T00:29:52.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf39a44700 1 -- 172.21.15.121:0/215378842 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf341a7400 con 0x7fcf3406ab10 2024-06-10T00:29:52.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf38a42700 1 -- 172.21.15.121:0/215378842 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf2800cbb0 con 0x7fcf3406ab10 2024-06-10T00:29:52.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf38a42700 1 -- 172.21.15.121:0/215378842 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf28018680 con 0x7fcf3406ab10 2024-06-10T00:29:52.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.029+0000 7fcf39a44700 1 -- 172.21.15.121:0/215378842 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf341a7990 con 0x7fcf3406ab10 2024-06-10T00:29:52.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.032+0000 7fcf38a42700 1 -- 172.21.15.121:0/215378842 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fcf28010430 con 0x7fcf3406ab10 2024-06-10T00:29:52.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.033+0000 7fcf38a42700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf1c0a2d90 0x7fcf1c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.033+0000 7fcf32ffd700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf1c0a2d90 0x7fcf1c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.033+0000 7fcf38a42700 1 -- 172.21.15.121:0/215378842 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcf280cfa20 con 0x7fcf3406ab10 2024-06-10T00:29:52.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.033+0000 7fcf32ffd700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf1c0a2d90 0x7fcf1c0a5250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcf24000c00 tx=0x7fcf24005fb0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:52.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf442c7b80 1 Processor -- start 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf442c7b80 1 -- start start 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558189137f60 0x558189138340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x558189132260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581891327a0 0x558189132bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x558189024f60 con 0x558189138880 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x558189083850 con 0x5581891327a0 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558188feb460 con 0x558189137f60 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x558189132260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x558189132260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42676/0 (socket says 172.21.15.121:42676) 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf32ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558189137f60 0x558189138340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf327fc700 1 -- 172.21.15.121:0/760853097 learned_addr learned my addr 172.21.15.121:0/760853097 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf337fe700 1 --2- 172.21.15.121:0/760853097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581891327a0 0x558189132bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf327fc700 1 -- 172.21.15.121:0/760853097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558189137f60 msgr2=0x558189138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:52.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf327fc700 1 --2- 172.21.15.121:0/760853097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558189137f60 0x558189138340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf327fc700 1 -- 172.21.15.121:0/760853097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581891327a0 msgr2=0x558189132bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:52.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf327fc700 1 --2- 172.21.15.121:0/760853097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581891327a0 0x558189132bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf327fc700 1 -- 172.21.15.121:0/760853097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558188fd28d0 con 0x558189138880 2024-06-10T00:29:52.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.034+0000 7fcf327fc700 1 --2- 172.21.15.121:0/760853097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x558189132260 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fcf28098580 tx=0x7fcf2809c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ba940577b104c9 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:52.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf19ffb700 1 -- 172.21.15.121:0/760853097 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf280a4820 con 0x558189138880 2024-06-10T00:29:52.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf19ffb700 1 -- 172.21.15.121:0/760853097 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf28099eb0 con 0x558189138880 2024-06-10T00:29:52.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf19ffb700 1 -- 172.21.15.121:0/760853097 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf28010920 con 0x558189138880 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf442c7b80 1 -- 172.21.15.121:0/760853097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 msgr2=0x558189132260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/760853097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x558189132260 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fcf28098580 tx=0x7fcf2809c330 comp rx=0 tx=0).stop 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf442c7b80 1 -- 172.21.15.121:0/760853097 shutdown_connections 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/760853097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581891327a0 0x558189132bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/760853097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558189137f60 0x558189138340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/760853097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x558189132260 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf442c7b80 1 -- 172.21.15.121:0/760853097 >> 172.21.15.121:0/760853097 conn(0x55818905f080 msgr2=0x55818912d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf442c7b80 1 -- 172.21.15.121:0/760853097 shutdown_connections 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.035+0000 7fcf442c7b80 1 -- 172.21.15.121:0/760853097 wait complete. 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf442c7b80 1 -- start start 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf442c7b80 1 Processor -- start 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x55818913e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558189137f60 0x55818913b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x55818913be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x558189023fc0 con 0x558189138880 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55818907f810 con 0x558189137f60 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55818913f260 con 0x5581891327a0 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf442c7b80 10 client.0 _ll_register_callbacks cb 0x5581890195b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf337fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x55818913be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf32ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x55818913e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf327fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558189137f60 0x55818913b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf32ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x55818913e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:44056/0 (socket says 172.21.15.121:44056) 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf337fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x55818913be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42682/0 (socket says 172.21.15.121:42682) 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.036+0000 7fcf32ffd700 1 -- 172.21.15.121:0/1794893796 learned_addr learned my addr 172.21.15.121:0/1794893796 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf32ffd700 1 -- 172.21.15.121:0/1794893796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558189137f60 msgr2=0x55818913b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf32ffd700 1 --2- 172.21.15.121:0/1794893796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558189137f60 0x55818913b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf32ffd700 1 -- 172.21.15.121:0/1794893796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 msgr2=0x55818913be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf32ffd700 1 --2- 172.21.15.121:0/1794893796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x55818913be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf32ffd700 1 -- 172.21.15.121:0/1794893796 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558189049750 con 0x5581891327a0 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf32ffd700 1 --2- 172.21.15.121:0/1794893796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x55818913e610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcf240099c0 tx=0x7fcf24009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7768e322308e1234 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf12ffd700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1794893796 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf24024bf0 con 0x5581891327a0 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1794893796 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf28099eb0 con 0x5581891327a0 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1794893796 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcf24024d90 con 0x5581891327a0 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1794893796 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcf2403e050 con 0x5581891327a0 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf442c7b80 10 client.25330 fetch_fsmap learned FSMap version 27 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.037+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1794893796 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558189149a90 con 0x5581891327a0 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.038+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1794893796 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf28099eb0 con 0x5581891327a0 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.038+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1794893796 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7fcf2400f6c0 con 0x5581891327a0 2024-06-10T00:29:52.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.038+0000 7fcf442c7b80 10 client.25330 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.038+0000 7fcf442c7b80 10 client.25330 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.038+0000 7fcf442c7b80 20 client.25330 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.038+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1794893796 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55818914c120 0x55818914e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.038+0000 7fcf442c7b80 4 client.25330 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.038+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1794893796 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558189090d50 con 0x55818914c120 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.038+0000 7fcf327fc700 1 --2- 172.21.15.121:0/1794893796 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55818914c120 0x55818914e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.039+0000 7fcf12ffd700 10 client.25330 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.039+0000 7fcf327fc700 1 --2- 172.21.15.121:0/1794893796 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55818914c120 0x55818914e510 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e521c2c4027055d3 server_cookie=d36cb74fbf3e567c in_seq=0 out_seq=0 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.039+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1794893796 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x558189090d50 con 0x55818914c120 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.039+0000 7fcf12ffd700 10 client.25330 handle_command_reply: tid=0 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 1 client.25330 shutdown 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1794893796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 msgr2=0x55818913e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1794893796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x55818913e610 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcf240099c0 tx=0x7fcf24009310 comp rx=0 tx=0).stop 2024-06-10T00:29:52.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1794893796 shutdown_connections 2024-06-10T00:29:52.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1794893796 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55818914c120 0x55818914e510 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1794893796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x55818913be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1794893796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x55818913e610 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf12ffd700 0 client.25330 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:29:52.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1794893796 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558189137f60 0x55818913b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1794893796 >> 172.21.15.121:0/1794893796 conn(0x55818905f080 msgr2=0x55818912c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:52.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1794893796 shutdown_connections 2024-06-10T00:29:52.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1794893796 wait complete. 2024-06-10T00:29:52.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.040+0000 7fcf442c7b80 20 client.25330 trim_cache size 0 max 16384 2024-06-10T00:29:52.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf442c7b80 1 Processor -- start 2024-06-10T00:29:52.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf442c7b80 1 -- start start 2024-06-10T00:29:52.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581891327a0 0x558189139530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558189137f60 0x55818913a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x55818913be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcf341b2f80 con 0x558189138880 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcf341b0810 con 0x5581891327a0 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf341ae660 con 0x558189137f60 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf337fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x55818913be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf337fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x55818913be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42686/0 (socket says 172.21.15.121:42686) 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf337fe700 1 -- 172.21.15.121:0/3266828584 learned_addr learned my addr 172.21.15.121:0/3266828584 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf327fc700 1 --2- 172.21.15.121:0/3266828584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558189137f60 0x55818913a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.055+0000 7fcf32ffd700 1 --2- 172.21.15.121:0/3266828584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581891327a0 0x558189139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.056+0000 7fcf337fe700 1 -- 172.21.15.121:0/3266828584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558189137f60 msgr2=0x55818913a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.056+0000 7fcf337fe700 1 --2- 172.21.15.121:0/3266828584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558189137f60 0x55818913a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.056+0000 7fcf337fe700 1 -- 172.21.15.121:0/3266828584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581891327a0 msgr2=0x558189139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.056+0000 7fcf337fe700 1 --2- 172.21.15.121:0/3266828584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581891327a0 0x558189139530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.056+0000 7fcf337fe700 1 -- 172.21.15.121:0/3266828584 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558189139eb0 con 0x558189138880 2024-06-10T00:29:52.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.056+0000 7fcf327fc700 1 --2- 172.21.15.121:0/3266828584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558189137f60 0x55818913a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:52.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.056+0000 7fcf32ffd700 1 --2- 172.21.15.121:0/3266828584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581891327a0 0x558189139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:52.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.056+0000 7fcf337fe700 1 --2- 172.21.15.121:0/3266828584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x55818913be80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c00eaf0 tx=0x7fcf2c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9fa9e506177b051 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:52.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.056+0000 7fcf197fa700 1 -- 172.21.15.121:0/3266828584 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c017820 con 0x558189138880 2024-06-10T00:29:52.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.056+0000 7fcf197fa700 1 -- 172.21.15.121:0/3266828584 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf2c007500 con 0x558189138880 2024-06-10T00:29:52.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.056+0000 7fcf197fa700 1 -- 172.21.15.121:0/3266828584 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c01f600 con 0x558189138880 2024-06-10T00:29:52.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.057+0000 7fcf442c7b80 1 -- 172.21.15.121:0/3266828584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 msgr2=0x55818913be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.057+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/3266828584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x55818913be80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c00eaf0 tx=0x7fcf2c009920 comp rx=0 tx=0).stop 2024-06-10T00:29:52.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.057+0000 7fcf442c7b80 1 -- 172.21.15.121:0/3266828584 shutdown_connections 2024-06-10T00:29:52.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.057+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/3266828584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189138880 0x55818913be80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.057+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/3266828584 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5581891327a0 0x558189139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.057+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/3266828584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558189137f60 0x55818913a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.057+0000 7fcf442c7b80 1 -- 172.21.15.121:0/3266828584 >> 172.21.15.121:0/3266828584 conn(0x55818905f080 msgr2=0x558189136a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.057+0000 7fcf442c7b80 1 -- 172.21.15.121:0/3266828584 shutdown_connections 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.057+0000 7fcf442c7b80 1 -- 172.21.15.121:0/3266828584 wait complete. 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.057+0000 7fcf442c7b80 1 -- start start 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.057+0000 7fcf442c7b80 1 Processor -- start 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x558189144a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189137f60 0x5581891534d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf442c7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558189138880 0x558189153a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcf341aecd0 con 0x558189137f60 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcf341b0140 con 0x558189138880 2024-06-10T00:29:52.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf442c7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf341af3a0 con 0x5581891327a0 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf442c7b80 10 client.0 _ll_register_callbacks cb 0x5581890195b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189137f60 0x5581891534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf32ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x558189144a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf327fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189137f60 0x5581891534d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42700/0 (socket says 172.21.15.121:42700) 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf327fc700 1 -- 172.21.15.121:0/1459384822 learned_addr learned my addr 172.21.15.121:0/1459384822 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf32ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x558189144a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:44082/0 (socket says 172.21.15.121:44082) 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf337fe700 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558189138880 0x558189153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf327fc700 1 -- 172.21.15.121:0/1459384822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 msgr2=0x558189144a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf327fc700 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x558189144a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf327fc700 1 -- 172.21.15.121:0/1459384822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558189138880 msgr2=0x558189153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf327fc700 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558189138880 0x558189153a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf327fc700 1 -- 172.21.15.121:0/1459384822 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558189153fd0 con 0x558189137f60 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf32ffd700 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x558189144a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:52.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf327fc700 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189137f60 0x5581891534d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcf28000c00 tx=0x7fcf280092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c837acc85d98a1b server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.058+0000 7fcf12ffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1459384822 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf280b2070 con 0x558189137f60 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1459384822 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcf280a4390 con 0x558189137f60 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1459384822 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf2c007500 con 0x558189137f60 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1459384822 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcf280a4820 con 0x558189137f60 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf442c7b80 10 client.15801 fetch_fsmap learned FSMap version 27 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1459384822 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5581891b6b60 con 0x558189137f60 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1459384822 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf2c007500 con 0x558189137f60 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1459384822 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7fcf280ba2b0 con 0x558189137f60 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf442c7b80 10 client.15801 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf442c7b80 10 client.15801 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf442c7b80 20 client.15801 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5581891b6ef0 0x5581891b92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:52.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf442c7b80 4 client.15801 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1459384822 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55818908b2d0 con 0x5581891b6ef0 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.059+0000 7fcf32ffd700 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5581891b6ef0 0x5581891b92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf12ffd700 10 client.15801 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf32ffd700 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5581891b6ef0 0x5581891b92e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5d63937ff3448964 server_cookie=4fd86f79d59cffc4 in_seq=0 out_seq=0 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf12ffd700 1 -- 172.21.15.121:0/1459384822 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55818908b2d0 con 0x5581891b6ef0 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf12ffd700 10 client.15801 handle_command_reply: tid=0 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 1 client.15801 shutdown 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1459384822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189137f60 msgr2=0x5581891534d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189137f60 0x5581891534d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fcf28000c00 tx=0x7fcf280092d0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1459384822 shutdown_connections 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5581891b6ef0 0x5581891b92e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf12ffd700 0 client.15801 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558189138880 0x558189153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581891327a0 0x558189144a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 1 --2- 172.21.15.121:0/1459384822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558189137f60 0x5581891534d0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1459384822 >> 172.21.15.121:0/1459384822 conn(0x55818905f080 msgr2=0x558189147c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1459384822 shutdown_connections 2024-06-10T00:29:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 1 -- 172.21.15.121:0/1459384822 wait complete. 2024-06-10T00:29:52.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.060+0000 7fcf442c7b80 20 client.15801 trim_cache size 0 max 16384 2024-06-10T00:29:52.066 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:29:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 -- 172.21.15.121:0/215378842 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf1c0a2d90 msgr2=0x7fcf1c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf1c0a2d90 0x7fcf1c0a5250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcf24000c00 tx=0x7fcf24005fb0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 -- 172.21.15.121:0/215378842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf3406ab10 msgr2=0x7fcf341019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf3406ab10 0x7fcf341019a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800e9a0 tx=0x7fcf2800ecb0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 -- 172.21.15.121:0/215378842 shutdown_connections 2024-06-10T00:29:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf1c0a2d90 0x7fcf1c0a5250 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf341084e0 0x7fcf34101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf3406a170 0x7fcf34101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 --2- 172.21.15.121:0/215378842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf3406ab10 0x7fcf341019a0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 -- 172.21.15.121:0/215378842 >> 172.21.15.121:0/215378842 conn(0x7fcf340782c0 msgr2=0x7fcf340ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:52.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 -- 172.21.15.121:0/215378842 shutdown_connections 2024-06-10T00:29:52.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.061+0000 7fcf39a44700 1 -- 172.21.15.121:0/215378842 wait complete. 2024-06-10T00:29:52.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbb0ff9700 1 -- 172.21.15.121:0/1943569834 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55d6428df420 con 0x55d64299ffb0 2024-06-10T00:29:52.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbb0ff9700 10 client.15783 handle_command_reply: tid=0 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 1 client.15783 shutdown 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1943569834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 msgr2=0x55d642995410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d642995410 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8000c00 tx=0x7fcbb800bd00 comp rx=0 tx=0).stop 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1943569834 shutdown_connections 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d64299ffb0 0x55d642a0dc20 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d64298c5f0 0x55d642995950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbb0ff9700 0 client.15783 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d642986460 0x55d642998930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 1 --2- 172.21.15.121:0/1943569834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d64298bcd0 0x55d642995410 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1943569834 >> 172.21.15.121:0/1943569834 conn(0x55d6428b0e50 msgr2=0x55d64298dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1943569834 shutdown_connections 2024-06-10T00:29:52.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 1 -- 172.21.15.121:0/1943569834 wait complete. 2024-06-10T00:29:52.155 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.153+0000 7fcbd9c4eb80 20 client.15783 trim_cache size 0 max 16384 2024-06-10T00:29:52.155 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:29:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 27, 2024-06-10T00:29:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:29:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 6, 2024-06-10T00:29:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:29:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "96a84dc5": { 2024-06-10T00:29:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 6, 2024-06-10T00:29:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:29:52.155 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 150.0, 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:52.156 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:52.157 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:29:52.157 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:29:52.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/597167349 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcbb40a2e30 msgr2=0x7fcbb40a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcbb40a2e30 0x7fcbb40a52f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8078680 tx=0x7fcbb8005fb0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/597167349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc806ab60 msgr2=0x7fcbc819e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:52.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc806ab60 0x7fcbc819e7b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcbc400d980 tx=0x7fcbc400dc90 comp rx=0 tx=0).stop 2024-06-10T00:29:52.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/597167349 shutdown_connections 2024-06-10T00:29:52.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcbb40a2e30 0x7fcbb40a52f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbc8108560 0x7fcbc8198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbc806ab60 0x7fcbc819e7b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 --2- 172.21.15.121:0/597167349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbc806b520 0x7fcbc819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:52.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/597167349 >> 172.21.15.121:0/597167349 conn(0x7fcbc80780f0 msgr2=0x7fcbc81000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:52.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/597167349 shutdown_connections 2024-06-10T00:29:52.158 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:52.154+0000 7fcbcf3cb700 1 -- 172.21.15.121:0/597167349 wait complete. 2024-06-10T00:29:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:52 smithi121 ceph-mon[32378]: pgmap v60: 129 pgs: 129 active+clean; 7.5 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 607 KiB/s wr, 59 op/s 2024-06-10T00:29:52.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:52 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:29:53.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:52 smithi135 ceph-mon[36424]: pgmap v60: 129 pgs: 129 active+clean; 7.5 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 607 KiB/s wr, 59 op/s 2024-06-10T00:29:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:52 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:29:53.118 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '96a84dc5', quiesce took 14.4 seconds. Will release after: 15.6 2024-06-10T00:29:53.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:52 smithi193 ceph-mon[36716]: pgmap v60: 129 pgs: 129 active+clean; 7.5 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 607 KiB/s wr, 59 op/s 2024-06-10T00:29:53.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:52 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:29:53.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-10T00:29:53.845 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:54.997 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:54 smithi121 ceph-mon[32378]: pgmap v61: 129 pgs: 129 active+clean; 33 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 2.8 MiB/s wr, 240 op/s 2024-06-10T00:29:55.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:54 smithi135 ceph-mon[36424]: pgmap v61: 129 pgs: 129 active+clean; 33 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 2.8 MiB/s wr, 240 op/s 2024-06-10T00:29:55.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:54 smithi193 ceph-mon[36716]: pgmap v61: 129 pgs: 129 active+clean; 33 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 2.8 MiB/s wr, 240 op/s 2024-06-10T00:29:55.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.673+0000 7f6dc0245700 1 -- 172.21.15.121:0/3373296014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db81066b0 msgr2=0x7f6db810e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:55.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.673+0000 7f6dc0245700 1 --2- 172.21.15.121:0/3373296014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db81066b0 0x7f6db810e280 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6db00099c0 tx=0x7f6db0009cd0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.673+0000 7f6dc0245700 1 -- 172.21.15.121:0/3373296014 shutdown_connections 2024-06-10T00:29:55.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.673+0000 7f6dc0245700 1 --2- 172.21.15.121:0/3373296014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db81066b0 0x7f6db810e280 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.673+0000 7f6dc0245700 1 --2- 172.21.15.121:0/3373296014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db8101620 0x7f6db8101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.673+0000 7f6dc0245700 1 --2- 172.21.15.121:0/3373296014 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db8101fd0 0x7f6db8105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.673+0000 7f6dc0245700 1 -- 172.21.15.121:0/3373296014 >> 172.21.15.121:0/3373296014 conn(0x7f6db8078360 msgr2=0x7f6db8078770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:55.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.673+0000 7f6dc0245700 1 -- 172.21.15.121:0/3373296014 shutdown_connections 2024-06-10T00:29:55.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.673+0000 7f6dc0245700 1 -- 172.21.15.121:0/3373296014 wait complete. 2024-06-10T00:29:55.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dc0245700 1 Processor -- start 2024-06-10T00:29:55.676 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dc0245700 1 -- start start 2024-06-10T00:29:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dc0245700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db8101620 0x7f6db819d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dc0245700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db8101fd0 0x7f6db819dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dc0245700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db81066b0 0x7f6db8197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dc0245700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6db8111710 con 0x7f6db81066b0 2024-06-10T00:29:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dc0245700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6db8111590 con 0x7f6db8101620 2024-06-10T00:29:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dc0245700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6db8111890 con 0x7f6db8101fd0 2024-06-10T00:29:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dbe7e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db81066b0 0x7f6db8197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dbe7e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db81066b0 0x7f6db8197970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42718/0 (socket says 172.21.15.121:42718) 2024-06-10T00:29:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/2161400785 learned_addr learned my addr 172.21.15.121:0/2161400785 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dbd7e0700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db8101fd0 0x7f6db819dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db8101620 0x7f6db819d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/2161400785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db8101fd0 msgr2=0x7f6db819dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:55.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db8101fd0 0x7f6db819dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/2161400785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db8101620 msgr2=0x7f6db819d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:55.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db8101620 0x7f6db819d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.674+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/2161400785 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db81980d0 con 0x7f6db81066b0 2024-06-10T00:29:55.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.675+0000 7f6dbd7e0700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db8101fd0 0x7f6db819dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:55.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.675+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db81066b0 0x7f6db8197970 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6db00059d0 tx=0x7f6db0003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:55.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.675+0000 7f6daeffd700 1 -- 172.21.15.121:0/2161400785 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db0027070 con 0x7f6db81066b0 2024-06-10T00:29:55.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.675+0000 7f6dc0245700 1 -- 172.21.15.121:0/2161400785 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db8198360 con 0x7f6db81066b0 2024-06-10T00:29:55.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.675+0000 7f6dc0245700 1 -- 172.21.15.121:0/2161400785 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db806abe0 con 0x7f6db81066b0 2024-06-10T00:29:55.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.675+0000 7f6daeffd700 1 -- 172.21.15.121:0/2161400785 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6db0004150 con 0x7f6db81066b0 2024-06-10T00:29:55.679 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.675+0000 7f6daeffd700 1 -- 172.21.15.121:0/2161400785 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db0031060 con 0x7f6db81066b0 2024-06-10T00:29:55.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.677+0000 7f6daeffd700 1 -- 172.21.15.121:0/2161400785 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6db002a070 con 0x7f6db81066b0 2024-06-10T00:29:55.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.678+0000 7f6daeffd700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6da40a2e10 0x7f6da40a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.678+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6da40a2e10 0x7f6da40a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.678+0000 7f6daeffd700 1 -- 172.21.15.121:0/2161400785 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f6db00d0570 con 0x7f6db81066b0 2024-06-10T00:29:55.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6da40a2e10 0x7f6da40a52d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6db8078a60 tx=0x7f6db40094f0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:55.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dcaac8b80 1 Processor -- start 2024-06-10T00:29:55.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dcaac8b80 1 -- start start 2024-06-10T00:29:55.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446c4f60 0x55c0446be9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 0x55c0446bf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c044610810 con 0x55c0446c4640 2024-06-10T00:29:55.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55c0445b8720 con 0x55c0446c4f60 2024-06-10T00:29:55.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c04460c7d0 con 0x55c0446bef20 2024-06-10T00:29:55.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dbdfe1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dbdfe1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446c4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42720/0 (socket says 172.21.15.121:42720) 2024-06-10T00:29:55.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dbdfe1700 1 -- 172.21.15.121:0/3701750052 learned_addr learned my addr 172.21.15.121:0/3701750052 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:55.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.679+0000 7f6dbd7e0700 1 --2- 172.21.15.121:0/3701750052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446c4f60 0x55c0446be9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.680+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/3701750052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 0x55c0446bf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.680+0000 7f6dbdfe1700 1 -- 172.21.15.121:0/3701750052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 msgr2=0x55c0446bf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:55.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.680+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/3701750052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 0x55c0446bf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.680+0000 7f6dbdfe1700 1 -- 172.21.15.121:0/3701750052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446c4f60 msgr2=0x55c0446be9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:55.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.680+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/3701750052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446c4f60 0x55c0446be9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.680+0000 7f6dbdfe1700 1 -- 172.21.15.121:0/3701750052 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0445aef50 con 0x55c0446c4640 2024-06-10T00:29:55.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.680+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/3701750052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 0x55c0446bf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:29:55.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.680+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/3701750052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446c4a20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6db400ce50 tx=0x7f6db4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c55cb4297d27646 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:55.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.680+0000 7f6da3fff700 1 -- 172.21.15.121:0/3701750052 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db4027070 con 0x55c0446c4640 2024-06-10T00:29:55.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.680+0000 7f6da3fff700 1 -- 172.21.15.121:0/3701750052 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6db4017bb0 con 0x55c0446c4640 2024-06-10T00:29:55.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.680+0000 7f6da3fff700 1 -- 172.21.15.121:0/3701750052 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db40163a0 con 0x55c0446c4640 2024-06-10T00:29:55.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.681+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/3701750052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 msgr2=0x55c0446c4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:55.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.681+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/3701750052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446c4a20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6db400ce50 tx=0x7f6db4011000 comp rx=0 tx=0).stop 2024-06-10T00:29:55.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.681+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/3701750052 shutdown_connections 2024-06-10T00:29:55.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.681+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/3701750052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 0x55c0446bf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.681+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/3701750052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446c4a20 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.681+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/3701750052 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446c4f60 0x55c0446be9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.681+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/3701750052 >> 172.21.15.121:0/3701750052 conn(0x55c0445e9cc0 msgr2=0x55c0446b9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:55.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.681+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/3701750052 shutdown_connections 2024-06-10T00:29:55.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.681+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/3701750052 wait complete. 2024-06-10T00:29:55.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.681+0000 7f6dcaac8b80 1 -- start start 2024-06-10T00:29:55.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.681+0000 7f6dcaac8b80 1 Processor -- start 2024-06-10T00:29:55.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446bef20 0x55c0446cac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446c4640 0x55c0446c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c0446cb9d0 con 0x55c0446c4f60 2024-06-10T00:29:55.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55c0445b0bf0 con 0x55c0446bef20 2024-06-10T00:29:55.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c0446cbb50 con 0x55c0446c4640 2024-06-10T00:29:55.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dcaac8b80 10 client.0 _ll_register_callbacks cb 0x55c0445aa920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:55.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dbe7e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dbe7e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446c84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42732/0 (socket says 172.21.15.121:42732) 2024-06-10T00:29:55.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/580073902 learned_addr learned my addr 172.21.15.121:0/580073902 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:55.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446bef20 0x55c0446cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dbd7e0700 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446c4640 0x55c0446c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/580073902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446c4640 msgr2=0x55c0446c7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:55.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446c4640 0x55c0446c7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/580073902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446bef20 msgr2=0x55c0446cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:55.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446bef20 0x55c0446cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/580073902 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c04455f8d0 con 0x55c0446c4f60 2024-06-10T00:29:55.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446c84c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6db009c490 tx=0x7f6db009c1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ddee1060faf615 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:55.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.682+0000 7f6da17fa700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:55.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6da17fa700 1 -- 172.21.15.121:0/580073902 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db00a0b40 con 0x55c0446c4f60 2024-06-10T00:29:55.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6da17fa700 1 -- 172.21.15.121:0/580073902 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f6db00d01a0 con 0x55c0446c4f60 2024-06-10T00:29:55.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/580073902 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6db4017bb0 con 0x55c0446c4f60 2024-06-10T00:29:55.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6da17fa700 1 -- 172.21.15.121:0/580073902 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6db00d03a0 con 0x55c0446c4f60 2024-06-10T00:29:55.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6dcaac8b80 10 client.15813 fetch_fsmap learned FSMap version 27 2024-06-10T00:29:55.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/580073902 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c0446d6250 con 0x55c0446c4f60 2024-06-10T00:29:55.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6da17fa700 1 -- 172.21.15.121:0/580073902 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6db4017bb0 con 0x55c0446c4f60 2024-06-10T00:29:55.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6da17fa700 1 -- 172.21.15.121:0/580073902 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f6db000fb80 con 0x55c0446c4f60 2024-06-10T00:29:55.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6dcaac8b80 10 client.15813 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:29:55.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6dcaac8b80 10 client.15813 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:29:55.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6dcaac8b80 20 client.15813 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:55.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55c0446d8810 0x55c0446dac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6dcaac8b80 4 client.15813 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:55.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.683+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/580073902 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c04461dd10 con 0x55c0446d8810 2024-06-10T00:29:55.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.684+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55c0446d8810 0x55c0446dac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.684+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55c0446d8810 0x55c0446dac30 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ce02fbc113b3ec98 server_cookie=ac082ccfe94e21c7 in_seq=0 out_seq=0 2024-06-10T00:29:55.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.684+0000 7f6da17fa700 10 client.15813 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:29:55.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6da17fa700 1 -- 172.21.15.121:0/580073902 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55c04461dd10 con 0x55c0446d8810 2024-06-10T00:29:55.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6da17fa700 10 client.15813 handle_command_reply: tid=0 2024-06-10T00:29:55.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 1 client.15813 shutdown 2024-06-10T00:29:55.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/580073902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 msgr2=0x55c0446c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:55.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446c84c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6db009c490 tx=0x7f6db009c1e0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/580073902 shutdown_connections 2024-06-10T00:29:55.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55c0446d8810 0x55c0446dac30 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446c84c0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446bef20 0x55c0446cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/580073902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446c4640 0x55c0446c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6da17fa700 0 client.15813 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:29:55.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/580073902 >> 172.21.15.121:0/580073902 conn(0x55c0445e9cc0 msgr2=0x55c0446bd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:55.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/580073902 shutdown_connections 2024-06-10T00:29:55.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/580073902 wait complete. 2024-06-10T00:29:55.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.685+0000 7f6dcaac8b80 20 client.15813 trim_cache size 0 max 16384 2024-06-10T00:29:55.705 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.700+0000 7f6dcaac8b80 1 Processor -- start 2024-06-10T00:29:55.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.700+0000 7f6dcaac8b80 1 -- start start 2024-06-10T00:29:55.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.700+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446bef20 0x55c0446be9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.700+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446e0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.700+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446c4f60 0x55c0446e0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.700+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c0115e0 con 0x55c0446c4640 2024-06-10T00:29:55.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.700+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c00ee70 con 0x55c0446bef20 2024-06-10T00:29:55.706 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.700+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c00ccc0 con 0x55c0446c4f60 2024-06-10T00:29:55.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.700+0000 7f6dbd7e0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446e0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6dbd7e0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446e0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42740/0 (socket says 172.21.15.121:42740) 2024-06-10T00:29:55.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6dbd7e0700 1 -- 172.21.15.121:0/2376039241 learned_addr learned my addr 172.21.15.121:0/2376039241 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:55.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/2376039241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446bef20 0x55c0446be9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/2376039241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446c4f60 0x55c0446e0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6dbd7e0700 1 -- 172.21.15.121:0/2376039241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446c4f60 msgr2=0x55c0446e0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:55.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6dbd7e0700 1 --2- 172.21.15.121:0/2376039241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446c4f60 0x55c0446e0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6dbd7e0700 1 -- 172.21.15.121:0/2376039241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446bef20 msgr2=0x55c0446be9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:55.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6dbd7e0700 1 --2- 172.21.15.121:0/2376039241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446bef20 0x55c0446be9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6dbd7e0700 1 -- 172.21.15.121:0/2376039241 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0446db2b0 con 0x55c0446c4640 2024-06-10T00:29:55.707 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/2376039241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446bef20 0x55c0446be9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:29:55.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6dbd7e0700 1 --2- 172.21.15.121:0/2376039241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446e0500 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6da800cd40 tx=0x7f6da8011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6937b887edbfb8c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:55.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6da37fe700 1 -- 172.21.15.121:0/2376039241 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da8004260 con 0x55c0446c4640 2024-06-10T00:29:55.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6da37fe700 1 -- 172.21.15.121:0/2376039241 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da8004400 con 0x55c0446c4640 2024-06-10T00:29:55.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.701+0000 7f6da37fe700 1 -- 172.21.15.121:0/2376039241 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da801b980 con 0x55c0446c4640 2024-06-10T00:29:55.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.702+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/2376039241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 msgr2=0x55c0446e0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:55.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.702+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/2376039241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446e0500 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6da800cd40 tx=0x7f6da8011720 comp rx=0 tx=0).stop 2024-06-10T00:29:55.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.702+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/2376039241 shutdown_connections 2024-06-10T00:29:55.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.702+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/2376039241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446c4f60 0x55c0446e0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.702+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/2376039241 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446bef20 0x55c0446be9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.702+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/2376039241 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4640 0x55c0446e0500 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.702+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/2376039241 >> 172.21.15.121:0/2376039241 conn(0x55c0445e9cc0 msgr2=0x55c0446c6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.702+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/2376039241 shutdown_connections 2024-06-10T00:29:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.702+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/2376039241 wait complete. 2024-06-10T00:29:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.702+0000 7f6dcaac8b80 1 -- start start 2024-06-10T00:29:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.702+0000 7f6dcaac8b80 1 Processor -- start 2024-06-10T00:29:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 0x55c0446d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446c4640 0x55c0446dfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dcaac8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446e0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c00d330 con 0x55c0446c4f60 2024-06-10T00:29:55.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c00e7a0 con 0x55c0446c4640 2024-06-10T00:29:55.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dcaac8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6d9c00da00 con 0x55c0446bef20 2024-06-10T00:29:55.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dcaac8b80 10 client.0 _ll_register_callbacks cb 0x55c0445aa920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:55.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbe7e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446e0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbe7e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446e0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42750/0 (socket says 172.21.15.121:42750) 2024-06-10T00:29:55.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/1903103365 learned_addr learned my addr 172.21.15.121:0/1903103365 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:55.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 0x55c0446d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbd7e0700 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446c4640 0x55c0446dfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/1903103365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 msgr2=0x55c0446d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:55.710 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 0x55c0446d14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/1903103365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446c4640 msgr2=0x55c0446dfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446c4640 0x55c0446dfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbe7e2700 1 -- 172.21.15.121:0/1903103365 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0446e0a90 con 0x55c0446c4f60 2024-06-10T00:29:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 0x55c0446d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:29:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6dbe7e2700 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446e0360 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6db00cdf30 tx=0x7f6db0009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c0a303451dbbd5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6da17fa700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.703+0000 7f6da17fa700 1 -- 172.21.15.121:0/1903103365 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db00a0b40 con 0x55c0446c4f60 2024-06-10T00:29:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6da17fa700 1 -- 172.21.15.121:0/1903103365 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f6db0004300 con 0x55c0446c4f60 2024-06-10T00:29:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/1903103365 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6da8004400 con 0x55c0446c4f60 2024-06-10T00:29:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6da17fa700 1 -- 172.21.15.121:0/1903103365 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6db009ce90 con 0x55c0446c4f60 2024-06-10T00:29:55.711 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6dcaac8b80 10 client.15819 fetch_fsmap learned FSMap version 27 2024-06-10T00:29:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/1903103365 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c0446e0a90 con 0x55c0446c4f60 2024-06-10T00:29:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6da17fa700 1 -- 172.21.15.121:0/1903103365 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da8004400 con 0x55c0446c4f60 2024-06-10T00:29:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6da17fa700 1 -- 172.21.15.121:0/1903103365 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f6db00984e0 con 0x55c0446c4f60 2024-06-10T00:29:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6dcaac8b80 10 client.15819 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:29:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6dcaac8b80 10 client.15819 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:29:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6dcaac8b80 20 client.15819 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55c0447436f0 0x55c044745b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6dcaac8b80 4 client.15819 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:29:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/1903103365 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c044618290 con 0x55c0447436f0 2024-06-10T00:29:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.704+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55c0447436f0 0x55c044745b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:55.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6da17fa700 10 client.15819 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:29:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dbdfe1700 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55c0447436f0 0x55c044745b10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=49c86706e0a7af4e server_cookie=9e956d4dce22f0c2 in_seq=0 out_seq=0 2024-06-10T00:29:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6da17fa700 1 -- 172.21.15.121:0/1903103365 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c044618290 con 0x55c0447436f0 2024-06-10T00:29:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6da17fa700 10 client.15819 handle_command_reply: tid=0 2024-06-10T00:29:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 1 client.15819 shutdown 2024-06-10T00:29:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/1903103365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 msgr2=0x55c0446e0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446e0360 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6db00cdf30 tx=0x7f6db0009cd0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/1903103365 shutdown_connections 2024-06-10T00:29:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55c0447436f0 0x55c044745b10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c0446c4f60 0x55c0446e0360 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0446bef20 0x55c0446d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.713 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 1 --2- 172.21.15.121:0/1903103365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55c0446c4640 0x55c0446dfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6da17fa700 0 client.15819 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:29:55.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/1903103365 >> 172.21.15.121:0/1903103365 conn(0x55c0445e9cc0 msgr2=0x55c0446b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:55.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/1903103365 shutdown_connections 2024-06-10T00:29:55.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 1 -- 172.21.15.121:0/1903103365 wait complete. 2024-06-10T00:29:55.714 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.705+0000 7f6dcaac8b80 20 client.15819 trim_cache size 0 max 16384 2024-06-10T00:29:55.714 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:29:55.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 -- 172.21.15.121:0/2161400785 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6da40a2e10 msgr2=0x7f6da40a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:55.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6da40a2e10 0x7f6da40a52d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6db8078a60 tx=0x7f6db40094f0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 -- 172.21.15.121:0/2161400785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db81066b0 msgr2=0x7f6db8197970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:55.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db81066b0 0x7f6db8197970 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6db00059d0 tx=0x7f6db0003730 comp rx=0 tx=0).stop 2024-06-10T00:29:55.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 -- 172.21.15.121:0/2161400785 shutdown_connections 2024-06-10T00:29:55.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6da40a2e10 0x7f6da40a52d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db81066b0 0x7f6db8197970 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db8101620 0x7f6db819d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 --2- 172.21.15.121:0/2161400785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db8101fd0 0x7f6db819dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:55.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 -- 172.21.15.121:0/2161400785 >> 172.21.15.121:0/2161400785 conn(0x7f6db8078360 msgr2=0x7f6db80ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:55.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 -- 172.21.15.121:0/2161400785 shutdown_connections 2024-06-10T00:29:55.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:55.706+0000 7f6dc0245700 1 -- 172.21.15.121:0/2161400785 wait complete. 2024-06-10T00:29:56.517 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-10T00:29:56.758 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:29:56.879 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:56 smithi121 ceph-mon[32378]: pgmap v62: 129 pgs: 129 active+clean; 33 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-06-10T00:29:57.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:56 smithi135 ceph-mon[36424]: pgmap v62: 129 pgs: 129 active+clean; 33 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-06-10T00:29:57.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:56 smithi193 ceph-mon[36716]: pgmap v62: 129 pgs: 129 active+clean; 33 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-06-10T00:29:57.292 INFO:tasks.workunit.client.0.smithi121.stdout: 9 24046 507 11606 324 1023 160 2024-06-10T00:29:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.570+0000 7f946e9b7700 1 -- 172.21.15.121:0/1312210993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9468101440 msgr2=0x7f946810f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.570+0000 7f946e9b7700 1 --2- 172.21.15.121:0/1312210993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9468101440 0x7f946810f330 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f946000b0a0 tx=0x7f946000b3b0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.570+0000 7f946e9b7700 1 -- 172.21.15.121:0/1312210993 shutdown_connections 2024-06-10T00:29:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.570+0000 7f946e9b7700 1 --2- 172.21.15.121:0/1312210993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9468101440 0x7f946810f330 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.570+0000 7f946e9b7700 1 --2- 172.21.15.121:0/1312210993 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9468108510 0x7f94681088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.570+0000 7f946e9b7700 1 --2- 172.21.15.121:0/1312210993 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9468100a80 0x7f9468100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.570+0000 7f946e9b7700 1 -- 172.21.15.121:0/1312210993 >> 172.21.15.121:0/1312210993 conn(0x7f9468077a00 msgr2=0x7f94680fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.570+0000 7f946e9b7700 1 -- 172.21.15.121:0/1312210993 shutdown_connections 2024-06-10T00:29:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.570+0000 7f946e9b7700 1 -- 172.21.15.121:0/1312210993 wait complete. 2024-06-10T00:29:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.570+0000 7f946e9b7700 1 Processor -- start 2024-06-10T00:29:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.570+0000 7f946e9b7700 1 -- start start 2024-06-10T00:29:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f946e9b7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9468100a80 0x7f9468101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f946e9b7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9468101440 0x7f94681021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f946e9b7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9468108510 0x7f94681026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f946e9b7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f94681127f0 con 0x7f9468100a80 2024-06-10T00:29:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f946e9b7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f9468112670 con 0x7f9468108510 2024-06-10T00:29:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f946e9b7700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9468112970 con 0x7f9468101440 2024-06-10T00:29:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f9467fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9468100a80 0x7f9468101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f9467fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9468100a80 0x7f9468101c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42758/0 (socket says 172.21.15.121:42758) 2024-06-10T00:29:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f9467fff700 1 -- 172.21.15.121:0/1808118841 learned_addr learned my addr 172.21.15.121:0/1808118841 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:58.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f94677fe700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9468101440 0x7f94681021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f946cf54700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9468108510 0x7f94681026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f9467fff700 1 -- 172.21.15.121:0/1808118841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9468101440 msgr2=0x7f94681021a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f9467fff700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9468101440 0x7f94681021a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f9467fff700 1 -- 172.21.15.121:0/1808118841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9468108510 msgr2=0x7f94681026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f9467fff700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9468108510 0x7f94681026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f9467fff700 1 -- 172.21.15.121:0/1808118841 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94681a7170 con 0x7f9468100a80 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.571+0000 7f9467fff700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9468100a80 0x7f9468101c60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f945800d950 tx=0x7f945800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.572+0000 7f94657fa700 1 -- 172.21.15.121:0/1808118841 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458016070 con 0x7f9468100a80 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.572+0000 7f946e9b7700 1 -- 172.21.15.121:0/1808118841 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94681a7460 con 0x7f9468100a80 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.572+0000 7f94657fa700 1 -- 172.21.15.121:0/1808118841 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f945800fd70 con 0x7f9468100a80 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.572+0000 7f94657fa700 1 -- 172.21.15.121:0/1808118841 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945800e610 con 0x7f9468100a80 2024-06-10T00:29:58.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.572+0000 7f946e9b7700 1 -- 172.21.15.121:0/1808118841 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94681a79a0 con 0x7f9468100a80 2024-06-10T00:29:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.574+0000 7f94657fa700 1 -- 172.21.15.121:0/1808118841 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9458019070 con 0x7f9468100a80 2024-06-10T00:29:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.575+0000 7f94657fa700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f94500a2d90 0x7f94500a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.575+0000 7f94677fe700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f94500a2d90 0x7f94500a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.575+0000 7f94657fa700 1 -- 172.21.15.121:0/1808118841 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f94580cf2f0 con 0x7f9468100a80 2024-06-10T00:29:58.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.575+0000 7f94677fe700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f94500a2d90 0x7f94500a5250 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9468107b00 tx=0x7f945c009190 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.576+0000 7f947923ab80 1 Processor -- start 2024-06-10T00:29:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.576+0000 7f947923ab80 1 -- start start 2024-06-10T00:29:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.576+0000 7f947923ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5de7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.576+0000 7f947923ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5de19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f947923ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de1f20 0x55bef5de2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f947923ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55bef5d33810 con 0x55bef5de7640 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f947923ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55bef5cdb720 con 0x55bef5de7f60 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f947923ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bef5d2f7d0 con 0x55bef5de1f20 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f9467fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5de7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f9467fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5de7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42770/0 (socket says 172.21.15.121:42770) 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f9467fff700 1 -- 172.21.15.121:0/2405873329 learned_addr learned my addr 172.21.15.121:0/2405873329 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f94677fe700 1 --2- 172.21.15.121:0/2405873329 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5de19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f946cf54700 1 --2- 172.21.15.121:0/2405873329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de1f20 0x55bef5de2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f9467fff700 1 -- 172.21.15.121:0/2405873329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de1f20 msgr2=0x55bef5de2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f9467fff700 1 --2- 172.21.15.121:0/2405873329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de1f20 0x55bef5de2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f9467fff700 1 -- 172.21.15.121:0/2405873329 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 msgr2=0x55bef5de19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f9467fff700 1 --2- 172.21.15.121:0/2405873329 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5de19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f9467fff700 1 -- 172.21.15.121:0/2405873329 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bef5cd1f50 con 0x55bef5de7640 2024-06-10T00:29:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f946cf54700 1 --2- 172.21.15.121:0/2405873329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de1f20 0x55bef5de2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:29:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f9467fff700 1 --2- 172.21.15.121:0/2405873329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5de7a20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f94580a3b60 tx=0x7f945809ae00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bb2010c18001e5d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f944e7fc700 1 -- 172.21.15.121:0/2405873329 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458020070 con 0x55bef5de7640 2024-06-10T00:29:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f944e7fc700 1 -- 172.21.15.121:0/2405873329 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f945809b420 con 0x55bef5de7640 2024-06-10T00:29:58.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.577+0000 7f944e7fc700 1 -- 172.21.15.121:0/2405873329 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945809b680 con 0x55bef5de7640 2024-06-10T00:29:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.578+0000 7f947923ab80 1 -- 172.21.15.121:0/2405873329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 msgr2=0x55bef5de7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.578+0000 7f947923ab80 1 --2- 172.21.15.121:0/2405873329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5de7a20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f94580a3b60 tx=0x7f945809ae00 comp rx=0 tx=0).stop 2024-06-10T00:29:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.578+0000 7f947923ab80 1 -- 172.21.15.121:0/2405873329 shutdown_connections 2024-06-10T00:29:58.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.578+0000 7f947923ab80 1 --2- 172.21.15.121:0/2405873329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de1f20 0x55bef5de2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.578+0000 7f947923ab80 1 --2- 172.21.15.121:0/2405873329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5de7a20 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.578+0000 7f947923ab80 1 --2- 172.21.15.121:0/2405873329 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5de19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.578+0000 7f947923ab80 1 -- 172.21.15.121:0/2405873329 >> 172.21.15.121:0/2405873329 conn(0x55bef5d0ccc0 msgr2=0x55bef5ddc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.578+0000 7f947923ab80 1 -- 172.21.15.121:0/2405873329 shutdown_connections 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.578+0000 7f947923ab80 1 -- 172.21.15.121:0/2405873329 wait complete. 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f947923ab80 1 -- start start 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f947923ab80 1 Processor -- start 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f947923ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de1f20 0x55bef5dedd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f947923ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5deb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f947923ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5deb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f947923ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55bef5dee910 con 0x55bef5de7640 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f947923ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55bef5cd3bf0 con 0x55bef5de7f60 2024-06-10T00:29:58.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f947923ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bef5deea90 con 0x55bef5de1f20 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f947923ab80 10 client.0 _ll_register_callbacks cb 0x55bef5ccd920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f94677fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5deb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f94677fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5deb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42784/0 (socket says 172.21.15.121:42784) 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f94677fe700 1 -- 172.21.15.121:0/3128126047 learned_addr learned my addr 172.21.15.121:0/3128126047 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f946cf54700 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5deb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.579+0000 7f9467fff700 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de1f20 0x55bef5dedd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94677fe700 1 -- 172.21.15.121:0/3128126047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de1f20 msgr2=0x55bef5dedd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94677fe700 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de1f20 0x55bef5dedd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94677fe700 1 -- 172.21.15.121:0/3128126047 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 msgr2=0x55bef5deb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94677fe700 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5deb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94677fe700 1 -- 172.21.15.121:0/3128126047 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bef5c828d0 con 0x55bef5de7640 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94677fe700 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5deb010 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f945c0154b0 tx=0x7f945c004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb9e382772b619e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94477fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:58.584 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94477fe700 1 -- 172.21.15.121:0/3128126047 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c025070 con 0x55bef5de7640 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94477fe700 1 -- 172.21.15.121:0/3128126047 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f945c015b80 con 0x55bef5de7640 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f947923ab80 1 -- 172.21.15.121:0/3128126047 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f945809b420 con 0x55bef5de7640 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94477fe700 1 -- 172.21.15.121:0/3128126047 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f945c01b040 con 0x55bef5de7640 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f947923ab80 10 client.15831 fetch_fsmap learned FSMap version 27 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f947923ab80 1 -- 172.21.15.121:0/3128126047 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bef5df9230 con 0x55bef5de7640 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94477fe700 1 -- 172.21.15.121:0/3128126047 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f945809b420 con 0x55bef5de7640 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.580+0000 7f94477fe700 1 -- 172.21.15.121:0/3128126047 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f945c0066d0 con 0x55bef5de7640 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.581+0000 7f947923ab80 10 client.15831 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.581+0000 7f947923ab80 10 client.15831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.581+0000 7f947923ab80 20 client.15831 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.581+0000 7f947923ab80 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55bef5dfb810 0x55bef5dfdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.581+0000 7f947923ab80 4 client.15831 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.581+0000 7f947923ab80 1 -- 172.21.15.121:0/3128126047 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bef5d40d10 con 0x55bef5dfb810 2024-06-10T00:29:58.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.581+0000 7f9467fff700 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55bef5dfb810 0x55bef5dfdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.581+0000 7f94477fe700 10 client.15831 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.581+0000 7f9467fff700 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55bef5dfb810 0x55bef5dfdc30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=606fbb2f0d16d3c0 server_cookie=981b94c4151bb439 in_seq=0 out_seq=0 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f94477fe700 1 -- 172.21.15.121:0/3128126047 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55bef5d40d10 con 0x55bef5dfb810 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f94477fe700 10 client.15831 handle_command_reply: tid=0 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 1 client.15831 shutdown 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 1 -- 172.21.15.121:0/3128126047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 msgr2=0x55bef5deb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5deb010 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f945c0154b0 tx=0x7f945c004860 comp rx=0 tx=0).stop 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 1 -- 172.21.15.121:0/3128126047 shutdown_connections 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55bef5dfb810 0x55bef5dfdc30 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5deb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de1f20 0x55bef5dedd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 1 --2- 172.21.15.121:0/3128126047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de7640 0x55bef5deb010 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f94477fe700 0 client.15831 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:58.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 1 -- 172.21.15.121:0/3128126047 >> 172.21.15.121:0/3128126047 conn(0x55bef5d0ccc0 msgr2=0x55bef5de09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:58.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 1 -- 172.21.15.121:0/3128126047 shutdown_connections 2024-06-10T00:29:58.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 1 -- 172.21.15.121:0/3128126047 wait complete. 2024-06-10T00:29:58.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.582+0000 7f947923ab80 20 client.15831 trim_cache size 0 max 16384 2024-06-10T00:29:58.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.597+0000 7f947923ab80 1 Processor -- start 2024-06-10T00:29:58.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.597+0000 7f947923ab80 1 -- start start 2024-06-10T00:29:58.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f947923ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5de19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f947923ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de7640 0x55bef5dd6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f947923ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5de8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f947923ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f94681b3110 con 0x55bef5de1f20 2024-06-10T00:29:58.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f947923ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f94681b09a0 con 0x55bef5de7f60 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f947923ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f94681ae7f0 con 0x55bef5de7640 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f9467fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5de19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f9467fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5de19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42792/0 (socket says 172.21.15.121:42792) 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f9467fff700 1 -- 172.21.15.121:0/592771680 learned_addr learned my addr 172.21.15.121:0/592771680 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f94677fe700 1 --2- 172.21.15.121:0/592771680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de7640 0x55bef5dd6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f946cf54700 1 --2- 172.21.15.121:0/592771680 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5de8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f9467fff700 1 -- 172.21.15.121:0/592771680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de7640 msgr2=0x55bef5dd6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f9467fff700 1 --2- 172.21.15.121:0/592771680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de7640 0x55bef5dd6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f9467fff700 1 -- 172.21.15.121:0/592771680 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 msgr2=0x55bef5de8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f9467fff700 1 --2- 172.21.15.121:0/592771680 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5de8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f9467fff700 1 -- 172.21.15.121:0/592771680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bef5dfe290 con 0x55bef5de1f20 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f9467fff700 1 --2- 172.21.15.121:0/592771680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5de19e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9458097880 tx=0x7f9458010b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22f99c75d7754ba server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:58.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f944dffb700 1 -- 172.21.15.121:0/592771680 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9458020040 con 0x55bef5de1f20 2024-06-10T00:29:58.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f944dffb700 1 -- 172.21.15.121:0/592771680 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9458010e10 con 0x55bef5de1f20 2024-06-10T00:29:58.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.598+0000 7f944dffb700 1 -- 172.21.15.121:0/592771680 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94580b4270 con 0x55bef5de1f20 2024-06-10T00:29:58.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.599+0000 7f947923ab80 1 -- 172.21.15.121:0/592771680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 msgr2=0x55bef5de19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:58.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.599+0000 7f947923ab80 1 --2- 172.21.15.121:0/592771680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5de19e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9458097880 tx=0x7f9458010b10 comp rx=0 tx=0).stop 2024-06-10T00:29:58.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.599+0000 7f947923ab80 1 -- 172.21.15.121:0/592771680 shutdown_connections 2024-06-10T00:29:58.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.599+0000 7f947923ab80 1 --2- 172.21.15.121:0/592771680 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5de8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.599+0000 7f947923ab80 1 --2- 172.21.15.121:0/592771680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5de19e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.599+0000 7f947923ab80 1 --2- 172.21.15.121:0/592771680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de7640 0x55bef5dd6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.599+0000 7f947923ab80 1 -- 172.21.15.121:0/592771680 >> 172.21.15.121:0/592771680 conn(0x55bef5d0ccc0 msgr2=0x55bef5de8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:58.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.599+0000 7f947923ab80 1 -- 172.21.15.121:0/592771680 shutdown_connections 2024-06-10T00:29:58.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.599+0000 7f947923ab80 1 -- 172.21.15.121:0/592771680 wait complete. 2024-06-10T00:29:58.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.599+0000 7f947923ab80 1 -- start start 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f947923ab80 1 Processor -- start 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f947923ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5df3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f947923ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de7640 0x55bef5df4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f947923ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5e028d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f947923ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f94681aee60 con 0x55bef5de1f20 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f947923ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f94681b02d0 con 0x55bef5de7f60 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f947923ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f94681af530 con 0x55bef5de7640 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f947923ab80 10 client.0 _ll_register_callbacks cb 0x55bef5ccd920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f9467fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5df3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f9467fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5df3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42794/0 (socket says 172.21.15.121:42794) 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f9467fff700 1 -- 172.21.15.121:0/1499883999 learned_addr learned my addr 172.21.15.121:0/1499883999 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f94677fe700 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de7640 0x55bef5df4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f9467fff700 1 -- 172.21.15.121:0/1499883999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de7640 msgr2=0x55bef5df4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f9467fff700 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de7640 0x55bef5df4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f9467fff700 1 -- 172.21.15.121:0/1499883999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 msgr2=0x55bef5e028d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f946cf54700 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5e028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f9467fff700 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5e028d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f9467fff700 1 -- 172.21.15.121:0/1499883999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bef5e03020 con 0x55bef5de1f20 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f9467fff700 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5df3f00 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f945809b360 tx=0x7f9458095f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9fe1bf44bc604be server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f94477fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.600+0000 7f94477fe700 1 -- 172.21.15.121:0/1499883999 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94580b9060 con 0x55bef5de1f20 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f94477fe700 1 -- 172.21.15.121:0/1499883999 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f9458010710 con 0x55bef5de1f20 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f947923ab80 1 -- 172.21.15.121:0/1499883999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9458010e10 con 0x55bef5de1f20 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f94477fe700 1 -- 172.21.15.121:0/1499883999 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9458020070 con 0x55bef5de1f20 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f947923ab80 10 client.15837 fetch_fsmap learned FSMap version 27 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f947923ab80 1 -- 172.21.15.121:0/1499883999 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bef5e65eb0 con 0x55bef5de1f20 2024-06-10T00:29:58.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f94477fe700 1 -- 172.21.15.121:0/1499883999 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9458010e10 con 0x55bef5de1f20 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f94477fe700 1 -- 172.21.15.121:0/1499883999 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f945809a870 con 0x55bef5de1f20 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f947923ab80 10 client.15837 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f947923ab80 10 client.15837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f947923ab80 20 client.15837 populate_metadata read hostname 'smithi121' 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f947923ab80 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55bef5e66130 0x55bef5e68520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f947923ab80 4 client.15837 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f947923ab80 1 -- 172.21.15.121:0/1499883999 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bef5d3b290 con 0x55bef5e66130 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.601+0000 7f94677fe700 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55bef5e66130 0x55bef5e68520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f94677fe700 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55bef5e66130 0x55bef5e68520 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1b9d2627663020f server_cookie=94e29b82ff176121 in_seq=0 out_seq=0 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f94477fe700 10 client.15837 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f94477fe700 1 -- 172.21.15.121:0/1499883999 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bef5d3b290 con 0x55bef5e66130 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f94477fe700 10 client.15837 handle_command_reply: tid=0 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 1 client.15837 shutdown 2024-06-10T00:29:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 1 -- 172.21.15.121:0/1499883999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 msgr2=0x55bef5df3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5df3f00 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f945809b360 tx=0x7f9458095f90 comp rx=0 tx=0).stop 2024-06-10T00:29:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 1 -- 172.21.15.121:0/1499883999 shutdown_connections 2024-06-10T00:29:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55bef5e66130 0x55bef5e68520 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55bef5de7f60 0x55bef5e028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55bef5de1f20 0x55bef5df3f00 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 1 --2- 172.21.15.121:0/1499883999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bef5de7640 0x55bef5df4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f94477fe700 0 client.15837 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:29:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 1 -- 172.21.15.121:0/1499883999 >> 172.21.15.121:0/1499883999 conn(0x55bef5d0ccc0 msgr2=0x55bef5de9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 1 -- 172.21.15.121:0/1499883999 shutdown_connections 2024-06-10T00:29:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 1 -- 172.21.15.121:0/1499883999 wait complete. 2024-06-10T00:29:58.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.602+0000 7f947923ab80 20 client.15837 trim_cache size 0 max 16384 2024-06-10T00:29:58.610 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:29:58.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 -- 172.21.15.121:0/1808118841 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f94500a2d90 msgr2=0x7f94500a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:58.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f94500a2d90 0x7f94500a5250 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9468107b00 tx=0x7f945c009190 comp rx=0 tx=0).stop 2024-06-10T00:29:58.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 -- 172.21.15.121:0/1808118841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9468100a80 msgr2=0x7f9468101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:29:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9468100a80 0x7f9468101c60 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f945800d950 tx=0x7f945800dc60 comp rx=0 tx=0).stop 2024-06-10T00:29:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 -- 172.21.15.121:0/1808118841 shutdown_connections 2024-06-10T00:29:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f94500a2d90 0x7f94500a5250 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9468108510 0x7f94681026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9468100a80 0x7f9468101c60 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 --2- 172.21.15.121:0/1808118841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9468101440 0x7f94681021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:29:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 -- 172.21.15.121:0/1808118841 >> 172.21.15.121:0/1808118841 conn(0x7f9468077a00 msgr2=0x7f94680fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:29:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 -- 172.21.15.121:0/1808118841 shutdown_connections 2024-06-10T00:29:58.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:29:58.603+0000 7f946e9b7700 1 -- 172.21.15.121:0/1808118841 wait complete. 2024-06-10T00:29:58.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:29:58 smithi121 ceph-mon[32378]: pgmap v63: 129 pgs: 129 active+clean; 33 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-06-10T00:29:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:29:58 smithi135 ceph-mon[36424]: pgmap v63: 129 pgs: 129 active+clean; 33 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-06-10T00:29:59.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:29:58 smithi193 ceph-mon[36716]: pgmap v63: 129 pgs: 129 active+clean; 33 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 2.6 MiB/s wr, 216 op/s 2024-06-10T00:30:00.425 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-10T00:30:00.426 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-10T00:30:00.683 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:30:01.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:00 smithi135 ceph-mon[36424]: pgmap v64: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 2.6 MiB/s wr, 223 op/s 2024-06-10T00:30:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:00 smithi135 ceph-mon[36424]: overall HEALTH_OK 2024-06-10T00:30:01.071 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:00 smithi121 ceph-mon[32378]: pgmap v64: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 2.6 MiB/s wr, 223 op/s 2024-06-10T00:30:01.071 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:00 smithi121 ceph-mon[32378]: overall HEALTH_OK 2024-06-10T00:30:01.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:00 smithi193 ceph-mon[36716]: pgmap v64: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 2.6 MiB/s wr, 223 op/s 2024-06-10T00:30:01.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:00 smithi193 ceph-mon[36716]: overall HEALTH_OK 2024-06-10T00:30:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.653+0000 7f543c735700 1 -- 172.21.15.121:0/1830849954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5434101390 msgr2=0x7f54341017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.653+0000 7f543c735700 1 --2- 172.21.15.121:0/1830849954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5434101390 0x7f54341017f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f5424009990 tx=0x7f5424009ca0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.653+0000 7f543c735700 1 -- 172.21.15.121:0/1830849954 shutdown_connections 2024-06-10T00:30:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.653+0000 7f543c735700 1 --2- 172.21.15.121:0/1830849954 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5434101d30 0x7f543410e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.653+0000 7f543c735700 1 --2- 172.21.15.121:0/1830849954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54341073d0 0x7f54341077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.653+0000 7f543c735700 1 --2- 172.21.15.121:0/1830849954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5434101390 0x7f54341017f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.653+0000 7f543c735700 1 -- 172.21.15.121:0/1830849954 >> 172.21.15.121:0/1830849954 conn(0x7f54340782c0 msgr2=0x7f54340786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.653+0000 7f543c735700 1 -- 172.21.15.121:0/1830849954 shutdown_connections 2024-06-10T00:30:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.653+0000 7f543c735700 1 -- 172.21.15.121:0/1830849954 wait complete. 2024-06-10T00:30:02.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.653+0000 7f543c735700 1 Processor -- start 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.653+0000 7f543c735700 1 -- start start 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543c735700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5434101390 0x7f543419d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543c735700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5434101d30 0x7f543419dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543c735700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54341073d0 0x7f54341979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543c735700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5434111630 con 0x7f5434101390 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543c735700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f54341114b0 con 0x7f5434101d30 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543c735700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f54341117b0 con 0x7f54341073d0 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543a4d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5434101390 0x7f543419d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543a4d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5434101390 0x7f543419d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38404/0 (socket says 172.21.15.121:38404) 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543a4d1700 1 -- 172.21.15.121:0/4019490510 learned_addr learned my addr 172.21.15.121:0/4019490510 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f5439cd0700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5434101d30 0x7f543419dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543acd2700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54341073d0 0x7f54341979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543a4d1700 1 -- 172.21.15.121:0/4019490510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54341073d0 msgr2=0x7f54341979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543a4d1700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54341073d0 0x7f54341979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543a4d1700 1 -- 172.21.15.121:0/4019490510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5434101d30 msgr2=0x7f543419dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543a4d1700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5434101d30 0x7f543419dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543a4d1700 1 -- 172.21.15.121:0/4019490510 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5434198110 con 0x7f5434101390 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543acd2700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54341073d0 0x7f54341979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543a4d1700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5434101390 0x7f543419d7f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f543000c920 tx=0x7f543000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f542b7fe700 1 -- 172.21.15.121:0/4019490510 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5430007b00 con 0x7f5434101390 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543c735700 1 -- 172.21.15.121:0/4019490510 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5434198400 con 0x7f5434101390 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f542b7fe700 1 -- 172.21.15.121:0/4019490510 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5430004d10 con 0x7f5434101390 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f542b7fe700 1 -- 172.21.15.121:0/4019490510 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5430005630 con 0x7f5434101390 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.654+0000 7f543c735700 1 -- 172.21.15.121:0/4019490510 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54341a46c0 con 0x7f5434101390 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.657+0000 7f542b7fe700 1 -- 172.21.15.121:0/4019490510 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5430004750 con 0x7f5434101390 2024-06-10T00:30:02.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.657+0000 7f542b7fe700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f54200a2d90 0x7f54200a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.657+0000 7f5439cd0700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f54200a2d90 0x7f54200a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.657+0000 7f542b7fe700 1 -- 172.21.15.121:0/4019490510 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f54300cf2d0 con 0x7f5434101390 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.658+0000 7f5439cd0700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f54200a2d90 0x7f54200a5250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f542400b5c0 tx=0x7f5424005960 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.658+0000 7f5446fb8b80 1 Processor -- start 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.658+0000 7f5446fb8b80 1 -- start start 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.658+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c7ffe0 0x55e494c803c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.658+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c80900 0x55e494c7a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.658+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.658+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e494b6cfe0 con 0x55e494c7a820 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.658+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55e494bcb8d0 con 0x55e494c7ffe0 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.658+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e494b334e0 con 0x55e494c80900 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.658+0000 7f543acd2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f543acd2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38408/0 (socket says 172.21.15.121:38408) 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f543acd2700 1 -- 172.21.15.121:0/2670249851 learned_addr learned my addr 172.21.15.121:0/2670249851 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f543a4d1700 1 --2- 172.21.15.121:0/2670249851 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c7ffe0 0x55e494c803c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f5439cd0700 1 --2- 172.21.15.121:0/2670249851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c80900 0x55e494c7a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f543acd2700 1 -- 172.21.15.121:0/2670249851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c80900 msgr2=0x55e494c7a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f543acd2700 1 --2- 172.21.15.121:0/2670249851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c80900 0x55e494c7a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f543acd2700 1 -- 172.21.15.121:0/2670249851 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c7ffe0 msgr2=0x55e494c803c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f543acd2700 1 --2- 172.21.15.121:0/2670249851 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c7ffe0 0x55e494c803c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f543acd2700 1 -- 172.21.15.121:0/2670249851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e494b1a950 con 0x55e494c7a820 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f543acd2700 1 --2- 172.21.15.121:0/2670249851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7ac40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f542c0087c0 tx=0x7f542c008ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1876bc39d4cf44d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f5428ff9700 1 -- 172.21.15.121:0/2670249851 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c003c00 con 0x55e494c7a820 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f5428ff9700 1 -- 172.21.15.121:0/2670249851 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f542c003da0 con 0x55e494c7a820 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.659+0000 7f5428ff9700 1 -- 172.21.15.121:0/2670249851 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c012a30 con 0x55e494c7a820 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.660+0000 7f5446fb8b80 1 -- 172.21.15.121:0/2670249851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 msgr2=0x55e494c7ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:02.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.660+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/2670249851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7ac40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f542c0087c0 tx=0x7f542c008ad0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.660+0000 7f5446fb8b80 1 -- 172.21.15.121:0/2670249851 shutdown_connections 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.660+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/2670249851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7ac40 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.660+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/2670249851 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c7ffe0 0x55e494c803c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.660+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/2670249851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c80900 0x55e494c7a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.660+0000 7f5446fb8b80 1 -- 172.21.15.121:0/2670249851 >> 172.21.15.121:0/2670249851 conn(0x55e494ba7100 msgr2=0x55e494c752a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.660+0000 7f5446fb8b80 1 -- 172.21.15.121:0/2670249851 shutdown_connections 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.660+0000 7f5446fb8b80 1 -- 172.21.15.121:0/2670249851 wait complete. 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.660+0000 7f5446fb8b80 1 -- start start 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.660+0000 7f5446fb8b80 1 Processor -- start 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c7a820 0x55e494c86680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c839d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c80900 0x55e494c83f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e494b6c040 con 0x55e494c80900 2024-06-10T00:30:02.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55e494bc7890 con 0x55e494c7a820 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e494c872e0 con 0x55e494c7ffe0 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f5446fb8b80 10 client.0 _ll_register_callbacks cb 0x55e494b61630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f543acd2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c80900 0x55e494c83f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f543acd2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c80900 0x55e494c83f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38416/0 (socket says 172.21.15.121:38416) 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f543acd2700 1 -- 172.21.15.121:0/3590624089 learned_addr learned my addr 172.21.15.121:0/3590624089 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f5439cd0700 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c839d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f543a4d1700 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c7a820 0x55e494c86680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f543acd2700 1 -- 172.21.15.121:0/3590624089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 msgr2=0x55e494c839d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f543acd2700 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c839d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f543acd2700 1 -- 172.21.15.121:0/3590624089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c7a820 msgr2=0x55e494c86680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f543acd2700 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c7a820 0x55e494c86680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f543acd2700 1 -- 172.21.15.121:0/3590624089 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e494b917d0 con 0x55e494c80900 2024-06-10T00:30:02.670 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f5439cd0700 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c839d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f543a4d1700 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c7a820 0x55e494c86680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f543acd2700 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c80900 0x55e494c83f10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f542c008790 tx=0x7f542c004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a7eaa9c7a19690 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f541dffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f541dffb700 1 -- 172.21.15.121:0/3590624089 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c004720 con 0x55e494c80900 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f541dffb700 1 -- 172.21.15.121:0/3590624089 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f542c0126b0 con 0x55e494c80900 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.661+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3590624089 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f542c003da0 con 0x55e494c80900 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f541dffb700 1 -- 172.21.15.121:0/3590624089 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f542c02b050 con 0x55e494c80900 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f5446fb8b80 10 client.15849 fetch_fsmap learned FSMap version 27 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3590624089 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e494c91a70 con 0x55e494c80900 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f541dffb700 1 -- 172.21.15.121:0/3590624089 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f542c003da0 con 0x55e494c80900 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f541dffb700 1 -- 172.21.15.121:0/3590624089 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f542c013070 con 0x55e494c80900 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f5446fb8b80 10 client.15849 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f5446fb8b80 10 client.15849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:30:02.671 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f5446fb8b80 20 client.15849 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:02.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e494c93ff0 0x55e494c963e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f5446fb8b80 4 client.15849 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:30:02.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3590624089 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e494bd8dd0 con 0x55e494c93ff0 2024-06-10T00:30:02.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.662+0000 7f543a4d1700 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e494c93ff0 0x55e494c963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.663+0000 7f543a4d1700 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e494c93ff0 0x55e494c963e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2117e176c9843bb server_cookie=9196ac6a2fcbbdd in_seq=0 out_seq=0 2024-06-10T00:30:02.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.663+0000 7f541dffb700 10 client.15849 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:02.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.663+0000 7f541dffb700 1 -- 172.21.15.121:0/3590624089 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55e494bd8dd0 con 0x55e494c93ff0 2024-06-10T00:30:02.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.663+0000 7f541dffb700 10 client.15849 handle_command_reply: tid=0 2024-06-10T00:30:02.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 1 client.15849 shutdown 2024-06-10T00:30:02.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3590624089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c80900 msgr2=0x55e494c83f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:02.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c80900 0x55e494c83f10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f542c008790 tx=0x7f542c004230 comp rx=0 tx=0).stop 2024-06-10T00:30:02.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3590624089 shutdown_connections 2024-06-10T00:30:02.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e494c93ff0 0x55e494c963e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.673 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c80900 0x55e494c83f10 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c7a820 0x55e494c86680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3590624089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c839d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3590624089 >> 172.21.15.121:0/3590624089 conn(0x55e494ba7100 msgr2=0x55e494c74780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:02.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f541dffb700 0 client.15849 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:02.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3590624089 shutdown_connections 2024-06-10T00:30:02.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3590624089 wait complete. 2024-06-10T00:30:02.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.664+0000 7f5446fb8b80 20 client.15849 trim_cache size 0 max 16384 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.684+0000 7f5446fb8b80 1 Processor -- start 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.684+0000 7f5446fb8b80 1 -- start start 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.684+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.684+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c81570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.684+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c80900 0x55e494c84000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.684+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f54341afe30 con 0x55e494c7a820 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.684+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f54341ad6c0 con 0x55e494c80900 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.684+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f54341ab510 con 0x55e494c7ffe0 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.684+0000 7f543a4d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.684+0000 7f543a4d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38420/0 (socket says 172.21.15.121:38420) 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.684+0000 7f543a4d1700 1 -- 172.21.15.121:0/1897978516 learned_addr learned my addr 172.21.15.121:0/1897978516 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:02.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f543acd2700 1 --2- 172.21.15.121:0/1897978516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c80900 0x55e494c84000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f543a4d1700 1 -- 172.21.15.121:0/1897978516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 msgr2=0x55e494c81570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:02.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f543a4d1700 1 --2- 172.21.15.121:0/1897978516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c81570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f543a4d1700 1 -- 172.21.15.121:0/1897978516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c80900 msgr2=0x55e494c84000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:02.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f543a4d1700 1 --2- 172.21.15.121:0/1897978516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c80900 0x55e494c84000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f543a4d1700 1 -- 172.21.15.121:0/1897978516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e494c7b1a0 con 0x55e494c7a820 2024-06-10T00:30:02.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f543a4d1700 1 --2- 172.21.15.121:0/1897978516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7a2e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f54300978c0 tx=0x7f54300ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd11d47dc6e7bc25 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:02.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f541ffff700 1 -- 172.21.15.121:0/1897978516 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543001d040 con 0x55e494c7a820 2024-06-10T00:30:02.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f541ffff700 1 -- 172.21.15.121:0/1897978516 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f543009bb60 con 0x55e494c7a820 2024-06-10T00:30:02.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f541ffff700 1 -- 172.21.15.121:0/1897978516 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5430007ca0 con 0x55e494c7a820 2024-06-10T00:30:02.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f5446fb8b80 1 -- 172.21.15.121:0/1897978516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 msgr2=0x55e494c7a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:02.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.685+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/1897978516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7a2e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f54300978c0 tx=0x7f54300ce040 comp rx=0 tx=0).stop 2024-06-10T00:30:02.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 -- 172.21.15.121:0/1897978516 shutdown_connections 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/1897978516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c80900 0x55e494c84000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/1897978516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c7a2e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/1897978516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c81570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 -- 172.21.15.121:0/1897978516 >> 172.21.15.121:0/1897978516 conn(0x55e494ba7100 msgr2=0x55e494c851b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 -- 172.21.15.121:0/1897978516 shutdown_connections 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 -- 172.21.15.121:0/1897978516 wait complete. 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 -- start start 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 Processor -- start 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c8cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c89570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c80900 0x55e494c89ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f54341abb80 con 0x55e494c7a820 2024-06-10T00:30:02.692 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f54341acff0 con 0x55e494c80900 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f54341ac250 con 0x55e494c7ffe0 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f5446fb8b80 10 client.0 _ll_register_callbacks cb 0x55e494b61630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f543a4d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c8cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f543a4d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c8cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38422/0 (socket says 172.21.15.121:38422) 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f543a4d1700 1 -- 172.21.15.121:0/3270524225 learned_addr learned my addr 172.21.15.121:0/3270524225 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.686+0000 7f543acd2700 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c80900 0x55e494c89ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f543a4d1700 1 -- 172.21.15.121:0/3270524225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 msgr2=0x55e494c89570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f5439cd0700 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c89570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f543a4d1700 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c89570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f543a4d1700 1 -- 172.21.15.121:0/3270524225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c80900 msgr2=0x55e494c89ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f543a4d1700 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c80900 0x55e494c89ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f543a4d1700 1 -- 172.21.15.121:0/3270524225 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e494c8a2f0 con 0x55e494c7a820 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f543a4d1700 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c8cc50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f54300cc740 tx=0x7f543009f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96475977f394fa6 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:02.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f541dffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f541dffb700 1 -- 172.21.15.121:0/3270524225 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54300b8020 con 0x55e494c7a820 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f541dffb700 1 -- 172.21.15.121:0/3270524225 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f54300b4040 con 0x55e494c7a820 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3270524225 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f543009bb60 con 0x55e494c7a820 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f541dffb700 1 -- 172.21.15.121:0/3270524225 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f543001d070 con 0x55e494c7a820 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f5446fb8b80 10 client.15855 fetch_fsmap learned FSMap version 27 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3270524225 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e494cff990 con 0x55e494c7a820 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f541dffb700 1 -- 172.21.15.121:0/3270524225 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f543009bb60 con 0x55e494c7a820 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f541dffb700 1 -- 172.21.15.121:0/3270524225 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f54300b4240 con 0x55e494c7a820 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f5446fb8b80 10 client.15855 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f5446fb8b80 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f5446fb8b80 20 client.15855 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e494cfffc0 0x55e494d023e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:02.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f5446fb8b80 4 client.15855 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.687+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3270524225 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e494bd3350 con 0x55e494cfffc0 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.688+0000 7f5439cd0700 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e494cfffc0 0x55e494d023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.688+0000 7f5439cd0700 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e494cfffc0 0x55e494d023e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bca2a5adc1d1d0d server_cookie=56a5563fa6f221d in_seq=0 out_seq=0 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.688+0000 7f541dffb700 10 client.15855 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.689+0000 7f541dffb700 1 -- 172.21.15.121:0/3270524225 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e494bd3350 con 0x55e494cfffc0 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.689+0000 7f541dffb700 10 client.15855 handle_command_reply: tid=0 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.689+0000 7f5446fb8b80 1 client.15855 shutdown 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.689+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3270524225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 msgr2=0x55e494c8cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.689+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c8cc50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f54300cc740 tx=0x7f543009f810 comp rx=0 tx=0).stop 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3270524225 shutdown_connections 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55e494cfffc0 0x55e494d023e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55e494c80900 0x55e494c89ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f541dffb700 0 client.15855 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:02.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e494c7a820 0x55e494c8cc50 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f5446fb8b80 1 --2- 172.21.15.121:0/3270524225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e494c7ffe0 0x55e494c89570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3270524225 >> 172.21.15.121:0/3270524225 conn(0x55e494ba7100 msgr2=0x55e494c7f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:02.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3270524225 shutdown_connections 2024-06-10T00:30:02.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f5446fb8b80 1 -- 172.21.15.121:0/3270524225 wait complete. 2024-06-10T00:30:02.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f5446fb8b80 20 client.15855 trim_cache size 0 max 16384 2024-06-10T00:30:02.696 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:30:02.696 INFO:teuthology.orchestra.run.smithi121.stdout: "return_code": 0, 2024-06-10T00:30:02.697 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_tag": "e28d93fb-0300-4377-a6fc-edd35ebad086", 2024-06-10T00:30:02.697 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": "asynchronous" 2024-06-10T00:30:02.697 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:30:02.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f543c735700 1 -- 172.21.15.121:0/4019490510 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f54200a2d90 msgr2=0x7f54200a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:02.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f543c735700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f54200a2d90 0x7f54200a5250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f542400b5c0 tx=0x7f5424005960 comp rx=0 tx=0).stop 2024-06-10T00:30:02.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f543c735700 1 -- 172.21.15.121:0/4019490510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5434101390 msgr2=0x7f543419d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:02.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.690+0000 7f543c735700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5434101390 0x7f543419d7f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f543000c920 tx=0x7f543000cc30 comp rx=0 tx=0).stop 2024-06-10T00:30:02.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.691+0000 7f543c735700 1 -- 172.21.15.121:0/4019490510 shutdown_connections 2024-06-10T00:30:02.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.691+0000 7f543c735700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f54200a2d90 0x7f54200a5250 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.691+0000 7f543c735700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54341073d0 0x7f54341979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.691+0000 7f543c735700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5434101390 0x7f543419d7f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.691+0000 7f543c735700 1 --2- 172.21.15.121:0/4019490510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5434101d30 0x7f543419dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:02.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.691+0000 7f543c735700 1 -- 172.21.15.121:0/4019490510 >> 172.21.15.121:0/4019490510 conn(0x7f54340782c0 msgr2=0x7f54341001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:02.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.691+0000 7f543c735700 1 -- 172.21.15.121:0/4019490510 shutdown_connections 2024-06-10T00:30:02.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:02.691+0000 7f543c735700 1 -- 172.21.15.121:0/4019490510 wait complete. 2024-06-10T00:30:02.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:02 smithi121 ceph-mon[32378]: pgmap v65: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 2.3 MiB/s wr, 187 op/s 2024-06-10T00:30:03.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:02 smithi135 ceph-mon[36424]: pgmap v65: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 2.3 MiB/s wr, 187 op/s 2024-06-10T00:30:03.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:02 smithi193 ceph-mon[36716]: pgmap v65: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 2.3 MiB/s wr, 187 op/s 2024-06-10T00:30:03.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:30:03.734 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:30:04.002 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:03 smithi121 ceph-mon[32378]: scrub queued for path: / 2024-06-10T00:30:04.002 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:03 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:04.002 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:03 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:04.003 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:03 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:04.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:03 smithi135 ceph-mon[36424]: scrub queued for path: / 2024-06-10T00:30:04.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:03 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:04.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:03 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:04.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:03 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:04.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:03 smithi193 ceph-mon[36716]: scrub queued for path: / 2024-06-10T00:30:04.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:03 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:04.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:03 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:04.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:03 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:04.911 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:04 smithi121 ceph-mon[32378]: pgmap v66: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-06-10T00:30:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:04 smithi135 ceph-mon[36424]: pgmap v66: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-06-10T00:30:05.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:04 smithi193 ceph-mon[36716]: pgmap v66: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 2.3 MiB/s wr, 206 op/s 2024-06-10T00:30:05.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.435+0000 7f16ad1cd700 1 -- 172.21.15.121:0/384216368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16a806a170 msgr2=0x7f16a806a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:05.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.435+0000 7f16ad1cd700 1 --2- 172.21.15.121:0/384216368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16a806a170 0x7f16a806a5f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f1690009990 tx=0x7f1690009ca0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.435+0000 7f16ad1cd700 1 -- 172.21.15.121:0/384216368 shutdown_connections 2024-06-10T00:30:05.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.435+0000 7f16ad1cd700 1 --2- 172.21.15.121:0/384216368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16a806ab30 0x7f16a810f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.435+0000 7f16ad1cd700 1 --2- 172.21.15.121:0/384216368 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16a8108500 0x7f16a81088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.435+0000 7f16ad1cd700 1 --2- 172.21.15.121:0/384216368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16a806a170 0x7f16a806a5f0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.435+0000 7f16ad1cd700 1 -- 172.21.15.121:0/384216368 >> 172.21.15.121:0/384216368 conn(0x7f16a8078340 msgr2=0x7f16a8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:05.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.435+0000 7f16ad1cd700 1 -- 172.21.15.121:0/384216368 shutdown_connections 2024-06-10T00:30:05.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.435+0000 7f16ad1cd700 1 -- 172.21.15.121:0/384216368 wait complete. 2024-06-10T00:30:05.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16ad1cd700 1 Processor -- start 2024-06-10T00:30:05.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16ad1cd700 1 -- start start 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16ad1cd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16a806a170 0x7f16a819e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16ad1cd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16a806ab30 0x7f16a819ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16ad1cd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16a8108500 0x7f16a8198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16ad1cd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f16a8112780 con 0x7f16a806a170 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16ad1cd700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f16a8112600 con 0x7f16a8108500 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16ad1cd700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f16a8112900 con 0x7f16a806ab30 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16a6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16a806a170 0x7f16a819e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16a6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16a806a170 0x7f16a819e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38432/0 (socket says 172.21.15.121:38432) 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16a6d9d700 1 -- 172.21.15.121:0/4250282968 learned_addr learned my addr 172.21.15.121:0/4250282968 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16a659c700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16a806ab30 0x7f16a819ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.436+0000 7f16a759e700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16a8108500 0x7f16a8198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.437+0000 7f16a6d9d700 1 -- 172.21.15.121:0/4250282968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16a806ab30 msgr2=0x7f16a819ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:05.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.437+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16a806ab30 0x7f16a819ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.437+0000 7f16a6d9d700 1 -- 172.21.15.121:0/4250282968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16a8108500 msgr2=0x7f16a8198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.437+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16a8108500 0x7f16a8198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.437+0000 7f16a6d9d700 1 -- 172.21.15.121:0/4250282968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a8199200 con 0x7f16a806a170 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.437+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16a806a170 0x7f16a819e8e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f16a00060b0 tx=0x7f16a000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.437+0000 7f169ffff700 1 -- 172.21.15.121:0/4250282968 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a001a070 con 0x7f16a806a170 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.437+0000 7f16ad1cd700 1 -- 172.21.15.121:0/4250282968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a81994f0 con 0x7f16a806a170 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.437+0000 7f169ffff700 1 -- 172.21.15.121:0/4250282968 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a000dd20 con 0x7f16a806a170 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.437+0000 7f169ffff700 1 -- 172.21.15.121:0/4250282968 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0014720 con 0x7f16a806a170 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.437+0000 7f16ad1cd700 1 -- 172.21.15.121:0/4250282968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a81a57b0 con 0x7f16a806a170 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.439+0000 7f169ffff700 1 -- 172.21.15.121:0/4250282968 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f16a001d070 con 0x7f16a806a170 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.440+0000 7f169ffff700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f16940a2d90 0x7f16940a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.440+0000 7f16a659c700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f16940a2d90 0x7f16940a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.440+0000 7f169ffff700 1 -- 172.21.15.121:0/4250282968 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f16a00cef60 con 0x7f16a806a170 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.440+0000 7f16a659c700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f16940a2d90 0x7f16940a5250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f16a80789f0 tx=0x7f169000b560 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.440+0000 7f16b7a50b80 1 Processor -- start 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.440+0000 7f16b7a50b80 1 -- start start 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 0x557443369a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x5574433639e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x557443364340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5574432b5810 con 0x557443369f60 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55744325d720 con 0x557443369640 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5574432b17d0 con 0x557443363f20 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16a659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x5574433639e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16a659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x5574433639e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38448/0 (socket says 172.21.15.121:38448) 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16a659c700 1 -- 172.21.15.121:0/3218699287 learned_addr learned my addr 172.21.15.121:0/3218699287 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:05.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/3218699287 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 0x557443369a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16a759e700 1 --2- 172.21.15.121:0/3218699287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x557443364340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16a6d9d700 1 -- 172.21.15.121:0/3218699287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 msgr2=0x557443364340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/3218699287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x557443364340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16a6d9d700 1 -- 172.21.15.121:0/3218699287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 msgr2=0x5574433639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/3218699287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x5574433639e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16a6d9d700 1 -- 172.21.15.121:0/3218699287 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557443253f50 con 0x557443369640 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.441+0000 7f16a659c700 1 --2- 172.21.15.121:0/3218699287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x5574433639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.442+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/3218699287 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 0x557443369a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f16a00cc320 tx=0x7f16a009acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50e574cbcd19808c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.442+0000 7f169d7fa700 1 -- 172.21.15.121:0/3218699287 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a009b0f0 con 0x557443369640 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.442+0000 7f169d7fa700 1 -- 172.21.15.121:0/3218699287 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a009f4a0 con 0x557443369640 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.442+0000 7f169d7fa700 1 -- 172.21.15.121:0/3218699287 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0097760 con 0x557443369640 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.442+0000 7f16b7a50b80 1 -- 172.21.15.121:0/3218699287 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 msgr2=0x557443369a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.442+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/3218699287 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 0x557443369a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f16a00cc320 tx=0x7f16a009acb0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 -- 172.21.15.121:0/3218699287 shutdown_connections 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/3218699287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x557443364340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/3218699287 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 0x557443369a20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/3218699287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x5574433639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 -- 172.21.15.121:0/3218699287 >> 172.21.15.121:0/3218699287 conn(0x55744328ecc0 msgr2=0x55744335e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 -- 172.21.15.121:0/3218699287 shutdown_connections 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 -- 172.21.15.121:0/3218699287 wait complete. 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 -- start start 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 Processor -- start 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443363f20 0x55744336fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443369640 0x55744336d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x55744336d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5574433709d0 con 0x557443369f60 2024-06-10T00:30:05.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x557443255bf0 con 0x557443363f20 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557443370b50 con 0x557443369640 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16b7a50b80 10 client.0 _ll_register_callbacks cb 0x55744324f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16a759e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x55744336d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16a759e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x55744336d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38460/0 (socket says 172.21.15.121:38460) 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16a759e700 1 -- 172.21.15.121:0/1150498811 learned_addr learned my addr 172.21.15.121:0/1150498811 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.443+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443363f20 0x55744336fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f16a659c700 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443369640 0x55744336d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f16a759e700 1 -- 172.21.15.121:0/1150498811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443369640 msgr2=0x55744336d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f16a759e700 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443369640 0x55744336d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f16a759e700 1 -- 172.21.15.121:0/1150498811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443363f20 msgr2=0x55744336fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f16a759e700 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443363f20 0x55744336fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f16a759e700 1 -- 172.21.15.121:0/1150498811 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574432048d0 con 0x557443369f60 2024-06-10T00:30:05.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443363f20 0x55744336fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f16a759e700 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x55744336d570 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1698003930 tx=0x7f1698003c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd65cc1c76fde60 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f168e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f168e7fc700 1 -- 172.21.15.121:0/1150498811 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169800bae0 con 0x557443369f60 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f168e7fc700 1 -- 172.21.15.121:0/1150498811 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f169800bc80 con 0x557443369f60 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f16b7a50b80 1 -- 172.21.15.121:0/1150498811 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16a009f4a0 con 0x557443369f60 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f168e7fc700 1 -- 172.21.15.121:0/1150498811 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1698004830 con 0x557443369f60 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f16b7a50b80 10 client.15867 fetch_fsmap learned FSMap version 27 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f16b7a50b80 1 -- 172.21.15.121:0/1150498811 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55744337b360 con 0x557443369f60 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.444+0000 7f168e7fc700 1 -- 172.21.15.121:0/1150498811 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a009f4a0 con 0x557443369f60 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.445+0000 7f168e7fc700 1 -- 172.21.15.121:0/1150498811 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f1698011440 con 0x557443369f60 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.445+0000 7f16b7a50b80 10 client.15867 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:30:05.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.445+0000 7f16b7a50b80 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.445+0000 7f16b7a50b80 20 client.15867 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.445+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55744337d9f0 0x55744337fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.445+0000 7f16b7a50b80 4 client.15867 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.445+0000 7f16b7a50b80 1 -- 172.21.15.121:0/1150498811 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574432c2d10 con 0x55744337d9f0 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.445+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55744337d9f0 0x55744337fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.445+0000 7f168e7fc700 10 client.15867 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.445+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55744337d9f0 0x55744337fde0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbf60daabe224904 server_cookie=8770ed8d703669e0 in_seq=0 out_seq=0 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f168e7fc700 1 -- 172.21.15.121:0/1150498811 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5574432c2d10 con 0x55744337d9f0 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f168e7fc700 10 client.15867 handle_command_reply: tid=0 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 1 client.15867 shutdown 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 1 -- 172.21.15.121:0/1150498811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 msgr2=0x55744336d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:05.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x55744336d570 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1698003930 tx=0x7f1698003c40 comp rx=0 tx=0).stop 2024-06-10T00:30:05.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 1 -- 172.21.15.121:0/1150498811 shutdown_connections 2024-06-10T00:30:05.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55744337d9f0 0x55744337fde0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x55744336d570 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f168e7fc700 0 client.15867 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:05.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443363f20 0x55744336fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/1150498811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443369640 0x55744336d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 1 -- 172.21.15.121:0/1150498811 >> 172.21.15.121:0/1150498811 conn(0x55744328ecc0 msgr2=0x55744335e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:05.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 1 -- 172.21.15.121:0/1150498811 shutdown_connections 2024-06-10T00:30:05.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 1 -- 172.21.15.121:0/1150498811 wait complete. 2024-06-10T00:30:05.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.446+0000 7f16b7a50b80 20 client.15867 trim_cache size 0 max 16384 2024-06-10T00:30:05.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16b7a50b80 1 Processor -- start 2024-06-10T00:30:05.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16b7a50b80 1 -- start start 2024-06-10T00:30:05.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x5574433639e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369640 0x557443367730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369f60 0x5574433761c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f16a81b0e90 con 0x557443369640 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f16a81ae720 con 0x557443369f60 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f16a81ac570 con 0x557443363f20 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16a659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369640 0x557443367730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16a659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369640 0x557443367730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38462/0 (socket says 172.21.15.121:38462) 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16a659c700 1 -- 172.21.15.121:0/557158182 learned_addr learned my addr 172.21.15.121:0/557158182 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/557158182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x5574433639e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.467+0000 7f16a759e700 1 --2- 172.21.15.121:0/557158182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369f60 0x5574433761c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.468+0000 7f16a659c700 1 -- 172.21.15.121:0/557158182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 msgr2=0x5574433639e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.468+0000 7f16a659c700 1 --2- 172.21.15.121:0/557158182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x5574433639e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.468+0000 7f16a659c700 1 -- 172.21.15.121:0/557158182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369f60 msgr2=0x5574433761c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.468+0000 7f16a659c700 1 --2- 172.21.15.121:0/557158182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369f60 0x5574433761c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.468+0000 7f16a659c700 1 -- 172.21.15.121:0/557158182 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557443364300 con 0x557443369640 2024-06-10T00:30:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.468+0000 7f16a659c700 1 --2- 172.21.15.121:0/557158182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369640 0x557443367730 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1690005d10 tx=0x7f1690020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8906b0b5085e0b6b server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.468+0000 7f169cff9700 1 -- 172.21.15.121:0/557158182 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690037070 con 0x557443369640 2024-06-10T00:30:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.468+0000 7f169cff9700 1 -- 172.21.15.121:0/557158182 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1690024410 con 0x557443369640 2024-06-10T00:30:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.468+0000 7f169cff9700 1 -- 172.21.15.121:0/557158182 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690032660 con 0x557443369640 2024-06-10T00:30:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.469+0000 7f16b7a50b80 1 -- 172.21.15.121:0/557158182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369640 msgr2=0x557443367730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.469+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/557158182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369640 0x557443367730 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1690005d10 tx=0x7f1690020000 comp rx=0 tx=0).stop 2024-06-10T00:30:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.469+0000 7f16b7a50b80 1 -- 172.21.15.121:0/557158182 shutdown_connections 2024-06-10T00:30:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.469+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/557158182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369f60 0x5574433761c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.469+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/557158182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x5574433639e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.469+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/557158182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369640 0x557443367730 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.469+0000 7f16b7a50b80 1 -- 172.21.15.121:0/557158182 >> 172.21.15.121:0/557158182 conn(0x55744328ecc0 msgr2=0x55744336b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.469+0000 7f16b7a50b80 1 -- 172.21.15.121:0/557158182 shutdown_connections 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.469+0000 7f16b7a50b80 1 -- 172.21.15.121:0/557158182 wait complete. 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.469+0000 7f16b7a50b80 1 -- start start 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.469+0000 7f16b7a50b80 1 Processor -- start 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x557443385c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 0x5574433755b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16b7a50b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x557443375af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f16a81acbe0 con 0x557443369f60 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f16a81ae050 con 0x557443369640 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16b7a50b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f16a81ad2b0 con 0x557443363f20 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16b7a50b80 10 client.0 _ll_register_callbacks cb 0x55744324f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:05.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a759e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x557443375af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a6d9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x557443385c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a759e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x557443375af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:38478/0 (socket says 172.21.15.121:38478) 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a759e700 1 -- 172.21.15.121:0/193438956 learned_addr learned my addr 172.21.15.121:0/193438956 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a659c700 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 0x5574433755b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a759e700 1 -- 172.21.15.121:0/193438956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 msgr2=0x557443385c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a759e700 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x557443385c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a759e700 1 -- 172.21.15.121:0/193438956 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 msgr2=0x5574433755b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a759e700 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 0x5574433755b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a759e700 1 -- 172.21.15.121:0/193438956 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574433760e0 con 0x557443369f60 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a659c700 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 0x5574433755b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f16a759e700 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x557443375af0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f16980118b0 tx=0x7f1698005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9ed9d4e565b3cd server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.470+0000 7f168e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:30:05.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f168e7fc700 1 -- 172.21.15.121:0/193438956 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169802e070 con 0x557443369f60 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f16b7a50b80 1 -- 172.21.15.121:0/193438956 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1690024410 con 0x557443369f60 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f168e7fc700 1 -- 172.21.15.121:0/193438956 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f169800bae0 con 0x557443369f60 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f168e7fc700 1 -- 172.21.15.121:0/193438956 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f169803a050 con 0x557443369f60 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f16b7a50b80 10 client.15873 fetch_fsmap learned FSMap version 27 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f16b7a50b80 1 -- 172.21.15.121:0/193438956 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5574433ec4a0 con 0x557443369f60 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f168e7fc700 1 -- 172.21.15.121:0/193438956 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1690024410 con 0x557443369f60 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f168e7fc700 1 -- 172.21.15.121:0/193438956 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f1698033cc0 con 0x557443369f60 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f16b7a50b80 10 client.15873 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f16b7a50b80 10 client.15873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f16b7a50b80 20 client.15873 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55744337d9f0 0x5574433eec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f16b7a50b80 4 client.15873 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:30:05.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f16b7a50b80 1 -- 172.21.15.121:0/193438956 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5574432bd290 con 0x55744337d9f0 2024-06-10T00:30:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.471+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55744337d9f0 0x5574433eec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16a6d9d700 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55744337d9f0 0x5574433eec20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=529a8008f02ef8ee server_cookie=4754d86d0947a646 in_seq=0 out_seq=0 2024-06-10T00:30:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f168e7fc700 10 client.15873 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f168e7fc700 1 -- 172.21.15.121:0/193438956 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5574432bd290 con 0x55744337d9f0 2024-06-10T00:30:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f168e7fc700 10 client.15873 handle_command_reply: tid=0 2024-06-10T00:30:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 1 client.15873 shutdown 2024-06-10T00:30:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 1 -- 172.21.15.121:0/193438956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 msgr2=0x557443375af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:05.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x557443375af0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f16980118b0 tx=0x7f1698005ce0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 1 -- 172.21.15.121:0/193438956 shutdown_connections 2024-06-10T00:30:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55744337d9f0 0x5574433eec20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557443369f60 0x557443375af0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557443363f20 0x557443385c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f168e7fc700 0 client.15873 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 1 --2- 172.21.15.121:0/193438956 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557443369640 0x5574433755b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 1 -- 172.21.15.121:0/193438956 >> 172.21.15.121:0/193438956 conn(0x55744328ecc0 msgr2=0x557443357e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:05.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 1 -- 172.21.15.121:0/193438956 shutdown_connections 2024-06-10T00:30:05.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 1 -- 172.21.15.121:0/193438956 wait complete. 2024-06-10T00:30:05.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.472+0000 7f16b7a50b80 20 client.15873 trim_cache size 0 max 16384 2024-06-10T00:30:05.479 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:30:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-10T00:30:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": { 2024-06-10T00:30:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "e28d93fb-0300-4377-a6fc-edd35ebad086": { 2024-06-10T00:30:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/", 2024-06-10T00:30:05.479 INFO:teuthology.orchestra.run.smithi121.stdout: "tag": "e28d93fb-0300-4377-a6fc-edd35ebad086", 2024-06-10T00:30:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: "options": "recursive,force" 2024-06-10T00:30:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:30:05.480 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:30:05.480 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:30:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 -- 172.21.15.121:0/4250282968 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f16940a2d90 msgr2=0x7f16940a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:05.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f16940a2d90 0x7f16940a5250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f16a80789f0 tx=0x7f169000b560 comp rx=0 tx=0).stop 2024-06-10T00:30:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 -- 172.21.15.121:0/4250282968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16a806a170 msgr2=0x7f16a819e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16a806a170 0x7f16a819e8e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f16a00060b0 tx=0x7f16a000d3b0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 -- 172.21.15.121:0/4250282968 shutdown_connections 2024-06-10T00:30:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f16940a2d90 0x7f16940a5250 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f16a8108500 0x7f16a8198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16a806a170 0x7f16a819e8e0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 --2- 172.21.15.121:0/4250282968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16a806ab30 0x7f16a819ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 -- 172.21.15.121:0/4250282968 >> 172.21.15.121:0/4250282968 conn(0x7f16a8078340 msgr2=0x7f16a81000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 -- 172.21.15.121:0/4250282968 shutdown_connections 2024-06-10T00:30:05.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:05.473+0000 7f16ad1cd700 1 -- 172.21.15.121:0/4250282968 wait complete. 2024-06-10T00:30:06.447 INFO:tasks.cephfs.filesystem:scrub status for tag:e28d93fb-0300-4377-a6fc-edd35ebad086 - {'path': '/', 'tag': 'e28d93fb-0300-4377-a6fc-edd35ebad086', 'options': 'recursive,force'} 2024-06-10T00:30:07.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:06 smithi135 ceph-mon[36424]: pgmap v67: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-06-10T00:30:07.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:06 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:30:07.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:06 smithi193 ceph-mon[36716]: pgmap v67: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-06-10T00:30:07.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:06 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:30:07.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:06 smithi121 ceph-mon[32378]: pgmap v67: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-06-10T00:30:07.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:06 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:30:07.292 INFO:tasks.workunit.client.0.smithi121.stdout: 9 0 0 0 0 0 0 2024-06-10T00:30:08.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:30:08.973 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:30:09.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:08 smithi135 ceph-mon[36424]: pgmap v68: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-06-10T00:30:09.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:08 smithi121 ceph-mon[32378]: pgmap v68: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-06-10T00:30:09.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:08 smithi193 ceph-mon[36716]: pgmap v68: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-06-10T00:30:10.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.763+0000 7fcedf2d9700 1 -- 172.21.15.121:0/142858996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fced8101df0 msgr2=0x7fced81021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:10.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.763+0000 7fcedf2d9700 1 --2- 172.21.15.121:0/142858996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fced8101df0 0x7fced81021d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fced40099c0 tx=0x7fced4009cd0 comp rx=0 tx=0).stop 2024-06-10T00:30:10.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.763+0000 7fcedf2d9700 1 -- 172.21.15.121:0/142858996 shutdown_connections 2024-06-10T00:30:10.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.763+0000 7fcedf2d9700 1 --2- 172.21.15.121:0/142858996 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fced8106750 0x7fced810e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:10.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.763+0000 7fcedf2d9700 1 --2- 172.21.15.121:0/142858996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fced8101df0 0x7fced81021d0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:10.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.763+0000 7fcedf2d9700 1 --2- 172.21.15.121:0/142858996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fced81027a0 0x7fced8106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:10.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.763+0000 7fcedf2d9700 1 -- 172.21.15.121:0/142858996 >> 172.21.15.121:0/142858996 conn(0x7fced8077a00 msgr2=0x7fced80fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:10.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.763+0000 7fcedf2d9700 1 -- 172.21.15.121:0/142858996 shutdown_connections 2024-06-10T00:30:10.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.763+0000 7fcedf2d9700 1 -- 172.21.15.121:0/142858996 wait complete. 2024-06-10T00:30:10.766 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:10 smithi121 ceph-mon[32378]: pgmap v69: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-06-10T00:30:10.766 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:10 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:30:10.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedf2d9700 1 Processor -- start 2024-06-10T00:30:10.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedf2d9700 1 -- start start 2024-06-10T00:30:10.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedf2d9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fced8101df0 0x7fced819d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:10.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedf2d9700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fced81027a0 0x7fced819dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedf2d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fced8106750 0x7fced8197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedd075700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fced8101df0 0x7fced819d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedf2d9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fced81116c0 con 0x7fced8106750 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedf2d9700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fced8111540 con 0x7fced8101df0 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedd075700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fced8101df0 0x7fced819d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:40322/0 (socket says 172.21.15.121:40322) 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedf2d9700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fced8111840 con 0x7fced81027a0 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedd075700 1 -- 172.21.15.121:0/2030303297 learned_addr learned my addr 172.21.15.121:0/2030303297 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedd876700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fced8106750 0x7fced8197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.764+0000 7fcedc874700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fced81027a0 0x7fced819dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.765+0000 7fcedd075700 1 -- 172.21.15.121:0/2030303297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fced81027a0 msgr2=0x7fced819dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.765+0000 7fcedd075700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fced81027a0 0x7fced819dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:10.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.765+0000 7fcedd075700 1 -- 172.21.15.121:0/2030303297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fced8106750 msgr2=0x7fced8197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:10.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.765+0000 7fcedd075700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fced8106750 0x7fced8197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:10.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.765+0000 7fcedd075700 1 -- 172.21.15.121:0/2030303297 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced81980e0 con 0x7fced8101df0 2024-06-10T00:30:10.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.765+0000 7fcedd075700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fced8101df0 0x7fced819d7c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fced4005910 tx=0x7fced401a840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:10.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.765+0000 7fcece7fc700 1 -- 172.21.15.121:0/2030303297 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced402a070 con 0x7fced8101df0 2024-06-10T00:30:10.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.765+0000 7fcedf2d9700 1 -- 172.21.15.121:0/2030303297 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced8198370 con 0x7fced8101df0 2024-06-10T00:30:10.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.765+0000 7fcece7fc700 1 -- 172.21.15.121:0/2030303297 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced4003680 con 0x7fced8101df0 2024-06-10T00:30:10.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.765+0000 7fcece7fc700 1 -- 172.21.15.121:0/2030303297 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced4025420 con 0x7fced8101df0 2024-06-10T00:30:10.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.765+0000 7fcedf2d9700 1 -- 172.21.15.121:0/2030303297 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced8100a80 con 0x7fced8101df0 2024-06-10T00:30:10.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.766+0000 7fcedf2d9700 1 -- 172.21.15.121:0/2030303297 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fced8069180 con 0x7fced8101df0 2024-06-10T00:30:10.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.767+0000 7fcece7fc700 1 -- 172.21.15.121:0/2030303297 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fced400f4f0 con 0x7fced8101df0 2024-06-10T00:30:10.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.768+0000 7fcece7fc700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcec40a2d90 0x7fcec40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:10.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.768+0000 7fcedc874700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcec40a2d90 0x7fcec40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:10.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.768+0000 7fcece7fc700 1 -- 172.21.15.121:0/2030303297 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fced40d2380 con 0x7fced8101df0 2024-06-10T00:30:10.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.768+0000 7fcedc874700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcec40a2d90 0x7fcec40a5250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fced8103e30 tx=0x7fcec8009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:10.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:10.773+0000 7fcece7fc700 1 -- 172.21.15.121:0/2030303297 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fced4013070 con 0x7fced8101df0 2024-06-10T00:30:11.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.037+0000 7fcedf2d9700 1 -- 172.21.15.121:0/2030303297 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fced81014b0 con 0x7fced8101df0 2024-06-10T00:30:11.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.038+0000 7fcece7fc700 1 -- 172.21.15.121:0/2030303297 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11886 (secure 0 0 0) 0x7fced409eca0 con 0x7fced8101df0 2024-06-10T00:30:11.039 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:30:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:29:51.277987+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:30:11.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 -- 172.21.15.121:0/2030303297 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcec40a2d90 msgr2=0x7fcec40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:11.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcec40a2d90 0x7fcec40a5250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fced8103e30 tx=0x7fcec8009260 comp rx=0 tx=0).stop 2024-06-10T00:30:11.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 -- 172.21.15.121:0/2030303297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fced8101df0 msgr2=0x7fced819d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:11.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fced8101df0 0x7fced819d7c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fced4005910 tx=0x7fced401a840 comp rx=0 tx=0).stop 2024-06-10T00:30:11.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 -- 172.21.15.121:0/2030303297 shutdown_connections 2024-06-10T00:30:11.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcec40a2d90 0x7fcec40a5250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:11.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fced8106750 0x7fced8197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:11.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fced8101df0 0x7fced819d7c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:11.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 --2- 172.21.15.121:0/2030303297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fced81027a0 0x7fced819dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:11.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 -- 172.21.15.121:0/2030303297 >> 172.21.15.121:0/2030303297 conn(0x7fced8077a00 msgr2=0x7fced80fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:11.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 -- 172.21.15.121:0/2030303297 shutdown_connections 2024-06-10T00:30:11.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:11.040+0000 7fcedf2d9700 1 -- 172.21.15.121:0/2030303297 wait complete. 2024-06-10T00:30:11.042 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 27 2024-06-10T00:30:11.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:10 smithi193 ceph-mon[36716]: pgmap v69: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-06-10T00:30:11.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:10 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:30:11.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:10 smithi135 ceph-mon[36424]: pgmap v69: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 45 KiB/s wr, 27 op/s 2024-06-10T00:30:11.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:10 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:30:11.798 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db --set-id 96a84dc5 --release --await' 2024-06-10T00:30:11.798 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db --set-id 96a84dc5 --release --await 2024-06-10T00:30:11.798 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:11 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:30:11.799 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:11 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:30:11.799 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:11 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:30:11.799 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:11 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2030303297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:30:12.039 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:30:12.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:11 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:30:12.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:11 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:30:12.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:11 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:30:12.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:11 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2030303297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:30:12.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:11 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:30:12.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:11 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:30:12.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:11 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:30:12.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:11 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2030303297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:30:13.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:12 smithi135 ceph-mon[36424]: pgmap v70: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-06-10T00:30:13.277 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:12 smithi121 ceph-mon[32378]: pgmap v70: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-06-10T00:30:13.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:12 smithi193 ceph-mon[36716]: pgmap v70: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 20 op/s 2024-06-10T00:30:13.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.825+0000 7f985efa4700 1 -- 172.21.15.121:0/2581554693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102ed0 msgr2=0x7f985810f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:13.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.825+0000 7f985efa4700 1 --2- 172.21.15.121:0/2581554693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102ed0 0x7f985810f360 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9850009990 tx=0x7f9850009ca0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.825+0000 7f985efa4700 1 -- 172.21.15.121:0/2581554693 shutdown_connections 2024-06-10T00:30:13.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.825+0000 7f985efa4700 1 --2- 172.21.15.121:0/2581554693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102ed0 0x7f985810f360 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.825+0000 7f985efa4700 1 --2- 172.21.15.121:0/2581554693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9858108520 0x7f9858108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.825+0000 7f985efa4700 1 --2- 172.21.15.121:0/2581554693 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9858102510 0x7f9858102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.825+0000 7f985efa4700 1 -- 172.21.15.121:0/2581554693 >> 172.21.15.121:0/2581554693 conn(0x7f9858079560 msgr2=0x7f9858079970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.825+0000 7f985efa4700 1 -- 172.21.15.121:0/2581554693 shutdown_connections 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.825+0000 7f985efa4700 1 -- 172.21.15.121:0/2581554693 wait complete. 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.825+0000 7f985efa4700 1 Processor -- start 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.825+0000 7f985efa4700 1 -- start start 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985efa4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102510 0x7f985819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985efa4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9858102ed0 0x7f985819ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985efa4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9858108520 0x7f9858198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985efa4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9858112890 con 0x7f9858102510 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985efa4700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f9858112710 con 0x7f9858102ed0 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985efa4700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9858112a10 con 0x7f9858108520 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985cd40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102510 0x7f985819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985cd40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102510 0x7f985819e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36778/0 (socket says 172.21.15.121:36778) 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985cd40700 1 -- 172.21.15.121:0/2173923182 learned_addr learned my addr 172.21.15.121:0/2173923182 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:13.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985d541700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9858108520 0x7f9858198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f984ffff700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9858102ed0 0x7f985819ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985cd40700 1 -- 172.21.15.121:0/2173923182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9858108520 msgr2=0x7f9858198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985cd40700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9858108520 0x7f9858198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985cd40700 1 -- 172.21.15.121:0/2173923182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9858102ed0 msgr2=0x7f985819ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985cd40700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9858102ed0 0x7f985819ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985cd40700 1 -- 172.21.15.121:0/2173923182 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9858199250 con 0x7f9858102510 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.826+0000 7f985cd40700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102510 0x7f985819e930 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f985400ba50 tx=0x7f985400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.827+0000 7f984dffb700 1 -- 172.21.15.121:0/2173923182 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854019070 con 0x7f9858102510 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.827+0000 7f985efa4700 1 -- 172.21.15.121:0/2173923182 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9858199540 con 0x7f9858102510 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.827+0000 7f984dffb700 1 -- 172.21.15.121:0/2173923182 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9854014440 con 0x7f9858102510 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.827+0000 7f984dffb700 1 -- 172.21.15.121:0/2173923182 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98540075d0 con 0x7f9858102510 2024-06-10T00:30:13.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.827+0000 7f985efa4700 1 -- 172.21.15.121:0/2173923182 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98581a5800 con 0x7f9858102510 2024-06-10T00:30:13.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.829+0000 7f984dffb700 1 -- 172.21.15.121:0/2173923182 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f985401c070 con 0x7f9858102510 2024-06-10T00:30:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.829+0000 7f984dffb700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f98400a2d90 0x7f98400a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.830+0000 7f984dffb700 1 -- 172.21.15.121:0/2173923182 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f98540ceb30 con 0x7f9858102510 2024-06-10T00:30:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.830+0000 7f984ffff700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f98400a2d90 0x7f98400a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.830+0000 7f984ffff700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f98400a2d90 0x7f98400a5250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9848006fd0 tx=0x7f9848008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.830+0000 7f9869827b80 1 Processor -- start 2024-06-10T00:30:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f9869827b80 1 -- start start 2024-06-10T00:30:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f9869827b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7500 0x55ba1c7f78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f9869827b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7f1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f9869827b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7f2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f9869827b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ba1c6e4060 con 0x55ba1c7f7e20 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f9869827b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55ba1c742950 con 0x55ba1c7f7500 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f9869827b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ba1c6aa560 con 0x55ba1c7f1c50 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f984ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f985cd40700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7500 0x55ba1c7f78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f984ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7f1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36782/0 (socket says 172.21.15.121:36782) 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f984ffff700 1 -- 172.21.15.121:0/1163316775 learned_addr learned my addr 172.21.15.121:0/1163316775 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f985d541700 1 --2- 172.21.15.121:0/1163316775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f984ffff700 1 -- 172.21.15.121:0/1163316775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f1c50 msgr2=0x55ba1c7f2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f984ffff700 1 --2- 172.21.15.121:0/1163316775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7f2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f984ffff700 1 -- 172.21.15.121:0/1163316775 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7500 msgr2=0x55ba1c7f78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f984ffff700 1 --2- 172.21.15.121:0/1163316775 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7500 0x55ba1c7f78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f984ffff700 1 -- 172.21.15.121:0/1163316775 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba1c6919d0 con 0x55ba1c7f7e20 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f984ffff700 1 --2- 172.21.15.121:0/1163316775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7f1710 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f984800d720 tx=0x7f9848010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4eb69d9433ab2bc server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.831+0000 7f983effd700 1 -- 172.21.15.121:0/1163316775 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9848024070 con 0x55ba1c7f7e20 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.832+0000 7f983effd700 1 -- 172.21.15.121:0/1163316775 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f984800de40 con 0x55ba1c7f7e20 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.832+0000 7f983effd700 1 -- 172.21.15.121:0/1163316775 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984801faf0 con 0x55ba1c7f7e20 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.832+0000 7f9869827b80 1 -- 172.21.15.121:0/1163316775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7e20 msgr2=0x55ba1c7f1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.832+0000 7f9869827b80 1 --2- 172.21.15.121:0/1163316775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7f1710 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f984800d720 tx=0x7f9848010000 comp rx=0 tx=0).stop 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.832+0000 7f9869827b80 1 -- 172.21.15.121:0/1163316775 shutdown_connections 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.832+0000 7f9869827b80 1 --2- 172.21.15.121:0/1163316775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7f2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.832+0000 7f9869827b80 1 --2- 172.21.15.121:0/1163316775 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7500 0x55ba1c7f78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.832+0000 7f9869827b80 1 --2- 172.21.15.121:0/1163316775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7f1710 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.832+0000 7f9869827b80 1 -- 172.21.15.121:0/1163316775 >> 172.21.15.121:0/1163316775 conn(0x55ba1c71e180 msgr2=0x55ba1c7eca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.832+0000 7f9869827b80 1 -- 172.21.15.121:0/1163316775 shutdown_connections 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.832+0000 7f9869827b80 1 -- 172.21.15.121:0/1163316775 wait complete. 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f9869827b80 1 -- start start 2024-06-10T00:30:13.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f9869827b80 1 Processor -- start 2024-06-10T00:30:13.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f9869827b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7fdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f9869827b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7500 0x55ba1c7fae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f9869827b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7fb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f9869827b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ba1c6e30c0 con 0x55ba1c7f1c50 2024-06-10T00:30:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f9869827b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55ba1c73e910 con 0x55ba1c7f7500 2024-06-10T00:30:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f9869827b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ba1c7fe800 con 0x55ba1c7f7e20 2024-06-10T00:30:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f9869827b80 10 client.0 _ll_register_callbacks cb 0x55ba1c6d86b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f985cd40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7fdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f985cd40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7fdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36796/0 (socket says 172.21.15.121:36796) 2024-06-10T00:30:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f985cd40700 1 -- 172.21.15.121:0/3888524590 learned_addr learned my addr 172.21.15.121:0/3888524590 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.833+0000 7f985d541700 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7fb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f985cd40700 1 -- 172.21.15.121:0/3888524590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7e20 msgr2=0x55ba1c7fb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f985cd40700 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7fb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f985cd40700 1 -- 172.21.15.121:0/3888524590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7500 msgr2=0x55ba1c7fae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f985cd40700 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7500 0x55ba1c7fae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f985cd40700 1 -- 172.21.15.121:0/3888524590 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba1c708850 con 0x55ba1c7f1c50 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f985d541700 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7fb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f985cd40700 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7fdb90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f985400ba20 tx=0x7f985409ae80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9958fb50dc7e476 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f9837fff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f9837fff700 1 -- 172.21.15.121:0/3888524590 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985409b230 con 0x55ba1c7f1c50 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f9837fff700 1 -- 172.21.15.121:0/3888524590 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f98540a35b0 con 0x55ba1c7f1c50 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f9869827b80 1 -- 172.21.15.121:0/3888524590 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f984800de40 con 0x55ba1c7f1c50 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f9837fff700 1 -- 172.21.15.121:0/3888524590 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f98540a37b0 con 0x55ba1c7f1c50 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f9869827b80 10 client.15891 fetch_fsmap learned FSMap version 27 2024-06-10T00:30:13.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f9869827b80 1 -- 172.21.15.121:0/3888524590 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ba1c809030 con 0x55ba1c7f1c50 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f9837fff700 1 -- 172.21.15.121:0/3888524590 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f984800de40 con 0x55ba1c7f1c50 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.834+0000 7f9837fff700 1 -- 172.21.15.121:0/3888524590 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f98540b0070 con 0x55ba1c7f1c50 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.835+0000 7f9869827b80 10 client.15891 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.835+0000 7f9869827b80 10 client.15891 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.835+0000 7f9869827b80 20 client.15891 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.835+0000 7f9869827b80 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ba1c80b670 0x55ba1c80da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.835+0000 7f9869827b80 4 client.15891 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.835+0000 7f9869827b80 1 -- 172.21.15.121:0/3888524590 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba1c74fe50 con 0x55ba1c80b670 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.835+0000 7f984ffff700 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ba1c80b670 0x55ba1c80da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.835+0000 7f9837fff700 10 client.15891 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.835+0000 7f984ffff700 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ba1c80b670 0x55ba1c80da60 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a655dbc2044fb21 server_cookie=595080ae031f9e9 in_seq=0 out_seq=0 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9837fff700 1 -- 172.21.15.121:0/3888524590 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55ba1c74fe50 con 0x55ba1c80b670 2024-06-10T00:30:13.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9837fff700 10 client.15891 handle_command_reply: tid=0 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9869827b80 1 client.15891 shutdown 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9869827b80 1 -- 172.21.15.121:0/3888524590 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 msgr2=0x55ba1c7fdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9869827b80 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7fdb90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f985400ba20 tx=0x7f985409ae80 comp rx=0 tx=0).stop 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9869827b80 1 -- 172.21.15.121:0/3888524590 shutdown_connections 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9869827b80 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ba1c80b670 0x55ba1c80da60 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9869827b80 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7fb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9869827b80 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7fdb90 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9869827b80 1 --2- 172.21.15.121:0/3888524590 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7500 0x55ba1c7fae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9869827b80 1 -- 172.21.15.121:0/3888524590 >> 172.21.15.121:0/3888524590 conn(0x55ba1c71e180 msgr2=0x55ba1c7eb320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9837fff700 0 client.15891 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9869827b80 1 -- 172.21.15.121:0/3888524590 shutdown_connections 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.836+0000 7f9869827b80 1 -- 172.21.15.121:0/3888524590 wait complete. 2024-06-10T00:30:13.842 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.837+0000 7f9869827b80 20 client.15891 trim_cache size 0 max 16384 2024-06-10T00:30:13.857 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.856+0000 7f9869827b80 1 Processor -- start 2024-06-10T00:30:13.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f9869827b80 1 -- start start 2024-06-10T00:30:13.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f9869827b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7f1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f9869827b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7500 0x55ba1c7ecbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f9869827b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7ebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f9869827b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f98581b0e40 con 0x55ba1c7f7500 2024-06-10T00:30:13.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f9869827b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f98581ae6d0 con 0x55ba1c7f1c50 2024-06-10T00:30:13.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f9869827b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f98581ac520 con 0x55ba1c7f7e20 2024-06-10T00:30:13.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f984ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7500 0x55ba1c7ecbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f984ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7500 0x55ba1c7ecbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36812/0 (socket says 172.21.15.121:36812) 2024-06-10T00:30:13.858 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f984ffff700 1 -- 172.21.15.121:0/1490820344 learned_addr learned my addr 172.21.15.121:0/1490820344 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f985d541700 1 --2- 172.21.15.121:0/1490820344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7ebbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f985cd40700 1 --2- 172.21.15.121:0/1490820344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f984ffff700 1 -- 172.21.15.121:0/1490820344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7e20 msgr2=0x55ba1c7ebbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f984ffff700 1 --2- 172.21.15.121:0/1490820344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7ebbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f984ffff700 1 -- 172.21.15.121:0/1490820344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f1c50 msgr2=0x55ba1c7f1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f984ffff700 1 --2- 172.21.15.121:0/1490820344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7f1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f984ffff700 1 -- 172.21.15.121:0/1490820344 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba1c7fbd90 con 0x55ba1c7f7500 2024-06-10T00:30:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f985cd40700 1 --2- 172.21.15.121:0/1490820344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7f1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:30:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f984ffff700 1 --2- 172.21.15.121:0/1490820344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7500 0x55ba1c7ecbb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f984800e010 tx=0x7f984801f5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b79e293d034c6d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.857+0000 7f983e7fc700 1 -- 172.21.15.121:0/1490820344 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9848024040 con 0x55ba1c7f7500 2024-06-10T00:30:13.859 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.858+0000 7f983e7fc700 1 -- 172.21.15.121:0/1490820344 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f984801f400 con 0x55ba1c7f7500 2024-06-10T00:30:13.860 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.858+0000 7f983e7fc700 1 -- 172.21.15.121:0/1490820344 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9848029440 con 0x55ba1c7f7500 2024-06-10T00:30:13.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.858+0000 7f9869827b80 1 -- 172.21.15.121:0/1490820344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7500 msgr2=0x55ba1c7ecbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:13.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.858+0000 7f9869827b80 1 --2- 172.21.15.121:0/1490820344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7500 0x55ba1c7ecbb0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f984800e010 tx=0x7f984801f5f0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.858+0000 7f9869827b80 1 -- 172.21.15.121:0/1490820344 shutdown_connections 2024-06-10T00:30:13.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.858+0000 7f9869827b80 1 --2- 172.21.15.121:0/1490820344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7e20 0x55ba1c7ebbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.858+0000 7f9869827b80 1 --2- 172.21.15.121:0/1490820344 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f1c50 0x55ba1c7f1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.858+0000 7f9869827b80 1 --2- 172.21.15.121:0/1490820344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f7500 0x55ba1c7ecbb0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.858+0000 7f9869827b80 1 -- 172.21.15.121:0/1490820344 >> 172.21.15.121:0/1490820344 conn(0x55ba1c71e180 msgr2=0x55ba1c7f8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.858+0000 7f9869827b80 1 -- 172.21.15.121:0/1490820344 shutdown_connections 2024-06-10T00:30:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.858+0000 7f9869827b80 1 -- 172.21.15.121:0/1490820344 wait complete. 2024-06-10T00:30:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f9869827b80 1 -- start start 2024-06-10T00:30:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f9869827b80 1 Processor -- start 2024-06-10T00:30:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f9869827b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c804280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f9869827b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7500 0x55ba1c812cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f9869827b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7e20 0x55ba1c813270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f9869827b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f98581acb90 con 0x55ba1c7f1c50 2024-06-10T00:30:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f9869827b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f98581ae000 con 0x55ba1c7f7e20 2024-06-10T00:30:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f9869827b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f98581ad260 con 0x55ba1c7f7500 2024-06-10T00:30:13.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f9869827b80 10 client.0 _ll_register_callbacks cb 0x55ba1c6d86b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f985cd40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c804280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f985cd40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c804280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36818/0 (socket says 172.21.15.121:36818) 2024-06-10T00:30:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f985cd40700 1 -- 172.21.15.121:0/158469296 learned_addr learned my addr 172.21.15.121:0/158469296 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f984ffff700 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7500 0x55ba1c812cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f985d541700 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7e20 0x55ba1c813270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f985cd40700 1 -- 172.21.15.121:0/158469296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7500 msgr2=0x55ba1c812cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.859+0000 7f985cd40700 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7500 0x55ba1c812cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f985cd40700 1 -- 172.21.15.121:0/158469296 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7e20 msgr2=0x55ba1c813270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f985cd40700 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7e20 0x55ba1c813270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f985cd40700 1 -- 172.21.15.121:0/158469296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba1c813b00 con 0x55ba1c7f1c50 2024-06-10T00:30:13.865 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f985d541700 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7e20 0x55ba1c813270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:30:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f985cd40700 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c804280 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f98540971e0 tx=0x7f985409b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b39ed68e34087e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9837fff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:30:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9837fff700 1 -- 172.21.15.121:0/158469296 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854099ff0 con 0x55ba1c7f1c50 2024-06-10T00:30:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9837fff700 1 -- 172.21.15.121:0/158469296 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f985409a430 con 0x55ba1c7f1c50 2024-06-10T00:30:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9869827b80 1 -- 172.21.15.121:0/158469296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f984801f400 con 0x55ba1c7f1c50 2024-06-10T00:30:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9837fff700 1 -- 172.21.15.121:0/158469296 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9854096b20 con 0x55ba1c7f1c50 2024-06-10T00:30:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9869827b80 10 client.15897 fetch_fsmap learned FSMap version 27 2024-06-10T00:30:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9869827b80 1 -- 172.21.15.121:0/158469296 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ba1c876350 con 0x55ba1c7f1c50 2024-06-10T00:30:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9837fff700 1 -- 172.21.15.121:0/158469296 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f984801f400 con 0x55ba1c7f1c50 2024-06-10T00:30:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9837fff700 1 -- 172.21.15.121:0/158469296 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) ==== 8350+0+0 (secure 0 0 0) 0x7f9854080020 con 0x55ba1c7f1c50 2024-06-10T00:30:13.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9869827b80 10 client.15897 fetch_fsmap finished waiting for FSMap version 27 2024-06-10T00:30:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9869827b80 10 client.15897 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:30:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9869827b80 20 client.15897 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9869827b80 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ba1c876870 0x55ba1c878c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9869827b80 4 client.15897 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "96a84dc5", "release": true, "await": true}] 2024-06-10T00:30:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.860+0000 7f9869827b80 1 -- 172.21.15.121:0/158469296 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "set_id": "96a84dc5", "release": true, "await": true}) -- 0x55ba1c74a3d0 con 0x55ba1c876870 2024-06-10T00:30:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.861+0000 7f984ffff700 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ba1c876870 0x55ba1c878c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.861+0000 7f984ffff700 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ba1c876870 0x55ba1c878c60 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25e0a921e97df3fb server_cookie=c0cffdf02b148da4 in_seq=0 out_seq=0 2024-06-10T00:30:13.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.861+0000 7f9837fff700 10 client.15897 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:13.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9837fff700 1 -- 172.21.15.121:0/158469296 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55ba1c74a3d0 con 0x55ba1c876870 2024-06-10T00:30:13.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9837fff700 10 client.15897 handle_command_reply: tid=0 2024-06-10T00:30:13.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 1 client.15897 shutdown 2024-06-10T00:30:13.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 1 -- 172.21.15.121:0/158469296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 msgr2=0x55ba1c804280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c804280 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f98540971e0 tx=0x7f985409b260 comp rx=0 tx=0).stop 2024-06-10T00:30:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 1 -- 172.21.15.121:0/158469296 shutdown_connections 2024-06-10T00:30:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ba1c876870 0x55ba1c878c60 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ba1c7f7e20 0x55ba1c813270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ba1c7f1c50 0x55ba1c804280 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 1 --2- 172.21.15.121:0/158469296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba1c7f7500 0x55ba1c812cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9837fff700 0 client.15897 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 1 -- 172.21.15.121:0/158469296 >> 172.21.15.121:0/158469296 conn(0x55ba1c71e180 msgr2=0x55ba1c7f67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 1 -- 172.21.15.121:0/158469296 shutdown_connections 2024-06-10T00:30:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 1 -- 172.21.15.121:0/158469296 wait complete. 2024-06-10T00:30:13.978 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.977+0000 7f9869827b80 20 client.15897 trim_cache size 0 max 16384 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 27, 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 8, 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: "96a84dc5": { 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 8, 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:30:13.979 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 150.0, 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:30:13.980 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:30:13.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 -- 172.21.15.121:0/2173923182 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f98400a2d90 msgr2=0x7f98400a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:13.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f98400a2d90 0x7f98400a5250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9848006fd0 tx=0x7f9848008040 comp rx=0 tx=0).stop 2024-06-10T00:30:13.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 -- 172.21.15.121:0/2173923182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102510 msgr2=0x7f985819e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:13.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102510 0x7f985819e930 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f985400ba50 tx=0x7f985400bd60 comp rx=0 tx=0).stop 2024-06-10T00:30:13.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 -- 172.21.15.121:0/2173923182 shutdown_connections 2024-06-10T00:30:13.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f98400a2d90 0x7f98400a5250 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9858108520 0x7f9858198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102510 0x7f985819e930 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 --2- 172.21.15.121:0/2173923182 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9858102ed0 0x7f985819ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:13.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 -- 172.21.15.121:0/2173923182 >> 172.21.15.121:0/2173923182 conn(0x7f9858079560 msgr2=0x7f98581011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:13.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 -- 172.21.15.121:0/2173923182 shutdown_connections 2024-06-10T00:30:13.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:13.978+0000 7f985efa4700 1 -- 172.21.15.121:0/2173923182 wait complete. 2024-06-10T00:30:14.774 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '96a84dc5', total seconds elapsed: 36.1 2024-06-10T00:30:15.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:14 smithi121 ceph-mon[32378]: pgmap v71: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-06-10T00:30:15.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:14 smithi135 ceph-mon[36424]: pgmap v71: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-06-10T00:30:15.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:14 smithi193 ceph-mon[36716]: pgmap v71: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-06-10T00:30:17.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:16 smithi135 ceph-mon[36424]: pgmap v72: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:30:17.292 INFO:tasks.workunit.client.0.smithi121.stdout: 12 19371 236 6182 109 2184 177 2024-06-10T00:30:17.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:16 smithi193 ceph-mon[36716]: pgmap v72: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:30:17.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:16 smithi121 ceph-mon[32378]: pgmap v72: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:30:19.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:18 smithi135 ceph-mon[36424]: pgmap v73: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:30:19.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:18 smithi193 ceph-mon[36716]: pgmap v73: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:30:19.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:18 smithi121 ceph-mon[32378]: pgmap v73: 129 pgs: 129 active+clean; 33 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:30:21.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:20 smithi135 ceph-mon[36424]: pgmap v74: 129 pgs: 129 active+clean; 36 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 297 KiB/s wr, 17 op/s 2024-06-10T00:30:21.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:20 smithi193 ceph-mon[36716]: pgmap v74: 129 pgs: 129 active+clean; 36 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 297 KiB/s wr, 17 op/s 2024-06-10T00:30:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:20 smithi121 ceph-mon[32378]: pgmap v74: 129 pgs: 129 active+clean; 36 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 297 KiB/s wr, 17 op/s 2024-06-10T00:30:23.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:22 smithi135 ceph-mon[36424]: pgmap v75: 129 pgs: 129 active+clean; 36 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 297 KiB/s wr, 14 op/s 2024-06-10T00:30:23.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:22 smithi193 ceph-mon[36716]: pgmap v75: 129 pgs: 129 active+clean; 36 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 297 KiB/s wr, 14 op/s 2024-06-10T00:30:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:22 smithi121 ceph-mon[32378]: pgmap v75: 129 pgs: 129 active+clean; 36 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 297 KiB/s wr, 14 op/s 2024-06-10T00:30:25.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:24 smithi135 ceph-mon[36424]: pgmap v76: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 569 KiB/s wr, 31 op/s 2024-06-10T00:30:25.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:24 smithi193 ceph-mon[36716]: pgmap v76: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 569 KiB/s wr, 31 op/s 2024-06-10T00:30:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:24 smithi121 ceph-mon[32378]: pgmap v76: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 569 KiB/s wr, 31 op/s 2024-06-10T00:30:27.251 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:26 smithi121 ceph-mon[32378]: pgmap v77: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 569 KiB/s wr, 29 op/s 2024-06-10T00:30:27.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:26 smithi135 ceph-mon[36424]: pgmap v77: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 569 KiB/s wr, 29 op/s 2024-06-10T00:30:27.292 INFO:tasks.workunit.client.0.smithi121.stdout: 16 131454 189 74067 374 12866 414 2024-06-10T00:30:27.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:26 smithi193 ceph-mon[36716]: pgmap v77: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 569 KiB/s wr, 29 op/s 2024-06-10T00:30:28.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:27 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:30:28.263 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:27 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:30:28.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:27 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:30:29.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:28 smithi135 ceph-mon[36424]: pgmap v78: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 569 KiB/s wr, 29 op/s 2024-06-10T00:30:29.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:28 smithi193 ceph-mon[36716]: pgmap v78: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 569 KiB/s wr, 29 op/s 2024-06-10T00:30:29.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:28 smithi121 ceph-mon[32378]: pgmap v78: 129 pgs: 129 active+clean; 38 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 569 KiB/s wr, 29 op/s 2024-06-10T00:30:31.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:30 smithi135 ceph-mon[36424]: pgmap v79: 129 pgs: 129 active+clean; 65 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 140 op/s 2024-06-10T00:30:31.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:30 smithi193 ceph-mon[36716]: pgmap v79: 129 pgs: 129 active+clean; 65 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 140 op/s 2024-06-10T00:30:31.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:30 smithi121 ceph-mon[32378]: pgmap v79: 129 pgs: 129 active+clean; 65 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 140 op/s 2024-06-10T00:30:33.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:32 smithi135 ceph-mon[36424]: pgmap v80: 129 pgs: 129 active+clean; 65 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.6 MiB/s wr, 128 op/s 2024-06-10T00:30:33.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:32 smithi193 ceph-mon[36716]: pgmap v80: 129 pgs: 129 active+clean; 65 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.6 MiB/s wr, 128 op/s 2024-06-10T00:30:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:32 smithi121 ceph-mon[32378]: pgmap v80: 129 pgs: 129 active+clean; 65 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.6 MiB/s wr, 128 op/s 2024-06-10T00:30:35.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:34 smithi135 ceph-mon[36424]: pgmap v81: 129 pgs: 129 active+clean; 74 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 155 op/s 2024-06-10T00:30:35.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:34 smithi193 ceph-mon[36716]: pgmap v81: 129 pgs: 129 active+clean; 74 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 155 op/s 2024-06-10T00:30:35.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:34 smithi121 ceph-mon[32378]: pgmap v81: 129 pgs: 129 active+clean; 74 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.5 MiB/s wr, 155 op/s 2024-06-10T00:30:36.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:30:36.718 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:30:37.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:36 smithi121 ceph-mon[32378]: pgmap v82: 129 pgs: 129 active+clean; 74 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 138 op/s 2024-06-10T00:30:37.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:36 smithi135 ceph-mon[36424]: pgmap v82: 129 pgs: 129 active+clean; 74 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 138 op/s 2024-06-10T00:30:37.292 INFO:tasks.workunit.client.0.smithi121.stdout: 25 300932 429 193150 646 84040 1320 2024-06-10T00:30:37.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:36 smithi193 ceph-mon[36716]: pgmap v82: 129 pgs: 129 active+clean; 74 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 138 op/s 2024-06-10T00:30:38.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.619+0000 7fc202745700 1 -- 172.21.15.121:0/3309757082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1fc06a170 msgr2=0x7fc1fc06a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:38.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.619+0000 7fc202745700 1 --2- 172.21.15.121:0/3309757082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1fc06a170 0x7fc1fc06a5f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0009990 tx=0x7fc1f0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.619+0000 7fc202745700 1 -- 172.21.15.121:0/3309757082 shutdown_connections 2024-06-10T00:30:38.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.619+0000 7fc202745700 1 --2- 172.21.15.121:0/3309757082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1fc06ab30 0x7fc1fc10f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.619+0000 7fc202745700 1 --2- 172.21.15.121:0/3309757082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc108500 0x7fc1fc1088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.619+0000 7fc202745700 1 --2- 172.21.15.121:0/3309757082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1fc06a170 0x7fc1fc06a5f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.619+0000 7fc202745700 1 -- 172.21.15.121:0/3309757082 >> 172.21.15.121:0/3309757082 conn(0x7fc1fc078340 msgr2=0x7fc1fc078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:38.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.619+0000 7fc202745700 1 -- 172.21.15.121:0/3309757082 shutdown_connections 2024-06-10T00:30:38.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.619+0000 7fc202745700 1 -- 172.21.15.121:0/3309757082 wait complete. 2024-06-10T00:30:38.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.620+0000 7fc202745700 1 Processor -- start 2024-06-10T00:30:38.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.620+0000 7fc202745700 1 -- start start 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.620+0000 7fc202745700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1fc06a170 0x7fc1fc19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc202745700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc06ab30 0x7fc1fc19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc202745700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1fc108500 0x7fc1fc198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc202745700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc112760 con 0x7fc1fc108500 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc202745700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc1125e0 con 0x7fc1fc06ab30 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc202745700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc1128e0 con 0x7fc1fc06a170 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc200ce2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1fc108500 0x7fc1fc198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc1fb7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc06ab30 0x7fc1fc19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc200ce2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1fc108500 0x7fc1fc198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55394/0 (socket says 172.21.15.121:55394) 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc200ce2700 1 -- 172.21.15.121:0/520158893 learned_addr learned my addr 172.21.15.121:0/520158893 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc1fbfff700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1fc06a170 0x7fc1fc19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/520158893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1fc06a170 msgr2=0x7fc1fc19e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1fc06a170 0x7fc1fc19e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/520158893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1fc108500 msgr2=0x7fc1fc198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1fc108500 0x7fc1fc198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/520158893 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1fc1991c0 con 0x7fc1fc06ab30 2024-06-10T00:30:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc06ab30 0x7fc1fc19ede0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc1f00059d0 tx=0x7fc1f000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc1f97fa700 1 -- 172.21.15.121:0/520158893 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0027070 con 0x7fc1fc06ab30 2024-06-10T00:30:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc202745700 1 -- 172.21.15.121:0/520158893 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1fc199450 con 0x7fc1fc06ab30 2024-06-10T00:30:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.621+0000 7fc1f97fa700 1 -- 172.21.15.121:0/520158893 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f0003d70 con 0x7fc1fc06ab30 2024-06-10T00:30:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.622+0000 7fc1f97fa700 1 -- 172.21.15.121:0/520158893 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f000f460 con 0x7fc1fc06ab30 2024-06-10T00:30:38.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.622+0000 7fc202745700 1 -- 172.21.15.121:0/520158893 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1fc1a5770 con 0x7fc1fc06ab30 2024-06-10T00:30:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.624+0000 7fc1f97fa700 1 -- 172.21.15.121:0/520158893 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc1f002a070 con 0x7fc1fc06ab30 2024-06-10T00:30:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.625+0000 7fc1f97fa700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc1e40a2d90 0x7fc1e40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.625+0000 7fc1fbfff700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc1e40a2d90 0x7fc1e40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc1f97fa700 1 -- 172.21.15.121:0/520158893 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fc1f00d0780 con 0x7fc1fc06ab30 2024-06-10T00:30:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc1fbfff700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc1e40a2d90 0x7fc1e40a5250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc199ba0 tx=0x7fc1ec009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc20cfc8b80 1 Processor -- start 2024-06-10T00:30:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc20cfc8b80 1 -- start start 2024-06-10T00:30:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 0x55f2f4a05340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 0x55f2f49ff260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f49ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f2f48f1f60 con 0x55f2f4a05880 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55f2f4950850 con 0x55f2f4a04f60 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f2f48b8460 con 0x55f2f49ff7a0 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc1fb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 0x55f2f49ff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc1fbfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 0x55f2f4a05340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc1fb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 0x55f2f49ff260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55402/0 (socket says 172.21.15.121:55402) 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/3988140266 learned_addr learned my addr 172.21.15.121:0/3988140266 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc1fbfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 0x55f2f4a05340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:35330/0 (socket says 172.21.15.121:35330) 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.626+0000 7fc200ce2700 1 --2- 172.21.15.121:0/3988140266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f49ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.627+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/3988140266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 msgr2=0x55f2f49ffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.627+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/3988140266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f49ffbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.627+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/3988140266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 msgr2=0x55f2f4a05340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:38.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.627+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/3988140266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 0x55f2f4a05340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.627+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/3988140266 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2f489f8d0 con 0x55f2f4a05880 2024-06-10T00:30:38.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.627+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/3988140266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 0x55f2f49ff260 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc1f00a4ff0 tx=0x7fc1f0022930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152efd8eab55cfb8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:38.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.627+0000 7fc1e27fc700 1 -- 172.21.15.121:0/3988140266 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f009d040 con 0x55f2f4a05880 2024-06-10T00:30:38.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.627+0000 7fc1e27fc700 1 -- 172.21.15.121:0/3988140266 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f009c940 con 0x55f2f4a05880 2024-06-10T00:30:38.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.627+0000 7fc1e27fc700 1 -- 172.21.15.121:0/3988140266 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f00d0370 con 0x55f2f4a05880 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.628+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/3988140266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 msgr2=0x55f2f49ff260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.628+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/3988140266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 0x55f2f49ff260 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc1f00a4ff0 tx=0x7fc1f0022930 comp rx=0 tx=0).stop 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.628+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/3988140266 shutdown_connections 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.628+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/3988140266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f49ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.628+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/3988140266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 0x55f2f4a05340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.628+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/3988140266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 0x55f2f49ff260 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.628+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/3988140266 >> 172.21.15.121:0/3988140266 conn(0x55f2f492c080 msgr2=0x55f2f49fa220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.628+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/3988140266 shutdown_connections 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.628+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/3988140266 wait complete. 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.628+0000 7fc20cfc8b80 1 -- start start 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc20cfc8b80 1 Processor -- start 2024-06-10T00:30:38.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a0b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 0x55f2f4a089e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f4a05880 0x55f2f4a08f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f2f48f0fc0 con 0x55f2f4a04f60 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55f2f494c810 con 0x55f2f49ff7a0 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f2f4a10420 con 0x55f2f4a05880 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc20cfc8b80 10 client.0 _ll_register_callbacks cb 0x55f2f48e65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc1fbfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a0b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc1fb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 0x55f2f4a089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc1fbfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a0b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:35344/0 (socket says 172.21.15.121:35344) 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc1fbfff700 1 -- 172.21.15.121:0/2397200760 learned_addr learned my addr 172.21.15.121:0/2397200760 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc200ce2700 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f4a05880 0x55f2f4a08f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/2397200760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f4a05880 msgr2=0x55f2f4a08f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f4a05880 0x55f2f4a08f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/2397200760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f49ff7a0 msgr2=0x55f2f4a0b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a0b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/2397200760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2f4916750 con 0x55f2f4a04f60 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.629+0000 7fc1fbfff700 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a0b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 0x55f2f4a089e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc1f00a56e0 tx=0x7fc1f00d05a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a2b4faa1a57ef3 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc1db7fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc1db7fe700 1 -- 172.21.15.121:0/2397200760 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f00a5520 con 0x55f2f4a04f60 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc1db7fe700 1 -- 172.21.15.121:0/2397200760 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fc1f00870e0 con 0x55f2f4a04f60 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2397200760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1f009c940 con 0x55f2f4a04f60 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc1db7fe700 1 -- 172.21.15.121:0/2397200760 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc1f009d040 con 0x55f2f4a04f60 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc20cfc8b80 10 client.15909 fetch_fsmap learned FSMap version 28 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2397200760 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f2f4a1ab60 con 0x55f2f4a04f60 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc1db7fe700 1 -- 172.21.15.121:0/2397200760 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f009c940 con 0x55f2f4a04f60 2024-06-10T00:30:38.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc1db7fe700 1 -- 172.21.15.121:0/2397200760 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc1f00a5090 con 0x55f2f4a04f60 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc20cfc8b80 10 client.15909 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc20cfc8b80 10 client.15909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc20cfc8b80 20 client.15909 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f2f4a1d030 0x55f2f4a1f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc20cfc8b80 4 client.15909 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.630+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2397200760 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2f495dd50 con 0x55f2f4a1d030 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.631+0000 7fc1fbfff700 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f2f4a1d030 0x55f2f4a1f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.631+0000 7fc1db7fe700 10 client.15909 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.631+0000 7fc1fbfff700 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f2f4a1d030 0x55f2f4a1f420 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8bc6e843308386d server_cookie=6ac28cddea5407ed in_seq=0 out_seq=0 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.632+0000 7fc1db7fe700 1 -- 172.21.15.121:0/2397200760 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55f2f495dd50 con 0x55f2f4a1d030 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.632+0000 7fc1db7fe700 10 client.15909 handle_command_reply: tid=0 2024-06-10T00:30:38.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.632+0000 7fc20cfc8b80 1 client.15909 shutdown 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2397200760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 msgr2=0x55f2f4a089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 0x55f2f4a089e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc1f00a56e0 tx=0x7fc1f00d05a0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2397200760 shutdown_connections 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f2f4a1d030 0x55f2f4a1f420 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc1db7fe700 0 client.15909 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f4a05880 0x55f2f4a08f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a0b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/2397200760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 0x55f2f4a089e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2397200760 >> 172.21.15.121:0/2397200760 conn(0x55f2f492c080 msgr2=0x55f2f49f9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2397200760 shutdown_connections 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2397200760 wait complete. 2024-06-10T00:30:38.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.633+0000 7fc20cfc8b80 20 client.15909 trim_cache size 0 max 16384 2024-06-10T00:30:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc20cfc8b80 1 Processor -- start 2024-06-10T00:30:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc20cfc8b80 1 -- start start 2024-06-10T00:30:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f49ff260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 0x55f2f4a06430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a05880 0x55f2f49f9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc1aeae0 con 0x55f2f4a04f60 2024-06-10T00:30:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc1ac370 con 0x55f2f4a05880 2024-06-10T00:30:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc1aa1c0 con 0x55f2f49ff7a0 2024-06-10T00:30:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc1fb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 0x55f2f4a06430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc1fb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 0x55f2f4a06430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55428/0 (socket says 172.21.15.121:55428) 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/2774331309 learned_addr learned my addr 172.21.15.121:0/2774331309 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.656+0000 7fc200ce2700 1 --2- 172.21.15.121:0/2774331309 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a05880 0x55f2f49f9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.657+0000 7fc1fbfff700 1 --2- 172.21.15.121:0/2774331309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f49ff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.657+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/2774331309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 msgr2=0x55f2f49ff260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.657+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/2774331309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f49ff260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.657+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/2774331309 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a05880 msgr2=0x55f2f49f9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.657+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/2774331309 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a05880 0x55f2f49f9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.657+0000 7fc1fb7fe700 1 -- 172.21.15.121:0/2774331309 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2f4a07940 con 0x55f2f4a04f60 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.657+0000 7fc1fb7fe700 1 --2- 172.21.15.121:0/2774331309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 0x55f2f4a06430 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0000c00 tx=0x7fc1f009c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=278a9ce45c5ab1c3 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.657+0000 7fc1e1ffb700 1 -- 172.21.15.121:0/2774331309 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f00ba080 con 0x55f2f4a04f60 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.657+0000 7fc1e1ffb700 1 -- 172.21.15.121:0/2774331309 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f00a5520 con 0x55f2f4a04f60 2024-06-10T00:30:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.657+0000 7fc1e1ffb700 1 -- 172.21.15.121:0/2774331309 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f009c940 con 0x55f2f4a04f60 2024-06-10T00:30:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.658+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2774331309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 msgr2=0x55f2f4a06430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.658+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/2774331309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 0x55f2f4a06430 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0000c00 tx=0x7fc1f009c7e0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.658+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2774331309 shutdown_connections 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.658+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/2774331309 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a05880 0x55f2f49f9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.658+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/2774331309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f49ff260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.658+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/2774331309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a04f60 0x55f2f4a06430 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.658+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2774331309 >> 172.21.15.121:0/2774331309 conn(0x55f2f492c080 msgr2=0x55f2f4a05d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.658+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2774331309 shutdown_connections 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.658+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/2774331309 wait complete. 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.658+0000 7fc20cfc8b80 1 -- start start 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.658+0000 7fc20cfc8b80 1 Processor -- start 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a15d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 0x55f2f4a24500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc20cfc8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 0x55f2f4a24a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc1aa830 con 0x55f2f4a05880 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc1abca0 con 0x55f2f4a04f60 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc20cfc8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc1fc1aaf00 con 0x55f2f49ff7a0 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc20cfc8b80 10 client.0 _ll_register_callbacks cb 0x55f2f48e65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc1fbfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a15d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc200ce2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 0x55f2f4a24a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc1fb7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 0x55f2f4a24500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc1fbfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a15d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:51680/0 (socket says 172.21.15.121:51680) 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc1fb7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 0x55f2f4a24500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:35374/0 (socket says 172.21.15.121:35374) 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc1fbfff700 1 -- 172.21.15.121:0/1267642414 learned_addr learned my addr 172.21.15.121:0/1267642414 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc200ce2700 1 -- 172.21.15.121:0/1267642414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 msgr2=0x55f2f4a15d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc200ce2700 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a15d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc200ce2700 1 -- 172.21.15.121:0/1267642414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 msgr2=0x55f2f4a24500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:38.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc200ce2700 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 0x55f2f4a24500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc200ce2700 1 -- 172.21.15.121:0/1267642414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2f4a25270 con 0x55f2f4a05880 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc1fbfff700 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a15d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc200ce2700 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 0x55f2f4a24a40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x55f2f494c480 tx=0x7fc1f4007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98af9a93feded2f1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.659+0000 7fc1db7fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.660+0000 7fc1db7fe700 1 -- 172.21.15.121:0/1267642414 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4009e60 con 0x55f2f4a05880 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.660+0000 7fc1db7fe700 1 -- 172.21.15.121:0/1267642414 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fc1f4004db0 con 0x55f2f4a05880 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.660+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/1267642414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f2f49582d0 con 0x55f2f4a05880 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.660+0000 7fc1db7fe700 1 -- 172.21.15.121:0/1267642414 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc1f4003a00 con 0x55f2f4a05880 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.660+0000 7fc20cfc8b80 10 client.15915 fetch_fsmap learned FSMap version 28 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.660+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/1267642414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f2f4a87c10 con 0x55f2f4a05880 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.660+0000 7fc1db7fe700 1 -- 172.21.15.121:0/1267642414 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f2f49582d0 con 0x55f2f4a05880 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.660+0000 7fc1db7fe700 1 -- 172.21.15.121:0/1267642414 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc1f4004170 con 0x55f2f4a05880 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.661+0000 7fc20cfc8b80 10 client.15915 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:30:38.664 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.661+0000 7fc20cfc8b80 10 client.15915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:30:38.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.661+0000 7fc20cfc8b80 20 client.15915 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:38.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.661+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f2f4a883a0 0x55f2f4a8a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:38.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.661+0000 7fc20cfc8b80 4 client.15915 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:30:38.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.661+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/1267642414 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f2f490bf50 con 0x55f2f4a883a0 2024-06-10T00:30:38.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.661+0000 7fc1fbfff700 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f2f4a883a0 0x55f2f4a8a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:38.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.661+0000 7fc1db7fe700 10 client.15915 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:38.665 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.661+0000 7fc1fbfff700 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f2f4a883a0 0x55f2f4a8a7c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f5313e0f1313a01 server_cookie=eab6dc3deb9326d1 in_seq=0 out_seq=0 2024-06-10T00:30:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc1db7fe700 1 -- 172.21.15.121:0/1267642414 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f2f490bf50 con 0x55f2f4a883a0 2024-06-10T00:30:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc1db7fe700 10 client.15915 handle_command_reply: tid=0 2024-06-10T00:30:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 1 client.15915 shutdown 2024-06-10T00:30:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/1267642414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 msgr2=0x55f2f4a24a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 0x55f2f4a24a40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x55f2f494c480 tx=0x7fc1f4007f40 comp rx=0 tx=0).stop 2024-06-10T00:30:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/1267642414 shutdown_connections 2024-06-10T00:30:38.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f2f4a883a0 0x55f2f4a8a7c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f2f4a05880 0x55f2f4a24a40 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f2f49ff7a0 0x55f2f4a15d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc1db7fe700 0 client.15915 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 1 --2- 172.21.15.121:0/1267642414 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f2f4a04f60 0x55f2f4a24500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/1267642414 >> 172.21.15.121:0/1267642414 conn(0x55f2f492c080 msgr2=0x55f2f4a06530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/1267642414 shutdown_connections 2024-06-10T00:30:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 1 -- 172.21.15.121:0/1267642414 wait complete. 2024-06-10T00:30:38.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.663+0000 7fc20cfc8b80 20 client.15915 trim_cache size 0 max 16384 2024-06-10T00:30:38.667 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:30:38.667 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "no active scrubs running", 2024-06-10T00:30:38.668 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": {} 2024-06-10T00:30:38.668 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:30:38.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 -- 172.21.15.121:0/520158893 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc1e40a2d90 msgr2=0x7fc1e40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:38.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc1e40a2d90 0x7fc1e40a5250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc199ba0 tx=0x7fc1ec009260 comp rx=0 tx=0).stop 2024-06-10T00:30:38.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 -- 172.21.15.121:0/520158893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc06ab30 msgr2=0x7fc1fc19ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc06ab30 0x7fc1fc19ede0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc1f00059d0 tx=0x7fc1f000bab0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 -- 172.21.15.121:0/520158893 shutdown_connections 2024-06-10T00:30:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc1e40a2d90 0x7fc1e40a5250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1fc108500 0x7fc1fc198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc1fc06a170 0x7fc1fc19e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 --2- 172.21.15.121:0/520158893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1fc06ab30 0x7fc1fc19ede0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 -- 172.21.15.121:0/520158893 >> 172.21.15.121:0/520158893 conn(0x7fc1fc078340 msgr2=0x7fc1fc100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 -- 172.21.15.121:0/520158893 shutdown_connections 2024-06-10T00:30:38.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:38.664+0000 7fc202745700 1 -- 172.21.15.121:0/520158893 wait complete. 2024-06-10T00:30:39.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:38 smithi135 ceph-mon[36424]: pgmap v83: 129 pgs: 129 active+clean; 74 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 138 op/s 2024-06-10T00:30:39.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:38 smithi193 ceph-mon[36716]: pgmap v83: 129 pgs: 129 active+clean; 74 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 138 op/s 2024-06-10T00:30:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:38 smithi121 ceph-mon[32378]: pgmap v83: 129 pgs: 129 active+clean; 74 MiB data, 472 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 138 op/s 2024-06-10T00:30:39.540 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-10T00:30:39.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:30:39.972 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:30:41.255 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:41 smithi121 ceph-mon[32378]: pgmap v84: 129 pgs: 129 active+clean; 80 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 178 op/s 2024-06-10T00:30:41.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:41 smithi135 ceph-mon[36424]: pgmap v84: 129 pgs: 129 active+clean; 80 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 178 op/s 2024-06-10T00:30:41.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:41 smithi193 ceph-mon[36716]: pgmap v84: 129 pgs: 129 active+clean; 80 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.9 MiB/s wr, 178 op/s 2024-06-10T00:30:42.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.263+0000 7fd196900700 1 -- 172.21.15.121:0/1477229558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd190101d50 msgr2=0x7fd19010e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:42.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.263+0000 7fd196900700 1 --2- 172.21.15.121:0/1477229558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd190101d50 0x7fd19010e240 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd18800b0a0 tx=0x7fd18800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:30:42.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.263+0000 7fd196900700 1 -- 172.21.15.121:0/1477229558 shutdown_connections 2024-06-10T00:30:42.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.263+0000 7fd196900700 1 --2- 172.21.15.121:0/1477229558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd190101d50 0x7fd19010e240 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:42.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.263+0000 7fd196900700 1 --2- 172.21.15.121:0/1477229558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd190107400 0x7fd1901077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:42.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.263+0000 7fd196900700 1 --2- 172.21.15.121:0/1477229558 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd190101390 0x7fd190101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:42.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.263+0000 7fd196900700 1 -- 172.21.15.121:0/1477229558 >> 172.21.15.121:0/1477229558 conn(0x7fd190078340 msgr2=0x7fd190078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:42.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.263+0000 7fd196900700 1 -- 172.21.15.121:0/1477229558 shutdown_connections 2024-06-10T00:30:42.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.263+0000 7fd196900700 1 -- 172.21.15.121:0/1477229558 wait complete. 2024-06-10T00:30:42.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.264+0000 7fd196900700 1 Processor -- start 2024-06-10T00:30:42.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.264+0000 7fd196900700 1 -- start start 2024-06-10T00:30:42.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.264+0000 7fd196900700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd190101390 0x7fd1901999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:42.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.264+0000 7fd196900700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd190101d50 0x7fd190199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:42.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.264+0000 7fd196900700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd190107400 0x7fd1901a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:42.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.264+0000 7fd196900700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd190111670 con 0x7fd190101d50 2024-06-10T00:30:42.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.264+0000 7fd196900700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd1901114f0 con 0x7fd190101390 2024-06-10T00:30:42.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.264+0000 7fd196900700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd1901117f0 con 0x7fd190107400 2024-06-10T00:30:42.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.264+0000 7fd187fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd190101d50 0x7fd190199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:42.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd18ffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd190101390 0x7fd1901999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:42.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd187fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd190101d50 0x7fd190199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50650/0 (socket says 172.21.15.121:50650) 2024-06-10T00:30:42.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd187fff700 1 -- 172.21.15.121:0/1277719685 learned_addr learned my addr 172.21.15.121:0/1277719685 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:42.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd194e9d700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd190107400 0x7fd1901a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:42.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd187fff700 1 -- 172.21.15.121:0/1277719685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd190107400 msgr2=0x7fd1901a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:42.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd187fff700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd190107400 0x7fd1901a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:42.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd187fff700 1 -- 172.21.15.121:0/1277719685 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd190101390 msgr2=0x7fd1901999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:42.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd187fff700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd190101390 0x7fd1901999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:42.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd187fff700 1 -- 172.21.15.121:0/1277719685 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1901a4f70 con 0x7fd190101d50 2024-06-10T00:30:42.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd194e9d700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd190107400 0x7fd1901a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:30:42.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd187fff700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd190101d50 0x7fd190199ef0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd17800e9a0 tx=0x7fd17800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:42.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd18dffb700 1 -- 172.21.15.121:0/1277719685 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17800cbd0 con 0x7fd190101d50 2024-06-10T00:30:42.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd18dffb700 1 -- 172.21.15.121:0/1277719685 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd17800cd70 con 0x7fd190101d50 2024-06-10T00:30:42.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd18dffb700 1 -- 172.21.15.121:0/1277719685 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd178018750 con 0x7fd190101d50 2024-06-10T00:30:42.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd196900700 1 -- 172.21.15.121:0/1277719685 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1901a51a0 con 0x7fd190101d50 2024-06-10T00:30:42.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.265+0000 7fd196900700 1 -- 172.21.15.121:0/1277719685 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1901a5700 con 0x7fd190101d50 2024-06-10T00:30:42.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.266+0000 7fd196900700 1 -- 172.21.15.121:0/1277719685 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd190069180 con 0x7fd190101d50 2024-06-10T00:30:42.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.267+0000 7fd18dffb700 1 -- 172.21.15.121:0/1277719685 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd178010430 con 0x7fd190101d50 2024-06-10T00:30:42.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.268+0000 7fd18dffb700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd17c0a2e10 0x7fd17c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:42.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.268+0000 7fd18ffff700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd17c0a2e10 0x7fd17c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:42.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.268+0000 7fd18dffb700 1 -- 172.21.15.121:0/1277719685 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fd1780cfe90 con 0x7fd190101d50 2024-06-10T00:30:42.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.268+0000 7fd18ffff700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd17c0a2e10 0x7fd17c0a52d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd18000abd0 tx=0x7fd18000a380 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:42.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.271+0000 7fd18dffb700 1 -- 172.21.15.121:0/1277719685 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd17809c720 con 0x7fd190101d50 2024-06-10T00:30:42.538 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.538+0000 7fd196900700 1 -- 172.21.15.121:0/1277719685 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1901a5a20 con 0x7fd190101d50 2024-06-10T00:30:42.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.539+0000 7fd18dffb700 1 -- 172.21.15.121:0/1277719685 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11888 (secure 0 0 0) 0x7fd17809c020 con 0x7fd190101d50 2024-06-10T00:30:42.539 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:30:42.539 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:30:26.280580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:30:42.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 -- 172.21.15.121:0/1277719685 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd17c0a2e10 msgr2=0x7fd17c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:42.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd17c0a2e10 0x7fd17c0a52d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd18000abd0 tx=0x7fd18000a380 comp rx=0 tx=0).stop 2024-06-10T00:30:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 -- 172.21.15.121:0/1277719685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd190101d50 msgr2=0x7fd190199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd190101d50 0x7fd190199ef0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd17800e9a0 tx=0x7fd17800ecb0 comp rx=0 tx=0).stop 2024-06-10T00:30:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 -- 172.21.15.121:0/1277719685 shutdown_connections 2024-06-10T00:30:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd17c0a2e10 0x7fd17c0a52d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd190107400 0x7fd1901a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd190101390 0x7fd1901999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 --2- 172.21.15.121:0/1277719685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd190101d50 0x7fd190199ef0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 -- 172.21.15.121:0/1277719685 >> 172.21.15.121:0/1277719685 conn(0x7fd190078340 msgr2=0x7fd1900ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 -- 172.21.15.121:0/1277719685 shutdown_connections 2024-06-10T00:30:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:42.541+0000 7fd196900700 1 -- 172.21.15.121:0/1277719685 wait complete. 2024-06-10T00:30:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 28 2024-06-10T00:30:43.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:43 smithi135 ceph-mon[36424]: pgmap v85: 129 pgs: 129 active+clean; 80 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 67 op/s 2024-06-10T00:30:43.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:43 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1277719685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:30:43.323 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:43 smithi121 ceph-mon[32378]: pgmap v85: 129 pgs: 129 active+clean; 80 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 67 op/s 2024-06-10T00:30:43.323 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:43 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1277719685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:30:43.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-10T00:30:43.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:43 smithi193 ceph-mon[36716]: pgmap v85: 129 pgs: 129 active+clean; 80 MiB data, 480 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 67 op/s 2024-06-10T00:30:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:43 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1277719685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:30:43.658 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:30:45.127 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:45 smithi121 ceph-mon[32378]: pgmap v86: 129 pgs: 129 active+clean; 148 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.7 MiB/s wr, 444 op/s 2024-06-10T00:30:45.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:45 smithi135 ceph-mon[36424]: pgmap v86: 129 pgs: 129 active+clean; 148 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.7 MiB/s wr, 444 op/s 2024-06-10T00:30:45.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:45 smithi193 ceph-mon[36716]: pgmap v86: 129 pgs: 129 active+clean; 148 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.7 MiB/s wr, 444 op/s 2024-06-10T00:30:45.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.556+0000 7fba52acb700 1 -- 172.21.15.121:0/2476046742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba4c102540 msgr2=0x7fba4c1029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:45.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.556+0000 7fba52acb700 1 --2- 172.21.15.121:0/2476046742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba4c102540 0x7fba4c1029a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009990 tx=0x7fba3c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.556+0000 7fba52acb700 1 -- 172.21.15.121:0/2476046742 shutdown_connections 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.556+0000 7fba52acb700 1 --2- 172.21.15.121:0/2476046742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba4c102ee0 0x7fba4c10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.556+0000 7fba52acb700 1 --2- 172.21.15.121:0/2476046742 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba4c107450 0x7fba4c107830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.556+0000 7fba52acb700 1 --2- 172.21.15.121:0/2476046742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba4c102540 0x7fba4c1029a0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.556+0000 7fba52acb700 1 -- 172.21.15.121:0/2476046742 >> 172.21.15.121:0/2476046742 conn(0x7fba4c0795f0 msgr2=0x7fba4c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.556+0000 7fba52acb700 1 -- 172.21.15.121:0/2476046742 shutdown_connections 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.556+0000 7fba52acb700 1 -- 172.21.15.121:0/2476046742 wait complete. 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba52acb700 1 Processor -- start 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba52acb700 1 -- start start 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba52acb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba4c102540 0x7fba4c19e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba52acb700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba4c102ee0 0x7fba4c19edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba52acb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba4c107450 0x7fba4c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba52acb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c1127e0 con 0x7fba4c102540 2024-06-10T00:30:45.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba52acb700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c112660 con 0x7fba4c107450 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba52acb700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c112960 con 0x7fba4c102ee0 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba50867700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba4c102540 0x7fba4c19e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba50867700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba4c102540 0x7fba4c19e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50674/0 (socket says 172.21.15.121:50674) 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba50867700 1 -- 172.21.15.121:0/2386004090 learned_addr learned my addr 172.21.15.121:0/2386004090 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba4bfff700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba4c102ee0 0x7fba4c19edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.557+0000 7fba51068700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba4c107450 0x7fba4c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba4bfff700 1 -- 172.21.15.121:0/2386004090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba4c107450 msgr2=0x7fba4c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba4bfff700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba4c107450 0x7fba4c198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba4bfff700 1 -- 172.21.15.121:0/2386004090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba4c102540 msgr2=0x7fba4c19e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba4bfff700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba4c102540 0x7fba4c19e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba4bfff700 1 -- 172.21.15.121:0/2386004090 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4c199220 con 0x7fba4c102ee0 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba50867700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba4c102540 0x7fba4c19e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:45.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba4bfff700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba4c102ee0 0x7fba4c19edd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0059d0 tx=0x7fba3c00bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:45.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba49ffb700 1 -- 172.21.15.121:0/2386004090 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c027070 con 0x7fba4c102ee0 2024-06-10T00:30:45.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba52acb700 1 -- 172.21.15.121:0/2386004090 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4c1994b0 con 0x7fba4c102ee0 2024-06-10T00:30:45.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba49ffb700 1 -- 172.21.15.121:0/2386004090 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba3c003d70 con 0x7fba4c102ee0 2024-06-10T00:30:45.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba49ffb700 1 -- 172.21.15.121:0/2386004090 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c022e30 con 0x7fba4c102ee0 2024-06-10T00:30:45.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.558+0000 7fba52acb700 1 -- 172.21.15.121:0/2386004090 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4c1a5760 con 0x7fba4c102ee0 2024-06-10T00:30:45.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.561+0000 7fba49ffb700 1 -- 172.21.15.121:0/2386004090 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fba3c02a070 con 0x7fba4c102ee0 2024-06-10T00:30:45.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.562+0000 7fba49ffb700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba2c0a2d90 0x7fba2c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.562+0000 7fba49ffb700 1 -- 172.21.15.121:0/2386004090 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fba3c0d04a0 con 0x7fba4c102ee0 2024-06-10T00:30:45.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.562+0000 7fba50867700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba2c0a2d90 0x7fba2c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.562+0000 7fba50867700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba2c0a2d90 0x7fba2c0a5250 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fba4c19a240 tx=0x7fba4400b040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:45.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.562+0000 7fba5d34eb80 1 Processor -- start 2024-06-10T00:30:45.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba5d34eb80 1 -- start start 2024-06-10T00:30:45.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 0x563a32d70e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 0x563a32d6adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d6b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x563a32cb8810 con 0x563a32d71370 2024-06-10T00:30:45.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x563a32c60720 con 0x563a32d70a50 2024-06-10T00:30:45.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563a32cb47d0 con 0x563a32d6b330 2024-06-10T00:30:45.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba50867700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 0x563a32d70e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba4bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 0x563a32d6adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba50867700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 0x563a32d70e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:60414/0 (socket says 172.21.15.121:60414) 2024-06-10T00:30:45.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba50867700 1 -- 172.21.15.121:0/1839086987 learned_addr learned my addr 172.21.15.121:0/1839086987 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:45.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba4bfff700 1 -- 172.21.15.121:0/1839086987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 msgr2=0x563a32d6b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:45.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba51068700 1 --2- 172.21.15.121:0/1839086987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d6b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba4bfff700 1 --2- 172.21.15.121:0/1839086987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d6b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba4bfff700 1 -- 172.21.15.121:0/1839086987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 msgr2=0x563a32d70e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:45.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba4bfff700 1 --2- 172.21.15.121:0/1839086987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 0x563a32d70e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba4bfff700 1 -- 172.21.15.121:0/1839086987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a32c56f50 con 0x563a32d71370 2024-06-10T00:30:45.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.563+0000 7fba50867700 1 --2- 172.21.15.121:0/1839086987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 0x563a32d70e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:45.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.564+0000 7fba4bfff700 1 --2- 172.21.15.121:0/1839086987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 0x563a32d6adf0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0cd900 tx=0x7fba3c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e18bc953ba89c2 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:45.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.564+0000 7fba42ffd700 1 -- 172.21.15.121:0/1839086987 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0095d0 con 0x563a32d71370 2024-06-10T00:30:45.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.564+0000 7fba42ffd700 1 -- 172.21.15.121:0/1839086987 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba3c09ce60 con 0x563a32d71370 2024-06-10T00:30:45.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.564+0000 7fba42ffd700 1 -- 172.21.15.121:0/1839086987 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c09c9c0 con 0x563a32d71370 2024-06-10T00:30:45.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.565+0000 7fba5d34eb80 1 -- 172.21.15.121:0/1839086987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 msgr2=0x563a32d6adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:45.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.565+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/1839086987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 0x563a32d6adf0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0cd900 tx=0x7fba3c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.565+0000 7fba5d34eb80 1 -- 172.21.15.121:0/1839086987 shutdown_connections 2024-06-10T00:30:45.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.565+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/1839086987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d6b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.565+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/1839086987 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 0x563a32d70e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.565+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/1839086987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 0x563a32d6adf0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.565+0000 7fba5d34eb80 1 -- 172.21.15.121:0/1839086987 >> 172.21.15.121:0/1839086987 conn(0x563a32c91cc0 msgr2=0x563a32d61850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:45.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.566+0000 7fba5d34eb80 1 -- 172.21.15.121:0/1839086987 shutdown_connections 2024-06-10T00:30:45.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.566+0000 7fba5d34eb80 1 -- 172.21.15.121:0/1839086987 wait complete. 2024-06-10T00:30:45.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.566+0000 7fba5d34eb80 1 -- start start 2024-06-10T00:30:45.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.566+0000 7fba5d34eb80 1 Processor -- start 2024-06-10T00:30:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d77230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 0x563a32d74530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 0x563a32d74a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x563a32d77b00 con 0x563a32d71370 2024-06-10T00:30:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x563a32c58bf0 con 0x563a32d70a50 2024-06-10T00:30:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563a32d77c80 con 0x563a32d6b330 2024-06-10T00:30:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba5d34eb80 10 client.0 _ll_register_callbacks cb 0x563a32c52920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:45.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba51068700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 0x563a32d74a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba51068700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 0x563a32d74a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50686/0 (socket says 172.21.15.121:50686) 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba51068700 1 -- 172.21.15.121:0/3818984794 learned_addr learned my addr 172.21.15.121:0/3818984794 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba4bfff700 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 0x563a32d74530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba50867700 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d77230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba4bfff700 1 -- 172.21.15.121:0/3818984794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 msgr2=0x563a32d77230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba4bfff700 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d77230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba4bfff700 1 -- 172.21.15.121:0/3818984794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 msgr2=0x563a32d74a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba4bfff700 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 0x563a32d74a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba4bfff700 1 -- 172.21.15.121:0/3818984794 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a32c078d0 con 0x563a32d70a50 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.567+0000 7fba50867700 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d77230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.568+0000 7fba4bfff700 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 0x563a32d74530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0cd930 tx=0x7fba3c005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5428d379383935f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.568+0000 7fba3bfff700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:30:45.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.568+0000 7fba3bfff700 1 -- 172.21.15.121:0/3818984794 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0ba060 con 0x563a32d70a50 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.568+0000 7fba3bfff700 1 -- 172.21.15.121:0/3818984794 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fba3c09cbc0 con 0x563a32d70a50 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.568+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3818984794 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba3c09ce60 con 0x563a32d70a50 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.568+0000 7fba3bfff700 1 -- 172.21.15.121:0/3818984794 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fba3c0a3070 con 0x563a32d70a50 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.568+0000 7fba5d34eb80 10 client.25511 fetch_fsmap learned FSMap version 28 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.568+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3818984794 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563a32d77b00 con 0x563a32d70a50 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.569+0000 7fba3bfff700 1 -- 172.21.15.121:0/3818984794 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba3c09ce60 con 0x563a32d70a50 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.569+0000 7fba3bfff700 1 -- 172.21.15.121:0/3818984794 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fba3c09bc40 con 0x563a32d70a50 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.569+0000 7fba5d34eb80 10 client.25511 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.569+0000 7fba5d34eb80 10 client.25511 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.569+0000 7fba5d34eb80 20 client.25511 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.569+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x563a32d84870 0x563a32d86c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.569+0000 7fba5d34eb80 4 client.25511 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.569+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3818984794 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a32cc5d10 con 0x563a32d84870 2024-06-10T00:30:45.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.569+0000 7fba50867700 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x563a32d84870 0x563a32d86c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.570+0000 7fba50867700 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x563a32d84870 0x563a32d86c60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5b07fdf3e744a881 server_cookie=d49ddb16bcc1843b in_seq=0 out_seq=0 2024-06-10T00:30:45.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.570+0000 7fba3bfff700 10 client.25511 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:30:45.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba3bfff700 1 -- 172.21.15.121:0/3818984794 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x563a32cc5d10 con 0x563a32d84870 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba3bfff700 10 client.25511 handle_command_reply: tid=0 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba5d34eb80 1 client.25511 shutdown 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3818984794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 msgr2=0x563a32d74530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 0x563a32d74530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0cd930 tx=0x7fba3c005fb0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3818984794 shutdown_connections 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x563a32d84870 0x563a32d86c60 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d71370 0x563a32d74a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d77230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba3bfff700 0 client.25511 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3818984794 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d70a50 0x563a32d74530 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3818984794 >> 172.21.15.121:0/3818984794 conn(0x563a32c91cc0 msgr2=0x563a32d69de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:45.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3818984794 shutdown_connections 2024-06-10T00:30:45.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.571+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3818984794 wait complete. 2024-06-10T00:30:45.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.572+0000 7fba5d34eb80 20 client.25511 trim_cache size 0 max 16384 2024-06-10T00:30:45.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.601+0000 7fba5d34eb80 1 Processor -- start 2024-06-10T00:30:45.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.601+0000 7fba5d34eb80 1 -- start start 2024-06-10T00:30:45.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.601+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d6adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.601+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.601+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d71370 0x563a32d80da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.601+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c1aebb0 con 0x563a32d70a50 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.601+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c1ac440 con 0x563a32d71370 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.601+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c1aa2b0 con 0x563a32d6b330 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba4bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba51068700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d71370 0x563a32d80da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba4bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d720a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50690/0 (socket says 172.21.15.121:50690) 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba4bfff700 1 -- 172.21.15.121:0/418052080 learned_addr learned my addr 172.21.15.121:0/418052080 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba50867700 1 --2- 172.21.15.121:0/418052080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d6adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba4bfff700 1 -- 172.21.15.121:0/418052080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 msgr2=0x563a32d6adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba4bfff700 1 --2- 172.21.15.121:0/418052080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d6adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba4bfff700 1 -- 172.21.15.121:0/418052080 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d71370 msgr2=0x563a32d80da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba4bfff700 1 --2- 172.21.15.121:0/418052080 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d71370 0x563a32d80da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba4bfff700 1 -- 172.21.15.121:0/418052080 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a32d87290 con 0x563a32d70a50 2024-06-10T00:30:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba51068700 1 --2- 172.21.15.121:0/418052080 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d71370 0x563a32d80da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba4bfff700 1 --2- 172.21.15.121:0/418052080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d720a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fba3c098bc0 tx=0x7fba3c09ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0771df876e554c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba427fc700 1 -- 172.21.15.121:0/418052080 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0ba070 con 0x563a32d70a50 2024-06-10T00:30:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba427fc700 1 -- 172.21.15.121:0/418052080 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba3c09bdb0 con 0x563a32d70a50 2024-06-10T00:30:45.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.602+0000 7fba427fc700 1 -- 172.21.15.121:0/418052080 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0a5110 con 0x563a32d70a50 2024-06-10T00:30:45.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.603+0000 7fba5d34eb80 1 -- 172.21.15.121:0/418052080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 msgr2=0x563a32d720a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:45.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.603+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/418052080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d720a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fba3c098bc0 tx=0x7fba3c09ac70 comp rx=0 tx=0).stop 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.603+0000 7fba5d34eb80 1 -- 172.21.15.121:0/418052080 shutdown_connections 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.603+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/418052080 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d71370 0x563a32d80da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.603+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/418052080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d6b330 0x563a32d6adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.603+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/418052080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d720a0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.603+0000 7fba5d34eb80 1 -- 172.21.15.121:0/418052080 >> 172.21.15.121:0/418052080 conn(0x563a32c91cc0 msgr2=0x563a32d5ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.603+0000 7fba5d34eb80 1 -- 172.21.15.121:0/418052080 shutdown_connections 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.603+0000 7fba5d34eb80 1 -- 172.21.15.121:0/418052080 wait complete. 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.603+0000 7fba5d34eb80 1 -- start start 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba5d34eb80 1 Processor -- start 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d6b330 0x563a32d7d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d8bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba5d34eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d71370 0x563a32d8c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c1aa900 con 0x563a32d70a50 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c1abd70 con 0x563a32d6b330 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba5d34eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba4c1aafd0 con 0x563a32d71370 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba5d34eb80 10 client.0 _ll_register_callbacks cb 0x563a32c52920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba4bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d8bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba4bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d8bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50692/0 (socket says 172.21.15.121:50692) 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba4bfff700 1 -- 172.21.15.121:0/3822165739 learned_addr learned my addr 172.21.15.121:0/3822165739 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba50867700 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d6b330 0x563a32d7d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba51068700 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d71370 0x563a32d8c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba4bfff700 1 -- 172.21.15.121:0/3822165739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d71370 msgr2=0x563a32d8c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba4bfff700 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d71370 0x563a32d8c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba4bfff700 1 -- 172.21.15.121:0/3822165739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d6b330 msgr2=0x563a32d7d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.604+0000 7fba4bfff700 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d6b330 0x563a32d7d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.605+0000 7fba4bfff700 1 -- 172.21.15.121:0/3822165739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a32d8ca20 con 0x563a32d70a50 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.605+0000 7fba4bfff700 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d8bda0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fba3c09bfb0 tx=0x7fba3c003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61eb485334070dad server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.605+0000 7fba3bfff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.605+0000 7fba3bfff700 1 -- 172.21.15.121:0/3822165739 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0ba040 con 0x563a32d70a50 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.605+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3822165739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba3c09bdb0 con 0x563a32d70a50 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.605+0000 7fba3bfff700 1 -- 172.21.15.121:0/3822165739 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fba3c003aa0 con 0x563a32d70a50 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.605+0000 7fba3bfff700 1 -- 172.21.15.121:0/3822165739 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fba3c0a5340 con 0x563a32d70a50 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.605+0000 7fba5d34eb80 10 client.15939 fetch_fsmap learned FSMap version 28 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.605+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3822165739 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563a32d8ca20 con 0x563a32d70a50 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.605+0000 7fba3bfff700 1 -- 172.21.15.121:0/3822165739 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba3c09bdb0 con 0x563a32d70a50 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.605+0000 7fba3bfff700 1 -- 172.21.15.121:0/3822165739 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fba3c09db20 con 0x563a32d70a50 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.606+0000 7fba5d34eb80 10 client.15939 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.606+0000 7fba5d34eb80 10 client.15939 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:30:45.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.606+0000 7fba5d34eb80 20 client.15939 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:45.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.606+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x563a32def620 0x563a32df1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:45.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.606+0000 7fba5d34eb80 4 client.15939 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:30:45.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.606+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3822165739 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a32cc0290 con 0x563a32def620 2024-06-10T00:30:45.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.606+0000 7fba50867700 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x563a32def620 0x563a32df1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:45.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.606+0000 7fba3bfff700 10 client.15939 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:30:45.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.606+0000 7fba50867700 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x563a32def620 0x563a32df1a10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=779d7443a6042a99 server_cookie=cab107d7ae32eb31 in_seq=0 out_seq=0 2024-06-10T00:30:45.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.607+0000 7fba3bfff700 1 -- 172.21.15.121:0/3822165739 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a32cc0290 con 0x563a32def620 2024-06-10T00:30:45.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.607+0000 7fba3bfff700 10 client.15939 handle_command_reply: tid=0 2024-06-10T00:30:45.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.607+0000 7fba5d34eb80 1 client.15939 shutdown 2024-06-10T00:30:45.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3822165739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 msgr2=0x563a32d8bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:45.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d8bda0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fba3c09bfb0 tx=0x7fba3c003f60 comp rx=0 tx=0).stop 2024-06-10T00:30:45.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3822165739 shutdown_connections 2024-06-10T00:30:45.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x563a32def620 0x563a32df1a10 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a32d71370 0x563a32d8c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba3bfff700 0 client.15939 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:30:45.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563a32d6b330 0x563a32d7d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba5d34eb80 1 --2- 172.21.15.121:0/3822165739 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563a32d70a50 0x563a32d8bda0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3822165739 >> 172.21.15.121:0/3822165739 conn(0x563a32c91cc0 msgr2=0x563a32d72020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:45.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3822165739 shutdown_connections 2024-06-10T00:30:45.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba5d34eb80 1 -- 172.21.15.121:0/3822165739 wait complete. 2024-06-10T00:30:45.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.608+0000 7fba5d34eb80 20 client.15939 trim_cache size 0 max 16384 2024-06-10T00:30:45.612 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:30:45.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 -- 172.21.15.121:0/2386004090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba2c0a2d90 msgr2=0x7fba2c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:45.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba2c0a2d90 0x7fba2c0a5250 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fba4c19a240 tx=0x7fba4400b040 comp rx=0 tx=0).stop 2024-06-10T00:30:45.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 -- 172.21.15.121:0/2386004090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba4c102ee0 msgr2=0x7fba4c19edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:45.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba4c102ee0 0x7fba4c19edd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0059d0 tx=0x7fba3c00bab0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 -- 172.21.15.121:0/2386004090 shutdown_connections 2024-06-10T00:30:45.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fba2c0a2d90 0x7fba2c0a5250 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fba4c107450 0x7fba4c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba4c102540 0x7fba4c19e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 --2- 172.21.15.121:0/2386004090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba4c102ee0 0x7fba4c19edd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:45.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 -- 172.21.15.121:0/2386004090 >> 172.21.15.121:0/2386004090 conn(0x7fba4c0795f0 msgr2=0x7fba4c100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:45.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 -- 172.21.15.121:0/2386004090 shutdown_connections 2024-06-10T00:30:45.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:45.609+0000 7fba52acb700 1 -- 172.21.15.121:0/2386004090 wait complete. 2024-06-10T00:30:46.448 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-10T00:30:46.721 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:30:47.108 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:47 smithi121 ceph-mon[32378]: pgmap v87: 129 pgs: 129 active+clean; 148 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 417 op/s 2024-06-10T00:30:47.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:47 smithi135 ceph-mon[36424]: pgmap v87: 129 pgs: 129 active+clean; 148 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 417 op/s 2024-06-10T00:30:47.292 INFO:tasks.workunit.client.0.smithi121.stdout: 36 254258 441 174819 558 103649 1068 2024-06-10T00:30:47.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:47 smithi193 ceph-mon[36716]: pgmap v87: 129 pgs: 129 active+clean; 148 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 417 op/s 2024-06-10T00:30:49.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:49 smithi121 ceph-mon[32378]: pgmap v88: 129 pgs: 129 active+clean; 148 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 417 op/s 2024-06-10T00:30:49.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:49 smithi193 ceph-mon[36716]: pgmap v88: 129 pgs: 129 active+clean; 148 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 417 op/s 2024-06-10T00:30:49.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:49 smithi135 ceph-mon[36424]: pgmap v88: 129 pgs: 129 active+clean; 148 MiB data, 560 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.8 MiB/s wr, 417 op/s 2024-06-10T00:30:50.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.104+0000 7f8b9e92f700 1 -- 172.21.15.121:0/1872259507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b98100b50 msgr2=0x7f8b98100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:50.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.104+0000 7f8b9e92f700 1 --2- 172.21.15.121:0/1872259507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b98100b50 0x7f8b98100fd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c009990 tx=0x7f8b8c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.104+0000 7f8b9e92f700 1 -- 172.21.15.121:0/1872259507 shutdown_connections 2024-06-10T00:30:50.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.104+0000 7f8b9e92f700 1 --2- 172.21.15.121:0/1872259507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b98101510 0x7f8b9810d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.104+0000 7f8b9e92f700 1 --2- 172.21.15.121:0/1872259507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b98106b90 0x7f8b98106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.104+0000 7f8b9e92f700 1 --2- 172.21.15.121:0/1872259507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b98100b50 0x7f8b98100fd0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.104+0000 7f8b9e92f700 1 -- 172.21.15.121:0/1872259507 >> 172.21.15.121:0/1872259507 conn(0x7f8b98077a00 msgr2=0x7f8b980fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:50.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.104+0000 7f8b9e92f700 1 -- 172.21.15.121:0/1872259507 shutdown_connections 2024-06-10T00:30:50.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.104+0000 7f8b9e92f700 1 -- 172.21.15.121:0/1872259507 wait complete. 2024-06-10T00:30:50.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.105+0000 7f8b9e92f700 1 Processor -- start 2024-06-10T00:30:50.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.105+0000 7f8b9e92f700 1 -- start start 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.105+0000 7f8b9e92f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b98100b50 0x7f8b9819c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b9e92f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b98101510 0x7f8b9819cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b9e92f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b98106b90 0x7f8b98196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b977fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b98101510 0x7f8b9819cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b977fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b98101510 0x7f8b9819cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:40100/0 (socket says 172.21.15.121:40100) 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b97fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b98100b50 0x7f8b9819c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b97fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b98100b50 0x7f8b9819c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:53364/0 (socket says 172.21.15.121:53364) 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b9cecc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b98106b90 0x7f8b98196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b9cecc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b98106b90 0x7f8b98196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40462/0 (socket says 172.21.15.121:40462) 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b9e92f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8b98110620 con 0x7f8b98106b90 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b9e92f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8b981104a0 con 0x7f8b98101510 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b977fe700 1 -- 172.21.15.121:0/3142618663 learned_addr learned my addr 172.21.15.121:0/3142618663 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.106+0000 7f8b9e92f700 1 -- 172.21.15.121:0/3142618663 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8b981107a0 con 0x7f8b98100b50 2024-06-10T00:30:50.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b977fe700 1 -- 172.21.15.121:0/3142618663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b98100b50 msgr2=0x7f8b9819c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:50.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b977fe700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b98100b50 0x7f8b9819c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b977fe700 1 -- 172.21.15.121:0/3142618663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b98106b90 msgr2=0x7f8b98196880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:50.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b977fe700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b98106b90 0x7f8b98196880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b977fe700 1 -- 172.21.15.121:0/3142618663 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b98197060 con 0x7f8b98101510 2024-06-10T00:30:50.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b9cecc700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b98106b90 0x7f8b98196880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:50.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b977fe700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b98101510 0x7f8b9819cbf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c005910 tx=0x7f8b8c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:50.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b957fa700 1 -- 172.21.15.121:0/3142618663 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c027070 con 0x7f8b98101510 2024-06-10T00:30:50.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b957fa700 1 -- 172.21.15.121:0/3142618663 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b8c003d70 con 0x7f8b98101510 2024-06-10T00:30:50.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b957fa700 1 -- 172.21.15.121:0/3142618663 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c00f460 con 0x7f8b98101510 2024-06-10T00:30:50.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b9e92f700 1 -- 172.21.15.121:0/3142618663 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b981972f0 con 0x7f8b98101510 2024-06-10T00:30:50.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.107+0000 7f8b9e92f700 1 -- 172.21.15.121:0/3142618663 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b981a3510 con 0x7f8b98101510 2024-06-10T00:30:50.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.110+0000 7f8b957fa700 1 -- 172.21.15.121:0/3142618663 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8b8c02a070 con 0x7f8b98101510 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.111+0000 7f8b957fa700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8b800a4f90 0x7f8b800a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.111+0000 7f8b957fa700 1 -- 172.21.15.121:0/3142618663 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8b8c0d0560 con 0x7f8b98101510 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.111+0000 7f8b97fff700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8b800a4f90 0x7f8b800a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8b97fff700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8b800a4f90 0x7f8b800a7450 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8b88009bd0 tx=0x7f8b88009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8ba91b2b80 1 Processor -- start 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8ba91b2b80 1 -- start start 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f5f60 0x55ced59f6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f6880 0x55ced59f0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f07a0 0x55ced59f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ced58e2f60 con 0x55ced59f5f60 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55ced5941850 con 0x55ced59f6880 2024-06-10T00:30:50.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ced58a9460 con 0x55ced59f07a0 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8b977fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f6880 0x55ced59f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8b9cecc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f07a0 0x55ced59f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8b97fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f5f60 0x55ced59f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8b977fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f6880 0x55ced59f0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:40106/0 (socket says 172.21.15.121:40106) 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.112+0000 7f8b977fe700 1 -- 172.21.15.121:0/2451870605 learned_addr learned my addr 172.21.15.121:0/2451870605 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.113+0000 7f8b9cecc700 1 -- 172.21.15.121:0/2451870605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f6880 msgr2=0x55ced59f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.113+0000 7f8b9cecc700 1 --2- 172.21.15.121:0/2451870605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f6880 0x55ced59f0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.113+0000 7f8b9cecc700 1 -- 172.21.15.121:0/2451870605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f5f60 msgr2=0x55ced59f6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.113+0000 7f8b9cecc700 1 --2- 172.21.15.121:0/2451870605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f5f60 0x55ced59f6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.113+0000 7f8b9cecc700 1 -- 172.21.15.121:0/2451870605 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ced58908d0 con 0x55ced59f07a0 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.113+0000 7f8b977fe700 1 --2- 172.21.15.121:0/2451870605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f6880 0x55ced59f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.113+0000 7f8b9cecc700 1 --2- 172.21.15.121:0/2451870605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f07a0 0x55ced59f0bc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8b90003db0 tx=0x7f8b90009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ab8c564a66dd67 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.113+0000 7f8b7e7fc700 1 -- 172.21.15.121:0/2451870605 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b90019030 con 0x55ced59f07a0 2024-06-10T00:30:50.120 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.113+0000 7f8b7e7fc700 1 -- 172.21.15.121:0/2451870605 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b90004bb0 con 0x55ced59f07a0 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.113+0000 7f8b7e7fc700 1 -- 172.21.15.121:0/2451870605 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b90007cd0 con 0x55ced59f07a0 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.114+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2451870605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f07a0 msgr2=0x55ced59f0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.114+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2451870605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f07a0 0x55ced59f0bc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8b90003db0 tx=0x7f8b90009920 comp rx=0 tx=0).stop 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.114+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2451870605 shutdown_connections 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.114+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2451870605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f07a0 0x55ced59f0bc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.114+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2451870605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f5f60 0x55ced59f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.114+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2451870605 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f6880 0x55ced59f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.114+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2451870605 >> 172.21.15.121:0/2451870605 conn(0x55ced591d080 msgr2=0x55ced59eb220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.114+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2451870605 shutdown_connections 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.114+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2451870605 wait complete. 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.115+0000 7f8ba91b2b80 1 -- start start 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.115+0000 7f8ba91b2b80 1 Processor -- start 2024-06-10T00:30:50.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.115+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59fc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.115+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f5f60 0x55ced59f9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.115+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f6880 0x55ced59f9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.115+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ced58e1fc0 con 0x55ced59f07a0 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.115+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55ced593d810 con 0x55ced59f5f60 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.115+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ced5a01670 con 0x55ced59f6880 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.115+0000 7f8ba91b2b80 10 client.0 _ll_register_callbacks cb 0x55ced58d75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.115+0000 7f8b97fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59fc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b97fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59fc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40474/0 (socket says 172.21.15.121:40474) 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b97fff700 1 -- 172.21.15.121:0/2062326061 learned_addr learned my addr 172.21.15.121:0/2062326061 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b9cecc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f6880 0x55ced59f9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b977fe700 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f5f60 0x55ced59f9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b9cecc700 1 -- 172.21.15.121:0/2062326061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f5f60 msgr2=0x55ced59f9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:50.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b9cecc700 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f5f60 0x55ced59f9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b9cecc700 1 -- 172.21.15.121:0/2062326061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 msgr2=0x55ced59fc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b9cecc700 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59fc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b9cecc700 1 -- 172.21.15.121:0/2062326061 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ced5907750 con 0x55ced59f6880 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b977fe700 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f5f60 0x55ced59f9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b97fff700 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59fc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b9cecc700 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f6880 0x55ced59f9e40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009e90 tx=0x7f8b90007860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d744a1e858d755e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b777fe700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b777fe700 1 -- 172.21.15.121:0/2062326061 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b90023070 con 0x55ced59f6880 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2062326061 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b90004bb0 con 0x55ced59f6880 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.116+0000 7f8b777fe700 1 -- 172.21.15.121:0/2062326061 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8b9000e4f0 con 0x55ced59f6880 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8b777fe700 1 -- 172.21.15.121:0/2062326061 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8b90031050 con 0x55ced59f6880 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8ba91b2b80 10 client.25453 fetch_fsmap learned FSMap version 28 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2062326061 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ced5a0be00 con 0x55ced59f6880 2024-06-10T00:30:50.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8b777fe700 1 -- 172.21.15.121:0/2062326061 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b90004bb0 con 0x55ced59f6880 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8b777fe700 1 -- 172.21.15.121:0/2062326061 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8b90028700 con 0x55ced59f6880 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8ba91b2b80 10 client.25453 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8ba91b2b80 10 client.25453 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8ba91b2b80 20 client.25453 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ced5a0e380 0x55ced5a10770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8ba91b2b80 4 client.25453 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2062326061 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ced594ed50 con 0x55ced5a0e380 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.117+0000 7f8b97fff700 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ced5a0e380 0x55ced5a10770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.118+0000 7f8b97fff700 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ced5a0e380 0x55ced5a10770 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf5d83d4cb817fc4 server_cookie=401d40830078f73b in_seq=0 out_seq=0 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.118+0000 7f8b777fe700 10 client.25453 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.118+0000 7f8b777fe700 1 -- 172.21.15.121:0/2062326061 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55ced594ed50 con 0x55ced5a0e380 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.118+0000 7f8b777fe700 10 client.25453 handle_command_reply: tid=0 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.118+0000 7f8ba91b2b80 1 client.25453 shutdown 2024-06-10T00:30:50.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2062326061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f6880 msgr2=0x55ced59f9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:50.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f6880 0x55ced59f9e40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009e90 tx=0x7f8b90007860 comp rx=0 tx=0).stop 2024-06-10T00:30:50.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2062326061 shutdown_connections 2024-06-10T00:30:50.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ced5a0e380 0x55ced5a10770 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f6880 0x55ced59f9e40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59fc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2062326061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced59f5f60 0x55ced59f9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2062326061 >> 172.21.15.121:0/2062326061 conn(0x55ced591d080 msgr2=0x55ced59ea700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:50.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8b777fe700 0 client.25453 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:30:50.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2062326061 shutdown_connections 2024-06-10T00:30:50.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2062326061 wait complete. 2024-06-10T00:30:50.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.119+0000 7f8ba91b2b80 20 client.25453 trim_cache size 0 max 16384 2024-06-10T00:30:50.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8ba91b2b80 1 Processor -- start 2024-06-10T00:30:50.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8ba91b2b80 1 -- start start 2024-06-10T00:30:50.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 0x55ced59f0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59f74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f5f60 0x55ced59fb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8b981aeb60 con 0x55ced59f07a0 2024-06-10T00:30:50.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8b981ac3f0 con 0x55ced591d080 2024-06-10T00:30:50.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8b981aa240 con 0x55ced59f5f60 2024-06-10T00:30:50.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8b977fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59f74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8b977fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59f74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40486/0 (socket says 172.21.15.121:40486) 2024-06-10T00:30:50.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8b97fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 0x55ced59f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8b977fe700 1 -- 172.21.15.121:0/460944995 learned_addr learned my addr 172.21.15.121:0/460944995 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.136+0000 7f8b9cecc700 1 --2- 172.21.15.121:0/460944995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f5f60 0x55ced59fb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.137+0000 7f8b97fff700 1 -- 172.21.15.121:0/460944995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f5f60 msgr2=0x55ced59fb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.137+0000 7f8b97fff700 1 --2- 172.21.15.121:0/460944995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f5f60 0x55ced59fb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.137+0000 7f8b97fff700 1 -- 172.21.15.121:0/460944995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 msgr2=0x55ced59f74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.137+0000 7f8b97fff700 1 --2- 172.21.15.121:0/460944995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59f74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.137+0000 7f8b97fff700 1 -- 172.21.15.121:0/460944995 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ced59fcde0 con 0x55ced591d080 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.137+0000 7f8b977fe700 1 --2- 172.21.15.121:0/460944995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59f74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.137+0000 7f8b97fff700 1 --2- 172.21.15.121:0/460944995 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 0x55ced59f0260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55ced59fad60 tx=0x7f8b88013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43335e5f36252ffd server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.137+0000 7f8b7dffb700 1 -- 172.21.15.121:0/460944995 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b88011e60 con 0x55ced591d080 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.137+0000 7f8b7dffb700 1 -- 172.21.15.121:0/460944995 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b8802a890 con 0x55ced591d080 2024-06-10T00:30:50.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.137+0000 7f8b7dffb700 1 -- 172.21.15.121:0/460944995 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8801fbc0 con 0x55ced591d080 2024-06-10T00:30:50.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.138+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/460944995 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 msgr2=0x55ced59f0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:50.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.138+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/460944995 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 0x55ced59f0260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55ced59fad60 tx=0x7f8b88013000 comp rx=0 tx=0).stop 2024-06-10T00:30:50.143 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.138+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/460944995 shutdown_connections 2024-06-10T00:30:50.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.138+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/460944995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f5f60 0x55ced59fb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.138+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/460944995 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 0x55ced59f0260 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.138+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/460944995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced59f74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.138+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/460944995 >> 172.21.15.121:0/460944995 conn(0x55ced59f41d0 msgr2=0x55ced59fb130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:50.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.138+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/460944995 shutdown_connections 2024-06-10T00:30:50.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.138+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/460944995 wait complete. 2024-06-10T00:30:50.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.139+0000 7f8ba91b2b80 1 -- start start 2024-06-10T00:30:50.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.139+0000 7f8ba91b2b80 1 Processor -- start 2024-06-10T00:30:50.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.139+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 0x55ced5a115d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.139+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced5a11b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.139+0000 7f8ba91b2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f5f60 0x55ced5a12050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.139+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8b981aa8b0 con 0x55ced59f07a0 2024-06-10T00:30:50.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.139+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8b981abd20 con 0x55ced591d080 2024-06-10T00:30:50.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.139+0000 7f8ba91b2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8b981aaf80 con 0x55ced59f5f60 2024-06-10T00:30:50.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8ba91b2b80 10 client.0 _ll_register_callbacks cb 0x55ced58d75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:50.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b977fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced5a11b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b977fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced5a11b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40488/0 (socket says 172.21.15.121:40488) 2024-06-10T00:30:50.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b97fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 0x55ced5a115d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b977fe700 1 -- 172.21.15.121:0/2564840694 learned_addr learned my addr 172.21.15.121:0/2564840694 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:50.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b9cecc700 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f5f60 0x55ced5a12050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b977fe700 1 -- 172.21.15.121:0/2564840694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f5f60 msgr2=0x55ced5a12050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b977fe700 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f5f60 0x55ced5a12050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b977fe700 1 -- 172.21.15.121:0/2564840694 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 msgr2=0x55ced5a115d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b977fe700 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 0x55ced5a115d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b977fe700 1 -- 172.21.15.121:0/2564840694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ced5a77d40 con 0x55ced59f07a0 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b97fff700 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 0x55ced5a115d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b977fe700 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced5a11b10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c09c710 tx=0x7f8b8c09ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6f82cfbe2f0d94 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b777fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b777fe700 1 -- 172.21.15.121:0/2564840694 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c0b4070 con 0x55ced59f07a0 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8b777fe700 1 -- 172.21.15.121:0/2564840694 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8b8c0d00f0 con 0x55ced59f07a0 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.140+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2564840694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b8802a890 con 0x55ced59f07a0 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.141+0000 7f8b777fe700 1 -- 172.21.15.121:0/2564840694 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8b8c09c0e0 con 0x55ced59f07a0 2024-06-10T00:30:50.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.141+0000 7f8ba91b2b80 10 client.15957 fetch_fsmap learned FSMap version 28 2024-06-10T00:30:50.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.141+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2564840694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ced5a79020 con 0x55ced59f07a0 2024-06-10T00:30:50.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.141+0000 7f8b777fe700 1 -- 172.21.15.121:0/2564840694 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b8802a890 con 0x55ced59f07a0 2024-06-10T00:30:50.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.141+0000 7f8b777fe700 1 -- 172.21.15.121:0/2564840694 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8b8c098040 con 0x55ced59f07a0 2024-06-10T00:30:50.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.141+0000 7f8ba91b2b80 10 client.15957 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:30:50.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.141+0000 7f8ba91b2b80 10 client.15957 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:30:50.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.141+0000 7f8ba91b2b80 20 client.15957 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:50.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.141+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ced59f6880 0x55ced5a7b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:50.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.141+0000 7f8ba91b2b80 4 client.15957 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:30:50.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.141+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2564840694 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ced59492d0 con 0x55ced59f6880 2024-06-10T00:30:50.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.142+0000 7f8b97fff700 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ced59f6880 0x55ced5a7b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:50.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.142+0000 7f8b97fff700 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ced59f6880 0x55ced5a7b7c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f7d2baf278ad7c7 server_cookie=2fa9fd85c3df8eed in_seq=0 out_seq=0 2024-06-10T00:30:50.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.142+0000 7f8b777fe700 10 client.15957 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:30:50.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.142+0000 7f8b777fe700 1 -- 172.21.15.121:0/2564840694 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ced59492d0 con 0x55ced59f6880 2024-06-10T00:30:50.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.142+0000 7f8b777fe700 10 client.15957 handle_command_reply: tid=0 2024-06-10T00:30:50.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.142+0000 7f8ba91b2b80 1 client.15957 shutdown 2024-06-10T00:30:50.149 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.142+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2564840694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 msgr2=0x55ced5a11b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:50.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.142+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced5a11b10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c09c710 tx=0x7f8b8c09ccc0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.143+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2564840694 shutdown_connections 2024-06-10T00:30:50.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.143+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55ced59f6880 0x55ced5a7b7c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.143+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ced59f5f60 0x55ced5a12050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.143+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ced591d080 0x55ced5a115d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.143+0000 7f8ba91b2b80 1 --2- 172.21.15.121:0/2564840694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ced59f07a0 0x55ced5a11b10 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.143+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2564840694 >> 172.21.15.121:0/2564840694 conn(0x55ced59f41d0 msgr2=0x55ced59f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:50.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.143+0000 7f8b777fe700 0 client.15957 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:30:50.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.143+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2564840694 shutdown_connections 2024-06-10T00:30:50.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.143+0000 7f8ba91b2b80 1 -- 172.21.15.121:0/2564840694 wait complete. 2024-06-10T00:30:50.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.143+0000 7f8ba91b2b80 20 client.15957 trim_cache size 0 max 16384 2024-06-10T00:30:50.151 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:30:50.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 -- 172.21.15.121:0/3142618663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8b800a4f90 msgr2=0x7f8b800a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:50.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8b800a4f90 0x7f8b800a7450 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8b88009bd0 tx=0x7f8b88009260 comp rx=0 tx=0).stop 2024-06-10T00:30:50.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 -- 172.21.15.121:0/3142618663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b98101510 msgr2=0x7f8b9819cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:50.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b98101510 0x7f8b9819cbf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c005910 tx=0x7f8b8c00bab0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 -- 172.21.15.121:0/3142618663 shutdown_connections 2024-06-10T00:30:50.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8b800a4f90 0x7f8b800a7450 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b98106b90 0x7f8b98196880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b98100b50 0x7f8b9819c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 --2- 172.21.15.121:0/3142618663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8b98101510 0x7f8b9819cbf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:50.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 -- 172.21.15.121:0/3142618663 >> 172.21.15.121:0/3142618663 conn(0x7f8b98077a00 msgr2=0x7f8b98103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:50.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 -- 172.21.15.121:0/3142618663 shutdown_connections 2024-06-10T00:30:50.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:50.144+0000 7f8b9e92f700 1 -- 172.21.15.121:0/3142618663 wait complete. 2024-06-10T00:30:51.059 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-10T00:30:51.380 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:30:51.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:51 smithi193 ceph-mon[36716]: pgmap v89: 129 pgs: 129 active+clean; 154 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.6 MiB/s wr, 479 op/s 2024-06-10T00:30:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:51 smithi121 ceph-mon[32378]: pgmap v89: 129 pgs: 129 active+clean; 154 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.6 MiB/s wr, 479 op/s 2024-06-10T00:30:51.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:51 smithi135 ceph-mon[36424]: pgmap v89: 129 pgs: 129 active+clean; 154 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 7.6 MiB/s wr, 479 op/s 2024-06-10T00:30:53.166 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:53 smithi121 ceph-mon[32378]: pgmap v90: 129 pgs: 129 active+clean; 154 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 439 op/s 2024-06-10T00:30:53.309 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:53 smithi135 ceph-mon[36424]: pgmap v90: 129 pgs: 129 active+clean; 154 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 439 op/s 2024-06-10T00:30:53.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:53 smithi193 ceph-mon[36716]: pgmap v90: 129 pgs: 129 active+clean; 154 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 7.0 MiB/s wr, 439 op/s 2024-06-10T00:30:53.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.423+0000 7f8fcf36f700 1 -- 172.21.15.121:0/1799700224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc806a170 msgr2=0x7f8fc806a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:53.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.423+0000 7f8fcf36f700 1 --2- 172.21.15.121:0/1799700224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc806a170 0x7f8fc806a5f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8009990 tx=0x7f8fb8009ca0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.423+0000 7f8fcf36f700 1 -- 172.21.15.121:0/1799700224 shutdown_connections 2024-06-10T00:30:53.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.423+0000 7f8fcf36f700 1 --2- 172.21.15.121:0/1799700224 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8fc806ab30 0x7f8fc810f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.423+0000 7f8fcf36f700 1 --2- 172.21.15.121:0/1799700224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fc8108500 0x7f8fc81088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.423+0000 7f8fcf36f700 1 --2- 172.21.15.121:0/1799700224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc806a170 0x7f8fc806a5f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.423+0000 7f8fcf36f700 1 -- 172.21.15.121:0/1799700224 >> 172.21.15.121:0/1799700224 conn(0x7f8fc8078340 msgr2=0x7f8fc8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:53.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.424+0000 7f8fcf36f700 1 -- 172.21.15.121:0/1799700224 shutdown_connections 2024-06-10T00:30:53.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.424+0000 7f8fcf36f700 1 -- 172.21.15.121:0/1799700224 wait complete. 2024-06-10T00:30:53.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.424+0000 7f8fcf36f700 1 Processor -- start 2024-06-10T00:30:53.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.425+0000 7f8fcf36f700 1 -- start start 2024-06-10T00:30:53.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcf36f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fc806a170 0x7f8fc819e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcf36f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8fc806ab30 0x7f8fc819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcf36f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc8108500 0x7f8fc8198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcd10b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fc806a170 0x7f8fc819e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcd10b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fc806a170 0x7f8fc819e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:53414/0 (socket says 172.21.15.121:53414) 2024-06-10T00:30:53.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcd10b700 1 -- 172.21.15.121:0/592520031 learned_addr learned my addr 172.21.15.121:0/592520031 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcf36f700 1 -- 172.21.15.121:0/592520031 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc8112760 con 0x7f8fc8108500 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcf36f700 1 -- 172.21.15.121:0/592520031 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc81125e0 con 0x7f8fc806ab30 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcf36f700 1 -- 172.21.15.121:0/592520031 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc81128e0 con 0x7f8fc806a170 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcd90c700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc8108500 0x7f8fc8198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcd10b700 1 -- 172.21.15.121:0/592520031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8fc806ab30 msgr2=0x7f8fc819eda0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8fc806ab30 0x7f8fc819eda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcd10b700 1 -- 172.21.15.121:0/592520031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc8108500 msgr2=0x7f8fc8198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc8108500 0x7f8fc8198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcd10b700 1 -- 172.21.15.121:0/592520031 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc8199180 con 0x7f8fc806a170 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.426+0000 7f8fcd90c700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc8108500 0x7f8fc8198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.427+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fc806a170 0x7f8fc819e860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400e9a0 tx=0x7f8fc400ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.427+0000 7f8fbe7fc700 1 -- 172.21.15.121:0/592520031 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc400cbd0 con 0x7f8fc806a170 2024-06-10T00:30:53.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.427+0000 7f8fcf36f700 1 -- 172.21.15.121:0/592520031 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc8199470 con 0x7f8fc806a170 2024-06-10T00:30:53.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.427+0000 7f8fcf36f700 1 -- 172.21.15.121:0/592520031 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fc81a5730 con 0x7f8fc806a170 2024-06-10T00:30:53.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.427+0000 7f8fbe7fc700 1 -- 172.21.15.121:0/592520031 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fc400cd70 con 0x7f8fc806a170 2024-06-10T00:30:53.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.427+0000 7f8fbe7fc700 1 -- 172.21.15.121:0/592520031 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4018750 con 0x7f8fc806a170 2024-06-10T00:30:53.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.430+0000 7f8fbe7fc700 1 -- 172.21.15.121:0/592520031 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8fc40188f0 con 0x7f8fc806a170 2024-06-10T00:30:53.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.431+0000 7f8fbe7fc700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8fb40a2e10 0x7f8fb40a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.431+0000 7f8fcc90a700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8fb40a2e10 0x7f8fb40a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.431+0000 7f8fbe7fc700 1 -- 172.21.15.121:0/592520031 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8fc40cf950 con 0x7f8fc806a170 2024-06-10T00:30:53.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.431+0000 7f8fd9bf2b80 1 Processor -- start 2024-06-10T00:30:53.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.431+0000 7f8fd9bf2b80 1 -- start start 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcc90a700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8fb40a2e10 0x7f8fb40a52d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80098e0 tx=0x7f8fb8005ab0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3dae50 0x561d7a3db230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3db770 0x561d7a3cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 0x561d7a3cd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561d7a31e810 con 0x561d7a3dae50 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x561d7a2c6720 con 0x561d7a3db770 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561d7a31a7d0 con 0x561d7a3ccf20 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcd10b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3dae50 0x561d7a3db230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcc90a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3db770 0x561d7a3cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcd10b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3dae50 0x561d7a3db230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40520/0 (socket says 172.21.15.121:40520) 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcd10b700 1 -- 172.21.15.121:0/1307830661 learned_addr learned my addr 172.21.15.121:0/1307830661 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:53.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcd90c700 1 --2- 172.21.15.121:0/1307830661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 0x561d7a3cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcd10b700 1 -- 172.21.15.121:0/1307830661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 msgr2=0x561d7a3cd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/1307830661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 0x561d7a3cd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcd10b700 1 -- 172.21.15.121:0/1307830661 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3db770 msgr2=0x561d7a3cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/1307830661 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3db770 0x561d7a3cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcd10b700 1 -- 172.21.15.121:0/1307830661 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d7a2bcf50 con 0x561d7a3dae50 2024-06-10T00:30:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcd90c700 1 --2- 172.21.15.121:0/1307830661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 0x561d7a3cd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/1307830661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3dae50 0x561d7a3db230 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400f010 tx=0x7f8fc409b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a6dd5dcf842498 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fb37fe700 1 -- 172.21.15.121:0/1307830661 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40a4750 con 0x561d7a3dae50 2024-06-10T00:30:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fb37fe700 1 -- 172.21.15.121:0/1307830661 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fc4018be0 con 0x561d7a3dae50 2024-06-10T00:30:53.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.432+0000 7f8fb37fe700 1 -- 172.21.15.121:0/1307830661 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc409af30 con 0x561d7a3dae50 2024-06-10T00:30:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.433+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1307830661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3dae50 msgr2=0x561d7a3db230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:53.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.433+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/1307830661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3dae50 0x561d7a3db230 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400f010 tx=0x7f8fc409b730 comp rx=0 tx=0).stop 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.433+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1307830661 shutdown_connections 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.433+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/1307830661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 0x561d7a3cd360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.433+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/1307830661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3dae50 0x561d7a3db230 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.433+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/1307830661 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3db770 0x561d7a3cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.433+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1307830661 >> 172.21.15.121:0/1307830661 conn(0x561d7a2f7cc0 msgr2=0x561d7a3c7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.433+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1307830661 shutdown_connections 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.433+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1307830661 wait complete. 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fd9bf2b80 1 -- start start 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fd9bf2b80 1 Processor -- start 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 0x561d7a3e1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3dae50 0x561d7a3de820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3ded60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561d7a3e21f0 con 0x561d7a3db770 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x561d7a2bebf0 con 0x561d7a3dae50 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561d7a3e2370 con 0x561d7a3ccf20 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fd9bf2b80 10 client.0 _ll_register_callbacks cb 0x561d7a2b8920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fcd90c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3ded60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fcd10b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 0x561d7a3e1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fcd90c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3ded60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40534/0 (socket says 172.21.15.121:40534) 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fcd10b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 0x561d7a3e1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:53434/0 (socket says 172.21.15.121:53434) 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fcd90c700 1 -- 172.21.15.121:0/1009827639 learned_addr learned my addr 172.21.15.121:0/1009827639 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.434+0000 7f8fcc90a700 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3dae50 0x561d7a3de820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fcd10b700 1 -- 172.21.15.121:0/1009827639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3dae50 msgr2=0x561d7a3de820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3dae50 0x561d7a3de820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fcd10b700 1 -- 172.21.15.121:0/1009827639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 msgr2=0x561d7a3ded60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3ded60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fcd10b700 1 -- 172.21.15.121:0/1009827639 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d7a26d8d0 con 0x561d7a3ccf20 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fcd90c700 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3ded60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fb0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 0x561d7a3e1520 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4097f40 tx=0x7f8fc4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c958e9c8b36102e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/1009827639 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4099de0 con 0x561d7a3ccf20 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1009827639 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fc4018be0 con 0x561d7a3ccf20 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/1009827639 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8fc409bf50 con 0x561d7a3ccf20 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.435+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/1009827639 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8fc40ba2b0 con 0x561d7a3ccf20 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.436+0000 7f8fd9bf2b80 10 client.25471 fetch_fsmap learned FSMap version 28 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.436+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1009827639 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561d7a3e21f0 con 0x561d7a3ccf20 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.436+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/1009827639 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fc4018be0 con 0x561d7a3ccf20 2024-06-10T00:30:53.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.436+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/1009827639 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8fc40b4ce0 con 0x561d7a3ccf20 2024-06-10T00:30:53.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.436+0000 7f8fd9bf2b80 10 client.25471 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:30:53.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.436+0000 7f8fd9bf2b80 10 client.25471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:30:53.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.436+0000 7f8fd9bf2b80 20 client.25471 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:53.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.436+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561d7a3eee80 0x561d7a3f1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.436+0000 7f8fd9bf2b80 4 client.25471 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:30:53.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.436+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1009827639 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d7a32bd10 con 0x561d7a3eee80 2024-06-10T00:30:53.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.437+0000 7f8fcc90a700 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561d7a3eee80 0x561d7a3f1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.437+0000 7f8fb0ff9700 10 client.25471 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:53.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.437+0000 7f8fcc90a700 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561d7a3eee80 0x561d7a3f1270 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8c87809df6391ce server_cookie=869882c97305405a in_seq=0 out_seq=0 2024-06-10T00:30:53.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/1009827639 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x561d7a32bd10 con 0x561d7a3eee80 2024-06-10T00:30:53.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fb0ff9700 10 client.25471 handle_command_reply: tid=0 2024-06-10T00:30:53.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 1 client.25471 shutdown 2024-06-10T00:30:53.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1009827639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 msgr2=0x561d7a3e1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:53.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 0x561d7a3e1520 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4097f40 tx=0x7f8fc4000bc0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1009827639 shutdown_connections 2024-06-10T00:30:53.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561d7a3eee80 0x561d7a3f1270 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3ded60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3ccf20 0x561d7a3e1520 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/1009827639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3dae50 0x561d7a3de820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fb0ff9700 0 client.25471 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:53.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1009827639 >> 172.21.15.121:0/1009827639 conn(0x561d7a2f7cc0 msgr2=0x561d7a3cb9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:53.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1009827639 shutdown_connections 2024-06-10T00:30:53.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/1009827639 wait complete. 2024-06-10T00:30:53.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.439+0000 7f8fd9bf2b80 20 client.25471 trim_cache size 0 max 16384 2024-06-10T00:30:53.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.458+0000 7f8fd9bf2b80 1 Processor -- start 2024-06-10T00:30:53.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.458+0000 7f8fd9bf2b80 1 -- start start 2024-06-10T00:30:53.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3ccf20 0x561d7a3cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3dae50 0x561d7a3dc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3f6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac011590 con 0x561d7a3db770 2024-06-10T00:30:53.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac00ee20 con 0x561d7a3ccf20 2024-06-10T00:30:53.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac00cc70 con 0x561d7a3dae50 2024-06-10T00:30:53.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fcd90c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fcd90c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3f6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40550/0 (socket says 172.21.15.121:40550) 2024-06-10T00:30:53.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fcd90c700 1 -- 172.21.15.121:0/4186670119 learned_addr learned my addr 172.21.15.121:0/4186670119 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:53.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fcc90a700 1 --2- 172.21.15.121:0/4186670119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3dae50 0x561d7a3dc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/4186670119 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3ccf20 0x561d7a3cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fcd90c700 1 -- 172.21.15.121:0/4186670119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3dae50 msgr2=0x561d7a3dc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fcd90c700 1 --2- 172.21.15.121:0/4186670119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3dae50 0x561d7a3dc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fcd90c700 1 -- 172.21.15.121:0/4186670119 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3ccf20 msgr2=0x561d7a3cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fcd90c700 1 --2- 172.21.15.121:0/4186670119 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3ccf20 0x561d7a3cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fcd90c700 1 -- 172.21.15.121:0/4186670119 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d7a3d9390 con 0x561d7a3db770 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.459+0000 7f8fcd90c700 1 --2- 172.21.15.121:0/4186670119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3f6810 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0008770 tx=0x7f8fc0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f1eedd392ccbee server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.460+0000 7f8fb2ffd700 1 -- 172.21.15.121:0/4186670119 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0017410 con 0x561d7a3db770 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.460+0000 7f8fb2ffd700 1 -- 172.21.15.121:0/4186670119 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fc0017a90 con 0x561d7a3db770 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.460+0000 7f8fb2ffd700 1 -- 172.21.15.121:0/4186670119 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0007bc0 con 0x561d7a3db770 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.460+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/4186670119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 msgr2=0x561d7a3f6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.460+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/4186670119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3f6810 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0008770 tx=0x7f8fc0009920 comp rx=0 tx=0).stop 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.460+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/4186670119 shutdown_connections 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.460+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/4186670119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3db770 0x561d7a3f6810 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.460+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/4186670119 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3ccf20 0x561d7a3cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.460+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/4186670119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3dae50 0x561d7a3dc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/4186670119 >> 172.21.15.121:0/4186670119 conn(0x561d7a2f7cc0 msgr2=0x561d7a3c79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/4186670119 shutdown_connections 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/4186670119 wait complete. 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 1 -- start start 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 1 Processor -- start 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3ccf20 0x561d7a3e7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3dae50 0x561d7a3f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3db770 0x561d7a3f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac00d2e0 con 0x561d7a3ccf20 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac00e750 con 0x561d7a3dae50 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac00d9b0 con 0x561d7a3db770 2024-06-10T00:30:53.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.461+0000 7f8fd9bf2b80 10 client.0 _ll_register_callbacks cb 0x561d7a2b8920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fcd10b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3ccf20 0x561d7a3e7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fcd10b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3ccf20 0x561d7a3e7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40552/0 (socket says 172.21.15.121:40552) 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fcd90c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3db770 0x561d7a3f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fcd10b700 1 -- 172.21.15.121:0/2365728004 learned_addr learned my addr 172.21.15.121:0/2365728004 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fcc90a700 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3dae50 0x561d7a3f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fcd90c700 1 -- 172.21.15.121:0/2365728004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3dae50 msgr2=0x561d7a3f61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fcd90c700 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3dae50 0x561d7a3f61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fcd90c700 1 -- 172.21.15.121:0/2365728004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3ccf20 msgr2=0x561d7a3e7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fcd90c700 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3ccf20 0x561d7a3e7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fcd90c700 1 -- 172.21.15.121:0/2365728004 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d7a3f6f20 con 0x561d7a3db770 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fcd90c700 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3db770 0x561d7a3f66e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0008770 tx=0x7f8fc0018e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0f9490f787573d3 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fb0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:30:53.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/2365728004 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0004ce0 con 0x561d7a3db770 2024-06-10T00:30:53.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/2365728004 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fc0017a90 con 0x561d7a3db770 2024-06-10T00:30:53.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.462+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/2365728004 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8fc0003a00 con 0x561d7a3db770 2024-06-10T00:30:53.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/2365728004 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8fc0003c00 con 0x561d7a3db770 2024-06-10T00:30:53.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fd9bf2b80 10 client.25477 fetch_fsmap learned FSMap version 28 2024-06-10T00:30:53.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/2365728004 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561d7a459830 con 0x561d7a3db770 2024-06-10T00:30:53.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/2365728004 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fc0017a90 con 0x561d7a3db770 2024-06-10T00:30:53.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/2365728004 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8fc002a470 con 0x561d7a3db770 2024-06-10T00:30:53.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fd9bf2b80 10 client.25477 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:30:53.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fd9bf2b80 10 client.25477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:30:53.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fd9bf2b80 20 client.25477 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:53.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561d7a459eb0 0x561d7a45c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:53.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fd9bf2b80 4 client.25477 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:30:53.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/2365728004 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561d7a326290 con 0x561d7a459eb0 2024-06-10T00:30:53.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.463+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561d7a459eb0 0x561d7a45c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:53.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.464+0000 7f8fb0ff9700 10 client.25477 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:53.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.464+0000 7f8fcd10b700 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561d7a459eb0 0x561d7a45c2d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fb53d5ae8cbfe6 server_cookie=a3e543eb3209983c in_seq=0 out_seq=0 2024-06-10T00:30:53.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.469+0000 7f8fb0ff9700 1 -- 172.21.15.121:0/2365728004 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561d7a326290 con 0x561d7a459eb0 2024-06-10T00:30:53.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.469+0000 7f8fb0ff9700 10 client.25477 handle_command_reply: tid=0 2024-06-10T00:30:53.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.469+0000 7f8fd9bf2b80 1 client.25477 shutdown 2024-06-10T00:30:53.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/2365728004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3db770 msgr2=0x561d7a3f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:53.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3db770 0x561d7a3f66e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0008770 tx=0x7f8fc0018e80 comp rx=0 tx=0).stop 2024-06-10T00:30:53.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/2365728004 shutdown_connections 2024-06-10T00:30:53.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561d7a459eb0 0x561d7a45c2d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d7a3db770 0x561d7a3f66e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fb0ff9700 0 client.25477 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:53.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561d7a3ccf20 0x561d7a3e7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fd9bf2b80 1 --2- 172.21.15.121:0/2365728004 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561d7a3dae50 0x561d7a3f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/2365728004 >> 172.21.15.121:0/2365728004 conn(0x561d7a2f7cc0 msgr2=0x561d7a3c0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:53.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/2365728004 shutdown_connections 2024-06-10T00:30:53.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fd9bf2b80 1 -- 172.21.15.121:0/2365728004 wait complete. 2024-06-10T00:30:53.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.470+0000 7f8fd9bf2b80 20 client.25477 trim_cache size 0 max 16384 2024-06-10T00:30:53.471 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:30:53.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 -- 172.21.15.121:0/592520031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8fb40a2e10 msgr2=0x7f8fb40a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:53.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8fb40a2e10 0x7f8fb40a52d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8fb80098e0 tx=0x7f8fb8005ab0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 -- 172.21.15.121:0/592520031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fc806a170 msgr2=0x7f8fc819e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:53.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fc806a170 0x7f8fc819e860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400e9a0 tx=0x7f8fc400ed60 comp rx=0 tx=0).stop 2024-06-10T00:30:53.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 -- 172.21.15.121:0/592520031 shutdown_connections 2024-06-10T00:30:53.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8fb40a2e10 0x7f8fb40a52d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc8108500 0x7f8fc8198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fc806a170 0x7f8fc819e860 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 --2- 172.21.15.121:0/592520031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8fc806ab30 0x7f8fc819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:53.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 -- 172.21.15.121:0/592520031 >> 172.21.15.121:0/592520031 conn(0x7f8fc8078340 msgr2=0x7f8fc80fffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:53.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 -- 172.21.15.121:0/592520031 shutdown_connections 2024-06-10T00:30:53.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:53.471+0000 7f8fcf36f700 1 -- 172.21.15.121:0/592520031 wait complete. 2024-06-10T00:30:55.252 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-10T00:30:55.253 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-10T00:30:55.278 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:55 smithi121 ceph-mon[32378]: pgmap v91: 129 pgs: 129 active+clean; 162 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.9 MiB/s wr, 526 op/s 2024-06-10T00:30:55.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:55 smithi193 ceph-mon[36716]: pgmap v91: 129 pgs: 129 active+clean; 162 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.9 MiB/s wr, 526 op/s 2024-06-10T00:30:55.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:55 smithi135 ceph-mon[36424]: pgmap v91: 129 pgs: 129 active+clean; 162 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 7.9 MiB/s wr, 526 op/s 2024-06-10T00:30:55.531 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:30:57.019 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:57 smithi121 ceph-mon[32378]: pgmap v92: 129 pgs: 129 active+clean; 162 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 148 op/s 2024-06-10T00:30:57.292 INFO:tasks.workunit.client.0.smithi121.stdout: 48 720158 656 490247 715 340932 1331 2024-06-10T00:30:57.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:57 smithi193 ceph-mon[36716]: pgmap v92: 129 pgs: 129 active+clean; 162 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 148 op/s 2024-06-10T00:30:57.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:57 smithi135 ceph-mon[36424]: pgmap v92: 129 pgs: 129 active+clean; 162 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 148 op/s 2024-06-10T00:30:57.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.556+0000 7fd29fc73700 1 -- 172.21.15.121:0/3984544734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd298101d40 msgr2=0x7fd29810e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:57.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.556+0000 7fd29fc73700 1 --2- 172.21.15.121:0/3984544734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd298101d40 0x7fd29810e230 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd29000b0a0 tx=0x7fd29000b3b0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.556+0000 7fd29fc73700 1 -- 172.21.15.121:0/3984544734 shutdown_connections 2024-06-10T00:30:57.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.556+0000 7fd29fc73700 1 --2- 172.21.15.121:0/3984544734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd298101d40 0x7fd29810e230 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.556+0000 7fd29fc73700 1 --2- 172.21.15.121:0/3984544734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2981073f0 0x7fd2981077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.556+0000 7fd29fc73700 1 --2- 172.21.15.121:0/3984544734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd298101380 0x7fd298101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.556+0000 7fd29fc73700 1 -- 172.21.15.121:0/3984544734 >> 172.21.15.121:0/3984544734 conn(0x7fd298078300 msgr2=0x7fd298078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:57.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.557+0000 7fd29fc73700 1 -- 172.21.15.121:0/3984544734 shutdown_connections 2024-06-10T00:30:57.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.557+0000 7fd29fc73700 1 -- 172.21.15.121:0/3984544734 wait complete. 2024-06-10T00:30:57.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.557+0000 7fd29fc73700 1 Processor -- start 2024-06-10T00:30:57.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29fc73700 1 -- start start 2024-06-10T00:30:57.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29fc73700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd298101380 0x7fd29819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29fc73700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd298101d40 0x7fd29819edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29fc73700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2981073f0 0x7fd298198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29fc73700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd2981116d0 con 0x7fd298101d40 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29fc73700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd298111550 con 0x7fd2981073f0 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29fc73700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd298111850 con 0x7fd298101380 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29da0f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd298101380 0x7fd29819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29d20e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd298101d40 0x7fd29819edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29da0f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd298101380 0x7fd29819e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:53466/0 (socket says 172.21.15.121:53466) 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29da0f700 1 -- 172.21.15.121:0/2674381031 learned_addr learned my addr 172.21.15.121:0/2674381031 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.558+0000 7fd29e210700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2981073f0 0x7fd298198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.559+0000 7fd29d20e700 1 -- 172.21.15.121:0/2674381031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd298101380 msgr2=0x7fd29819e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.559+0000 7fd29d20e700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd298101380 0x7fd29819e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.559+0000 7fd29d20e700 1 -- 172.21.15.121:0/2674381031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2981073f0 msgr2=0x7fd298198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.559+0000 7fd29d20e700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2981073f0 0x7fd298198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.559+0000 7fd29d20e700 1 -- 172.21.15.121:0/2674381031 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2981991b0 con 0x7fd298101d40 2024-06-10T00:30:57.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.559+0000 7fd29da0f700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd298101380 0x7fd29819e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:57.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.559+0000 7fd29e210700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2981073f0 0x7fd298198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:57.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.559+0000 7fd29d20e700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd298101d40 0x7fd29819edd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd294009c50 tx=0x7fd294009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:57.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.559+0000 7fd28effd700 1 -- 172.21.15.121:0/2674381031 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29401e070 con 0x7fd298101d40 2024-06-10T00:30:57.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.559+0000 7fd29fc73700 1 -- 172.21.15.121:0/2674381031 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2981994a0 con 0x7fd298101d40 2024-06-10T00:30:57.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.559+0000 7fd29fc73700 1 -- 172.21.15.121:0/2674381031 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd298102f90 con 0x7fd298101d40 2024-06-10T00:30:57.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.560+0000 7fd28effd700 1 -- 172.21.15.121:0/2674381031 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2940223b0 con 0x7fd298101d40 2024-06-10T00:30:57.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.560+0000 7fd28effd700 1 -- 172.21.15.121:0/2674381031 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294028060 con 0x7fd298101d40 2024-06-10T00:30:57.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.563+0000 7fd28effd700 1 -- 172.21.15.121:0/2674381031 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd294021070 con 0x7fd298101d40 2024-06-10T00:30:57.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.564+0000 7fd28effd700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd2840a2e50 0x7fd2840a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.564+0000 7fd2aa4f6b80 1 Processor -- start 2024-06-10T00:30:57.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.565+0000 7fd2aa4f6b80 1 -- start start 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.565+0000 7fd29da0f700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd2840a2e50 0x7fd2840a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.565+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd826fe0 0x5648dd8273c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.565+0000 7fd28effd700 1 -- 172.21.15.121:0/2674381031 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fd2940d2ba0 con 0x7fd298101d40 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.565+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd827900 0x5648dd8212e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.565+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 0x5648dd821c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.565+0000 7fd2aa4f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5648dd713fe0 con 0x5648dd821820 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.565+0000 7fd2aa4f6b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5648dd7728d0 con 0x5648dd826fe0 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.565+0000 7fd2aa4f6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5648dd6da4e0 con 0x5648dd827900 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29e210700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 0x5648dd821c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29e210700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 0x5648dd821c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40576/0 (socket says 172.21.15.121:40576) 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29e210700 1 -- 172.21.15.121:0/1391967059 learned_addr learned my addr 172.21.15.121:0/1391967059 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:57.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29da0f700 1 --2- 172.21.15.121:0/1391967059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd826fe0 0x5648dd8273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29e210700 1 -- 172.21.15.121:0/1391967059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd827900 msgr2=0x5648dd8212e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:30:57.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29e210700 1 --2- 172.21.15.121:0/1391967059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd827900 0x5648dd8212e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29e210700 1 -- 172.21.15.121:0/1391967059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd826fe0 msgr2=0x5648dd8273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:57.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29e210700 1 --2- 172.21.15.121:0/1391967059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd826fe0 0x5648dd8273c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29e210700 1 -- 172.21.15.121:0/1391967059 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648dd6c1950 con 0x5648dd821820 2024-06-10T00:30:57.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29da0f700 1 --2- 172.21.15.121:0/1391967059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd826fe0 0x5648dd8273c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:30:57.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29e210700 1 --2- 172.21.15.121:0/1391967059 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 0x5648dd821c40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd29000cf70 tx=0x7fd29000cfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccfe766af82a5eb1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:57.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.566+0000 7fd29da0f700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd2840a2e50 0x7fd2840a5310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd298199b90 tx=0x7fd28800bf90 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:57.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.567+0000 7fd283fff700 1 -- 172.21.15.121:0/1391967059 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290021070 con 0x5648dd821820 2024-06-10T00:30:57.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.567+0000 7fd283fff700 1 -- 172.21.15.121:0/1391967059 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd290009490 con 0x5648dd821820 2024-06-10T00:30:57.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.567+0000 7fd283fff700 1 -- 172.21.15.121:0/1391967059 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2900256b0 con 0x5648dd821820 2024-06-10T00:30:57.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.568+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/1391967059 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 msgr2=0x5648dd821c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:57.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.568+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/1391967059 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 0x5648dd821c40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd29000cf70 tx=0x7fd29000cfa0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.568+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/1391967059 shutdown_connections 2024-06-10T00:30:57.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.568+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/1391967059 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 0x5648dd821c40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.568+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/1391967059 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd826fe0 0x5648dd8273c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.568+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/1391967059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd827900 0x5648dd8212e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.568+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/1391967059 >> 172.21.15.121:0/1391967059 conn(0x5648dd74e100 msgr2=0x5648dd81c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:57.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.568+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/1391967059 shutdown_connections 2024-06-10T00:30:57.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.568+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/1391967059 wait complete. 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.569+0000 7fd2aa4f6b80 1 -- start start 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.569+0000 7fd2aa4f6b80 1 Processor -- start 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.569+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 0x5648dd82d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.569+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 0x5648dd82aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.569+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd827900 0x5648dd82afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd2aa4f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5648dd713040 con 0x5648dd821820 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd29d20e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 0x5648dd82aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd29e210700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd827900 0x5648dd82afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd29e210700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd827900 0x5648dd82afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:40222/0 (socket says 172.21.15.121:40222) 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd29e210700 1 -- 172.21.15.121:0/740801348 learned_addr learned my addr 172.21.15.121:0/740801348 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd29da0f700 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 0x5648dd82d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd29d20e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 0x5648dd82aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:53492/0 (socket says 172.21.15.121:53492) 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd2aa4f6b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5648dd76e890 con 0x5648dd827900 2024-06-10T00:30:57.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/740801348 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5648dd82e280 con 0x5648dd826fe0 2024-06-10T00:30:57.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd2aa4f6b80 10 client.0 _ll_register_callbacks cb 0x5648dd708630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:57.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd29d20e700 1 -- 172.21.15.121:0/740801348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd827900 msgr2=0x5648dd82afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:57.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd29d20e700 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd827900 0x5648dd82afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd29d20e700 1 -- 172.21.15.121:0/740801348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 msgr2=0x5648dd82d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:57.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd29d20e700 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 0x5648dd82d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.570+0000 7fd29d20e700 1 -- 172.21.15.121:0/740801348 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648dd7387d0 con 0x5648dd826fe0 2024-06-10T00:30:57.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.571+0000 7fd29d20e700 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 0x5648dd82aa60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd2940cff60 tx=0x7fd29409e710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=165b3ac82573ae65 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:57.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.571+0000 7fd2817fa700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:30:57.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.571+0000 7fd2817fa700 1 -- 172.21.15.121:0/740801348 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2940a30e0 con 0x5648dd826fe0 2024-06-10T00:30:57.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.571+0000 7fd2817fa700 1 -- 172.21.15.121:0/740801348 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fd29409e180 con 0x5648dd826fe0 2024-06-10T00:30:57.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.571+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/740801348 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd290009490 con 0x5648dd826fe0 2024-06-10T00:30:57.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.571+0000 7fd2817fa700 1 -- 172.21.15.121:0/740801348 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd2940a79a0 con 0x5648dd826fe0 2024-06-10T00:30:57.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.571+0000 7fd2aa4f6b80 10 client.25486 fetch_fsmap learned FSMap version 28 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.571+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/740801348 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5648dd838b20 con 0x5648dd826fe0 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.572+0000 7fd2817fa700 1 -- 172.21.15.121:0/740801348 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd290009490 con 0x5648dd826fe0 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.572+0000 7fd2817fa700 1 -- 172.21.15.121:0/740801348 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd2940a4300 con 0x5648dd826fe0 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.572+0000 7fd2aa4f6b80 10 client.25486 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.572+0000 7fd2aa4f6b80 10 client.25486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.572+0000 7fd2aa4f6b80 20 client.25486 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.572+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5648dd83b160 0x5648dd83d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.572+0000 7fd2aa4f6b80 4 client.25486 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.572+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/740801348 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648dd77fdd0 con 0x5648dd83b160 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.572+0000 7fd29da0f700 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5648dd83b160 0x5648dd83d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.573+0000 7fd2817fa700 10 client.25486 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:57.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.573+0000 7fd29da0f700 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5648dd83b160 0x5648dd83d550 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6955012af22d0bb8 server_cookie=22ab4ba32d38ecfa in_seq=0 out_seq=0 2024-06-10T00:30:57.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2817fa700 1 -- 172.21.15.121:0/740801348 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5648dd77fdd0 con 0x5648dd83b160 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2817fa700 10 client.25486 handle_command_reply: tid=0 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2aa4f6b80 1 client.25486 shutdown 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/740801348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 msgr2=0x5648dd82aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 0x5648dd82aa60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd2940cff60 tx=0x7fd29409e710 comp rx=0 tx=0).stop 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/740801348 shutdown_connections 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5648dd83b160 0x5648dd83d550 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd827900 0x5648dd82afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd821820 0x5648dd82d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/740801348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 0x5648dd82aa60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/740801348 >> 172.21.15.121:0/740801348 conn(0x5648dd74e100 msgr2=0x5648dd81b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.576+0000 7fd2817fa700 0 client.25486 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.577+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/740801348 shutdown_connections 2024-06-10T00:30:57.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.577+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/740801348 wait complete. 2024-06-10T00:30:57.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.577+0000 7fd2aa4f6b80 20 client.25486 trim_cache size 0 max 16384 2024-06-10T00:30:57.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.599+0000 7fd2aa4f6b80 1 Processor -- start 2024-06-10T00:30:57.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd2aa4f6b80 1 -- start start 2024-06-10T00:30:57.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd821820 0x5648dd837430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 0x5648dd82afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd827900 0x5648dd81bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd2aa4f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c011670 con 0x5648dd827900 2024-06-10T00:30:57.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd2aa4f6b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c00ef00 con 0x5648dd821820 2024-06-10T00:30:57.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd2aa4f6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c00cd50 con 0x5648dd826fe0 2024-06-10T00:30:57.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd29da0f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd821820 0x5648dd837430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd29da0f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd821820 0x5648dd837430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:40232/0 (socket says 172.21.15.121:40232) 2024-06-10T00:30:57.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd29da0f700 1 -- 172.21.15.121:0/891604604 learned_addr learned my addr 172.21.15.121:0/891604604 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:57.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd29d20e700 1 --2- 172.21.15.121:0/891604604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 0x5648dd82afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd29e210700 1 --2- 172.21.15.121:0/891604604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd827900 0x5648dd81bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd29d20e700 1 -- 172.21.15.121:0/891604604 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd821820 msgr2=0x5648dd837430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:57.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd29d20e700 1 --2- 172.21.15.121:0/891604604 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd821820 0x5648dd837430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd29d20e700 1 -- 172.21.15.121:0/891604604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd827900 msgr2=0x5648dd81bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:57.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd29d20e700 1 --2- 172.21.15.121:0/891604604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd827900 0x5648dd81bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd29d20e700 1 -- 172.21.15.121:0/891604604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648dd81da20 con 0x5648dd826fe0 2024-06-10T00:30:57.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.600+0000 7fd29da0f700 1 --2- 172.21.15.121:0/891604604 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd821820 0x5648dd837430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:30:57.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.601+0000 7fd29d20e700 1 --2- 172.21.15.121:0/891604604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 0x5648dd82afd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd294022d10 tx=0x7fd29409afd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baecd0f73765daf9 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:57.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.601+0000 7fd2837fe700 1 -- 172.21.15.121:0/891604604 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2940a4270 con 0x5648dd826fe0 2024-06-10T00:30:57.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.601+0000 7fd2837fe700 1 -- 172.21.15.121:0/891604604 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2940a4410 con 0x5648dd826fe0 2024-06-10T00:30:57.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.601+0000 7fd2837fe700 1 -- 172.21.15.121:0/891604604 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29409e3e0 con 0x5648dd826fe0 2024-06-10T00:30:57.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.602+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/891604604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 msgr2=0x5648dd82afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:57.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.602+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/891604604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 0x5648dd82afd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd294022d10 tx=0x7fd29409afd0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.602+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/891604604 shutdown_connections 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.602+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/891604604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd827900 0x5648dd81bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.602+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/891604604 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd821820 0x5648dd837430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.602+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/891604604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd826fe0 0x5648dd82afd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.602+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/891604604 >> 172.21.15.121:0/891604604 conn(0x5648dd74e100 msgr2=0x5648dd82cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.602+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/891604604 shutdown_connections 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.602+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/891604604 wait complete. 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.602+0000 7fd2aa4f6b80 1 -- start start 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.602+0000 7fd2aa4f6b80 1 Processor -- start 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd821820 0x5648dd8338d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd826fe0 0x5648dd833e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd2aa4f6b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd827900 0x5648dd830640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd2aa4f6b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c00d3c0 con 0x5648dd826fe0 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd2aa4f6b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c00e830 con 0x5648dd827900 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd2aa4f6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c00da90 con 0x5648dd821820 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd2aa4f6b80 10 client.0 _ll_register_callbacks cb 0x5648dd708630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd29da0f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd821820 0x5648dd8338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd29da0f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd821820 0x5648dd8338d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:53510/0 (socket says 172.21.15.121:53510) 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd29da0f700 1 -- 172.21.15.121:0/3312199499 learned_addr learned my addr 172.21.15.121:0/3312199499 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd29d20e700 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd826fe0 0x5648dd833e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd29e210700 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd827900 0x5648dd830640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd29da0f700 1 -- 172.21.15.121:0/3312199499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd827900 msgr2=0x5648dd830640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd29da0f700 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd827900 0x5648dd830640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd29da0f700 1 -- 172.21.15.121:0/3312199499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd826fe0 msgr2=0x5648dd833e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd29da0f700 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd826fe0 0x5648dd833e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.603+0000 7fd29da0f700 1 -- 172.21.15.121:0/3312199499 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648dd818890 con 0x5648dd821820 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd29da0f700 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd821820 0x5648dd8338d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd28800be50 tx=0x7fd288004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b3464732c8e8ea2 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2817fa700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2817fa700 1 -- 172.21.15.121:0/3312199499 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28801a040 con 0x5648dd821820 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2817fa700 1 -- 172.21.15.121:0/3312199499 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fd288022640 con 0x5648dd821820 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/3312199499 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2940a4410 con 0x5648dd821820 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2817fa700 1 -- 172.21.15.121:0/3312199499 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd288010e80 con 0x5648dd821820 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2aa4f6b80 10 client.25492 fetch_fsmap learned FSMap version 28 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/3312199499 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5648dd827610 con 0x5648dd821820 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2817fa700 1 -- 172.21.15.121:0/3312199499 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2940a4410 con 0x5648dd821820 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2817fa700 1 -- 172.21.15.121:0/3312199499 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fd288020040 con 0x5648dd821820 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2aa4f6b80 10 client.25492 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2aa4f6b80 10 client.25492 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:30:57.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2aa4f6b80 20 client.25492 populate_metadata read hostname 'smithi121' 2024-06-10T00:30:57.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5648dd8af700 0x5648dd8b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:30:57.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2aa4f6b80 4 client.25492 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-10T00:30:57.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.604+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/3312199499 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5648dd77a350 con 0x5648dd8af700 2024-06-10T00:30:57.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.605+0000 7fd29d20e700 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5648dd8af700 0x5648dd8b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:30:57.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.605+0000 7fd29d20e700 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5648dd8af700 0x5648dd8b1b20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3cc3fc61db01a90 server_cookie=e673866f3b33679b in_seq=0 out_seq=0 2024-06-10T00:30:57.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.605+0000 7fd2817fa700 10 client.25492 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.627+0000 7fd2817fa700 1 -- 172.21.15.121:0/3312199499 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5648dd77a350 con 0x5648dd8af700 2024-06-10T00:30:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.627+0000 7fd2817fa700 10 client.25492 handle_command_reply: tid=0 2024-06-10T00:30:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.627+0000 7fd2aa4f6b80 1 client.25492 shutdown 2024-06-10T00:30:57.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/3312199499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd821820 msgr2=0x5648dd8338d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd821820 0x5648dd8338d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd28800be50 tx=0x7fd288004750 comp rx=0 tx=0).stop 2024-06-10T00:30:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/3312199499 shutdown_connections 2024-06-10T00:30:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5648dd8af700 0x5648dd8b1b20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5648dd827900 0x5648dd830640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648dd821820 0x5648dd8338d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2aa4f6b80 1 --2- 172.21.15.121:0/3312199499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5648dd826fe0 0x5648dd833e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/3312199499 >> 172.21.15.121:0/3312199499 conn(0x5648dd74e100 msgr2=0x5648dd825bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2817fa700 0 client.25492 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:30:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/3312199499 shutdown_connections 2024-06-10T00:30:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2aa4f6b80 1 -- 172.21.15.121:0/3312199499 wait complete. 2024-06-10T00:30:57.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.628+0000 7fd2aa4f6b80 20 client.25492 trim_cache size 0 max 16384 2024-06-10T00:30:57.630 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:30:57.630 INFO:teuthology.orchestra.run.smithi121.stdout: "return_code": 0, 2024-06-10T00:30:57.630 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_tag": "8ead0358-83cc-4173-b698-45a22774ed67", 2024-06-10T00:30:57.630 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": "asynchronous" 2024-06-10T00:30:57.630 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 -- 172.21.15.121:0/2674381031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd2840a2e50 msgr2=0x7fd2840a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd2840a2e50 0x7fd2840a5310 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd298199b90 tx=0x7fd28800bf90 comp rx=0 tx=0).stop 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 -- 172.21.15.121:0/2674381031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd298101d40 msgr2=0x7fd29819edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd298101d40 0x7fd29819edd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd294009c50 tx=0x7fd294009f60 comp rx=0 tx=0).stop 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 -- 172.21.15.121:0/2674381031 shutdown_connections 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd2840a2e50 0x7fd2840a5310 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2981073f0 0x7fd298198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd298101380 0x7fd29819e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 --2- 172.21.15.121:0/2674381031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd298101d40 0x7fd29819edd0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 -- 172.21.15.121:0/2674381031 >> 172.21.15.121:0/2674381031 conn(0x7fd298078300 msgr2=0x7fd298100560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 -- 172.21.15.121:0/2674381031 shutdown_connections 2024-06-10T00:30:57.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:30:57.629+0000 7fd29fc73700 1 -- 172.21.15.121:0/2674381031 wait complete. 2024-06-10T00:30:58.815 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:30:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:30:59.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: pgmap v93: 129 pgs: 129 active+clean; 162 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 148 op/s 2024-06-10T00:30:59.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub queued for path: / 2024-06-10T00:30:59.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.383 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:30:59 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.420 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: pgmap v93: 129 pgs: 129 active+clean; 162 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 148 op/s 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub queued for path: / 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.421 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.422 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.423 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.423 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.423 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.423 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:30:59.423 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.423 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:30:59 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: pgmap v93: 129 pgs: 129 active+clean; 162 MiB data, 585 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 148 op/s 2024-06-10T00:30:59.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub queued for path: / 2024-06-10T00:30:59.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:30:59.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:30:59.510 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:30:59 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:31:01.072 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:01 smithi121 ceph-mon[32378]: pgmap v94: 129 pgs: 129 active+clean; 168 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 318 op/s 2024-06-10T00:31:01.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:01 smithi193 ceph-mon[36716]: pgmap v94: 129 pgs: 129 active+clean; 168 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 318 op/s 2024-06-10T00:31:01.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.488+0000 7fc7a6a36700 1 -- 172.21.15.121:0/2044543692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a006be40 msgr2=0x7fc7a006c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:01.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.488+0000 7fc7a6a36700 1 --2- 172.21.15.121:0/2044543692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a006be40 0x7fc7a006c220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc790009990 tx=0x7fc790009ca0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.489+0000 7fc7a6a36700 1 -- 172.21.15.121:0/2044543692 shutdown_connections 2024-06-10T00:31:01.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.489+0000 7fc7a6a36700 1 --2- 172.21.15.121:0/2044543692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7a011bd30 0x7fc7a011e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.489+0000 7fc7a6a36700 1 --2- 172.21.15.121:0/2044543692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a006be40 0x7fc7a006c220 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.489+0000 7fc7a6a36700 1 --2- 172.21.15.121:0/2044543692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7a006c760 0x7fc7a011b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.489+0000 7fc7a6a36700 1 -- 172.21.15.121:0/2044543692 >> 172.21.15.121:0/2044543692 conn(0x7fc7a006f630 msgr2=0x7fc7a0109490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:01.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.490+0000 7fc7a6a36700 1 -- 172.21.15.121:0/2044543692 shutdown_connections 2024-06-10T00:31:01.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.490+0000 7fc7a6a36700 1 -- 172.21.15.121:0/2044543692 wait complete. 2024-06-10T00:31:01.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.491+0000 7fc7a6a36700 1 Processor -- start 2024-06-10T00:31:01.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.491+0000 7fc7a6a36700 1 -- start start 2024-06-10T00:31:01.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.491+0000 7fc7a6a36700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7a006be40 0x7fc7a01ad780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.491+0000 7fc7a6a36700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a006c760 0x7fc7a01adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.491+0000 7fc7a6a36700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7a011bd30 0x7fc7a01a7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.491+0000 7fc7a6a36700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a0121620 con 0x7fc7a006be40 2024-06-10T00:31:01.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.491+0000 7fc7a6a36700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a01214a0 con 0x7fc7a011bd30 2024-06-10T00:31:01.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.491+0000 7fc7a6a36700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a01217a0 con 0x7fc7a006c760 2024-06-10T00:31:01.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.492+0000 7fc79ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7a006be40 0x7fc7a01ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.492+0000 7fc79ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7a006be40 0x7fc7a01ad780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50662/0 (socket says 172.21.15.121:50662) 2024-06-10T00:31:01.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.492+0000 7fc79ffff700 1 -- 172.21.15.121:0/1340614210 learned_addr learned my addr 172.21.15.121:0/1340614210 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:01.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.492+0000 7fc7a4fd3700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7a011bd30 0x7fc7a01a7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.492+0000 7fc79f7fe700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a006c760 0x7fc7a01adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.492+0000 7fc7a4fd3700 1 -- 172.21.15.121:0/1340614210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a006c760 msgr2=0x7fc7a01adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:01.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.492+0000 7fc7a4fd3700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a006c760 0x7fc7a01adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.492+0000 7fc7a4fd3700 1 -- 172.21.15.121:0/1340614210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7a006be40 msgr2=0x7fc7a01ad780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:01.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.492+0000 7fc7a4fd3700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7a006be40 0x7fc7a01ad780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.492+0000 7fc7a4fd3700 1 -- 172.21.15.121:0/1340614210 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a01a80a0 con 0x7fc7a011bd30 2024-06-10T00:31:01.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.493+0000 7fc7a4fd3700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7a011bd30 0x7fc7a01a7940 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc798007ee0 tx=0x7fc79800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:01.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.493+0000 7fc79d7fa700 1 -- 172.21.15.121:0/1340614210 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79800ddd0 con 0x7fc7a011bd30 2024-06-10T00:31:01.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.493+0000 7fc7a6a36700 1 -- 172.21.15.121:0/1340614210 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7a01a8390 con 0x7fc7a011bd30 2024-06-10T00:31:01.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.493+0000 7fc7a6a36700 1 -- 172.21.15.121:0/1340614210 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7a01b4650 con 0x7fc7a011bd30 2024-06-10T00:31:01.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.493+0000 7fc79d7fa700 1 -- 172.21.15.121:0/1340614210 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc798009e20 con 0x7fc7a011bd30 2024-06-10T00:31:01.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.493+0000 7fc79d7fa700 1 -- 172.21.15.121:0/1340614210 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798015400 con 0x7fc7a011bd30 2024-06-10T00:31:01.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.497+0000 7fc79d7fa700 1 -- 172.21.15.121:0/1340614210 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc7980155a0 con 0x7fc7a011bd30 2024-06-10T00:31:01.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.498+0000 7fc79d7fa700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc7880a2e90 0x7fc7880a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.499+0000 7fc79d7fa700 1 -- 172.21.15.121:0/1340614210 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fc7980cf070 con 0x7fc7a011bd30 2024-06-10T00:31:01.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.500+0000 7fc79ffff700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc7880a2e90 0x7fc7880a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.501+0000 7fc79ffff700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc7880a2e90 0x7fc7880a5350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc790009750 tx=0x7fc790005ab0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:01.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.503+0000 7fc7b12b9b80 1 Processor -- start 2024-06-10T00:31:01.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.504+0000 7fc7b12b9b80 1 -- start start 2024-06-10T00:31:01.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.504+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7cf60 0x556742b7d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.504+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7d880 0x556742b77260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.504+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b777a0 0x556742b77bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.505+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556742a69f60 con 0x556742b7cf60 2024-06-10T00:31:01.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.505+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x556742ac8850 con 0x556742b777a0 2024-06-10T00:31:01.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.505+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556742a30460 con 0x556742b7d880 2024-06-10T00:31:01.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.505+0000 7fc79f7fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7d880 0x556742b77260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.505+0000 7fc7a4fd3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b777a0 0x556742b77bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.505+0000 7fc79f7fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7d880 0x556742b77260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:40828/0 (socket says 172.21.15.121:40828) 2024-06-10T00:31:01.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.505+0000 7fc79f7fe700 1 -- 172.21.15.121:0/1635235349 learned_addr learned my addr 172.21.15.121:0/1635235349 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:01.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.505+0000 7fc79ffff700 1 --2- 172.21.15.121:0/1635235349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7cf60 0x556742b7d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.507+0000 7fc79ffff700 1 -- 172.21.15.121:0/1635235349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7d880 msgr2=0x556742b77260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:01.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.507+0000 7fc79ffff700 1 --2- 172.21.15.121:0/1635235349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7d880 0x556742b77260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.507+0000 7fc79ffff700 1 -- 172.21.15.121:0/1635235349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b777a0 msgr2=0x556742b77bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:01.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.507+0000 7fc79ffff700 1 --2- 172.21.15.121:0/1635235349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b777a0 0x556742b77bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.507+0000 7fc79ffff700 1 -- 172.21.15.121:0/1635235349 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556742a178d0 con 0x556742b7cf60 2024-06-10T00:31:01.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.507+0000 7fc79ffff700 1 --2- 172.21.15.121:0/1635235349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7cf60 0x556742b7d340 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc7900098e0 tx=0x7fc79001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945ec1eae0a1b2aa server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:01.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.507+0000 7fc7867fc700 1 -- 172.21.15.121:0/1635235349 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790003680 con 0x556742b7cf60 2024-06-10T00:31:01.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.507+0000 7fc7867fc700 1 -- 172.21.15.121:0/1635235349 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc790003d00 con 0x556742b7cf60 2024-06-10T00:31:01.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.507+0000 7fc7867fc700 1 -- 172.21.15.121:0/1635235349 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79000fcc0 con 0x556742b7cf60 2024-06-10T00:31:01.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:01 smithi135 ceph-mon[36424]: pgmap v94: 129 pgs: 129 active+clean; 168 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.4 MiB/s wr, 318 op/s 2024-06-10T00:31:01.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.508+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/1635235349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7cf60 msgr2=0x556742b7d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:01.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.508+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/1635235349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7cf60 0x556742b7d340 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fc7900098e0 tx=0x7fc79001f000 comp rx=0 tx=0).stop 2024-06-10T00:31:01.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.509+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/1635235349 shutdown_connections 2024-06-10T00:31:01.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.509+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/1635235349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b777a0 0x556742b77bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.509+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/1635235349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7cf60 0x556742b7d340 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.509+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/1635235349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7d880 0x556742b77260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.509+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/1635235349 >> 172.21.15.121:0/1635235349 conn(0x556742aa4080 msgr2=0x556742b72220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:01.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.509+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/1635235349 shutdown_connections 2024-06-10T00:31:01.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.509+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/1635235349 wait complete. 2024-06-10T00:31:01.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.509+0000 7fc7b12b9b80 1 -- start start 2024-06-10T00:31:01.510 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.510+0000 7fc7b12b9b80 1 Processor -- start 2024-06-10T00:31:01.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.510+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 0x556742b83630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.510+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 0x556742b80930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.510+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7d880 0x556742b80e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.510+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556742a68fc0 con 0x556742b777a0 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.510+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x556742ac4810 con 0x556742b7d880 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.510+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556742b95210 con 0x556742b7cf60 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc79f7fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 0x556742b80930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc7b12b9b80 10 client.0 _ll_register_callbacks cb 0x556742a5e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc79f7fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 0x556742b80930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:40838/0 (socket says 172.21.15.121:40838) 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc79f7fe700 1 -- 172.21.15.121:0/4129575457 learned_addr learned my addr 172.21.15.121:0/4129575457 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc79ffff700 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 0x556742b83630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc7a4fd3700 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7d880 0x556742b80e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc79f7fe700 1 -- 172.21.15.121:0/4129575457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7d880 msgr2=0x556742b80e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc79f7fe700 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7d880 0x556742b80e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc79f7fe700 1 -- 172.21.15.121:0/4129575457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 msgr2=0x556742b83630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:01.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc79f7fe700 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 0x556742b83630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc79f7fe700 1 -- 172.21.15.121:0/4129575457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556742a8e750 con 0x556742b7cf60 2024-06-10T00:31:01.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.511+0000 7fc7a4fd3700 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7d880 0x556742b80e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:31:01.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.513+0000 7fc79f7fe700 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 0x556742b80930 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc794012790 tx=0x7fc794012aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9639a8a66dfa5944 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:01.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.513+0000 7fc77f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:31:01.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.513+0000 7fc77f7fe700 1 -- 172.21.15.121:0/4129575457 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7940117d0 con 0x556742b7cf60 2024-06-10T00:31:01.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.513+0000 7fc77f7fe700 1 -- 172.21.15.121:0/4129575457 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fc794011970 con 0x556742b7cf60 2024-06-10T00:31:01.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.513+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/4129575457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc790003d00 con 0x556742b7cf60 2024-06-10T00:31:01.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.513+0000 7fc77f7fe700 1 -- 172.21.15.121:0/4129575457 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc794011bb0 con 0x556742b7cf60 2024-06-10T00:31:01.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.513+0000 7fc7b12b9b80 10 client.25504 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:01.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.513+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/4129575457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556742b9f8b0 con 0x556742b7cf60 2024-06-10T00:31:01.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.514+0000 7fc77f7fe700 1 -- 172.21.15.121:0/4129575457 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc790003d00 con 0x556742b7cf60 2024-06-10T00:31:01.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.514+0000 7fc77f7fe700 1 -- 172.21.15.121:0/4129575457 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc79401c4b0 con 0x556742b7cf60 2024-06-10T00:31:01.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.514+0000 7fc7b12b9b80 10 client.25504 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:01.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.514+0000 7fc7b12b9b80 10 client.25504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:31:01.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.514+0000 7fc7b12b9b80 20 client.25504 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:01.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.514+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x556742ba1ef0 0x556742ba42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.514+0000 7fc7b12b9b80 4 client.25504 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:31:01.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.514+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/4129575457 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556742ad5d50 con 0x556742ba1ef0 2024-06-10T00:31:01.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.516+0000 7fc79ffff700 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x556742ba1ef0 0x556742ba42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.518+0000 7fc79ffff700 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x556742ba1ef0 0x556742ba42e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4080456bc89b5c55 server_cookie=f1ac13ede9f456d4 in_seq=0 out_seq=0 2024-06-10T00:31:01.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.518+0000 7fc77f7fe700 10 client.25504 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:01.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.528+0000 7fc77f7fe700 1 -- 172.21.15.121:0/4129575457 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x556742ad5d50 con 0x556742ba1ef0 2024-06-10T00:31:01.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.528+0000 7fc77f7fe700 10 client.25504 handle_command_reply: tid=0 2024-06-10T00:31:01.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.528+0000 7fc7b12b9b80 1 client.25504 shutdown 2024-06-10T00:31:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.528+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/4129575457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 msgr2=0x556742b80930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.528+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 0x556742b80930 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc794012790 tx=0x7fc794012aa0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.529+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/4129575457 shutdown_connections 2024-06-10T00:31:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.529+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x556742ba1ef0 0x556742ba42e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.529+0000 7fc77f7fe700 0 client.25504 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.529+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7d880 0x556742b80e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.529+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 0x556742b83630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.529+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/4129575457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 0x556742b80930 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.529+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/4129575457 >> 172.21.15.121:0/4129575457 conn(0x556742aa4080 msgr2=0x556742b71700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:01.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.529+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/4129575457 shutdown_connections 2024-06-10T00:31:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.529+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/4129575457 wait complete. 2024-06-10T00:31:01.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.529+0000 7fc7b12b9b80 20 client.25504 trim_cache size 0 max 16384 2024-06-10T00:31:01.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.567+0000 7fc7b12b9b80 1 Processor -- start 2024-06-10T00:31:01.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.567+0000 7fc7b12b9b80 1 -- start start 2024-06-10T00:31:01.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.567+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 0x556742b7e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.567+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 0x556742b83170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.567+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7d880 0x556742b71e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.567+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c0115f0 con 0x556742b777a0 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.567+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c00ee80 con 0x556742b7d880 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.567+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c00ccd0 con 0x556742b7cf60 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc79ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 0x556742b7e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc79ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 0x556742b7e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50698/0 (socket says 172.21.15.121:50698) 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc79ffff700 1 -- 172.21.15.121:0/3490737744 learned_addr learned my addr 172.21.15.121:0/3490737744 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc79f7fe700 1 --2- 172.21.15.121:0/3490737744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 0x556742b83170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc7a4fd3700 1 --2- 172.21.15.121:0/3490737744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7d880 0x556742b71e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc79ffff700 1 -- 172.21.15.121:0/3490737744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 msgr2=0x556742b83170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc79ffff700 1 --2- 172.21.15.121:0/3490737744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 0x556742b83170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc79ffff700 1 -- 172.21.15.121:0/3490737744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7d880 msgr2=0x556742b71e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:01.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc79ffff700 1 --2- 172.21.15.121:0/3490737744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7d880 0x556742b71e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc79ffff700 1 -- 172.21.15.121:0/3490737744 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556742b7d680 con 0x556742b777a0 2024-06-10T00:31:01.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc79f7fe700 1 --2- 172.21.15.121:0/3490737744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 0x556742b83170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:31:01.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc79ffff700 1 --2- 172.21.15.121:0/3490737744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 0x556742b7e4f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc790005850 tx=0x7fc79000f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1172d54f4e59ab08 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:01.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.568+0000 7fc785ffb700 1 -- 172.21.15.121:0/3490737744 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79000f910 con 0x556742b777a0 2024-06-10T00:31:01.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.569+0000 7fc785ffb700 1 -- 172.21.15.121:0/3490737744 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc790003680 con 0x556742b777a0 2024-06-10T00:31:01.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.569+0000 7fc785ffb700 1 -- 172.21.15.121:0/3490737744 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79002e460 con 0x556742b777a0 2024-06-10T00:31:01.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.569+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3490737744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 msgr2=0x556742b7e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:01.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.569+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/3490737744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 0x556742b7e4f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc790005850 tx=0x7fc79000f8a0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.569+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3490737744 shutdown_connections 2024-06-10T00:31:01.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.569+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/3490737744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7d880 0x556742b71e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.569+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/3490737744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b777a0 0x556742b7e4f0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.569+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/3490737744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b7cf60 0x556742b83170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.569+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3490737744 >> 172.21.15.121:0/3490737744 conn(0x556742aa4080 msgr2=0x556742b7ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.569+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3490737744 shutdown_connections 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.569+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3490737744 wait complete. 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.570+0000 7fc7b12b9b80 1 -- start start 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.570+0000 7fc7b12b9b80 1 Processor -- start 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.570+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b777a0 0x556742b9ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.570+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7cf60 0x556742ba93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.570+0000 7fc7b12b9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7d880 0x556742ba98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.570+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c00d340 con 0x556742b7d880 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.570+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c00e7b0 con 0x556742b7cf60 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc7b12b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc78c00da10 con 0x556742b777a0 2024-06-10T00:31:01.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc79ffff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b777a0 0x556742b9ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc79f7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7cf60 0x556742ba93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc79ffff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b777a0 0x556742b9ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:40852/0 (socket says 172.21.15.121:40852) 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc79ffff700 1 -- 172.21.15.121:0/3096295336 learned_addr learned my addr 172.21.15.121:0/3096295336 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc7a4fd3700 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7d880 0x556742ba98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc7b12b9b80 10 client.0 _ll_register_callbacks cb 0x556742a5e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc7a4fd3700 1 -- 172.21.15.121:0/3096295336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b777a0 msgr2=0x556742b9ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc7a4fd3700 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b777a0 0x556742b9ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc7a4fd3700 1 -- 172.21.15.121:0/3096295336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7cf60 msgr2=0x556742ba93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc7a4fd3700 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7cf60 0x556742ba93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc7a4fd3700 1 -- 172.21.15.121:0/3096295336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556742baa110 con 0x556742b7d880 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc79ffff700 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b777a0 0x556742b9ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc7a4fd3700 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7d880 0x556742ba98e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc7980cc3d0 tx=0x7fc79800df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f44349d50a525ef server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc77f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:31:01.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc77f7fe700 1 -- 172.21.15.121:0/3096295336 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798004cd0 con 0x556742b7d880 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3096295336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc790003680 con 0x556742b7d880 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.571+0000 7fc77f7fe700 1 -- 172.21.15.121:0/3096295336 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fc7980cc650 con 0x556742b7d880 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.572+0000 7fc77f7fe700 1 -- 172.21.15.121:0/3096295336 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc7980cc850 con 0x556742b7d880 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.572+0000 7fc7b12b9b80 10 client.16011 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.572+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3096295336 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556742c0ca70 con 0x556742b7d880 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.572+0000 7fc77f7fe700 1 -- 172.21.15.121:0/3096295336 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc790003680 con 0x556742b7d880 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.572+0000 7fc77f7fe700 1 -- 172.21.15.121:0/3096295336 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc7980a38e0 con 0x556742b7d880 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.572+0000 7fc7b12b9b80 10 client.16011 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.572+0000 7fc7b12b9b80 10 client.16011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.572+0000 7fc7b12b9b80 20 client.16011 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.572+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x556742ba1ef0 0x556742c0f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.572+0000 7fc7b12b9b80 4 client.16011 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:31:01.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.572+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3096295336 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556742ad02d0 con 0x556742ba1ef0 2024-06-10T00:31:01.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.573+0000 7fc79ffff700 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x556742ba1ef0 0x556742c0f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:01.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.573+0000 7fc77f7fe700 10 client.16011 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:01.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.573+0000 7fc79ffff700 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x556742ba1ef0 0x556742c0f220 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f0ed45b30f5ad77 server_cookie=1f0dcbcd09487a1 in_seq=0 out_seq=0 2024-06-10T00:31:01.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.577+0000 7fc77f7fe700 1 -- 172.21.15.121:0/3096295336 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x556742ad02d0 con 0x556742ba1ef0 2024-06-10T00:31:01.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.577+0000 7fc77f7fe700 10 client.16011 handle_command_reply: tid=0 2024-06-10T00:31:01.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.577+0000 7fc7b12b9b80 1 client.16011 shutdown 2024-06-10T00:31:01.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3096295336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7d880 msgr2=0x556742ba98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:01.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7d880 0x556742ba98e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc7980cc3d0 tx=0x7fc79800df70 comp rx=0 tx=0).stop 2024-06-10T00:31:01.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3096295336 shutdown_connections 2024-06-10T00:31:01.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x556742ba1ef0 0x556742c0f220 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556742b7d880 0x556742ba98e0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556742b777a0 0x556742b9ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc77f7fe700 0 client.16011 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:01.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc7b12b9b80 1 --2- 172.21.15.121:0/3096295336 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556742b7cf60 0x556742ba93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3096295336 >> 172.21.15.121:0/3096295336 conn(0x556742aa4080 msgr2=0x556742b7e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:01.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3096295336 shutdown_connections 2024-06-10T00:31:01.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc7b12b9b80 1 -- 172.21.15.121:0/3096295336 wait complete. 2024-06-10T00:31:01.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.578+0000 7fc7b12b9b80 20 client.16011 trim_cache size 0 max 16384 2024-06-10T00:31:01.579 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:31:01.579 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "scrub active (383 inodes in the stack)", 2024-06-10T00:31:01.579 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": { 2024-06-10T00:31:01.579 INFO:teuthology.orchestra.run.smithi121.stdout: "8ead0358-83cc-4173-b698-45a22774ed67": { 2024-06-10T00:31:01.580 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/", 2024-06-10T00:31:01.580 INFO:teuthology.orchestra.run.smithi121.stdout: "tag": "8ead0358-83cc-4173-b698-45a22774ed67", 2024-06-10T00:31:01.580 INFO:teuthology.orchestra.run.smithi121.stdout: "options": "recursive,force" 2024-06-10T00:31:01.580 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:31:01.580 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:31:01.580 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:31:01.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 -- 172.21.15.121:0/1340614210 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc7880a2e90 msgr2=0x7fc7880a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:01.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc7880a2e90 0x7fc7880a5350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc790009750 tx=0x7fc790005ab0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 -- 172.21.15.121:0/1340614210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7a011bd30 msgr2=0x7fc7a01a7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:01.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7a011bd30 0x7fc7a01a7940 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc798007ee0 tx=0x7fc79800d3b0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 -- 172.21.15.121:0/1340614210 shutdown_connections 2024-06-10T00:31:01.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc7880a2e90 0x7fc7880a5350 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7a011bd30 0x7fc7a01a7940 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7a006be40 0x7fc7a01ad780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 --2- 172.21.15.121:0/1340614210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7a006c760 0x7fc7a01adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:01.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 -- 172.21.15.121:0/1340614210 >> 172.21.15.121:0/1340614210 conn(0x7fc7a006f630 msgr2=0x7fc7a011d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:01.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 -- 172.21.15.121:0/1340614210 shutdown_connections 2024-06-10T00:31:01.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:01.579+0000 7fc7a6a36700 1 -- 172.21.15.121:0/1340614210 wait complete. 2024-06-10T00:31:02.525 INFO:tasks.cephfs.filesystem:scrub status for tag:8ead0358-83cc-4173-b698-45a22774ed67 - {'path': '/', 'tag': '8ead0358-83cc-4173-b698-45a22774ed67', 'options': 'recursive,force'} 2024-06-10T00:31:03.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:03 smithi193 ceph-mon[36716]: pgmap v95: 129 pgs: 129 active+clean; 168 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 256 op/s 2024-06-10T00:31:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:03 smithi121 ceph-mon[32378]: pgmap v95: 129 pgs: 129 active+clean; 168 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 256 op/s 2024-06-10T00:31:03.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:03 smithi135 ceph-mon[36424]: pgmap v95: 129 pgs: 129 active+clean; 168 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 256 op/s 2024-06-10T00:31:04.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:04 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:31:04.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:04 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:31:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:04 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:31:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:05 smithi193 ceph-mon[36716]: pgmap v96: 129 pgs: 129 active+clean; 233 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.5 MiB/s wr, 695 op/s 2024-06-10T00:31:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:05 smithi121 ceph-mon[32378]: pgmap v96: 129 pgs: 129 active+clean; 233 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.5 MiB/s wr, 695 op/s 2024-06-10T00:31:05.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:05 smithi135 ceph-mon[36424]: pgmap v96: 129 pgs: 129 active+clean; 233 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.5 MiB/s wr, 695 op/s 2024-06-10T00:31:07.292 INFO:tasks.workunit.client.0.smithi121.stdout: 52 465869 326 335104 321 233470 502 2024-06-10T00:31:07.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:07 smithi193 ceph-mon[36716]: pgmap v97: 129 pgs: 129 active+clean; 233 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.6 MiB/s wr, 609 op/s 2024-06-10T00:31:07.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:07 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:31:07.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:07 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:31:07.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:07 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:31:07.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:07 smithi121 ceph-mon[32378]: pgmap v97: 129 pgs: 129 active+clean; 233 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.6 MiB/s wr, 609 op/s 2024-06-10T00:31:07.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:07 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:31:07.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:07 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:31:07.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:07 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:31:07.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:07 smithi135 ceph-mon[36424]: pgmap v97: 129 pgs: 129 active+clean; 233 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.6 MiB/s wr, 609 op/s 2024-06-10T00:31:07.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:07 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:31:07.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:07 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:31:07.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:07 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:31:09.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:09 smithi193 ceph-mon[36716]: pgmap v98: 129 pgs: 129 active+clean; 233 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.6 MiB/s wr, 609 op/s 2024-06-10T00:31:09.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:09 smithi121 ceph-mon[32378]: pgmap v98: 129 pgs: 129 active+clean; 233 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.6 MiB/s wr, 609 op/s 2024-06-10T00:31:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:09 smithi135 ceph-mon[36424]: pgmap v98: 129 pgs: 129 active+clean; 233 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.6 MiB/s wr, 609 op/s 2024-06-10T00:31:11.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:11 smithi193 ceph-mon[36716]: pgmap v99: 129 pgs: 129 active+clean; 254 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.5 MiB/s wr, 747 op/s 2024-06-10T00:31:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:11 smithi121 ceph-mon[32378]: pgmap v99: 129 pgs: 129 active+clean; 254 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.5 MiB/s wr, 747 op/s 2024-06-10T00:31:11.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:11 smithi135 ceph-mon[36424]: pgmap v99: 129 pgs: 129 active+clean; 254 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.5 MiB/s wr, 747 op/s 2024-06-10T00:31:12.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:12 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:31:12.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:12 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:31:12.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:12 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:31:12.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:12 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:31:12.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:12 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:31:12.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:12 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:31:12.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:12 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:31:12.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:12 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:31:12.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:12 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:31:12.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:12 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:31:12.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:12 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:31:12.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:12 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:31:13.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:13 smithi193 ceph-mon[36716]: pgmap v100: 129 pgs: 129 active+clean; 254 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.8 MiB/s wr, 577 op/s 2024-06-10T00:31:13.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:13 smithi121 ceph-mon[32378]: pgmap v100: 129 pgs: 129 active+clean; 254 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.8 MiB/s wr, 577 op/s 2024-06-10T00:31:13.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:13 smithi135 ceph-mon[36424]: pgmap v100: 129 pgs: 129 active+clean; 254 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 7.8 MiB/s wr, 577 op/s 2024-06-10T00:31:15.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:15 smithi193 ceph-mon[36716]: pgmap v101: 129 pgs: 129 active+clean; 262 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-06-10T00:31:15.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:15 smithi121 ceph-mon[32378]: pgmap v101: 129 pgs: 129 active+clean; 262 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-06-10T00:31:15.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:15 smithi135 ceph-mon[36424]: pgmap v101: 129 pgs: 129 active+clean; 262 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 8.5 MiB/s wr, 604 op/s 2024-06-10T00:31:17.293 INFO:tasks.workunit.client.0.smithi121.stdout: 76 49244 1370 36151 1216 16899 313 2024-06-10T00:31:17.382 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:17 smithi193 ceph-mon[36716]: pgmap v102: 129 pgs: 129 active+clean; 262 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-06-10T00:31:17.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:17 smithi121 ceph-mon[32378]: pgmap v102: 129 pgs: 129 active+clean; 262 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-06-10T00:31:17.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:17 smithi135 ceph-mon[36424]: pgmap v102: 129 pgs: 129 active+clean; 262 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-06-10T00:31:19.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:19 smithi193 ceph-mon[36716]: pgmap v103: 129 pgs: 129 active+clean; 262 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-06-10T00:31:19.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:19 smithi121 ceph-mon[32378]: pgmap v103: 129 pgs: 129 active+clean; 262 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-06-10T00:31:19.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:19 smithi135 ceph-mon[36424]: pgmap v103: 129 pgs: 129 active+clean; 262 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-06-10T00:31:21.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:21 smithi193 ceph-mon[36716]: pgmap v104: 129 pgs: 129 active+clean; 270 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 192 op/s 2024-06-10T00:31:21.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:21 smithi121 ceph-mon[32378]: pgmap v104: 129 pgs: 129 active+clean; 270 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 192 op/s 2024-06-10T00:31:21.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:21 smithi135 ceph-mon[36424]: pgmap v104: 129 pgs: 129 active+clean; 270 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.4 MiB/s wr, 192 op/s 2024-06-10T00:31:23.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:23 smithi193 ceph-mon[36716]: pgmap v105: 129 pgs: 129 active+clean; 270 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-06-10T00:31:23.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:23 smithi121 ceph-mon[32378]: pgmap v105: 129 pgs: 129 active+clean; 270 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-06-10T00:31:23.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:23 smithi135 ceph-mon[36424]: pgmap v105: 129 pgs: 129 active+clean; 270 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 54 op/s 2024-06-10T00:31:25.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:25 smithi193 ceph-mon[36716]: pgmap v106: 129 pgs: 129 active+clean; 350 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.5 MiB/s wr, 399 op/s 2024-06-10T00:31:25.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:25 smithi121 ceph-mon[32378]: pgmap v106: 129 pgs: 129 active+clean; 350 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.5 MiB/s wr, 399 op/s 2024-06-10T00:31:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:25 smithi135 ceph-mon[36424]: pgmap v106: 129 pgs: 129 active+clean; 350 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 8.5 MiB/s wr, 399 op/s 2024-06-10T00:31:27.293 INFO:tasks.workunit.client.0.smithi121.stdout: 86 721960 584 520436 583 223811 358 2024-06-10T00:31:27.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:27 smithi193 ceph-mon[36716]: pgmap v107: 129 pgs: 129 active+clean; 350 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.8 MiB/s wr, 372 op/s 2024-06-10T00:31:27.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:27 smithi121 ceph-mon[32378]: pgmap v107: 129 pgs: 129 active+clean; 350 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.8 MiB/s wr, 372 op/s 2024-06-10T00:31:27.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:27 smithi135 ceph-mon[36424]: pgmap v107: 129 pgs: 129 active+clean; 350 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.8 MiB/s wr, 372 op/s 2024-06-10T00:31:29.077 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:29 smithi121 ceph-mon[32378]: pgmap v108: 129 pgs: 129 active+clean; 350 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.8 MiB/s wr, 372 op/s 2024-06-10T00:31:29.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:29 smithi193 ceph-mon[36716]: pgmap v108: 129 pgs: 129 active+clean; 350 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.8 MiB/s wr, 372 op/s 2024-06-10T00:31:29.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:29 smithi135 ceph-mon[36424]: pgmap v108: 129 pgs: 129 active+clean; 350 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 7.8 MiB/s wr, 372 op/s 2024-06-10T00:31:31.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:31 smithi193 ceph-mon[36716]: pgmap v109: 129 pgs: 129 active+clean; 388 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 11 MiB/s wr, 561 op/s 2024-06-10T00:31:31.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:31 smithi121 ceph-mon[32378]: pgmap v109: 129 pgs: 129 active+clean; 388 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 11 MiB/s wr, 561 op/s 2024-06-10T00:31:31.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:31 smithi135 ceph-mon[36424]: pgmap v109: 129 pgs: 129 active+clean; 388 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 11 MiB/s wr, 561 op/s 2024-06-10T00:31:32.526 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:31:32.789 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:31:33.175 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:33 smithi121 ceph-mon[32378]: pgmap v110: 129 pgs: 129 active+clean; 388 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 10 MiB/s wr, 534 op/s 2024-06-10T00:31:33.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:33 smithi193 ceph-mon[36716]: pgmap v110: 129 pgs: 129 active+clean; 388 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 10 MiB/s wr, 534 op/s 2024-06-10T00:31:33.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:33 smithi135 ceph-mon[36424]: pgmap v110: 129 pgs: 129 active+clean; 388 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 10 MiB/s wr, 534 op/s 2024-06-10T00:31:34.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.921+0000 7f14d15aa700 1 -- 172.21.15.121:0/1109102351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14cc06a6e0 msgr2=0x7f14cc06ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:34.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.921+0000 7f14d15aa700 1 --2- 172.21.15.121:0/1109102351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14cc06a6e0 0x7f14cc06ab40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f14bc009990 tx=0x7f14bc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.922+0000 7f14d15aa700 1 -- 172.21.15.121:0/1109102351 shutdown_connections 2024-06-10T00:31:34.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.922+0000 7f14d15aa700 1 --2- 172.21.15.121:0/1109102351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14cc104ed0 0x7f14cc105360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.922+0000 7f14d15aa700 1 --2- 172.21.15.121:0/1109102351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14cc1073c0 0x7f14cc06a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.922+0000 7f14d15aa700 1 --2- 172.21.15.121:0/1109102351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14cc06a6e0 0x7f14cc06ab40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.922+0000 7f14d15aa700 1 -- 172.21.15.121:0/1109102351 >> 172.21.15.121:0/1109102351 conn(0x7f14cc078340 msgr2=0x7f14cc078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:34.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.922+0000 7f14d15aa700 1 -- 172.21.15.121:0/1109102351 shutdown_connections 2024-06-10T00:31:34.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.925+0000 7f14d15aa700 1 -- 172.21.15.121:0/1109102351 wait complete. 2024-06-10T00:31:34.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.926+0000 7f14d15aa700 1 Processor -- start 2024-06-10T00:31:34.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.927+0000 7f14d15aa700 1 -- start start 2024-06-10T00:31:34.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.927+0000 7f14d15aa700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14cc06a6e0 0x7f14cc072e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.927+0000 7f14d15aa700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14cc104ed0 0x7f14cc073390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.927+0000 7f14d15aa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14cc1073c0 0x7f14cc10bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.927+0000 7f14d15aa700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc1083c0 con 0x7f14cc1073c0 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.927+0000 7f14d15aa700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc108240 con 0x7f14cc104ed0 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.927+0000 7f14d15aa700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc108540 con 0x7f14cc06a6e0 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.928+0000 7f14cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14cc1073c0 0x7f14cc10bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.928+0000 7f14cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14cc1073c0 0x7f14cc10bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45530/0 (socket says 172.21.15.121:45530) 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.928+0000 7f14cb7fe700 1 -- 172.21.15.121:0/3690602295 learned_addr learned my addr 172.21.15.121:0/3690602295 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.928+0000 7f14cb7fe700 1 -- 172.21.15.121:0/3690602295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14cc06a6e0 msgr2=0x7f14cc072e50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.928+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14cc104ed0 0x7f14cc073390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.928+0000 7f14cb7fe700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14cc06a6e0 0x7f14cc072e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.928+0000 7f14cb7fe700 1 -- 172.21.15.121:0/3690602295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14cc104ed0 msgr2=0x7f14cc073390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.928+0000 7f14cb7fe700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14cc104ed0 0x7f14cc073390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.929+0000 7f14cb7fe700 1 -- 172.21.15.121:0/3690602295 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc10c320 con 0x7f14cc1073c0 2024-06-10T00:31:34.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.929+0000 7f14cb7fe700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14cc1073c0 0x7f14cc10bc50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f14c400b970 tx=0x7f14c400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:34.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.929+0000 7f14b3fff700 1 -- 172.21.15.121:0/3690602295 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c4019070 con 0x7f14cc1073c0 2024-06-10T00:31:34.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.930+0000 7f14b3fff700 1 -- 172.21.15.121:0/3690602295 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14c4014440 con 0x7f14cc1073c0 2024-06-10T00:31:34.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.930+0000 7f14d15aa700 1 -- 172.21.15.121:0/3690602295 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc10c610 con 0x7f14cc1073c0 2024-06-10T00:31:34.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.930+0000 7f14d15aa700 1 -- 172.21.15.121:0/3690602295 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14cc113f40 con 0x7f14cc1073c0 2024-06-10T00:31:34.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.930+0000 7f14b3fff700 1 -- 172.21.15.121:0/3690602295 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c40075d0 con 0x7f14cc1073c0 2024-06-10T00:31:34.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.935+0000 7f14b3fff700 1 -- 172.21.15.121:0/3690602295 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f14c401c070 con 0x7f14cc1073c0 2024-06-10T00:31:34.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.936+0000 7f14b3fff700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f14b40a2e50 0x7f14b40a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.936+0000 7f14b3fff700 1 -- 172.21.15.121:0/3690602295 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f14c40cea10 con 0x7f14cc1073c0 2024-06-10T00:31:34.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14dbe2db80 1 Processor -- start 2024-06-10T00:31:34.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14caffd700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f14b40a2e50 0x7f14b40a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14dbe2db80 1 -- start start 2024-06-10T00:31:34.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba120d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 0x559ba1207260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 0x559ba1207bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x559ba10f9f60 con 0x559ba12077a0 2024-06-10T00:31:34.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x559ba1158850 con 0x559ba120d880 2024-06-10T00:31:34.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559ba10c0460 con 0x559ba120cf60 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14ca7fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 0x559ba1207260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 0x559ba1207bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14ca7fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 0x559ba1207260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:54470/0 (socket says 172.21.15.121:54470) 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14ca7fc700 1 -- 172.21.15.121:0/2081040719 learned_addr learned my addr 172.21.15.121:0/2081040719 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14caffd700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f14b40a2e50 0x7f14b40a5310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f14c0005950 tx=0x7f14c00058e0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.937+0000 7f14caffd700 1 --2- 172.21.15.121:0/2081040719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba120d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.938+0000 7f14ca7fc700 1 -- 172.21.15.121:0/2081040719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 msgr2=0x559ba120d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.938+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/2081040719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba120d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.938+0000 7f14ca7fc700 1 -- 172.21.15.121:0/2081040719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 msgr2=0x559ba1207bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.938+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/2081040719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 0x559ba1207bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.938+0000 7f14ca7fc700 1 -- 172.21.15.121:0/2081040719 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ba10a78d0 con 0x559ba120d880 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.938+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/2081040719 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 0x559ba1207260 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f14bc005e00 tx=0x7f14bc005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db8f72e55f46188b server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.938+0000 7f14b17fa700 1 -- 172.21.15.121:0/2081040719 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc026070 con 0x559ba120d880 2024-06-10T00:31:34.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.938+0000 7f14b17fa700 1 -- 172.21.15.121:0/2081040719 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14bc00b610 con 0x559ba120d880 2024-06-10T00:31:34.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.938+0000 7f14b17fa700 1 -- 172.21.15.121:0/2081040719 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc003a80 con 0x559ba120d880 2024-06-10T00:31:34.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.939+0000 7f14dbe2db80 1 -- 172.21.15.121:0/2081040719 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 msgr2=0x559ba1207260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:34.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.939+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/2081040719 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 0x559ba1207260 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f14bc005e00 tx=0x7f14bc005c00 comp rx=0 tx=0).stop 2024-06-10T00:31:34.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.939+0000 7f14dbe2db80 1 -- 172.21.15.121:0/2081040719 shutdown_connections 2024-06-10T00:31:34.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.939+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/2081040719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 0x559ba1207bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.939+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/2081040719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba120d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.939+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/2081040719 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 0x559ba1207260 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.939+0000 7f14dbe2db80 1 -- 172.21.15.121:0/2081040719 >> 172.21.15.121:0/2081040719 conn(0x559ba1134080 msgr2=0x559ba1202220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:34.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.939+0000 7f14dbe2db80 1 -- 172.21.15.121:0/2081040719 shutdown_connections 2024-06-10T00:31:34.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.939+0000 7f14dbe2db80 1 -- 172.21.15.121:0/2081040719 wait complete. 2024-06-10T00:31:34.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.940+0000 7f14dbe2db80 1 -- start start 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.940+0000 7f14dbe2db80 1 Processor -- start 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.940+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 0x559ba12135a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.940+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba12108a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.940+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 0x559ba1210de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.940+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x559ba10f8fc0 con 0x559ba120d880 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.940+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x559ba1154810 con 0x559ba12077a0 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.940+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559ba1214200 con 0x559ba120cf60 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14dbe2db80 10 client.0 _ll_register_callbacks cb 0x559ba10ee5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14caffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 0x559ba12135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14caffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 0x559ba12135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:54472/0 (socket says 172.21.15.121:54472) 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14caffd700 1 -- 172.21.15.121:0/3156302663 learned_addr learned my addr 172.21.15.121:0/3156302663 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba12108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14cb7fe700 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 0x559ba1210de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14caffd700 1 -- 172.21.15.121:0/3156302663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 msgr2=0x559ba12108a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14caffd700 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba12108a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14caffd700 1 -- 172.21.15.121:0/3156302663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 msgr2=0x559ba1210de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14caffd700 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 0x559ba1210de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14caffd700 1 -- 172.21.15.121:0/3156302663 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ba111e750 con 0x559ba12077a0 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba12108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14caffd700 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 0x559ba12135a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f14c001d6e0 tx=0x7f14c0004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=127d7bb28e926db7 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14aeff5700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14aeff5700 1 -- 172.21.15.121:0/3156302663 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c002a060 con 0x559ba12077a0 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3156302663 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14bc00b610 con 0x559ba12077a0 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.941+0000 7f14aeff5700 1 -- 172.21.15.121:0/3156302663 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f14c0010840 con 0x559ba12077a0 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14aeff5700 1 -- 172.21.15.121:0/3156302663 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f14c0031050 con 0x559ba12077a0 2024-06-10T00:31:34.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14dbe2db80 10 client.25595 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:34.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3156302663 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559ba121e9f0 con 0x559ba12077a0 2024-06-10T00:31:34.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14aeff5700 1 -- 172.21.15.121:0/3156302663 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14bc00b610 con 0x559ba12077a0 2024-06-10T00:31:34.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14aeff5700 1 -- 172.21.15.121:0/3156302663 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f14c0024030 con 0x559ba12077a0 2024-06-10T00:31:34.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14dbe2db80 10 client.25595 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:34.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14dbe2db80 10 client.25595 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:31:34.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14dbe2db80 20 client.25595 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:34.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x559ba1221030 0x559ba1223490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14dbe2db80 4 client.25595 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:31:34.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3156302663 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ba1165d50 con 0x559ba1221030 2024-06-10T00:31:34.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.942+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x559ba1221030 0x559ba1223490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.943+0000 7f14aeff5700 10 client.25595 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:34.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.943+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x559ba1221030 0x559ba1223490 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39561c89237f5a5c server_cookie=1f50e93a74bcbc2f in_seq=0 out_seq=0 2024-06-10T00:31:34.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14aeff5700 1 -- 172.21.15.121:0/3156302663 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x559ba1165d50 con 0x559ba1221030 2024-06-10T00:31:34.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14aeff5700 10 client.25595 handle_command_reply: tid=0 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14dbe2db80 1 client.25595 shutdown 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3156302663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 msgr2=0x559ba12135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 0x559ba12135a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f14c001d6e0 tx=0x7f14c0004750 comp rx=0 tx=0).stop 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3156302663 shutdown_connections 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x559ba1221030 0x559ba1223490 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 0x559ba1210de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14aeff5700 0 client.25595 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 0x559ba12135a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3156302663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba12108a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3156302663 >> 172.21.15.121:0/3156302663 conn(0x559ba1134080 msgr2=0x559ba1201700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3156302663 shutdown_connections 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.947+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3156302663 wait complete. 2024-06-10T00:31:34.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.948+0000 7f14dbe2db80 20 client.25595 trim_cache size 0 max 16384 2024-06-10T00:31:34.969 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14dbe2db80 1 Processor -- start 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14dbe2db80 1 -- start start 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 0x559ba1207260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba12127e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 0x559ba1201e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f14b80115e0 con 0x559ba120d880 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f14b800ee70 con 0x559ba12077a0 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14b800ccc0 con 0x559ba120cf60 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 0x559ba1201e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 0x559ba1201e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45564/0 (socket says 172.21.15.121:45564) 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14cb7fe700 1 -- 172.21.15.121:0/2383590320 learned_addr learned my addr 172.21.15.121:0/2383590320 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14caffd700 1 --2- 172.21.15.121:0/2383590320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 0x559ba1207260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/2383590320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba12127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.970 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14cb7fe700 1 -- 172.21.15.121:0/2383590320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 msgr2=0x559ba12127e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:34.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14cb7fe700 1 --2- 172.21.15.121:0/2383590320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba12127e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14cb7fe700 1 -- 172.21.15.121:0/2383590320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 msgr2=0x559ba1207260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:34.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14cb7fe700 1 --2- 172.21.15.121:0/2383590320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 0x559ba1207260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.969+0000 7f14cb7fe700 1 -- 172.21.15.121:0/2383590320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ba120ee10 con 0x559ba120d880 2024-06-10T00:31:34.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.970+0000 7f14cb7fe700 1 --2- 172.21.15.121:0/2383590320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 0x559ba1201e80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009a00 tx=0x7f14c400c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6767f600194aac server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:34.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.970+0000 7f14b0ff9700 1 -- 172.21.15.121:0/2383590320 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c4097780 con 0x559ba120d880 2024-06-10T00:31:34.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.970+0000 7f14b0ff9700 1 -- 172.21.15.121:0/2383590320 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14c409a5b0 con 0x559ba120d880 2024-06-10T00:31:34.971 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.970+0000 7f14b0ff9700 1 -- 172.21.15.121:0/2383590320 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c40a0290 con 0x559ba120d880 2024-06-10T00:31:34.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 -- 172.21.15.121:0/2383590320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 msgr2=0x559ba1201e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:34.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/2383590320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 0x559ba1201e80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009a00 tx=0x7f14c400c420 comp rx=0 tx=0).stop 2024-06-10T00:31:34.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 -- 172.21.15.121:0/2383590320 shutdown_connections 2024-06-10T00:31:34.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/2383590320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba120d880 0x559ba1201e80 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/2383590320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba12077a0 0x559ba1207260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/2383590320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba12127e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 -- 172.21.15.121:0/2383590320 >> 172.21.15.121:0/2383590320 conn(0x559ba1134080 msgr2=0x559ba12120d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:34.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 -- 172.21.15.121:0/2383590320 shutdown_connections 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 -- 172.21.15.121:0/2383590320 wait complete. 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 -- start start 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 Processor -- start 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 0x559ba1219520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba1219a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 0x559ba12164f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f14b800d330 con 0x559ba12077a0 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f14b800e7a0 con 0x559ba120d880 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.971+0000 7f14dbe2db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14b800da00 con 0x559ba120cf60 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14dbe2db80 10 client.0 _ll_register_callbacks cb 0x559ba10ee5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14caffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 0x559ba1219520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14cb7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 0x559ba12164f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14caffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 0x559ba1219520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45570/0 (socket says 172.21.15.121:45570) 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14caffd700 1 -- 172.21.15.121:0/3130486528 learned_addr learned my addr 172.21.15.121:0/3130486528 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba1219a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14caffd700 1 -- 172.21.15.121:0/3130486528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 msgr2=0x559ba1219a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14caffd700 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba1219a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14caffd700 1 -- 172.21.15.121:0/3130486528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 msgr2=0x559ba12164f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14caffd700 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 0x559ba12164f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14caffd700 1 -- 172.21.15.121:0/3130486528 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ba121a160 con 0x559ba12077a0 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14cb7fe700 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 0x559ba12164f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14caffd700 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 0x559ba1219520 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f14c0025e50 tx=0x7f14c00256a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a05acfe2d2c027a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14aeff5700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14aeff5700 1 -- 172.21.15.121:0/3130486528 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c003a070 con 0x559ba12077a0 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14aeff5700 1 -- 172.21.15.121:0/3130486528 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f14c0004a30 con 0x559ba12077a0 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3130486528 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14c409a5b0 con 0x559ba12077a0 2024-06-10T00:31:34.976 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14aeff5700 1 -- 172.21.15.121:0/3130486528 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f14c0045050 con 0x559ba12077a0 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14dbe2db80 10 client.16026 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.972+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3130486528 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559ba128c3e0 con 0x559ba12077a0 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.973+0000 7f14aeff5700 1 -- 172.21.15.121:0/3130486528 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14c409a5b0 con 0x559ba12077a0 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.973+0000 7f14aeff5700 1 -- 172.21.15.121:0/3130486528 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f14c0024040 con 0x559ba12077a0 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.973+0000 7f14dbe2db80 10 client.16026 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.973+0000 7f14dbe2db80 10 client.16026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.973+0000 7f14dbe2db80 20 client.16026 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.973+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x559ba1221030 0x559ba128eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.973+0000 7f14dbe2db80 4 client.16026 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.973+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3130486528 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559ba11602d0 con 0x559ba1221030 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.973+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x559ba1221030 0x559ba128eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.973+0000 7f14ca7fc700 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x559ba1221030 0x559ba128eb90 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96312f8e158fcea6 server_cookie=1ec9c2bb24f89def in_seq=0 out_seq=0 2024-06-10T00:31:34.977 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.973+0000 7f14aeff5700 10 client.16026 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:34.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14aeff5700 1 -- 172.21.15.121:0/3130486528 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559ba11602d0 con 0x559ba1221030 2024-06-10T00:31:34.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14aeff5700 10 client.16026 handle_command_reply: tid=0 2024-06-10T00:31:34.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14dbe2db80 1 client.16026 shutdown 2024-06-10T00:31:34.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3130486528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 msgr2=0x559ba1219520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:34.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 0x559ba1219520 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f14c0025e50 tx=0x7f14c00256a0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3130486528 shutdown_connections 2024-06-10T00:31:34.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x559ba1221030 0x559ba128eb90 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x559ba120d880 0x559ba12164f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559ba12077a0 0x559ba1219520 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14aeff5700 0 client.16026 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:34.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14dbe2db80 1 --2- 172.21.15.121:0/3130486528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559ba120cf60 0x559ba1219a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3130486528 >> 172.21.15.121:0/3130486528 conn(0x559ba1134080 msgr2=0x559ba1212960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:34.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.979+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3130486528 shutdown_connections 2024-06-10T00:31:34.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14dbe2db80 1 -- 172.21.15.121:0/3130486528 wait complete. 2024-06-10T00:31:34.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14dbe2db80 20 client.16026 trim_cache size 0 max 16384 2024-06-10T00:31:34.981 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:31:34.981 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "no active scrubs running", 2024-06-10T00:31:34.981 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": {} 2024-06-10T00:31:34.981 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:31:34.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14d15aa700 1 -- 172.21.15.121:0/3690602295 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f14b40a2e50 msgr2=0x7f14b40a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:34.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14d15aa700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f14b40a2e50 0x7f14b40a5310 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f14c0005950 tx=0x7f14c00058e0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14d15aa700 1 -- 172.21.15.121:0/3690602295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14cc1073c0 msgr2=0x7f14cc10bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:34.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14d15aa700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14cc1073c0 0x7f14cc10bc50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f14c400b970 tx=0x7f14c400bd30 comp rx=0 tx=0).stop 2024-06-10T00:31:34.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14d15aa700 1 -- 172.21.15.121:0/3690602295 shutdown_connections 2024-06-10T00:31:34.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14d15aa700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f14b40a2e50 0x7f14b40a5310 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14d15aa700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14cc1073c0 0x7f14cc10bc50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14d15aa700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14cc06a6e0 0x7f14cc072e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14d15aa700 1 --2- 172.21.15.121:0/3690602295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f14cc104ed0 0x7f14cc073390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14d15aa700 1 -- 172.21.15.121:0/3690602295 >> 172.21.15.121:0/3690602295 conn(0x7f14cc078340 msgr2=0x7f14cc103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.980+0000 7f14d15aa700 1 -- 172.21.15.121:0/3690602295 shutdown_connections 2024-06-10T00:31:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:34.981+0000 7f14d15aa700 1 -- 172.21.15.121:0/3690602295 wait complete. 2024-06-10T00:31:35.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:35 smithi193 ceph-mon[36716]: pgmap v111: 129 pgs: 129 active+clean; 393 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 11 MiB/s wr, 592 op/s 2024-06-10T00:31:35.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:35 smithi121 ceph-mon[32378]: pgmap v111: 129 pgs: 129 active+clean; 393 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 11 MiB/s wr, 592 op/s 2024-06-10T00:31:35.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:35 smithi135 ceph-mon[36424]: pgmap v111: 129 pgs: 129 active+clean; 393 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 11 MiB/s wr, 592 op/s 2024-06-10T00:31:36.918 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-10T00:31:36.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:31:37.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:37 smithi121 ceph-mon[32378]: pgmap v112: 129 pgs: 129 active+clean; 393 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-06-10T00:31:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:31:37.294 INFO:tasks.workunit.client.0.smithi121.stdout: 100 1403420 963 1008469 778 417541 668 2024-06-10T00:31:37.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:37 smithi193 ceph-mon[36716]: pgmap v112: 129 pgs: 129 active+clean; 393 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-06-10T00:31:37.535 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:37 smithi135 ceph-mon[36424]: pgmap v112: 129 pgs: 129 active+clean; 393 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-06-10T00:31:39.334 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:39 smithi121 ceph-mon[32378]: pgmap v113: 129 pgs: 129 active+clean; 393 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-06-10T00:31:39.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:39 smithi193 ceph-mon[36716]: pgmap v113: 129 pgs: 129 active+clean; 393 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-06-10T00:31:39.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:39 smithi135 ceph-mon[36424]: pgmap v113: 129 pgs: 129 active+clean; 393 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.8 MiB/s wr, 248 op/s 2024-06-10T00:31:39.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.643+0000 7f1f19c5c700 1 -- 172.21.15.121:0/2554790187 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f14102510 msgr2=0x7f1f14102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:39.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.643+0000 7f1f19c5c700 1 --2- 172.21.15.121:0/2554790187 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f14102510 0x7f1f14102970 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f1f04009990 tx=0x7f1f04009ca0 comp rx=0 tx=0).stop 2024-06-10T00:31:39.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.643+0000 7f1f19c5c700 1 -- 172.21.15.121:0/2554790187 shutdown_connections 2024-06-10T00:31:39.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.643+0000 7f1f19c5c700 1 --2- 172.21.15.121:0/2554790187 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f14102eb0 0x7f1f1410f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:39.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.643+0000 7f1f19c5c700 1 --2- 172.21.15.121:0/2554790187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f14107420 0x7f1f14107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:39.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.643+0000 7f1f19c5c700 1 --2- 172.21.15.121:0/2554790187 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f14102510 0x7f1f14102970 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:39.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.643+0000 7f1f19c5c700 1 -- 172.21.15.121:0/2554790187 >> 172.21.15.121:0/2554790187 conn(0x7f1f140795f0 msgr2=0x7f1f14079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:39.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.643+0000 7f1f19c5c700 1 -- 172.21.15.121:0/2554790187 shutdown_connections 2024-06-10T00:31:39.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.643+0000 7f1f19c5c700 1 -- 172.21.15.121:0/2554790187 wait complete. 2024-06-10T00:31:39.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f19c5c700 1 Processor -- start 2024-06-10T00:31:39.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f19c5c700 1 -- start start 2024-06-10T00:31:39.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f19c5c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f14102510 0x7f1f14074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:39.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f19c5c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f14102eb0 0x7f1f14074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:39.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f19c5c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f14107420 0x7f1f14075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:39.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f19c5c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f14112830 con 0x7f1f14102510 2024-06-10T00:31:39.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f19c5c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f1f141126b0 con 0x7f1f14102eb0 2024-06-10T00:31:39.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f19c5c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1f141129b0 con 0x7f1f14107420 2024-06-10T00:31:39.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f137fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f14102510 0x7f1f14074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:39.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f12ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f14102eb0 0x7f1f14074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:39.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f137fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f14102510 0x7f1f14074790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45842/0 (socket says 172.21.15.121:45842) 2024-06-10T00:31:39.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f137fe700 1 -- 172.21.15.121:0/3937564389 learned_addr learned my addr 172.21.15.121:0/3937564389 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:39.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f13fff700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f14107420 0x7f1f14075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:39.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f137fe700 1 -- 172.21.15.121:0/3937564389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f14107420 msgr2=0x7f1f14075210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:39.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f137fe700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f14107420 0x7f1f14075210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:39.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f137fe700 1 -- 172.21.15.121:0/3937564389 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f14102eb0 msgr2=0x7f1f14074cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:39.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f137fe700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f14102eb0 0x7f1f14074cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:39.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f137fe700 1 -- 172.21.15.121:0/3937564389 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f141a7260 con 0x7f1f14102510 2024-06-10T00:31:39.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.644+0000 7f1f12ffd700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f14102eb0 0x7f1f14074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:31:39.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.645+0000 7f1f137fe700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f14102510 0x7f1f14074790 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00b500 tx=0x7f1efc00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:39.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.645+0000 7f1f10ff9700 1 -- 172.21.15.121:0/3937564389 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1efc015020 con 0x7f1f14102510 2024-06-10T00:31:39.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.645+0000 7f1f19c5c700 1 -- 172.21.15.121:0/3937564389 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f141a7550 con 0x7f1f14102510 2024-06-10T00:31:39.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.645+0000 7f1f10ff9700 1 -- 172.21.15.121:0/3937564389 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1efc011d10 con 0x7f1f14102510 2024-06-10T00:31:39.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.645+0000 7f1f10ff9700 1 -- 172.21.15.121:0/3937564389 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1efc010db0 con 0x7f1f14102510 2024-06-10T00:31:39.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.645+0000 7f1f19c5c700 1 -- 172.21.15.121:0/3937564389 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f141a7ab0 con 0x7f1f14102510 2024-06-10T00:31:39.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.646+0000 7f1f19c5c700 1 -- 172.21.15.121:0/3937564389 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ef4005330 con 0x7f1f14102510 2024-06-10T00:31:39.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.647+0000 7f1f10ff9700 1 -- 172.21.15.121:0/3937564389 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1efc018020 con 0x7f1f14102510 2024-06-10T00:31:39.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.648+0000 7f1f10ff9700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1f000a2d90 0x7f1f000a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:39.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.648+0000 7f1f12ffd700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1f000a2d90 0x7f1f000a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:39.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.648+0000 7f1f10ff9700 1 -- 172.21.15.121:0/3937564389 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f1efc0ced20 con 0x7f1f14102510 2024-06-10T00:31:39.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.648+0000 7f1f12ffd700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1f000a2d90 0x7f1f000a5250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400b2e0 tx=0x7f1f04005960 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:39.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.652+0000 7f1f10ff9700 1 -- 172.21.15.121:0/3937564389 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1efc01f020 con 0x7f1f14102510 2024-06-10T00:31:39.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.960+0000 7f1f19c5c700 1 -- 172.21.15.121:0/3937564389 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ef40058f0 con 0x7f1f14102510 2024-06-10T00:31:39.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.961+0000 7f1f10ff9700 1 -- 172.21.15.121:0/3937564389 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11888 (secure 0 0 0) 0x7f1efc09b6c0 con 0x7f1f14102510 2024-06-10T00:31:39.961 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:31:39.962 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:30:26.280580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:31:39.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.963+0000 7f1f19c5c700 1 -- 172.21.15.121:0/3937564389 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1f000a2d90 msgr2=0x7f1f000a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:39.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.963+0000 7f1f19c5c700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1f000a2d90 0x7f1f000a5250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400b2e0 tx=0x7f1f04005960 comp rx=0 tx=0).stop 2024-06-10T00:31:39.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.963+0000 7f1f19c5c700 1 -- 172.21.15.121:0/3937564389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f14102510 msgr2=0x7f1f14074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:39.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.963+0000 7f1f19c5c700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f14102510 0x7f1f14074790 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00b500 tx=0x7f1efc00b810 comp rx=0 tx=0).stop 2024-06-10T00:31:39.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.964+0000 7f1f19c5c700 1 -- 172.21.15.121:0/3937564389 shutdown_connections 2024-06-10T00:31:39.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.964+0000 7f1f19c5c700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1f000a2d90 0x7f1f000a5250 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:39.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.964+0000 7f1f19c5c700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1f14107420 0x7f1f14075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:39.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.964+0000 7f1f19c5c700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f14102510 0x7f1f14074790 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:39.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.964+0000 7f1f19c5c700 1 --2- 172.21.15.121:0/3937564389 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1f14102eb0 0x7f1f14074cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:39.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.964+0000 7f1f19c5c700 1 -- 172.21.15.121:0/3937564389 >> 172.21.15.121:0/3937564389 conn(0x7f1f140795f0 msgr2=0x7f1f14100600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:39.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.964+0000 7f1f19c5c700 1 -- 172.21.15.121:0/3937564389 shutdown_connections 2024-06-10T00:31:39.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:39.964+0000 7f1f19c5c700 1 -- 172.21.15.121:0/3937564389 wait complete. 2024-06-10T00:31:39.965 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 28 2024-06-10T00:31:40.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:40 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3937564389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:31:40.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:40 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3937564389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:31:40.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:40 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3937564389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:31:41.097 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-10T00:31:41.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:41 smithi193 ceph-mon[36716]: pgmap v114: 129 pgs: 129 active+clean; 401 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-06-10T00:31:41.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:41 smithi121 ceph-mon[32378]: pgmap v114: 129 pgs: 129 active+clean; 401 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-06-10T00:31:41.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:41 smithi135 ceph-mon[36424]: pgmap v114: 129 pgs: 129 active+clean; 401 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 324 op/s 2024-06-10T00:31:41.635 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:31:43.096 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:43 smithi121 ceph-mon[32378]: pgmap v115: 129 pgs: 129 active+clean; 401 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-10T00:31:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:43 smithi193 ceph-mon[36716]: pgmap v115: 129 pgs: 129 active+clean; 401 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-10T00:31:43.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:43 smithi135 ceph-mon[36424]: pgmap v115: 129 pgs: 129 active+clean; 401 MiB data, 861 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 134 op/s 2024-06-10T00:31:45.339 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:45 smithi121 ceph-mon[32378]: pgmap v116: 129 pgs: 129 active+clean; 463 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 455 op/s 2024-06-10T00:31:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:45 smithi193 ceph-mon[36716]: pgmap v116: 129 pgs: 129 active+clean; 463 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 455 op/s 2024-06-10T00:31:45.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.462+0000 7f90c9ffb700 1 -- 172.21.15.121:0/623092884 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc07d640 con 0x7f90c4094ee0 2024-06-10T00:31:45.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.462+0000 7f90d1295700 1 -- 172.21.15.121:0/623092884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90c4094ee0 msgr2=0x7f90c4095360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:45.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.462+0000 7f90d1295700 1 --2- 172.21.15.121:0/623092884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90c4094ee0 0x7f90c4095360 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0517a0 tx=0x7f90cc068aa0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.463+0000 7f90d1295700 1 -- 172.21.15.121:0/623092884 shutdown_connections 2024-06-10T00:31:45.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.463+0000 7f90d1295700 1 --2- 172.21.15.121:0/623092884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c40958a0 0x7f90c40a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.463+0000 7f90d1295700 1 --2- 172.21.15.121:0/623092884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c409aca0 0x7f90c409b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.463+0000 7f90d1295700 1 --2- 172.21.15.121:0/623092884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90c4094ee0 0x7f90c4095360 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.463+0000 7f90d1295700 1 -- 172.21.15.121:0/623092884 >> 172.21.15.121:0/623092884 conn(0x7f90c400a7a0 msgr2=0x7f90c400abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:45.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.463+0000 7f90d1295700 1 -- 172.21.15.121:0/623092884 shutdown_connections 2024-06-10T00:31:45.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.463+0000 7f90d1295700 1 -- 172.21.15.121:0/623092884 wait complete. 2024-06-10T00:31:45.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.464+0000 7f90d1295700 1 Processor -- start 2024-06-10T00:31:45.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.464+0000 7f90d1295700 1 -- start start 2024-06-10T00:31:45.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.465+0000 7f90d1295700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90c4094ee0 0x7f90c41386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.465+0000 7f90d1295700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c40958a0 0x7f90c4138c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.465+0000 7f90d1295700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c409aca0 0x7f90c41328a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.465+0000 7f90d1295700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f90c40ac680 con 0x7f90c409aca0 2024-06-10T00:31:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.465+0000 7f90d1295700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f90c40ac500 con 0x7f90c4094ee0 2024-06-10T00:31:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.465+0000 7f90d1295700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f90c40ac800 con 0x7f90c40958a0 2024-06-10T00:31:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.465+0000 7f90ca7fc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c40958a0 0x7f90c4138c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.465+0000 7f90ca7fc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c40958a0 0x7f90c4138c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:53508/0 (socket says 172.21.15.121:53508) 2024-06-10T00:31:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.465+0000 7f90ca7fc700 1 -- 172.21.15.121:0/1276090308 learned_addr learned my addr 172.21.15.121:0/1276090308 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.465+0000 7f90caffd700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90c4094ee0 0x7f90c41386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.465+0000 7f90cb7fe700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c409aca0 0x7f90c41328a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.466+0000 7f90ca7fc700 1 -- 172.21.15.121:0/1276090308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90c4094ee0 msgr2=0x7f90c41386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.466+0000 7f90ca7fc700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90c4094ee0 0x7f90c41386e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.466+0000 7f90ca7fc700 1 -- 172.21.15.121:0/1276090308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c409aca0 msgr2=0x7f90c41328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.466+0000 7f90ca7fc700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c409aca0 0x7f90c41328a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.466+0000 7f90ca7fc700 1 -- 172.21.15.121:0/1276090308 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c4133000 con 0x7f90c40958a0 2024-06-10T00:31:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.466+0000 7f90ca7fc700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c40958a0 0x7f90c4138c20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f90cc065ce0 tx=0x7f90cc06aff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.466+0000 7f90b3fff700 1 -- 172.21.15.121:0/1276090308 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc084040 con 0x7f90c40958a0 2024-06-10T00:31:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.466+0000 7f90b3fff700 1 -- 172.21.15.121:0/1276090308 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90cc074dd0 con 0x7f90c40958a0 2024-06-10T00:31:45.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.466+0000 7f90b3fff700 1 -- 172.21.15.121:0/1276090308 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc07d7e0 con 0x7f90c40958a0 2024-06-10T00:31:45.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.467+0000 7f90d1295700 1 -- 172.21.15.121:0/1276090308 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90c4133290 con 0x7f90c40958a0 2024-06-10T00:31:45.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.467+0000 7f90d1295700 1 -- 172.21.15.121:0/1276090308 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90c4133690 con 0x7f90c40958a0 2024-06-10T00:31:45.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.470+0000 7f90b3fff700 1 -- 172.21.15.121:0/1276090308 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f90cc090060 con 0x7f90c40958a0 2024-06-10T00:31:45.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.472+0000 7f90b3fff700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f90b40a2e50 0x7f90b40a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.472+0000 7f90b3fff700 1 -- 172.21.15.121:0/1276090308 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f90cc13c090 con 0x7f90c40958a0 2024-06-10T00:31:45.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.473+0000 7f90caffd700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f90b40a2e50 0x7f90b40a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.473+0000 7f90dbb18b80 1 Processor -- start 2024-06-10T00:31:45.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.473+0000 7f90dbb18b80 1 -- start start 2024-06-10T00:31:45.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fcf8f0 0x561739fcfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fd0210 0x561739fc9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fca080 0x561739fca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561739f1b810 con 0x561739fca080 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x561739ec3720 con 0x561739fd0210 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561739f177d0 con 0x561739fcf8f0 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fca080 0x561739fca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fca080 0x561739fca4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45864/0 (socket says 172.21.15.121:45864) 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90cb7fe700 1 -- 172.21.15.121:0/3276273802 learned_addr learned my addr 172.21.15.121:0/3276273802 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90ca7fc700 1 --2- 172.21.15.121:0/3276273802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fd0210 0x561739fc9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90cb7fe700 1 -- 172.21.15.121:0/3276273802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fcf8f0 msgr2=0x561739fcfcd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90cb7fe700 1 --2- 172.21.15.121:0/3276273802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fcf8f0 0x561739fcfcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90cb7fe700 1 -- 172.21.15.121:0/3276273802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fd0210 msgr2=0x561739fc9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90cb7fe700 1 --2- 172.21.15.121:0/3276273802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fd0210 0x561739fc9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.474+0000 7f90cb7fe700 1 -- 172.21.15.121:0/3276273802 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561739eb9f50 con 0x561739fca080 2024-06-10T00:31:45.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.475+0000 7f90cb7fe700 1 --2- 172.21.15.121:0/3276273802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fca080 0x561739fca4c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f90c000bc30 tx=0x7f90c000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1d1764416246a7 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:45.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.475+0000 7f90b17fa700 1 -- 172.21.15.121:0/3276273802 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c000aad0 con 0x561739fca080 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.475+0000 7f90b17fa700 1 -- 172.21.15.121:0/3276273802 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90c0004d10 con 0x561739fca080 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.476+0000 7f90b17fa700 1 -- 172.21.15.121:0/3276273802 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c00153f0 con 0x561739fca080 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.476+0000 7f90dbb18b80 1 -- 172.21.15.121:0/3276273802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fca080 msgr2=0x561739fca4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.476+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/3276273802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fca080 0x561739fca4c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f90c000bc30 tx=0x7f90c000bf40 comp rx=0 tx=0).stop 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.476+0000 7f90caffd700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f90b40a2e50 0x7f90b40a5310 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f90c41339e0 tx=0x7f90bc00d540 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.476+0000 7f90dbb18b80 1 -- 172.21.15.121:0/3276273802 shutdown_connections 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.476+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/3276273802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fca080 0x561739fca4c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.476+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/3276273802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fcf8f0 0x561739fcfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.476+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/3276273802 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fd0210 0x561739fc9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.476+0000 7f90dbb18b80 1 -- 172.21.15.121:0/3276273802 >> 172.21.15.121:0/3276273802 conn(0x561739ef4cc0 msgr2=0x561739fc4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.476+0000 7f90dbb18b80 1 -- 172.21.15.121:0/3276273802 shutdown_connections 2024-06-10T00:31:45.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.477+0000 7f90dbb18b80 1 -- 172.21.15.121:0/3276273802 wait complete. 2024-06-10T00:31:45.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.477+0000 7f90dbb18b80 1 -- start start 2024-06-10T00:31:45.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.478+0000 7f90dbb18b80 1 Processor -- start 2024-06-10T00:31:45.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.478+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fca080 0x561739fd5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.478+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fcf8f0 0x561739fd3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 0x561739fd36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561739fd6c70 con 0x561739fd0210 2024-06-10T00:31:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x561739ebbbf0 con 0x561739fca080 2024-06-10T00:31:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561739fd6df0 con 0x561739fcf8f0 2024-06-10T00:31:45.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 0x561739fd36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90caffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fca080 0x561739fd5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90caffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fca080 0x561739fd5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:42490/0 (socket says 172.21.15.121:42490) 2024-06-10T00:31:45.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90caffd700 1 -- 172.21.15.121:0/2680372817 learned_addr learned my addr 172.21.15.121:0/2680372817 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:45.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90dbb18b80 10 client.0 _ll_register_callbacks cb 0x561739eb5920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:45.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90caffd700 1 -- 172.21.15.121:0/2680372817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fcf8f0 msgr2=0x561739fd3160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:31:45.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90caffd700 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fcf8f0 0x561739fd3160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.479+0000 7f90caffd700 1 -- 172.21.15.121:0/2680372817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 msgr2=0x561739fd36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:45.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.480+0000 7f90caffd700 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 0x561739fd36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.480+0000 7f90caffd700 1 -- 172.21.15.121:0/2680372817 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561739e6a8d0 con 0x561739fca080 2024-06-10T00:31:45.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.480+0000 7f90cb7fe700 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 0x561739fd36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:31:45.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.480+0000 7f90aeff5700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:31:45.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.480+0000 7f90caffd700 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fca080 0x561739fd5e60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f90bc01b6e0 tx=0x7f90bc004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ae45f03a2bbc05 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:45.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.480+0000 7f90aeff5700 1 -- 172.21.15.121:0/2680372817 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc028060 con 0x561739fca080 2024-06-10T00:31:45.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.480+0000 7f90dbb18b80 1 -- 172.21.15.121:0/2680372817 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90c0004d10 con 0x561739fca080 2024-06-10T00:31:45.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.480+0000 7f90aeff5700 1 -- 172.21.15.121:0/2680372817 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f90bc020070 con 0x561739fca080 2024-06-10T00:31:45.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.480+0000 7f90aeff5700 1 -- 172.21.15.121:0/2680372817 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f90bc028060 con 0x561739fca080 2024-06-10T00:31:45.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.480+0000 7f90dbb18b80 10 client.25619 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:45.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.480+0000 7f90dbb18b80 1 -- 172.21.15.121:0/2680372817 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561739fe1550 con 0x561739fca080 2024-06-10T00:31:45.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.481+0000 7f90aeff5700 1 -- 172.21.15.121:0/2680372817 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90c0004d10 con 0x561739fca080 2024-06-10T00:31:45.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.481+0000 7f90aeff5700 1 -- 172.21.15.121:0/2680372817 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f90bc0236e0 con 0x561739fca080 2024-06-10T00:31:45.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.481+0000 7f90dbb18b80 10 client.25619 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:45.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.481+0000 7f90dbb18b80 10 client.25619 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:31:45.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.481+0000 7f90dbb18b80 20 client.25619 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:45.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.481+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x561739fe3b90 0x561739fe5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.481+0000 7f90dbb18b80 4 client.25619 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:31:45.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.481+0000 7f90dbb18b80 1 -- 172.21.15.121:0/2680372817 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561739f28d10 con 0x561739fe3b90 2024-06-10T00:31:45.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.481+0000 7f90ca7fc700 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x561739fe3b90 0x561739fe5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.481+0000 7f90aeff5700 10 client.25619 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:31:45.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.482+0000 7f90ca7fc700 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x561739fe3b90 0x561739fe5f80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dfd6b4b5128c930f server_cookie=a91ef098b812503f in_seq=0 out_seq=0 2024-06-10T00:31:45.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.483+0000 7f90aeff5700 1 -- 172.21.15.121:0/2680372817 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x561739f28d10 con 0x561739fe3b90 2024-06-10T00:31:45.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90aeff5700 10 client.25619 handle_command_reply: tid=0 2024-06-10T00:31:45.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 1 client.25619 shutdown 2024-06-10T00:31:45.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 1 -- 172.21.15.121:0/2680372817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fca080 msgr2=0x561739fd5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:45.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fca080 0x561739fd5e60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f90bc01b6e0 tx=0x7f90bc004750 comp rx=0 tx=0).stop 2024-06-10T00:31:45.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 1 -- 172.21.15.121:0/2680372817 shutdown_connections 2024-06-10T00:31:45.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x561739fe3b90 0x561739fe5f80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 0x561739fd36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fca080 0x561739fd5e60 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f90bc01b6e0 tx=0x7f90bc004750 comp rx=0 tx=0).stop 2024-06-10T00:31:45.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/2680372817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fcf8f0 0x561739fd3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 1 -- 172.21.15.121:0/2680372817 >> 172.21.15.121:0/2680372817 conn(0x561739ef4cc0 msgr2=0x561739fc8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:45.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90aeff5700 0 client.25619 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:31:45.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 1 -- 172.21.15.121:0/2680372817 shutdown_connections 2024-06-10T00:31:45.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 1 -- 172.21.15.121:0/2680372817 wait complete. 2024-06-10T00:31:45.485 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.484+0000 7f90dbb18b80 20 client.25619 trim_cache size 0 max 16384 2024-06-10T00:31:45.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:45 smithi135 ceph-mon[36424]: pgmap v116: 129 pgs: 129 active+clean; 463 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.9 MiB/s wr, 455 op/s 2024-06-10T00:31:45.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.515+0000 7f90dbb18b80 1 Processor -- start 2024-06-10T00:31:45.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90dbb18b80 1 -- start start 2024-06-10T00:31:45.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fcf8f0 0x561739fd36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fd0210 0x561739fd0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fc44e0 0x561739fc4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f90c414ade0 con 0x561739fcf8f0 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f90c4148670 con 0x561739fc44e0 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f90c4146460 con 0x561739fd0210 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90caffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fcf8f0 0x561739fd36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90caffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fcf8f0 0x561739fd36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45888/0 (socket says 172.21.15.121:45888) 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90caffd700 1 -- 172.21.15.121:0/936390095 learned_addr learned my addr 172.21.15.121:0/936390095 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90ca7fc700 1 --2- 172.21.15.121:0/936390095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fd0210 0x561739fd0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90cb7fe700 1 --2- 172.21.15.121:0/936390095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fc44e0 0x561739fc4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90caffd700 1 -- 172.21.15.121:0/936390095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fd0210 msgr2=0x561739fd0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90caffd700 1 --2- 172.21.15.121:0/936390095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fd0210 0x561739fd0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90caffd700 1 -- 172.21.15.121:0/936390095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fc44e0 msgr2=0x561739fc4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:45.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90caffd700 1 --2- 172.21.15.121:0/936390095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fc44e0 0x561739fc4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90caffd700 1 -- 172.21.15.121:0/936390095 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561739fc60d0 con 0x561739fcf8f0 2024-06-10T00:31:45.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.516+0000 7f90ca7fc700 1 --2- 172.21.15.121:0/936390095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fd0210 0x561739fd0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:31:45.518 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.517+0000 7f90caffd700 1 --2- 172.21.15.121:0/936390095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fcf8f0 0x561739fd36d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f90bc01be30 tx=0x7f90bc00d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3889ee8cecf1d90d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:45.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.517+0000 7f90b0ff9700 1 -- 172.21.15.121:0/936390095 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc028060 con 0x561739fcf8f0 2024-06-10T00:31:45.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.517+0000 7f90b0ff9700 1 -- 172.21.15.121:0/936390095 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90bc021e10 con 0x561739fcf8f0 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.517+0000 7f90b0ff9700 1 -- 172.21.15.121:0/936390095 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc02b350 con 0x561739fcf8f0 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.518+0000 7f90dbb18b80 1 -- 172.21.15.121:0/936390095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fcf8f0 msgr2=0x561739fd36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.518+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/936390095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fcf8f0 0x561739fd36d0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f90bc01be30 tx=0x7f90bc00d340 comp rx=0 tx=0).stop 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.518+0000 7f90dbb18b80 1 -- 172.21.15.121:0/936390095 shutdown_connections 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.518+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/936390095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fc44e0 0x561739fc4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.518+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/936390095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fcf8f0 0x561739fd36d0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.518+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/936390095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fd0210 0x561739fd0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.518+0000 7f90dbb18b80 1 -- 172.21.15.121:0/936390095 >> 172.21.15.121:0/936390095 conn(0x561739ef4cc0 msgr2=0x561739fd0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.518+0000 7f90dbb18b80 1 -- 172.21.15.121:0/936390095 shutdown_connections 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.518+0000 7f90dbb18b80 1 -- 172.21.15.121:0/936390095 wait complete. 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.519+0000 7f90dbb18b80 1 -- start start 2024-06-10T00:31:45.524 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.519+0000 7f90dbb18b80 1 Processor -- start 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fc44e0 0x561739fdc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fcf8f0 0x561739feb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90dbb18b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 0x561739feb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f90c4146b30 con 0x561739fd0210 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f90c4147fa0 con 0x561739fcf8f0 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90dbb18b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f90c4147200 con 0x561739fc44e0 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90caffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fc44e0 0x561739fdc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90ca7fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fcf8f0 0x561739feb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90cb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 0x561739feb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90caffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fc44e0 0x561739fdc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:53534/0 (socket says 172.21.15.121:53534) 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90caffd700 1 -- 172.21.15.121:0/4030095707 learned_addr learned my addr 172.21.15.121:0/4030095707 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90ca7fc700 1 -- 172.21.15.121:0/4030095707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fc44e0 msgr2=0x561739fdc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:45.525 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90ca7fc700 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fc44e0 0x561739fdc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90ca7fc700 1 -- 172.21.15.121:0/4030095707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 msgr2=0x561739feb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90ca7fc700 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 0x561739feb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90ca7fc700 1 -- 172.21.15.121:0/4030095707 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56173a04d720 con 0x561739fcf8f0 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90caffd700 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fc44e0 0x561739fdc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.520+0000 7f90cb7fe700 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 0x561739feb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.521+0000 7f90ca7fc700 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fcf8f0 0x561739feb220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f90cc139450 tx=0x7f90cc128040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=254f9f54878cf57f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.521+0000 7f90aeff5700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.521+0000 7f90aeff5700 1 -- 172.21.15.121:0/4030095707 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc126e60 con 0x561739fcf8f0 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.521+0000 7f90dbb18b80 10 client.0 _ll_register_callbacks cb 0x561739eb5920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.521+0000 7f90aeff5700 1 -- 172.21.15.121:0/4030095707 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f90cc1282b0 con 0x561739fcf8f0 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.521+0000 7f90dbb18b80 1 -- 172.21.15.121:0/4030095707 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90bc021e10 con 0x561739fcf8f0 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.521+0000 7f90aeff5700 1 -- 172.21.15.121:0/4030095707 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f90cc10c660 con 0x561739fcf8f0 2024-06-10T00:31:45.526 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.521+0000 7f90dbb18b80 10 client.25625 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.521+0000 7f90dbb18b80 1 -- 172.21.15.121:0/4030095707 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({fsmap=28}) -- 0x56173a04e700 con 0x561739fcf8f0 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.522+0000 7f90aeff5700 1 -- 172.21.15.121:0/4030095707 <== mon.2 v2:172.21.15.135:3300/0 4 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f90cc108510 con 0x561739fcf8f0 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.522+0000 7f90dbb18b80 10 client.25625 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.522+0000 7f90dbb18b80 10 client.25625 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.522+0000 7f90dbb18b80 20 client.25625 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.522+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x56173a04ec00 0x56173a051040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.522+0000 7f90dbb18b80 4 client.25625 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.522+0000 7f90dbb18b80 1 -- 172.21.15.121:0/4030095707 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f90bc021e10 con 0x56173a04ec00 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.522+0000 7f90caffd700 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x56173a04ec00 0x56173a051040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.522+0000 7f90caffd700 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x56173a04ec00 0x56173a051040 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c9c7e3739d3f614 server_cookie=79c4af1758756871 in_seq=0 out_seq=0 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.522+0000 7f90aeff5700 10 client.25625 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90aeff5700 1 -- 172.21.15.121:0/4030095707 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f90bc021e10 con 0x56173a04ec00 2024-06-10T00:31:45.527 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90aeff5700 10 client.25625 handle_command_reply: tid=0 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 1 client.25625 shutdown 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 1 -- 172.21.15.121:0/4030095707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fcf8f0 msgr2=0x561739feb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fcf8f0 0x561739feb220 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f90cc139450 tx=0x7f90cc128040 comp rx=0 tx=0).stop 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 1 -- 172.21.15.121:0/4030095707 shutdown_connections 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x56173a04ec00 0x56173a051040 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561739fd0210 0x561739feb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90aeff5700 0 client.25625 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561739fc44e0 0x561739fdc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 1 --2- 172.21.15.121:0/4030095707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561739fcf8f0 0x561739feb220 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 1 -- 172.21.15.121:0/4030095707 >> 172.21.15.121:0/4030095707 conn(0x561739ef4cc0 msgr2=0x561739fc92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 1 -- 172.21.15.121:0/4030095707 shutdown_connections 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 1 -- 172.21.15.121:0/4030095707 wait complete. 2024-06-10T00:31:45.528 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.523+0000 7f90dbb18b80 20 client.25625 trim_cache size 0 max 16384 2024-06-10T00:31:45.529 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:31:45.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 -- 172.21.15.121:0/1276090308 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f90b40a2e50 msgr2=0x7f90b40a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:45.529 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f90b40a2e50 0x7f90b40a5310 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f90c41339e0 tx=0x7f90bc00d540 comp rx=0 tx=0).stop 2024-06-10T00:31:45.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 -- 172.21.15.121:0/1276090308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c40958a0 msgr2=0x7f90c4138c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:45.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c40958a0 0x7f90c4138c20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f90cc065ce0 tx=0x7f90cc06aff0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 -- 172.21.15.121:0/1276090308 shutdown_connections 2024-06-10T00:31:45.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f90b40a2e50 0x7f90b40a5310 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90c409aca0 0x7f90c41328a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f90c4094ee0 0x7f90c41386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 --2- 172.21.15.121:0/1276090308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f90c40958a0 0x7f90c4138c20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:45.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 -- 172.21.15.121:0/1276090308 >> 172.21.15.121:0/1276090308 conn(0x7f90c400a7a0 msgr2=0x7f90c4093ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:45.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 -- 172.21.15.121:0/1276090308 shutdown_connections 2024-06-10T00:31:45.530 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:45.524+0000 7f90d1295700 1 -- 172.21.15.121:0/1276090308 wait complete. 2024-06-10T00:31:46.940 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-10T00:31:47.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:47 smithi121 ceph-mon[32378]: pgmap v117: 129 pgs: 129 active+clean; 463 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 397 op/s 2024-06-10T00:31:47.201 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:31:47.294 INFO:tasks.workunit.client.0.smithi121.stdout: 111 1405681 639 983843 522 423406 1071 2024-06-10T00:31:47.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:47 smithi135 ceph-mon[36424]: pgmap v117: 129 pgs: 129 active+clean; 463 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 397 op/s 2024-06-10T00:31:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:47 smithi193 ceph-mon[36716]: pgmap v117: 129 pgs: 129 active+clean; 463 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 397 op/s 2024-06-10T00:31:49.157 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:49 smithi121 ceph-mon[32378]: pgmap v118: 129 pgs: 129 active+clean; 463 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 397 op/s 2024-06-10T00:31:49.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:49 smithi135 ceph-mon[36424]: pgmap v118: 129 pgs: 129 active+clean; 463 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 397 op/s 2024-06-10T00:31:49.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.618+0000 7f46233ea700 1 -- 172.21.15.121:0/4000075952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c1067d0 msgr2=0x7f461c106c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:49.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.618+0000 7f46233ea700 1 --2- 172.21.15.121:0/4000075952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c1067d0 0x7f461c106c80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4614009990 tx=0x7f4614009ca0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.618+0000 7f46233ea700 1 -- 172.21.15.121:0/4000075952 shutdown_connections 2024-06-10T00:31:49.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.618+0000 7f46233ea700 1 --2- 172.21.15.121:0/4000075952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c1067d0 0x7f461c106c80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.618+0000 7f46233ea700 1 --2- 172.21.15.121:0/4000075952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c1013c0 0x7f461c1017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.618+0000 7f46233ea700 1 --2- 172.21.15.121:0/4000075952 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c101ce0 0x7f461c1061b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.618+0000 7f46233ea700 1 -- 172.21.15.121:0/4000075952 >> 172.21.15.121:0/4000075952 conn(0x7f461c0fce20 msgr2=0x7f461c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:49.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.621+0000 7f46233ea700 1 -- 172.21.15.121:0/4000075952 shutdown_connections 2024-06-10T00:31:49.625 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.625+0000 7f46233ea700 1 -- 172.21.15.121:0/4000075952 wait complete. 2024-06-10T00:31:49.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.626+0000 7f46233ea700 1 Processor -- start 2024-06-10T00:31:49.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.627+0000 7f46233ea700 1 -- start start 2024-06-10T00:31:49.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.627+0000 7f46233ea700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c1013c0 0x7f461c072e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.627+0000 7f46233ea700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c101ce0 0x7f461c0733a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.627+0000 7f46233ea700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c1067d0 0x7f461c10a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.627+0000 7f46233ea700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f461c118dc0 con 0x7f461c101ce0 2024-06-10T00:31:49.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.627+0000 7f46233ea700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f461c118c40 con 0x7f461c1013c0 2024-06-10T00:31:49.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.627+0000 7f46233ea700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f461c118f40 con 0x7f461c1067d0 2024-06-10T00:31:49.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.627+0000 7f4622be9700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c1067d0 0x7f461c10a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.627+0000 7f4622be9700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c1067d0 0x7f461c10a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:46040/0 (socket says 172.21.15.121:46040) 2024-06-10T00:31:49.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.627+0000 7f4622be9700 1 -- 172.21.15.121:0/389734861 learned_addr learned my addr 172.21.15.121:0/389734861 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:49.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.628+0000 7f4621be7700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c101ce0 0x7f461c0733a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.629+0000 7f4621be7700 1 -- 172.21.15.121:0/389734861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c1067d0 msgr2=0x7f461c10a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:49.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.629+0000 7f4621be7700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c1067d0 0x7f461c10a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.629+0000 7f4621be7700 1 -- 172.21.15.121:0/389734861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c1013c0 msgr2=0x7f461c072e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:49.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.629+0000 7f4621be7700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c1013c0 0x7f461c072e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.629+0000 7f4621be7700 1 -- 172.21.15.121:0/389734861 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c10af50 con 0x7f461c101ce0 2024-06-10T00:31:49.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:49 smithi193 ceph-mon[36716]: pgmap v118: 129 pgs: 129 active+clean; 463 MiB data, 950 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 397 op/s 2024-06-10T00:31:49.632 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.629+0000 7f4621be7700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c101ce0 0x7f461c0733a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f461c102140 tx=0x7f460c00b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:49.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.630+0000 7f46137fe700 1 -- 172.21.15.121:0/389734861 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c015020 con 0x7f461c101ce0 2024-06-10T00:31:49.633 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.630+0000 7f46137fe700 1 -- 172.21.15.121:0/389734861 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f460c00be40 con 0x7f461c101ce0 2024-06-10T00:31:49.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.630+0000 7f46137fe700 1 -- 172.21.15.121:0/389734861 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c010d00 con 0x7f461c101ce0 2024-06-10T00:31:49.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.630+0000 7f46233ea700 1 -- 172.21.15.121:0/389734861 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461c10b240 con 0x7f461c101ce0 2024-06-10T00:31:49.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.630+0000 7f46233ea700 1 -- 172.21.15.121:0/389734861 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461c112b20 con 0x7f461c101ce0 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.639+0000 7f462eecfb80 1 Processor -- start 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.640+0000 7f462eecfb80 1 -- start start 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.640+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed01e0 0x556729ed05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.640+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ed51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.640+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed56e0 0x556729ed5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.641+0000 7f462eecfb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556729dc1f60 con 0x556729ed0b00 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.641+0000 7f462eecfb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x556729e20850 con 0x556729ed56e0 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.641+0000 7f462eecfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556729d88460 con 0x556729ed01e0 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.641+0000 7f46223e8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ed51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.641+0000 7f46223e8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ed51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56318/0 (socket says 172.21.15.121:56318) 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.641+0000 7f46223e8700 1 -- 172.21.15.121:0/781159755 learned_addr learned my addr 172.21.15.121:0/781159755 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:49.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.645+0000 7f4622be9700 1 --2- 172.21.15.121:0/781159755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed01e0 0x556729ed05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.647+0000 7f46223e8700 1 -- 172.21.15.121:0/781159755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed01e0 msgr2=0x556729ed05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:49.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.647+0000 7f46223e8700 1 --2- 172.21.15.121:0/781159755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed01e0 0x556729ed05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.647+0000 7f46223e8700 1 -- 172.21.15.121:0/781159755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed56e0 msgr2=0x556729ed5b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-10T00:31:49.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.647+0000 7f46223e8700 1 --2- 172.21.15.121:0/781159755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed56e0 0x556729ed5b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.647+0000 7f46223e8700 1 -- 172.21.15.121:0/781159755 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556729d6f8d0 con 0x556729ed0b00 2024-06-10T00:31:49.648 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.647+0000 7f46223e8700 1 --2- 172.21.15.121:0/781159755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ed51a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4618009cf0 tx=0x7f4618007500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4abbf037c66f95 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:49.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.648+0000 7f4607fff700 1 -- 172.21.15.121:0/781159755 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618007e50 con 0x556729ed0b00 2024-06-10T00:31:49.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.648+0000 7f46137fe700 1 -- 172.21.15.121:0/389734861 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f460c018020 con 0x7f461c101ce0 2024-06-10T00:31:49.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.649+0000 7f4607fff700 1 -- 172.21.15.121:0/781159755 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4618004bb0 con 0x556729ed0b00 2024-06-10T00:31:49.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.649+0000 7f4607fff700 1 -- 172.21.15.121:0/781159755 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618013630 con 0x556729ed0b00 2024-06-10T00:31:49.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.650+0000 7f462eecfb80 1 -- 172.21.15.121:0/781159755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 msgr2=0x556729ed51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:49.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.650+0000 7f462eecfb80 1 --2- 172.21.15.121:0/781159755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ed51a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4618009cf0 tx=0x7f4618007500 comp rx=0 tx=0).stop 2024-06-10T00:31:49.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.654+0000 7f46137fe700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f46080a2e90 0x7f46080a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.654+0000 7f4622be9700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f46080a2e90 0x7f46080a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.654+0000 7f462eecfb80 1 -- 172.21.15.121:0/781159755 shutdown_connections 2024-06-10T00:31:49.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.654+0000 7f462eecfb80 1 --2- 172.21.15.121:0/781159755 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed56e0 0x556729ed5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.654+0000 7f462eecfb80 1 --2- 172.21.15.121:0/781159755 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed01e0 0x556729ed05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.654+0000 7f462eecfb80 1 --2- 172.21.15.121:0/781159755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ed51a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.654+0000 7f462eecfb80 1 -- 172.21.15.121:0/781159755 >> 172.21.15.121:0/781159755 conn(0x556729dfc080 msgr2=0x556729eca830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:49.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.655+0000 7f46137fe700 1 -- 172.21.15.121:0/389734861 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f460c0ced50 con 0x7f461c101ce0 2024-06-10T00:31:49.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.655+0000 7f4622be9700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f46080a2e90 0x7f46080a5350 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f461400b5c0 tx=0x7f46140095d0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:49.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.656+0000 7f462eecfb80 1 -- 172.21.15.121:0/781159755 shutdown_connections 2024-06-10T00:31:49.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.657+0000 7f462eecfb80 1 -- 172.21.15.121:0/781159755 wait complete. 2024-06-10T00:31:49.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.658+0000 7f462eecfb80 1 -- start start 2024-06-10T00:31:49.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.658+0000 7f462eecfb80 1 Processor -- start 2024-06-10T00:31:49.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.659+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed01e0 0x556729edb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.659+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed0b00 0x556729ed8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.659+0000 7f46223e8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed01e0 0x556729edb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.659+0000 7f46223e8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed01e0 0x556729edb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56322/0 (socket says 172.21.15.121:56322) 2024-06-10T00:31:49.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.659+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed56e0 0x556729ed9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.659+0000 7f46223e8700 1 -- 172.21.15.121:0/3450926555 learned_addr learned my addr 172.21.15.121:0/3450926555 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:49.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.659+0000 7f462eecfb80 1 -- 172.21.15.121:0/3450926555 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556729dc0fc0 con 0x556729ed01e0 2024-06-10T00:31:49.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.659+0000 7f462eecfb80 1 -- 172.21.15.121:0/3450926555 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x556729e1c810 con 0x556729ed56e0 2024-06-10T00:31:49.666 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.659+0000 7f462eecfb80 1 -- 172.21.15.121:0/3450926555 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556729edc630 con 0x556729ed0b00 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.659+0000 7f462eecfb80 10 client.0 _ll_register_callbacks cb 0x556729db65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.659+0000 7f4622be9700 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed56e0 0x556729ed9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.660+0000 7f4622be9700 1 -- 172.21.15.121:0/3450926555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed0b00 msgr2=0x556729ed8bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.660+0000 7f4622be9700 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed0b00 0x556729ed8bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.660+0000 7f4622be9700 1 -- 172.21.15.121:0/3450926555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed01e0 msgr2=0x556729edb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.660+0000 7f4622be9700 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed01e0 0x556729edb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.660+0000 7f4622be9700 1 -- 172.21.15.121:0/3450926555 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556729de6750 con 0x556729ed56e0 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.661+0000 7f4622be9700 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed56e0 0x556729ed9110 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f46140098e0 tx=0x7f4614009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd48be02f0454537 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.661+0000 7f4605ffb700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.661+0000 7f4605ffb700 1 -- 172.21.15.121:0/3450926555 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614036060 con 0x556729ed56e0 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.661+0000 7f462eecfb80 1 -- 172.21.15.121:0/3450926555 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4618004bb0 con 0x556729ed56e0 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.661+0000 7f4605ffb700 1 -- 172.21.15.121:0/3450926555 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f461400f670 con 0x556729ed56e0 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.661+0000 7f4605ffb700 1 -- 172.21.15.121:0/3450926555 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f461403d050 con 0x556729ed56e0 2024-06-10T00:31:49.667 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.661+0000 7f462eecfb80 10 client.25628 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.661+0000 7f462eecfb80 1 -- 172.21.15.121:0/3450926555 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556729ee6af0 con 0x556729ed56e0 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.662+0000 7f4605ffb700 1 -- 172.21.15.121:0/3450926555 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4618004bb0 con 0x556729ed56e0 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.662+0000 7f4605ffb700 1 -- 172.21.15.121:0/3450926555 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f46140319c0 con 0x556729ed56e0 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.662+0000 7f462eecfb80 10 client.25628 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.662+0000 7f462eecfb80 10 client.25628 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.662+0000 7f462eecfb80 20 client.25628 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.662+0000 7f462eecfb80 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556729ee9540 0x556729eeb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.663+0000 7f462eecfb80 4 client.25628 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.663+0000 7f462eecfb80 1 -- 172.21.15.121:0/3450926555 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556729e2dd50 con 0x556729ee9540 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.663+0000 7f46223e8700 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556729ee9540 0x556729eeb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.665+0000 7f46223e8700 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556729ee9540 0x556729eeb9a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26518717cbb2bde5 server_cookie=1cf37ef9785cbff8 in_seq=0 out_seq=0 2024-06-10T00:31:49.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.665+0000 7f4605ffb700 10 client.25628 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:31:49.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.668+0000 7f4605ffb700 1 -- 172.21.15.121:0/3450926555 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x556729e2dd50 con 0x556729ee9540 2024-06-10T00:31:49.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.668+0000 7f4605ffb700 10 client.25628 handle_command_reply: tid=0 2024-06-10T00:31:49.669 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.668+0000 7f462eecfb80 1 client.25628 shutdown 2024-06-10T00:31:49.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.672+0000 7f462eecfb80 1 -- 172.21.15.121:0/3450926555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed56e0 msgr2=0x556729ed9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:49.672 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.672+0000 7f462eecfb80 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed56e0 0x556729ed9110 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f46140098e0 tx=0x7f4614009310 comp rx=0 tx=0).stop 2024-06-10T00:31:49.674 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.674+0000 7f462eecfb80 1 -- 172.21.15.121:0/3450926555 shutdown_connections 2024-06-10T00:31:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.674+0000 7f462eecfb80 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556729ee9540 0x556729eeb9a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.674+0000 7f462eecfb80 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed56e0 0x556729ed9110 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.674+0000 7f462eecfb80 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed01e0 0x556729edb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.674+0000 7f462eecfb80 1 --2- 172.21.15.121:0/3450926555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed0b00 0x556729ed8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.674+0000 7f462eecfb80 1 -- 172.21.15.121:0/3450926555 >> 172.21.15.121:0/3450926555 conn(0x556729dfc080 msgr2=0x556729ec47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.675+0000 7f4605ffb700 0 client.25628 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:31:49.675 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.675+0000 7f462eecfb80 1 -- 172.21.15.121:0/3450926555 shutdown_connections 2024-06-10T00:31:49.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.677+0000 7f462eecfb80 1 -- 172.21.15.121:0/3450926555 wait complete. 2024-06-10T00:31:49.678 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.678+0000 7f462eecfb80 20 client.25628 trim_cache size 0 max 16384 2024-06-10T00:31:49.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.716+0000 7f462eecfb80 1 Processor -- start 2024-06-10T00:31:49.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.719+0000 7f462eecfb80 1 -- start start 2024-06-10T00:31:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.719+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed01e0 0x556729eca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.719+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ec48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.719+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed56e0 0x556729ed3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.719+0000 7f462eecfb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1b7410 con 0x556729ed0b00 2024-06-10T00:31:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.719+0000 7f462eecfb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1b4ca0 con 0x556729ed01e0 2024-06-10T00:31:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.719+0000 7f462eecfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1b2af0 con 0x556729ed56e0 2024-06-10T00:31:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.719+0000 7f4621be7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ec48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.719+0000 7f46223e8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed01e0 0x556729eca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.720+0000 7f46223e8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed01e0 0x556729eca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:37640/0 (socket says 172.21.15.121:37640) 2024-06-10T00:31:49.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.720+0000 7f46223e8700 1 -- 172.21.15.121:0/275073806 learned_addr learned my addr 172.21.15.121:0/275073806 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:49.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.720+0000 7f4621be7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ec48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56336/0 (socket says 172.21.15.121:56336) 2024-06-10T00:31:49.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.720+0000 7f4622be9700 1 --2- 172.21.15.121:0/275073806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed56e0 0x556729ed3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.720+0000 7f46223e8700 1 -- 172.21.15.121:0/275073806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed56e0 msgr2=0x556729ed3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.720+0000 7f46223e8700 1 --2- 172.21.15.121:0/275073806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed56e0 0x556729ed3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.720+0000 7f46223e8700 1 -- 172.21.15.121:0/275073806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 msgr2=0x556729ec48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.720+0000 7f46223e8700 1 --2- 172.21.15.121:0/275073806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ec48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.720+0000 7f46223e8700 1 -- 172.21.15.121:0/275073806 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556729ea5430 con 0x556729ed01e0 2024-06-10T00:31:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.721+0000 7f46223e8700 1 --2- 172.21.15.121:0/275073806 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed01e0 0x556729eca950 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4618009c10 tx=0x7f4618000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdb34afa9535f3ed server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.721+0000 7f46077fe700 1 -- 172.21.15.121:0/275073806 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618004bb0 con 0x556729ed01e0 2024-06-10T00:31:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.722+0000 7f46077fe700 1 -- 172.21.15.121:0/275073806 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f461802b070 con 0x556729ed01e0 2024-06-10T00:31:49.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.722+0000 7f46077fe700 1 -- 172.21.15.121:0/275073806 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618026880 con 0x556729ed01e0 2024-06-10T00:31:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.723+0000 7f462eecfb80 1 -- 172.21.15.121:0/275073806 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed01e0 msgr2=0x556729eca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.723+0000 7f462eecfb80 1 --2- 172.21.15.121:0/275073806 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed01e0 0x556729eca950 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4618009c10 tx=0x7f4618000f40 comp rx=0 tx=0).stop 2024-06-10T00:31:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.723+0000 7f462eecfb80 1 -- 172.21.15.121:0/275073806 shutdown_connections 2024-06-10T00:31:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.723+0000 7f462eecfb80 1 --2- 172.21.15.121:0/275073806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed56e0 0x556729ed3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.723+0000 7f462eecfb80 1 --2- 172.21.15.121:0/275073806 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed01e0 0x556729eca950 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.723+0000 7f462eecfb80 1 --2- 172.21.15.121:0/275073806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ec48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.723+0000 7f462eecfb80 1 -- 172.21.15.121:0/275073806 >> 172.21.15.121:0/275073806 conn(0x556729dfc080 msgr2=0x556729eca240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:49.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.728+0000 7f462eecfb80 1 -- 172.21.15.121:0/275073806 shutdown_connections 2024-06-10T00:31:49.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.728+0000 7f462eecfb80 1 -- 172.21.15.121:0/275073806 wait complete. 2024-06-10T00:31:49.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.728+0000 7f462eecfb80 1 -- start start 2024-06-10T00:31:49.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.728+0000 7f462eecfb80 1 Processor -- start 2024-06-10T00:31:49.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.730+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed01e0 0x556729ef18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.730+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ee1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.730+0000 7f462eecfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed56e0 0x556729ee1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.730+0000 7f462eecfb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1b3160 con 0x556729ed0b00 2024-06-10T00:31:49.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.730+0000 7f462eecfb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1b45d0 con 0x556729ed01e0 2024-06-10T00:31:49.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.730+0000 7f462eecfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f461c1b3830 con 0x556729ed56e0 2024-06-10T00:31:49.731 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.730+0000 7f462eecfb80 10 client.0 _ll_register_callbacks cb 0x556729db65b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:49.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.731+0000 7f4621be7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ee1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.731+0000 7f4621be7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ee1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56338/0 (socket says 172.21.15.121:56338) 2024-06-10T00:31:49.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.731+0000 7f4621be7700 1 -- 172.21.15.121:0/593590538 learned_addr learned my addr 172.21.15.121:0/593590538 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:49.733 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.731+0000 7f4622be9700 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed56e0 0x556729ee1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.731+0000 7f4621be7700 1 -- 172.21.15.121:0/593590538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed56e0 msgr2=0x556729ee1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:49.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.731+0000 7f4621be7700 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed56e0 0x556729ee1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.731+0000 7f4621be7700 1 -- 172.21.15.121:0/593590538 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed01e0 msgr2=0x556729ef18b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:31:49.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.731+0000 7f4621be7700 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed01e0 0x556729ef18b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.731+0000 7f4621be7700 1 -- 172.21.15.121:0/593590538 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556729ee2520 con 0x556729ed0b00 2024-06-10T00:31:49.734 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.731+0000 7f4621be7700 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ee1940 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f460c009fd0 tx=0x7f460c09a9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ce962b2690c4f7 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:49.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.736+0000 7f4605ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:31:49.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.736+0000 7f4605ffb700 1 -- 172.21.15.121:0/593590538 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c09b3f0 con 0x556729ed0b00 2024-06-10T00:31:49.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.736+0000 7f4605ffb700 1 -- 172.21.15.121:0/593590538 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f460c0b9bd0 con 0x556729ed0b00 2024-06-10T00:31:49.736 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.736+0000 7f462eecfb80 1 -- 172.21.15.121:0/593590538 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f461802b070 con 0x556729ed0b00 2024-06-10T00:31:49.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.737+0000 7f4605ffb700 1 -- 172.21.15.121:0/593590538 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f460c09af90 con 0x556729ed0b00 2024-06-10T00:31:49.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.737+0000 7f462eecfb80 10 client.16062 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:49.737 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.737+0000 7f462eecfb80 1 -- 172.21.15.121:0/593590538 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556729f532e0 con 0x556729ed0b00 2024-06-10T00:31:49.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.742+0000 7f4605ffb700 1 -- 172.21.15.121:0/593590538 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f461802b070 con 0x556729ed0b00 2024-06-10T00:31:49.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.742+0000 7f4605ffb700 1 -- 172.21.15.121:0/593590538 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f460c09a3e0 con 0x556729ed0b00 2024-06-10T00:31:49.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.743+0000 7f462eecfb80 10 client.16062 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:49.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.743+0000 7f462eecfb80 10 client.16062 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:31:49.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.743+0000 7f462eecfb80 20 client.16062 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:49.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.743+0000 7f462eecfb80 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556729f539a0 0x556729f55dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:49.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.743+0000 7f462eecfb80 4 client.16062 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:31:49.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.743+0000 7f462eecfb80 1 -- 172.21.15.121:0/593590538 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556729e282d0 con 0x556729f539a0 2024-06-10T00:31:49.747 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.743+0000 7f46223e8700 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556729f539a0 0x556729f55dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:49.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.745+0000 7f4605ffb700 10 client.16062 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:31:49.748 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.745+0000 7f46223e8700 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556729f539a0 0x556729f55dc0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=506cee97fd961908 server_cookie=88b1cb1c7e56a37f in_seq=0 out_seq=0 2024-06-10T00:31:49.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.747+0000 7f4605ffb700 1 -- 172.21.15.121:0/593590538 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556729e282d0 con 0x556729f539a0 2024-06-10T00:31:49.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.747+0000 7f4605ffb700 10 client.16062 handle_command_reply: tid=0 2024-06-10T00:31:49.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.750+0000 7f462eecfb80 1 client.16062 shutdown 2024-06-10T00:31:49.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.757+0000 7f462eecfb80 1 -- 172.21.15.121:0/593590538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 msgr2=0x556729ee1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:49.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.757+0000 7f462eecfb80 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ee1940 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f460c009fd0 tx=0x7f460c09a9a0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.762+0000 7f462eecfb80 1 -- 172.21.15.121:0/593590538 shutdown_connections 2024-06-10T00:31:49.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.762+0000 7f462eecfb80 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x556729f539a0 0x556729f55dc0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.762+0000 7f462eecfb80 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556729ed56e0 0x556729ee1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.762+0000 7f462eecfb80 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x556729ed01e0 0x556729ef18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.762+0000 7f462eecfb80 1 --2- 172.21.15.121:0/593590538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556729ed0b00 0x556729ee1940 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.762+0000 7f462eecfb80 1 -- 172.21.15.121:0/593590538 >> 172.21.15.121:0/593590538 conn(0x556729dfc080 msgr2=0x556729ece4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:49.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.762+0000 7f4605ffb700 0 client.16062 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:31:49.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.766+0000 7f462eecfb80 1 -- 172.21.15.121:0/593590538 shutdown_connections 2024-06-10T00:31:49.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.769+0000 7f462eecfb80 1 -- 172.21.15.121:0/593590538 wait complete. 2024-06-10T00:31:49.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.773+0000 7f462eecfb80 20 client.16062 trim_cache size 0 max 16384 2024-06-10T00:31:49.777 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:31:49.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.781+0000 7f46117fa700 1 -- 172.21.15.121:0/389734861 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f46080a2e90 msgr2=0x7f46080a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:49.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.781+0000 7f46117fa700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f46080a2e90 0x7f46080a5350 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f461400b5c0 tx=0x7f46140095d0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.781+0000 7f46117fa700 1 -- 172.21.15.121:0/389734861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c101ce0 msgr2=0x7f461c0733a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:49.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.781+0000 7f46117fa700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c101ce0 0x7f461c0733a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f461c102140 tx=0x7f460c00b6c0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.781+0000 7f46117fa700 1 -- 172.21.15.121:0/389734861 shutdown_connections 2024-06-10T00:31:49.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.781+0000 7f46117fa700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f46080a2e90 0x7f46080a5350 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.781+0000 7f46117fa700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f461c1067d0 0x7f461c10a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.781+0000 7f46117fa700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f461c1013c0 0x7f461c072e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.781+0000 7f46117fa700 1 --2- 172.21.15.121:0/389734861 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461c101ce0 0x7f461c0733a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:49.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.781+0000 7f46117fa700 1 -- 172.21.15.121:0/389734861 >> 172.21.15.121:0/389734861 conn(0x7f461c0fce20 msgr2=0x7f461c0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:49.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.784+0000 7f46117fa700 1 -- 172.21.15.121:0/389734861 shutdown_connections 2024-06-10T00:31:49.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:49.786+0000 7f46117fa700 1 -- 172.21.15.121:0/389734861 wait complete. 2024-06-10T00:31:50.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-10T00:31:51.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:51 smithi121 ceph-mon[32378]: pgmap v119: 129 pgs: 129 active+clean; 493 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 9.1 MiB/s wr, 566 op/s 2024-06-10T00:31:51.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:51 smithi135 ceph-mon[36424]: pgmap v119: 129 pgs: 129 active+clean; 493 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 9.1 MiB/s wr, 566 op/s 2024-06-10T00:31:51.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:51 smithi193 ceph-mon[36716]: pgmap v119: 129 pgs: 129 active+clean; 493 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 9.1 MiB/s wr, 566 op/s 2024-06-10T00:31:52.437 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:31:53.432 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:53 smithi193 ceph-mon[36716]: pgmap v120: 129 pgs: 129 active+clean; 493 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.3 MiB/s wr, 489 op/s 2024-06-10T00:31:53.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:53 smithi121 ceph-mon[32378]: pgmap v120: 129 pgs: 129 active+clean; 493 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.3 MiB/s wr, 489 op/s 2024-06-10T00:31:53.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:53 smithi135 ceph-mon[36424]: pgmap v120: 129 pgs: 129 active+clean; 493 MiB data, 1005 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 8.3 MiB/s wr, 489 op/s 2024-06-10T00:31:54.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.338+0000 7f9b990cf700 1 -- 172.21.15.121:0/4250157643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b94101380 msgr2=0x7f9b94101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:54.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.338+0000 7f9b990cf700 1 --2- 172.21.15.121:0/4250157643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b94101380 0x7f9b94101800 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009990 tx=0x7f9b88009ca0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.338+0000 7f9b990cf700 1 -- 172.21.15.121:0/4250157643 shutdown_connections 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.338+0000 7f9b990cf700 1 --2- 172.21.15.121:0/4250157643 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b94101d40 0x7f9b9410e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.338+0000 7f9b990cf700 1 --2- 172.21.15.121:0/4250157643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b941073c0 0x7f9b941077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.338+0000 7f9b990cf700 1 --2- 172.21.15.121:0/4250157643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b94101380 0x7f9b94101800 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.338+0000 7f9b990cf700 1 -- 172.21.15.121:0/4250157643 >> 172.21.15.121:0/4250157643 conn(0x7f9b94078330 msgr2=0x7f9b94078740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.338+0000 7f9b990cf700 1 -- 172.21.15.121:0/4250157643 shutdown_connections 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.338+0000 7f9b990cf700 1 -- 172.21.15.121:0/4250157643 wait complete. 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.339+0000 7f9b990cf700 1 Processor -- start 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.339+0000 7f9b990cf700 1 -- start start 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b990cf700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b94101380 0x7f9b9419ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b990cf700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b94101d40 0x7f9b9419b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b990cf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b941073c0 0x7f9b9419f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b990cf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9b94111650 con 0x7f9b941073c0 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b990cf700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f9b941114d0 con 0x7f9b94101380 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b990cf700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9b941117d0 con 0x7f9b94101d40 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b94101d40 0x7f9b9419b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b9359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b941073c0 0x7f9b9419f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b92d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b94101380 0x7f9b9419ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b94101d40 0x7f9b9419b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:46096/0 (socket says 172.21.15.121:46096) 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b9359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b941073c0 0x7f9b9419f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56350/0 (socket says 172.21.15.121:56350) 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b9259c700 1 -- 172.21.15.121:0/1996001776 learned_addr learned my addr 172.21.15.121:0/1996001776 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b92d9d700 1 -- 172.21.15.121:0/1996001776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b94101d40 msgr2=0x7f9b9419b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b94101d40 0x7f9b9419b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b92d9d700 1 -- 172.21.15.121:0/1996001776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b941073c0 msgr2=0x7f9b9419f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b941073c0 0x7f9b9419f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b92d9d700 1 -- 172.21.15.121:0/1996001776 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9419fe90 con 0x7f9b94101380 2024-06-10T00:31:54.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b94101380 0x7f9b9419ae80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400e9f0 tx=0x7f9b8400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.340+0000 7f9b7bfff700 1 -- 172.21.15.121:0/1996001776 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b84004220 con 0x7f9b94101380 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.341+0000 7f9b990cf700 1 -- 172.21.15.121:0/1996001776 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b941a0180 con 0x7f9b94101380 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.341+0000 7f9b990cf700 1 -- 172.21.15.121:0/1996001776 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b941a7a00 con 0x7f9b94101380 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.341+0000 7f9b7bfff700 1 -- 172.21.15.121:0/1996001776 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b84004d10 con 0x7f9b94101380 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.341+0000 7f9b7bfff700 1 -- 172.21.15.121:0/1996001776 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8401c070 con 0x7f9b94101380 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.343+0000 7f9b7bfff700 1 -- 172.21.15.121:0/1996001776 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f9b84005020 con 0x7f9b94101380 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.344+0000 7f9b7bfff700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9b7c0a2d90 0x7f9b7c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.344+0000 7f9b9259c700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9b7c0a2d90 0x7f9b7c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.344+0000 7f9b7bfff700 1 -- 172.21.15.121:0/1996001776 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f9b840cf5d0 con 0x7f9b94101380 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.344+0000 7f9b9259c700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9b7c0a2d90 0x7f9b7c0a5250 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9b9410e5d0 tx=0x7f9b88009590 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9ba3952b80 1 Processor -- start 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9ba3952b80 1 -- start start 2024-06-10T00:31:54.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7640 0x561b7afc7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7f60 0x561b7afc19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc1f20 0x561b7afc2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561b7af13810 con 0x561b7afc7f60 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x561b7aebb720 con 0x561b7afc7640 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561b7af0f7d0 con 0x561b7afc1f20 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7f60 0x561b7afc19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7f60 0x561b7afc19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56354/0 (socket says 172.21.15.121:56354) 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9b92d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7640 0x561b7afc7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9b9259c700 1 -- 172.21.15.121:0/10611776 learned_addr learned my addr 172.21.15.121:0/10611776 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9b9359e700 1 --2- 172.21.15.121:0/10611776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc1f20 0x561b7afc2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9b9259c700 1 -- 172.21.15.121:0/10611776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc1f20 msgr2=0x561b7afc2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9b9259c700 1 --2- 172.21.15.121:0/10611776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc1f20 0x561b7afc2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9b9259c700 1 -- 172.21.15.121:0/10611776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7640 msgr2=0x561b7afc7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:54.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9b9259c700 1 --2- 172.21.15.121:0/10611776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7640 0x561b7afc7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9b9259c700 1 -- 172.21.15.121:0/10611776 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b7aeb1f50 con 0x561b7afc7f60 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.345+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/10611776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7640 0x561b7afc7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.346+0000 7f9b9259c700 1 --2- 172.21.15.121:0/10611776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7f60 0x561b7afc19e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800be70 tx=0x7f9b8801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec2c121ad23bf3a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.346+0000 7f9b797fa700 1 -- 172.21.15.121:0/10611776 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8800bea0 con 0x561b7afc7f60 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.346+0000 7f9b797fa700 1 -- 172.21.15.121:0/10611776 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b88003b60 con 0x561b7afc7f60 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.346+0000 7f9b797fa700 1 -- 172.21.15.121:0/10611776 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8800fbc0 con 0x561b7afc7f60 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.347+0000 7f9ba3952b80 1 -- 172.21.15.121:0/10611776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7f60 msgr2=0x561b7afc19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.347+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/10611776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7f60 0x561b7afc19e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800be70 tx=0x7f9b8801f000 comp rx=0 tx=0).stop 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.347+0000 7f9ba3952b80 1 -- 172.21.15.121:0/10611776 shutdown_connections 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.347+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/10611776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc1f20 0x561b7afc2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.347+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/10611776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7640 0x561b7afc7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.347+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/10611776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7f60 0x561b7afc19e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.347+0000 7f9ba3952b80 1 -- 172.21.15.121:0/10611776 >> 172.21.15.121:0/10611776 conn(0x561b7aeeccc0 msgr2=0x561b7afbc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:54.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.347+0000 7f9ba3952b80 1 -- 172.21.15.121:0/10611776 shutdown_connections 2024-06-10T00:31:54.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.347+0000 7f9ba3952b80 1 -- 172.21.15.121:0/10611776 wait complete. 2024-06-10T00:31:54.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.347+0000 7f9ba3952b80 1 -- start start 2024-06-10T00:31:54.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9ba3952b80 1 Processor -- start 2024-06-10T00:31:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 0x561b7afcdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 0x561b7afcafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7f60 0x561b7afcb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561b7afce910 con 0x561b7afc1f20 2024-06-10T00:31:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x561b7aeb3bf0 con 0x561b7afc7f60 2024-06-10T00:31:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561b7afcea90 con 0x561b7afc7640 2024-06-10T00:31:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 0x561b7afcafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 0x561b7afcafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:46110/0 (socket says 172.21.15.121:46110) 2024-06-10T00:31:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9b9259c700 1 -- 172.21.15.121:0/1911066301 learned_addr learned my addr 172.21.15.121:0/1911066301 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:54.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 0x561b7afcdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9b9359e700 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7f60 0x561b7afcb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.348+0000 7f9ba3952b80 10 client.0 _ll_register_callbacks cb 0x561b7aead920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9b9259c700 1 -- 172.21.15.121:0/1911066301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7f60 msgr2=0x561b7afcb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9b9259c700 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7f60 0x561b7afcb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9b9259c700 1 -- 172.21.15.121:0/1911066301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 msgr2=0x561b7afcdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9b9259c700 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 0x561b7afcdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9b9259c700 1 -- 172.21.15.121:0/1911066301 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b7ae628d0 con 0x561b7afc7640 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 0x561b7afcdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9b9259c700 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 0x561b7afcafb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009ce0 tx=0x7f9b88005ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fd37ff1ed9c3440 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9b76ff5700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9b76ff5700 1 -- 172.21.15.121:0/1911066301 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8803c060 con 0x561b7afc7640 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9ba3952b80 1 -- 172.21.15.121:0/1911066301 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b88003b60 con 0x561b7afc7640 2024-06-10T00:31:54.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9b76ff5700 1 -- 172.21.15.121:0/1911066301 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f9b8800f800 con 0x561b7afc7640 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.349+0000 7f9b76ff5700 1 -- 172.21.15.121:0/1911066301 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9b8800bea0 con 0x561b7afc7640 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.350+0000 7f9ba3952b80 10 client.25567 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.350+0000 7f9ba3952b80 1 -- 172.21.15.121:0/1911066301 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561b7afce910 con 0x561b7afc7640 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.350+0000 7f9b76ff5700 1 -- 172.21.15.121:0/1911066301 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b88003b60 con 0x561b7afc7640 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.350+0000 7f9b76ff5700 1 -- 172.21.15.121:0/1911066301 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9b88037c90 con 0x561b7afc7640 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.350+0000 7f9ba3952b80 10 client.25567 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.350+0000 7f9ba3952b80 10 client.25567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.350+0000 7f9ba3952b80 20 client.25567 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.350+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561b7afdb530 0x561b7afdd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.350+0000 7f9ba3952b80 4 client.25567 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.350+0000 7f9ba3952b80 1 -- 172.21.15.121:0/1911066301 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b7af20d10 con 0x561b7afdb530 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.350+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561b7afdb530 0x561b7afdd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.358 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.351+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561b7afdb530 0x561b7afdd920 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ccc587d045b42fa server_cookie=630c6c48fd0255f2 in_seq=0 out_seq=0 2024-06-10T00:31:54.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.351+0000 7f9b76ff5700 10 client.25567 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.357+0000 7f9b76ff5700 1 -- 172.21.15.121:0/1911066301 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x561b7af20d10 con 0x561b7afdb530 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.357+0000 7f9b76ff5700 10 client.25567 handle_command_reply: tid=0 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.357+0000 7f9ba3952b80 1 client.25567 shutdown 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9ba3952b80 1 -- 172.21.15.121:0/1911066301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 msgr2=0x561b7afcafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 0x561b7afcafb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009ce0 tx=0x7f9b88005ee0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9ba3952b80 1 -- 172.21.15.121:0/1911066301 shutdown_connections 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561b7afdb530 0x561b7afdd920 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7f60 0x561b7afcb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 0x561b7afcdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/1911066301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 0x561b7afcafb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9ba3952b80 1 -- 172.21.15.121:0/1911066301 >> 172.21.15.121:0/1911066301 conn(0x561b7aeeccc0 msgr2=0x561b7afbc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:54.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9b76ff5700 0 client.25567 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:54.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9ba3952b80 1 -- 172.21.15.121:0/1911066301 shutdown_connections 2024-06-10T00:31:54.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9ba3952b80 1 -- 172.21.15.121:0/1911066301 wait complete. 2024-06-10T00:31:54.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.358+0000 7f9ba3952b80 20 client.25567 trim_cache size 0 max 16384 2024-06-10T00:31:54.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.374+0000 7f9ba3952b80 1 Processor -- start 2024-06-10T00:31:54.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.374+0000 7f9ba3952b80 1 -- start start 2024-06-10T00:31:54.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.374+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 0x561b7afc19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.374+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 0x561b7afcaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.374+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7f60 0x561b7afcb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.374+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9b800115f0 con 0x561b7afc1f20 2024-06-10T00:31:54.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.374+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8000ee80 con 0x561b7afc7f60 2024-06-10T00:31:54.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.374+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8000ccd0 con 0x561b7afc7640 2024-06-10T00:31:54.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b92d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 0x561b7afc19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b92d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 0x561b7afc19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56384/0 (socket says 172.21.15.121:56384) 2024-06-10T00:31:54.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b92d9d700 1 -- 172.21.15.121:0/69488282 learned_addr learned my addr 172.21.15.121:0/69488282 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:54.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b9259c700 1 --2- 172.21.15.121:0/69488282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 0x561b7afcaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b9359e700 1 --2- 172.21.15.121:0/69488282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7f60 0x561b7afcb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b92d9d700 1 -- 172.21.15.121:0/69488282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 msgr2=0x561b7afcaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:54.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/69488282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 0x561b7afcaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b92d9d700 1 -- 172.21.15.121:0/69488282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7f60 msgr2=0x561b7afcb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:54.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/69488282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7f60 0x561b7afcb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b92d9d700 1 -- 172.21.15.121:0/69488282 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b7afbcd60 con 0x561b7afc1f20 2024-06-10T00:31:54.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b9359e700 1 --2- 172.21.15.121:0/69488282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7f60 0x561b7afcb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:31:54.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/69488282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 0x561b7afc19e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400cab0 tx=0x7f9b8409e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac5c386903b14bd server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:54.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b78ff9700 1 -- 172.21.15.121:0/69488282 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8409fb10 con 0x561b7afc1f20 2024-06-10T00:31:54.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b78ff9700 1 -- 172.21.15.121:0/69488282 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b840a43d0 con 0x561b7afc1f20 2024-06-10T00:31:54.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.375+0000 7f9b78ff9700 1 -- 172.21.15.121:0/69488282 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b840a3f40 con 0x561b7afc1f20 2024-06-10T00:31:54.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.376+0000 7f9ba3952b80 1 -- 172.21.15.121:0/69488282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 msgr2=0x561b7afc19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:54.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.376+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/69488282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 0x561b7afc19e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400cab0 tx=0x7f9b8409e040 comp rx=0 tx=0).stop 2024-06-10T00:31:54.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.376+0000 7f9ba3952b80 1 -- 172.21.15.121:0/69488282 shutdown_connections 2024-06-10T00:31:54.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.376+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/69488282 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc7f60 0x561b7afcb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.376+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/69488282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc1f20 0x561b7afc19e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.376+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/69488282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7640 0x561b7afcaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.376+0000 7f9ba3952b80 1 -- 172.21.15.121:0/69488282 >> 172.21.15.121:0/69488282 conn(0x561b7aeeccc0 msgr2=0x561b7afc98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:54.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.376+0000 7f9ba3952b80 1 -- 172.21.15.121:0/69488282 shutdown_connections 2024-06-10T00:31:54.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.376+0000 7f9ba3952b80 1 -- 172.21.15.121:0/69488282 wait complete. 2024-06-10T00:31:54.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9ba3952b80 1 -- start start 2024-06-10T00:31:54.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9ba3952b80 1 Processor -- start 2024-06-10T00:31:54.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc1f20 0x561b7afd3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7640 0x561b7afd44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9ba3952b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7f60 0x561b7afe2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8000d340 con 0x561b7afc7640 2024-06-10T00:31:54.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8000e7b0 con 0x561b7afc1f20 2024-06-10T00:31:54.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9ba3952b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8000da10 con 0x561b7afc7f60 2024-06-10T00:31:54.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9ba3952b80 10 client.0 _ll_register_callbacks cb 0x561b7aead920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:54.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7640 0x561b7afd44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9b92d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc1f20 0x561b7afd3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9b9259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7640 0x561b7afd44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56398/0 (socket says 172.21.15.121:56398) 2024-06-10T00:31:54.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9b9259c700 1 -- 172.21.15.121:0/2428621090 learned_addr learned my addr 172.21.15.121:0/2428621090 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:54.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.377+0000 7f9b9359e700 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7f60 0x561b7afe2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9b9259c700 1 -- 172.21.15.121:0/2428621090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7f60 msgr2=0x561b7afe2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:54.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9b9259c700 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7f60 0x561b7afe2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9b9259c700 1 -- 172.21.15.121:0/2428621090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc1f20 msgr2=0x561b7afd3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:54.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9b9259c700 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc1f20 0x561b7afd3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9b9259c700 1 -- 172.21.15.121:0/2428621090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b7afe2e70 con 0x561b7afc7640 2024-06-10T00:31:54.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc1f20 0x561b7afd3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:31:54.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9b9259c700 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7640 0x561b7afd44d0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9b88005f50 tx=0x7f9b8800f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a61332bbd51117 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:54.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9b76ff5700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:31:54.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9b76ff5700 1 -- 172.21.15.121:0/2428621090 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88026070 con 0x561b7afc7640 2024-06-10T00:31:54.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9b76ff5700 1 -- 172.21.15.121:0/2428621090 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f9b8800fd50 con 0x561b7afc7640 2024-06-10T00:31:54.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9ba3952b80 1 -- 172.21.15.121:0/2428621090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b840a43d0 con 0x561b7afc7640 2024-06-10T00:31:54.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9b76ff5700 1 -- 172.21.15.121:0/2428621090 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9b8803c040 con 0x561b7afc7640 2024-06-10T00:31:54.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9ba3952b80 10 client.16080 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:54.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.378+0000 7f9ba3952b80 1 -- 172.21.15.121:0/2428621090 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561b7b045ac0 con 0x561b7afc7640 2024-06-10T00:31:54.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.379+0000 7f9b76ff5700 1 -- 172.21.15.121:0/2428621090 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b840a43d0 con 0x561b7afc7640 2024-06-10T00:31:54.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.379+0000 7f9b76ff5700 1 -- 172.21.15.121:0/2428621090 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9b88037630 con 0x561b7afc7640 2024-06-10T00:31:54.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.379+0000 7f9ba3952b80 10 client.16080 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:54.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.379+0000 7f9ba3952b80 10 client.16080 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:31:54.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.379+0000 7f9ba3952b80 20 client.16080 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:54.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.379+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561b7b045f70 0x561b7b048390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:54.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.379+0000 7f9ba3952b80 4 client.16080 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:31:54.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.379+0000 7f9ba3952b80 1 -- 172.21.15.121:0/2428621090 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b7af1b290 con 0x561b7b045f70 2024-06-10T00:31:54.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.379+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561b7b045f70 0x561b7b048390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:54.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.379+0000 7f9b76ff5700 10 client.16080 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:54.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.379+0000 7f9b92d9d700 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561b7b045f70 0x561b7b048390 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3a81d1bf5818a29 server_cookie=931a78b7916f2dfa in_seq=0 out_seq=0 2024-06-10T00:31:54.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9b76ff5700 1 -- 172.21.15.121:0/2428621090 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b7af1b290 con 0x561b7b045f70 2024-06-10T00:31:54.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9b76ff5700 10 client.16080 handle_command_reply: tid=0 2024-06-10T00:31:54.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 1 client.16080 shutdown 2024-06-10T00:31:54.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 1 -- 172.21.15.121:0/2428621090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7640 msgr2=0x561b7afd44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:54.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7640 0x561b7afd44d0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9b88005f50 tx=0x7f9b8800f8a0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 1 -- 172.21.15.121:0/2428621090 shutdown_connections 2024-06-10T00:31:54.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561b7b045f70 0x561b7b048390 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561b7afc7f60 0x561b7afe2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561b7afc1f20 0x561b7afd3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 1 --2- 172.21.15.121:0/2428621090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b7afc7640 0x561b7afd44d0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 1 -- 172.21.15.121:0/2428621090 >> 172.21.15.121:0/2428621090 conn(0x561b7aeeccc0 msgr2=0x561b7afc9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:54.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9b76ff5700 0 client.16080 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:54.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 1 -- 172.21.15.121:0/2428621090 shutdown_connections 2024-06-10T00:31:54.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 1 -- 172.21.15.121:0/2428621090 wait complete. 2024-06-10T00:31:54.389 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.383+0000 7f9ba3952b80 20 client.16080 trim_cache size 0 max 16384 2024-06-10T00:31:54.389 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:31:54.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 -- 172.21.15.121:0/1996001776 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9b7c0a2d90 msgr2=0x7f9b7c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:54.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9b7c0a2d90 0x7f9b7c0a5250 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9b9410e5d0 tx=0x7f9b88009590 comp rx=0 tx=0).stop 2024-06-10T00:31:54.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 -- 172.21.15.121:0/1996001776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b94101380 msgr2=0x7f9b9419ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:54.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b94101380 0x7f9b9419ae80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9b8400e9f0 tx=0x7f9b8400ed00 comp rx=0 tx=0).stop 2024-06-10T00:31:54.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 -- 172.21.15.121:0/1996001776 shutdown_connections 2024-06-10T00:31:54.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f9b7c0a2d90 0x7f9b7c0a5250 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b941073c0 0x7f9b9419f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b94101380 0x7f9b9419ae80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 --2- 172.21.15.121:0/1996001776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b94101d40 0x7f9b9419b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:54.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 -- 172.21.15.121:0/1996001776 >> 172.21.15.121:0/1996001776 conn(0x7f9b94078330 msgr2=0x7f9b9406aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:54.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 -- 172.21.15.121:0/1996001776 shutdown_connections 2024-06-10T00:31:54.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:54.384+0000 7f9b990cf700 1 -- 172.21.15.121:0/1996001776 wait complete. 2024-06-10T00:31:55.165 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:55 smithi121 ceph-mon[32378]: pgmap v121: 129 pgs: 129 active+clean; 508 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.8 MiB/s wr, 546 op/s 2024-06-10T00:31:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:55 smithi135 ceph-mon[36424]: pgmap v121: 129 pgs: 129 active+clean; 508 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.8 MiB/s wr, 546 op/s 2024-06-10T00:31:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:55 smithi193 ceph-mon[36716]: pgmap v121: 129 pgs: 129 active+clean; 508 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 9.8 MiB/s wr, 546 op/s 2024-06-10T00:31:56.456 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-10T00:31:56.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-10T00:31:57.061 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:31:57.294 INFO:tasks.workunit.client.0.smithi121.stdout: 140 1988889 1557 1372946 1533 646422 1878 2024-06-10T00:31:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:57 smithi121 ceph-mon[32378]: pgmap v122: 129 pgs: 129 active+clean; 508 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 225 op/s 2024-06-10T00:31:57.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:57 smithi135 ceph-mon[36424]: pgmap v122: 129 pgs: 129 active+clean; 508 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 225 op/s 2024-06-10T00:31:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:57 smithi193 ceph-mon[36716]: pgmap v122: 129 pgs: 129 active+clean; 508 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 225 op/s 2024-06-10T00:31:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.071+0000 7fcfa80a4700 1 -- 172.21.15.121:0/2920481954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa01099d0 msgr2=0x7fcfa0109e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.071+0000 7fcfa80a4700 1 --2- 172.21.15.121:0/2920481954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa01099d0 0x7fcfa0109e50 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009990 tx=0x7fcf94009ca0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.071+0000 7fcfa80a4700 1 -- 172.21.15.121:0/2920481954 shutdown_connections 2024-06-10T00:31:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.071+0000 7fcfa80a4700 1 --2- 172.21.15.121:0/2920481954 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfa011bc70 0x7fcfa011e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.071+0000 7fcfa80a4700 1 --2- 172.21.15.121:0/2920481954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa0110c70 0x7fcfa0109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.071+0000 7fcfa80a4700 1 --2- 172.21.15.121:0/2920481954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa01099d0 0x7fcfa0109e50 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.072+0000 7fcfa80a4700 1 -- 172.21.15.121:0/2920481954 >> 172.21.15.121:0/2920481954 conn(0x7fcfa006d600 msgr2=0x7fcfa006da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.072+0000 7fcfa80a4700 1 -- 172.21.15.121:0/2920481954 shutdown_connections 2024-06-10T00:31:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.072+0000 7fcfa80a4700 1 -- 172.21.15.121:0/2920481954 wait complete. 2024-06-10T00:31:59.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.073+0000 7fcfa80a4700 1 Processor -- start 2024-06-10T00:31:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.073+0000 7fcfa80a4700 1 -- start start 2024-06-10T00:31:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.073+0000 7fcfa80a4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa01099d0 0x7fcfa011a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.073+0000 7fcfa80a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa0110c70 0x7fcfa011a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.073+0000 7fcfa80a4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfa011bc70 0x7fcfa0113300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.073+0000 7fcfa80a4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa0121750 con 0x7fcfa0110c70 2024-06-10T00:31:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.073+0000 7fcfa80a4700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa01215d0 con 0x7fcfa011bc70 2024-06-10T00:31:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.073+0000 7fcfa80a4700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa01218d0 con 0x7fcfa01099d0 2024-06-10T00:31:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.074+0000 7fcfa6641700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfa011bc70 0x7fcfa0113300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.074+0000 7fcfa6641700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfa011bc70 0x7fcfa0113300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:54278/0 (socket says 172.21.15.121:54278) 2024-06-10T00:31:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.074+0000 7fcfa6641700 1 -- 172.21.15.121:0/608421373 learned_addr learned my addr 172.21.15.121:0/608421373 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.074+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa01099d0 0x7fcfa011a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.075+0000 7fcfa6641700 1 -- 172.21.15.121:0/608421373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa01099d0 msgr2=0x7fcfa011a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.075+0000 7fcfa6641700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa01099d0 0x7fcfa011a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.075+0000 7fcfa6641700 1 -- 172.21.15.121:0/608421373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa0110c70 msgr2=0x7fcfa011a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.075+0000 7fcfa6641700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa0110c70 0x7fcfa011a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.075+0000 7fcfa6641700 1 -- 172.21.15.121:0/608421373 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa0113c00 con 0x7fcfa011bc70 2024-06-10T00:31:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.075+0000 7fcfa6641700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfa011bc70 0x7fcfa0113300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcf980094f0 tx=0x7fcf98007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:59.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.076+0000 7fcf92ffd700 1 -- 172.21.15.121:0/608421373 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98004db0 con 0x7fcfa011bc70 2024-06-10T00:31:59.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.076+0000 7fcfa80a4700 1 -- 172.21.15.121:0/608421373 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa0113ef0 con 0x7fcfa011bc70 2024-06-10T00:31:59.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.076+0000 7fcfa80a4700 1 -- 172.21.15.121:0/608421373 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa0117ee0 con 0x7fcfa011bc70 2024-06-10T00:31:59.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.079+0000 7fcf92ffd700 1 -- 172.21.15.121:0/608421373 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf9800db60 con 0x7fcfa011bc70 2024-06-10T00:31:59.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.079+0000 7fcf92ffd700 1 -- 172.21.15.121:0/608421373 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98015660 con 0x7fcfa011bc70 2024-06-10T00:31:59.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.082+0000 7fcf92ffd700 1 -- 172.21.15.121:0/608421373 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fcf980158c0 con 0x7fcfa011bc70 2024-06-10T00:31:59.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.084+0000 7fcf92ffd700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf8c0a2ed0 0x7fcf8c0a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.085+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf8c0a2ed0 0x7fcf8c0a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.085+0000 7fcf92ffd700 1 -- 172.21.15.121:0/608421373 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcf980d0350 con 0x7fcfa011bc70 2024-06-10T00:31:59.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.086+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf8c0a2ed0 0x7fcf8c0a5390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c006fd0 tx=0x7fcf9c008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:59.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.087+0000 7fcfb2927b80 1 Processor -- start 2024-06-10T00:31:59.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.087+0000 7fcfb2927b80 1 -- start start 2024-06-10T00:31:59.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.088+0000 7fcfb2927b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 0x5580119ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.089+0000 7fcfa5e40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 0x5580119ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.089+0000 7fcfa5e40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 0x5580119ca3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46030/0 (socket says 172.21.15.121:46030) 2024-06-10T00:31:59.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.089+0000 7fcfa5e40700 1 -- 172.21.15.121:0/61674493 learned_addr learned my addr 172.21.15.121:0/61674493 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:59.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.089+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/61674493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119ca900 0x5580119c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.089+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/61674493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119c4820 0x5580119c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.089+0000 7fcfb2927b80 1 -- 172.21.15.121:0/61674493 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5580118b6fe0 con 0x5580119c9fe0 2024-06-10T00:31:59.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.089+0000 7fcfb2927b80 1 -- 172.21.15.121:0/61674493 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5580119158d0 con 0x5580119ca900 2024-06-10T00:31:59.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.089+0000 7fcfb2927b80 1 -- 172.21.15.121:0/61674493 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55801187d4e0 con 0x5580119c4820 2024-06-10T00:31:59.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.090+0000 7fcfa5e40700 1 -- 172.21.15.121:0/61674493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119c4820 msgr2=0x5580119c4c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:31:59.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.090+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/61674493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119c4820 0x5580119c4c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.090+0000 7fcfa5e40700 1 -- 172.21.15.121:0/61674493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119ca900 msgr2=0x5580119c42e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:31:59.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.090+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/61674493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119ca900 0x5580119c42e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.090+0000 7fcfa5e40700 1 -- 172.21.15.121:0/61674493 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558011864950 con 0x5580119c9fe0 2024-06-10T00:31:59.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.090+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/61674493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 0x5580119ca3c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c013bd0 tx=0x7fcf9c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b618ca57558e8a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:59.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.092+0000 7fcf8bfff700 1 -- 172.21.15.121:0/61674493 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c019040 con 0x5580119c9fe0 2024-06-10T00:31:59.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.092+0000 7fcf8bfff700 1 -- 172.21.15.121:0/61674493 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf9c022070 con 0x5580119c9fe0 2024-06-10T00:31:59.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.092+0000 7fcf8bfff700 1 -- 172.21.15.121:0/61674493 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c00c3a0 con 0x5580119c9fe0 2024-06-10T00:31:59.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.093+0000 7fcfb2927b80 1 -- 172.21.15.121:0/61674493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 msgr2=0x5580119ca3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:59.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.093+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/61674493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 0x5580119ca3c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c013bd0 tx=0x7fcf9c00f000 comp rx=0 tx=0).stop 2024-06-10T00:31:59.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.094+0000 7fcfb2927b80 1 -- 172.21.15.121:0/61674493 shutdown_connections 2024-06-10T00:31:59.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.094+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/61674493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119c4820 0x5580119c4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.094+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/61674493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 0x5580119ca3c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.094+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/61674493 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119ca900 0x5580119c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.094+0000 7fcfb2927b80 1 -- 172.21.15.121:0/61674493 >> 172.21.15.121:0/61674493 conn(0x5580118f1100 msgr2=0x5580119bf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:59.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.094+0000 7fcfb2927b80 1 -- 172.21.15.121:0/61674493 shutdown_connections 2024-06-10T00:31:59.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.094+0000 7fcfb2927b80 1 -- 172.21.15.121:0/61674493 wait complete. 2024-06-10T00:31:59.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.095+0000 7fcfb2927b80 1 -- start start 2024-06-10T00:31:59.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.096+0000 7fcfb2927b80 1 Processor -- start 2024-06-10T00:31:59.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.097+0000 7fcfb2927b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c4820 0x5580119d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.097+0000 7fcfa5e40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c4820 0x5580119d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.097+0000 7fcfa5e40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c4820 0x5580119d0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46034/0 (socket says 172.21.15.121:46034) 2024-06-10T00:31:59.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.097+0000 7fcfa5e40700 1 -- 172.21.15.121:0/3480745412 learned_addr learned my addr 172.21.15.121:0/3480745412 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:59.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.097+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c9fe0 0x5580119cd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.098+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119ca900 0x5580119cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.098+0000 7fcfb2927b80 1 -- 172.21.15.121:0/3480745412 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5580118b6040 con 0x5580119c4820 2024-06-10T00:31:59.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.098+0000 7fcfa563f700 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c9fe0 0x5580119cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.098+0000 7fcfb2927b80 1 -- 172.21.15.121:0/3480745412 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x558011911890 con 0x5580119c9fe0 2024-06-10T00:31:59.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.098+0000 7fcfa6641700 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119ca900 0x5580119cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.098+0000 7fcfb2927b80 1 -- 172.21.15.121:0/3480745412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5580119d1330 con 0x5580119ca900 2024-06-10T00:31:59.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.098+0000 7fcfb2927b80 10 client.0 _ll_register_callbacks cb 0x5580118ab630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.099+0000 7fcfa5e40700 1 -- 172.21.15.121:0/3480745412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119ca900 msgr2=0x5580119cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.099+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119ca900 0x5580119cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.099+0000 7fcfa5e40700 1 -- 172.21.15.121:0/3480745412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c9fe0 msgr2=0x5580119cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.099+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c9fe0 0x5580119cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.099+0000 7fcfa5e40700 1 -- 172.21.15.121:0/3480745412 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580118db7d0 con 0x5580119c4820 2024-06-10T00:31:59.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.100+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c4820 0x5580119d0650 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c01d9d0 tx=0x7fcf9c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0967fc42ee590b1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:59.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.100+0000 7fcf897fa700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:31:59.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.101+0000 7fcf897fa700 1 -- 172.21.15.121:0/3480745412 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c01d5f0 con 0x5580119c4820 2024-06-10T00:31:59.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.101+0000 7fcfb2927b80 1 -- 172.21.15.121:0/3480745412 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf9c022070 con 0x5580119c4820 2024-06-10T00:31:59.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.101+0000 7fcf897fa700 1 -- 172.21.15.121:0/3480745412 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcf9c011860 con 0x5580119c4820 2024-06-10T00:31:59.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.102+0000 7fcf897fa700 1 -- 172.21.15.121:0/3480745412 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcf9c03a050 con 0x5580119c4820 2024-06-10T00:31:59.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.103+0000 7fcfb2927b80 10 client.16089 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:59.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.103+0000 7fcfb2927b80 1 -- 172.21.15.121:0/3480745412 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5580119db930 con 0x5580119c4820 2024-06-10T00:31:59.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.104+0000 7fcf897fa700 1 -- 172.21.15.121:0/3480745412 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf9c022070 con 0x5580119c4820 2024-06-10T00:31:59.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.104+0000 7fcf897fa700 1 -- 172.21.15.121:0/3480745412 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcf9c01c030 con 0x5580119c4820 2024-06-10T00:31:59.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.104+0000 7fcfb2927b80 10 client.16089 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:59.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.104+0000 7fcfb2927b80 10 client.16089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:31:59.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.104+0000 7fcfb2927b80 20 client.16089 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:59.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.104+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5580119dde60 0x5580119e0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.104+0000 7fcfb2927b80 4 client.16089 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:31:59.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.104+0000 7fcfb2927b80 1 -- 172.21.15.121:0/3480745412 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558011922dd0 con 0x5580119dde60 2024-06-10T00:31:59.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.105+0000 7fcfa563f700 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5580119dde60 0x5580119e0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.107+0000 7fcfa563f700 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5580119dde60 0x5580119e0250 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8731024a2858dd9b server_cookie=d761eb86ddd79771 in_seq=0 out_seq=0 2024-06-10T00:31:59.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.107+0000 7fcf897fa700 10 client.16089 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:59.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.109+0000 7fcf897fa700 1 -- 172.21.15.121:0/3480745412 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x558011922dd0 con 0x5580119dde60 2024-06-10T00:31:59.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.109+0000 7fcf897fa700 10 client.16089 handle_command_reply: tid=0 2024-06-10T00:31:59.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.110+0000 7fcfb2927b80 1 client.16089 shutdown 2024-06-10T00:31:59.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.111+0000 7fcfb2927b80 1 -- 172.21.15.121:0/3480745412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c4820 msgr2=0x5580119d0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:59.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.111+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c4820 0x5580119d0650 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c01d9d0 tx=0x7fcf9c00a620 comp rx=0 tx=0).stop 2024-06-10T00:31:59.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.111+0000 7fcfb2927b80 1 -- 172.21.15.121:0/3480745412 shutdown_connections 2024-06-10T00:31:59.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.111+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5580119dde60 0x5580119e0250 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.111+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119ca900 0x5580119cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.111+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c4820 0x5580119d0650 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.111+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/3480745412 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c9fe0 0x5580119cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.111+0000 7fcfb2927b80 1 -- 172.21.15.121:0/3480745412 >> 172.21.15.121:0/3480745412 conn(0x5580118f1100 msgr2=0x5580119be780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:59.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.111+0000 7fcf897fa700 0 client.16089 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:59.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.111+0000 7fcfb2927b80 1 -- 172.21.15.121:0/3480745412 shutdown_connections 2024-06-10T00:31:59.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.112+0000 7fcfb2927b80 1 -- 172.21.15.121:0/3480745412 wait complete. 2024-06-10T00:31:59.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.112+0000 7fcfb2927b80 20 client.16089 trim_cache size 0 max 16384 2024-06-10T00:31:59.151 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.150+0000 7fcfb2927b80 1 Processor -- start 2024-06-10T00:31:59.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.151+0000 7fcfb2927b80 1 -- start start 2024-06-10T00:31:59.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.152+0000 7fcfb2927b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 0x5580119c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.152+0000 7fcfb2927b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119c9fe0 0x5580119cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.153 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.153+0000 7fcfb2927b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119ca900 0x5580119cb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.156 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.153+0000 7fcfa6641700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119ca900 0x5580119cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.154+0000 7fcfa563f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119c9fe0 0x5580119cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.155+0000 7fcfa5e40700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 0x5580119c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.156+0000 7fcfa6641700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119ca900 0x5580119cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46044/0 (socket says 172.21.15.121:46044) 2024-06-10T00:31:59.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.156+0000 7fcfa6641700 1 -- 172.21.15.121:0/1053818308 learned_addr learned my addr 172.21.15.121:0/1053818308 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:59.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.156+0000 7fcfb2927b80 1 -- 172.21.15.121:0/1053818308 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcf84011790 con 0x5580119ca900 2024-06-10T00:31:59.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.156+0000 7fcfb2927b80 1 -- 172.21.15.121:0/1053818308 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8400f020 con 0x5580119c4820 2024-06-10T00:31:59.157 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.156+0000 7fcfb2927b80 1 -- 172.21.15.121:0/1053818308 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8400ce70 con 0x5580119c9fe0 2024-06-10T00:31:59.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.158+0000 7fcfa6641700 1 -- 172.21.15.121:0/1053818308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119c9fe0 msgr2=0x5580119cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:59.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.158+0000 7fcfa6641700 1 --2- 172.21.15.121:0/1053818308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119c9fe0 0x5580119cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.158+0000 7fcfa6641700 1 -- 172.21.15.121:0/1053818308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 msgr2=0x5580119c42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:59.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.158+0000 7fcfa6641700 1 --2- 172.21.15.121:0/1053818308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 0x5580119c42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.159+0000 7fcfa6641700 1 -- 172.21.15.121:0/1053818308 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580119c9aa0 con 0x5580119ca900 2024-06-10T00:31:59.160 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.159+0000 7fcfa6641700 1 --2- 172.21.15.121:0/1053818308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119ca900 0x5580119cb430 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcf980cd710 tx=0x7fcf9800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fab31a561e4be89 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:59.161 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.162+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/1053818308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 0x5580119c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:31:59.162 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.162+0000 7fcf8b7fe700 1 -- 172.21.15.121:0/1053818308 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf980d0150 con 0x5580119ca900 2024-06-10T00:31:59.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.163+0000 7fcf8b7fe700 1 -- 172.21.15.121:0/1053818308 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf980a0830 con 0x5580119ca900 2024-06-10T00:31:59.163 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.163+0000 7fcf8b7fe700 1 -- 172.21.15.121:0/1053818308 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf980a4c60 con 0x5580119ca900 2024-06-10T00:31:59.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.164+0000 7fcfb2927b80 1 -- 172.21.15.121:0/1053818308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119ca900 msgr2=0x5580119cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:59.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.164+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/1053818308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119ca900 0x5580119cb430 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcf980cd710 tx=0x7fcf9800b040 comp rx=0 tx=0).stop 2024-06-10T00:31:59.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.165+0000 7fcfb2927b80 1 -- 172.21.15.121:0/1053818308 shutdown_connections 2024-06-10T00:31:59.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.165+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/1053818308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119ca900 0x5580119cb430 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.165+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/1053818308 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 0x5580119c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.165+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/1053818308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119c9fe0 0x5580119cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.165+0000 7fcfb2927b80 1 -- 172.21.15.121:0/1053818308 >> 172.21.15.121:0/1053818308 conn(0x5580118f1100 msgr2=0x5580119cfa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:59.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.167+0000 7fcfb2927b80 1 -- 172.21.15.121:0/1053818308 shutdown_connections 2024-06-10T00:31:59.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.167+0000 7fcfb2927b80 1 -- 172.21.15.121:0/1053818308 wait complete. 2024-06-10T00:31:59.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.168+0000 7fcfb2927b80 1 -- start start 2024-06-10T00:31:59.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.168+0000 7fcfb2927b80 1 Processor -- start 2024-06-10T00:31:59.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.169+0000 7fcfb2927b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 0x5580119d6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.169+0000 7fcfb2927b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 0x5580119d6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfa5e40700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 0x5580119d6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfa5e40700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 0x5580119d6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:54320/0 (socket says 172.21.15.121:54320) 2024-06-10T00:31:59.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfa5e40700 1 -- 172.21.15.121:0/753588236 learned_addr learned my addr 172.21.15.121:0/753588236 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:31:59.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119ca900 0x5580119d3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfb2927b80 1 -- 172.21.15.121:0/753588236 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8400d4e0 con 0x5580119c9fe0 2024-06-10T00:31:59.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfb2927b80 1 -- 172.21.15.121:0/753588236 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8400e950 con 0x5580119c4820 2024-06-10T00:31:59.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfb2927b80 1 -- 172.21.15.121:0/753588236 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8400dbb0 con 0x5580119ca900 2024-06-10T00:31:59.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfa563f700 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 0x5580119d6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfa6641700 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119ca900 0x5580119d3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfb2927b80 10 client.0 _ll_register_callbacks cb 0x5580118ab630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:31:59.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfa5e40700 1 -- 172.21.15.121:0/753588236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119ca900 msgr2=0x5580119d3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:59.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119ca900 0x5580119d3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfa5e40700 1 -- 172.21.15.121:0/753588236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 msgr2=0x5580119d6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:31:59.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 0x5580119d6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.170+0000 7fcfa5e40700 1 -- 172.21.15.121:0/753588236 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580119d3d30 con 0x5580119c4820 2024-06-10T00:31:59.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.171+0000 7fcfa563f700 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 0x5580119d6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:31:59.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.171+0000 7fcfa5e40700 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 0x5580119d6880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c008010 tx=0x7fcf9c027630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83463b33629a87f0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:31:59.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.171+0000 7fcf897fa700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:31:59.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.171+0000 7fcf897fa700 1 -- 172.21.15.121:0/753588236 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c01d5f0 con 0x5580119c4820 2024-06-10T00:31:59.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.171+0000 7fcf897fa700 1 -- 172.21.15.121:0/753588236 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcf9c0313e0 con 0x5580119c4820 2024-06-10T00:31:59.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.172+0000 7fcfb2927b80 1 -- 172.21.15.121:0/753588236 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf980a0830 con 0x5580119c4820 2024-06-10T00:31:59.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.176+0000 7fcf897fa700 1 -- 172.21.15.121:0/753588236 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcf9c04b050 con 0x5580119c4820 2024-06-10T00:31:59.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.176+0000 7fcfb2927b80 10 client.25664 fetch_fsmap learned FSMap version 28 2024-06-10T00:31:59.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.176+0000 7fcfb2927b80 1 -- 172.21.15.121:0/753588236 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558011a49270 con 0x5580119c4820 2024-06-10T00:31:59.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.177+0000 7fcf897fa700 1 -- 172.21.15.121:0/753588236 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf980a0830 con 0x5580119c4820 2024-06-10T00:31:59.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.177+0000 7fcf897fa700 1 -- 172.21.15.121:0/753588236 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcf9c046030 con 0x5580119c4820 2024-06-10T00:31:59.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.177+0000 7fcfb2927b80 10 client.25664 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:31:59.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.177+0000 7fcfb2927b80 10 client.25664 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:31:59.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.177+0000 7fcfb2927b80 20 client.25664 populate_metadata read hostname 'smithi121' 2024-06-10T00:31:59.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.177+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x558011a499a0 0x558011a4bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:31:59.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.177+0000 7fcfb2927b80 4 client.25664 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-10T00:31:59.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.177+0000 7fcfb2927b80 1 -- 172.21.15.121:0/753588236 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55801191d350 con 0x558011a499a0 2024-06-10T00:31:59.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.177+0000 7fcfa563f700 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x558011a499a0 0x558011a4bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:31:59.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.178+0000 7fcf897fa700 10 client.25664 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:59.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.178+0000 7fcfa563f700 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x558011a499a0 0x558011a4bdc0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fcf58292c0ed5e8 server_cookie=57a65e7d7872e948 in_seq=0 out_seq=0 2024-06-10T00:31:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.226+0000 7fcf897fa700 1 -- 172.21.15.121:0/753588236 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55801191d350 con 0x558011a499a0 2024-06-10T00:31:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.226+0000 7fcf897fa700 10 client.25664 handle_command_reply: tid=0 2024-06-10T00:31:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.226+0000 7fcfb2927b80 1 client.25664 shutdown 2024-06-10T00:31:59.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcfb2927b80 1 -- 172.21.15.121:0/753588236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 msgr2=0x5580119d6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 0x5580119d6880 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c008010 tx=0x7fcf9c027630 comp rx=0 tx=0).stop 2024-06-10T00:31:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcfb2927b80 1 -- 172.21.15.121:0/753588236 shutdown_connections 2024-06-10T00:31:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x558011a499a0 0x558011a4bdc0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580119ca900 0x5580119d3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5580119c4820 0x5580119d6880 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcfb2927b80 1 --2- 172.21.15.121:0/753588236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580119c9fe0 0x5580119d6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcfb2927b80 1 -- 172.21.15.121:0/753588236 >> 172.21.15.121:0/753588236 conn(0x5580118f1100 msgr2=0x5580119c9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcf897fa700 0 client.25664 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:31:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcfb2927b80 1 -- 172.21.15.121:0/753588236 shutdown_connections 2024-06-10T00:31:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcfb2927b80 1 -- 172.21.15.121:0/753588236 wait complete. 2024-06-10T00:31:59.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.227+0000 7fcfb2927b80 20 client.25664 trim_cache size 0 max 16384 2024-06-10T00:31:59.229 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:31:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "return_code": 0, 2024-06-10T00:31:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_tag": "7427e957-a8c9-4286-920b-de888b26f65e", 2024-06-10T00:31:59.229 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": "asynchronous" 2024-06-10T00:31:59.229 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:31:59.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.228+0000 7fcfa80a4700 1 -- 172.21.15.121:0/608421373 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf8c0a2ed0 msgr2=0x7fcf8c0a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.228+0000 7fcfa80a4700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf8c0a2ed0 0x7fcf8c0a5390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c006fd0 tx=0x7fcf9c008040 comp rx=0 tx=0).stop 2024-06-10T00:31:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.228+0000 7fcfa80a4700 1 -- 172.21.15.121:0/608421373 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfa011bc70 msgr2=0x7fcfa0113300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:31:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.228+0000 7fcfa80a4700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfa011bc70 0x7fcfa0113300 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcf980094f0 tx=0x7fcf98007f90 comp rx=0 tx=0).stop 2024-06-10T00:31:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.228+0000 7fcfa80a4700 1 -- 172.21.15.121:0/608421373 shutdown_connections 2024-06-10T00:31:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.228+0000 7fcfa80a4700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcf8c0a2ed0 0x7fcf8c0a5390 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.228+0000 7fcfa80a4700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfa011bc70 0x7fcfa0113300 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.228+0000 7fcfa80a4700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfa01099d0 0x7fcfa011a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.228+0000 7fcfa80a4700 1 --2- 172.21.15.121:0/608421373 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa0110c70 0x7fcfa011a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:31:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.228+0000 7fcfa80a4700 1 -- 172.21.15.121:0/608421373 >> 172.21.15.121:0/608421373 conn(0x7fcfa006d600 msgr2=0x7fcfa0071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:31:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.229+0000 7fcfa80a4700 1 -- 172.21.15.121:0/608421373 shutdown_connections 2024-06-10T00:31:59.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:31:59.229+0000 7fcfa80a4700 1 -- 172.21.15.121:0/608421373 wait complete. 2024-06-10T00:31:59.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:31:59 smithi121 ceph-mon[32378]: pgmap v123: 129 pgs: 129 active+clean; 508 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 225 op/s 2024-06-10T00:31:59.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:31:59 smithi135 ceph-mon[36424]: pgmap v123: 129 pgs: 129 active+clean; 508 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 225 op/s 2024-06-10T00:31:59.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:31:59 smithi193 ceph-mon[36716]: pgmap v123: 129 pgs: 129 active+clean; 508 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.2 MiB/s wr, 225 op/s 2024-06-10T00:32:00.074 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:32:00.371 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:32:00.451 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:00 smithi121 ceph-mon[32378]: scrub queued for path: / 2024-06-10T00:32:00.452 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:00 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:00.452 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:00 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:32:00.452 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:00 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:32:00.452 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:00 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:32:00.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:00 smithi135 ceph-mon[36424]: scrub queued for path: / 2024-06-10T00:32:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:00 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:00 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:32:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:00 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:32:00.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:00 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:32:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:00 smithi193 ceph-mon[36716]: scrub queued for path: / 2024-06-10T00:32:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:00 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:00 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:32:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:00 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:32:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:00 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:32:01.238 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:01 smithi121 ceph-mon[32378]: pgmap v124: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.7 MiB/s wr, 313 op/s 2024-06-10T00:32:01.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:01 smithi135 ceph-mon[36424]: pgmap v124: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.7 MiB/s wr, 313 op/s 2024-06-10T00:32:01.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:01 smithi193 ceph-mon[36716]: pgmap v124: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.7 MiB/s wr, 313 op/s 2024-06-10T00:32:03.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.031+0000 7f65fdd35700 1 -- 172.21.15.121:0/4183210278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65f81033a0 msgr2=0x7f65f8103780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:03.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.031+0000 7f65fdd35700 1 --2- 172.21.15.121:0/4183210278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65f81033a0 0x7f65f8103780 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009990 tx=0x7f65e8009ca0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.033+0000 7f65fdd35700 1 -- 172.21.15.121:0/4183210278 shutdown_connections 2024-06-10T00:32:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.033+0000 7f65fdd35700 1 --2- 172.21.15.121:0/4183210278 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f8101520 0x7f65f8111210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.033+0000 7f65fdd35700 1 --2- 172.21.15.121:0/4183210278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65f81033a0 0x7f65f8103780 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.033+0000 7f65fdd35700 1 --2- 172.21.15.121:0/4183210278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f8100a80 0x7f65f8100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.033+0000 7f65fdd35700 1 -- 172.21.15.121:0/4183210278 >> 172.21.15.121:0/4183210278 conn(0x7f65f8077a00 msgr2=0x7f65f80fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.033+0000 7f65fdd35700 1 -- 172.21.15.121:0/4183210278 shutdown_connections 2024-06-10T00:32:03.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.033+0000 7f65fdd35700 1 -- 172.21.15.121:0/4183210278 wait complete. 2024-06-10T00:32:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.034+0000 7f65fdd35700 1 Processor -- start 2024-06-10T00:32:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.034+0000 7f65fdd35700 1 -- start start 2024-06-10T00:32:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.035+0000 7f65fdd35700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f8100a80 0x7f65f810a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.035+0000 7f65fdd35700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f8101520 0x7f65f810acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.035+0000 7f65fdd35700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65f81033a0 0x7f65f8105900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.035+0000 7f65fdd35700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8114ac0 con 0x7f65f8100a80 2024-06-10T00:32:03.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.035+0000 7f65fdd35700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8114940 con 0x7f65f8101520 2024-06-10T00:32:03.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.035+0000 7f65fdd35700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8114c40 con 0x7f65f81033a0 2024-06-10T00:32:03.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.037+0000 7f65f77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f8100a80 0x7f65f810a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.037+0000 7f65f77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f8100a80 0x7f65f810a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46080/0 (socket says 172.21.15.121:46080) 2024-06-10T00:32:03.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.037+0000 7f65f77fe700 1 -- 172.21.15.121:0/4146888759 learned_addr learned my addr 172.21.15.121:0/4146888759 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:03.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.037+0000 7f65f7fff700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65f81033a0 0x7f65f8105900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.037+0000 7f65eeffd700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f8101520 0x7f65f810acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.037+0000 7f65f77fe700 1 -- 172.21.15.121:0/4146888759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65f81033a0 msgr2=0x7f65f8105900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.037+0000 7f65f77fe700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65f81033a0 0x7f65f8105900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.037+0000 7f65f77fe700 1 -- 172.21.15.121:0/4146888759 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f8101520 msgr2=0x7f65f810acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.037+0000 7f65f77fe700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f8101520 0x7f65f810acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.038 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.037+0000 7f65f77fe700 1 -- 172.21.15.121:0/4146888759 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f8106200 con 0x7f65f8100a80 2024-06-10T00:32:03.039 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.038+0000 7f65f77fe700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f8100a80 0x7f65f810a790 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f65e8003a30 tx=0x7f65e80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:03.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.039+0000 7f65f57fa700 1 -- 172.21.15.121:0/4146888759 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e800bb60 con 0x7f65f8100a80 2024-06-10T00:32:03.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.039+0000 7f65fdd35700 1 -- 172.21.15.121:0/4146888759 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f8106490 con 0x7f65f8100a80 2024-06-10T00:32:03.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.039+0000 7f65fdd35700 1 -- 172.21.15.121:0/4146888759 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f8110680 con 0x7f65f8100a80 2024-06-10T00:32:03.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.039+0000 7f65f57fa700 1 -- 172.21.15.121:0/4146888759 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e800bd00 con 0x7f65f8100a80 2024-06-10T00:32:03.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.039+0000 7f65f57fa700 1 -- 172.21.15.121:0/4146888759 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e8017870 con 0x7f65f8100a80 2024-06-10T00:32:03.049 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.046+0000 7f66085b8b80 1 Processor -- start 2024-06-10T00:32:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.047+0000 7f66085b8b80 1 -- start start 2024-06-10T00:32:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.047+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffdb640 0x55a67ffdba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.047+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 0x55a67ffd59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.047+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67ffd6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.047+0000 7f66085b8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a67ff27810 con 0x55a67ffdbf60 2024-06-10T00:32:03.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.047+0000 7f66085b8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55a67fecf720 con 0x55a67ffdb640 2024-06-10T00:32:03.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.047+0000 7f66085b8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a67ff237d0 con 0x55a67ffd5f20 2024-06-10T00:32:03.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.047+0000 7f65f57fa700 1 -- 172.21.15.121:0/4146888759 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f65e8017a10 con 0x7f65f8100a80 2024-06-10T00:32:03.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.047+0000 7f65eeffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67ffd6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.047+0000 7f65eeffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67ffd6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:45272/0 (socket says 172.21.15.121:45272) 2024-06-10T00:32:03.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.047+0000 7f65eeffd700 1 -- 172.21.15.121:0/126958330 learned_addr learned my addr 172.21.15.121:0/126958330 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:03.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.048+0000 7f65f77fe700 1 --2- 172.21.15.121:0/126958330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 0x55a67ffd59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.052 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.048+0000 7f65f7fff700 1 --2- 172.21.15.121:0/126958330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffdb640 0x55a67ffdba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.049+0000 7f65eeffd700 1 -- 172.21.15.121:0/126958330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffdb640 msgr2=0x55a67ffdba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:03.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.049+0000 7f65eeffd700 1 --2- 172.21.15.121:0/126958330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffdb640 0x55a67ffdba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.049+0000 7f65eeffd700 1 -- 172.21.15.121:0/126958330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 msgr2=0x55a67ffd59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:03.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.049+0000 7f65eeffd700 1 --2- 172.21.15.121:0/126958330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 0x55a67ffd59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.049+0000 7f65eeffd700 1 -- 172.21.15.121:0/126958330 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a67fec5f50 con 0x55a67ffd5f20 2024-06-10T00:32:03.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.049+0000 7f65f77fe700 1 --2- 172.21.15.121:0/126958330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 0x55a67ffd59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:32:03.053 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.050+0000 7f65eeffd700 1 --2- 172.21.15.121:0/126958330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67ffd6340 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f65e000bcf0 tx=0x7f65e00323e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=494145cb13965f6b server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:03.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.050+0000 7f65edffb700 1 -- 172.21.15.121:0/126958330 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e0032dd0 con 0x55a67ffd5f20 2024-06-10T00:32:03.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.050+0000 7f65edffb700 1 -- 172.21.15.121:0/126958330 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e0004510 con 0x55a67ffd5f20 2024-06-10T00:32:03.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.050+0000 7f65edffb700 1 -- 172.21.15.121:0/126958330 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e00053a0 con 0x55a67ffd5f20 2024-06-10T00:32:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.051+0000 7f66085b8b80 1 -- 172.21.15.121:0/126958330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 msgr2=0x55a67ffd6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.051+0000 7f66085b8b80 1 --2- 172.21.15.121:0/126958330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67ffd6340 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f65e000bcf0 tx=0x7f65e00323e0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.053+0000 7f65f57fa700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f65e40a2f90 0x7f65e40a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.053+0000 7f65f57fa700 1 -- 172.21.15.121:0/4146888759 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f65e80d0af0 con 0x7f65f8100a80 2024-06-10T00:32:03.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.054+0000 7f66085b8b80 1 -- 172.21.15.121:0/126958330 shutdown_connections 2024-06-10T00:32:03.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.054+0000 7f66085b8b80 1 --2- 172.21.15.121:0/126958330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67ffd6340 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.054+0000 7f66085b8b80 1 --2- 172.21.15.121:0/126958330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffdb640 0x55a67ffdba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.054+0000 7f66085b8b80 1 --2- 172.21.15.121:0/126958330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 0x55a67ffd59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.054+0000 7f66085b8b80 1 -- 172.21.15.121:0/126958330 >> 172.21.15.121:0/126958330 conn(0x55a67ff00cc0 msgr2=0x55a67ffd0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.054+0000 7f65f7fff700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f65e40a2f90 0x7f65e40a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.054+0000 7f66085b8b80 1 -- 172.21.15.121:0/126958330 shutdown_connections 2024-06-10T00:32:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.054+0000 7f66085b8b80 1 -- 172.21.15.121:0/126958330 wait complete. 2024-06-10T00:32:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.054+0000 7f66085b8b80 1 -- start start 2024-06-10T00:32:03.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.054+0000 7f65f7fff700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f65e40a2f90 0x7f65e40a5450 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009fd0 tx=0x7f65f000e040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:03.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.055+0000 7f66085b8b80 1 Processor -- start 2024-06-10T00:32:03.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.055+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffd5f20 0x55a67ffe1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.055+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdb640 0x55a67ffdf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.055+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffdbf60 0x55a67ffdf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.055+0000 7f66085b8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a67fff3900 con 0x55a67ffdb640 2024-06-10T00:32:03.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.055+0000 7f66085b8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55a67fec7bf0 con 0x55a67ffd5f20 2024-06-10T00:32:03.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.055+0000 7f66085b8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a67fff3a80 con 0x55a67ffdbf60 2024-06-10T00:32:03.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.055+0000 7f66085b8b80 10 client.0 _ll_register_callbacks cb 0x55a67fec1920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:03.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.055+0000 7f65f7fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffdbf60 0x55a67ffdf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.055+0000 7f65f7fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffdbf60 0x55a67ffdf5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:45274/0 (socket says 172.21.15.121:45274) 2024-06-10T00:32:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.055+0000 7f65f7fff700 1 -- 172.21.15.121:0/2194994938 learned_addr learned my addr 172.21.15.121:0/2194994938 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.056+0000 7f65f77fe700 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffd5f20 0x55a67ffe1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.056+0000 7f65eeffd700 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdb640 0x55a67ffdf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.057+0000 7f65f77fe700 1 -- 172.21.15.121:0/2194994938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffdbf60 msgr2=0x55a67ffdf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.057+0000 7f65f77fe700 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffdbf60 0x55a67ffdf5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.057+0000 7f65f77fe700 1 -- 172.21.15.121:0/2194994938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdb640 msgr2=0x55a67ffdf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.057+0000 7f65f77fe700 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdb640 0x55a67ffdf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.057+0000 7f65f77fe700 1 -- 172.21.15.121:0/2194994938 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a67fe768d0 con 0x55a67ffd5f20 2024-06-10T00:32:03.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.057+0000 7f65f77fe700 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffd5f20 0x55a67ffe1c70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f65e80cde50 tx=0x7f65e80054d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3e08dc3c776ba4c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:03.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.057+0000 7f65f7fff700 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffdbf60 0x55a67ffdf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:32:03.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.057+0000 7f65d6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:32:03.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.057+0000 7f65d6ffd700 1 -- 172.21.15.121:0/2194994938 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e80d7070 con 0x55a67ffd5f20 2024-06-10T00:32:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.057+0000 7f66085b8b80 1 -- 172.21.15.121:0/2194994938 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65e0004510 con 0x55a67ffd5f20 2024-06-10T00:32:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.057+0000 7f65d6ffd700 1 -- 172.21.15.121:0/2194994938 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f65e8099320 con 0x55a67ffd5f20 2024-06-10T00:32:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.059+0000 7f65d6ffd700 1 -- 172.21.15.121:0/2194994938 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f65e809c610 con 0x55a67ffd5f20 2024-06-10T00:32:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.059+0000 7f66085b8b80 10 client.25673 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.059+0000 7f66085b8b80 1 -- 172.21.15.121:0/2194994938 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a67fffe2e0 con 0x55a67ffd5f20 2024-06-10T00:32:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.060+0000 7f65d6ffd700 1 -- 172.21.15.121:0/2194994938 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e0004510 con 0x55a67ffd5f20 2024-06-10T00:32:03.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.060+0000 7f65d6ffd700 1 -- 172.21.15.121:0/2194994938 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f65e809cf10 con 0x55a67ffd5f20 2024-06-10T00:32:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.061+0000 7f66085b8b80 10 client.25673 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.061+0000 7f66085b8b80 10 client.25673 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:32:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.061+0000 7f66085b8b80 20 client.25673 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.061+0000 7f66085b8b80 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a680000850 0x55a680002c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.061+0000 7f66085b8b80 4 client.25673 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:32:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.061+0000 7f66085b8b80 1 -- 172.21.15.121:0/2194994938 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a67ff34d10 con 0x55a680000850 2024-06-10T00:32:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.062+0000 7f65eeffd700 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a680000850 0x55a680002c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.064+0000 7f65eeffd700 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a680000850 0x55a680002c40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=134f1e311152229f server_cookie=13230ecb78b4be9b in_seq=0 out_seq=0 2024-06-10T00:32:03.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.064+0000 7f65d6ffd700 10 client.25673 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.068+0000 7f65d6ffd700 1 -- 172.21.15.121:0/2194994938 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a67ff34d10 con 0x55a680000850 2024-06-10T00:32:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.068+0000 7f65d6ffd700 10 client.25673 handle_command_reply: tid=0 2024-06-10T00:32:03.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.068+0000 7f66085b8b80 1 client.25673 shutdown 2024-06-10T00:32:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.069+0000 7f66085b8b80 1 -- 172.21.15.121:0/2194994938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffd5f20 msgr2=0x55a67ffe1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.069+0000 7f66085b8b80 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffd5f20 0x55a67ffe1c70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f65e80cde50 tx=0x7f65e80054d0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.069+0000 7f66085b8b80 1 -- 172.21.15.121:0/2194994938 shutdown_connections 2024-06-10T00:32:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.069+0000 7f66085b8b80 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a680000850 0x55a680002c40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.069+0000 7f66085b8b80 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffdbf60 0x55a67ffdf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.069+0000 7f66085b8b80 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffd5f20 0x55a67ffe1c70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.069+0000 7f66085b8b80 1 --2- 172.21.15.121:0/2194994938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdb640 0x55a67ffdf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.069+0000 7f66085b8b80 1 -- 172.21.15.121:0/2194994938 >> 172.21.15.121:0/2194994938 conn(0x55a67ff00cc0 msgr2=0x55a67ffd49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:03.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.070+0000 7f65d6ffd700 0 client.25673 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:03.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.070+0000 7f66085b8b80 1 -- 172.21.15.121:0/2194994938 shutdown_connections 2024-06-10T00:32:03.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.073+0000 7f66085b8b80 1 -- 172.21.15.121:0/2194994938 wait complete. 2024-06-10T00:32:03.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.073+0000 7f66085b8b80 20 client.25673 trim_cache size 0 max 16384 2024-06-10T00:32:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.120+0000 7f66085b8b80 1 Processor -- start 2024-06-10T00:32:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.120+0000 7f66085b8b80 1 -- start start 2024-06-10T00:32:03.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.121+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffd5f20 0x55a67ffd59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.121+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffdb640 0x55a67ffde020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.121+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 0x55a67ffdcc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.121+0000 7f66085b8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f65d8011670 con 0x55a67ffdbf60 2024-06-10T00:32:03.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.122+0000 7f65f7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 0x55a67ffdcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.122+0000 7f65f7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 0x55a67ffdcc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46098/0 (socket says 172.21.15.121:46098) 2024-06-10T00:32:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.122+0000 7f65f7fff700 1 -- 172.21.15.121:0/1692010450 learned_addr learned my addr 172.21.15.121:0/1692010450 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.122+0000 7f65f77fe700 1 --2- 172.21.15.121:0/1692010450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffd5f20 0x55a67ffd59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.122+0000 7f66085b8b80 1 -- 172.21.15.121:0/1692010450 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f65d800ef00 con 0x55a67ffd5f20 2024-06-10T00:32:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.122+0000 7f66085b8b80 1 -- 172.21.15.121:0/1692010450 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f65d800cd50 con 0x55a67ffdb640 2024-06-10T00:32:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.122+0000 7f65f7fff700 1 -- 172.21.15.121:0/1692010450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffdb640 msgr2=0x55a67ffde020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.122+0000 7f65f7fff700 1 --2- 172.21.15.121:0/1692010450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffdb640 0x55a67ffde020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.122+0000 7f65f7fff700 1 -- 172.21.15.121:0/1692010450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffd5f20 msgr2=0x55a67ffd59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.122+0000 7f65f7fff700 1 --2- 172.21.15.121:0/1692010450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffd5f20 0x55a67ffd59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.122+0000 7f65f7fff700 1 -- 172.21.15.121:0/1692010450 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a67ffd0380 con 0x55a67ffdbf60 2024-06-10T00:32:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.123+0000 7f65f7fff700 1 --2- 172.21.15.121:0/1692010450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 0x55a67ffdcc70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f65f001a5c0 tx=0x7f65f000c050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f9932cda7488c06 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:03.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.123+0000 7f65ed7fa700 1 -- 172.21.15.121:0/1692010450 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0028070 con 0x55a67ffdbf60 2024-06-10T00:32:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.123+0000 7f65ed7fa700 1 -- 172.21.15.121:0/1692010450 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f00238f0 con 0x55a67ffdbf60 2024-06-10T00:32:03.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.123+0000 7f65ed7fa700 1 -- 172.21.15.121:0/1692010450 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0017a30 con 0x55a67ffdbf60 2024-06-10T00:32:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.124+0000 7f66085b8b80 1 -- 172.21.15.121:0/1692010450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 msgr2=0x55a67ffdcc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.124+0000 7f66085b8b80 1 --2- 172.21.15.121:0/1692010450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 0x55a67ffdcc70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f65f001a5c0 tx=0x7f65f000c050 comp rx=0 tx=0).stop 2024-06-10T00:32:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.124+0000 7f66085b8b80 1 -- 172.21.15.121:0/1692010450 shutdown_connections 2024-06-10T00:32:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.124+0000 7f66085b8b80 1 --2- 172.21.15.121:0/1692010450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdbf60 0x55a67ffdcc70 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.124+0000 7f66085b8b80 1 --2- 172.21.15.121:0/1692010450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffd5f20 0x55a67ffd59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.124+0000 7f66085b8b80 1 --2- 172.21.15.121:0/1692010450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffdb640 0x55a67ffde020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.124+0000 7f66085b8b80 1 -- 172.21.15.121:0/1692010450 >> 172.21.15.121:0/1692010450 conn(0x55a67ff00cc0 msgr2=0x55a67ffc9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:03.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.125+0000 7f66085b8b80 1 -- 172.21.15.121:0/1692010450 shutdown_connections 2024-06-10T00:32:03.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.127+0000 7f66085b8b80 1 -- 172.21.15.121:0/1692010450 wait complete. 2024-06-10T00:32:03.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.127+0000 7f66085b8b80 1 -- start start 2024-06-10T00:32:03.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.127+0000 7f66085b8b80 1 Processor -- start 2024-06-10T00:32:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67fff8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdb640 0x55a67fff93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f66085b8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffdbf60 0x55a67fff5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f66085b8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f65d800d3c0 con 0x55a67ffdb640 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f66085b8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f65d800e830 con 0x55a67ffdbf60 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f66085b8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f65d800da90 con 0x55a67ffd5f20 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f66085b8b80 10 client.0 _ll_register_callbacks cb 0x55a67fec1920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f65f77fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67fff8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f65f77fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67fff8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:45288/0 (socket says 172.21.15.121:45288) 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f65f77fe700 1 -- 172.21.15.121:0/1777576200 learned_addr learned my addr 172.21.15.121:0/1777576200 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f65f7fff700 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffdbf60 0x55a67fff5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f65eeffd700 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdb640 0x55a67fff93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.128+0000 7f65eeffd700 1 -- 172.21.15.121:0/1777576200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 msgr2=0x55a67fff8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.129+0000 7f65eeffd700 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67fff8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.129+0000 7f65eeffd700 1 -- 172.21.15.121:0/1777576200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffdbf60 msgr2=0x55a67fff5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:03.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.129+0000 7f65eeffd700 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffdbf60 0x55a67fff5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.131 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.129+0000 7f65eeffd700 1 -- 172.21.15.121:0/1777576200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a67fff9ac0 con 0x55a67ffdb640 2024-06-10T00:32:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.130+0000 7f65eeffd700 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdb640 0x55a67fff93c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f65e000bcc0 tx=0x7f65e0032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e914fa791da1b5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.130+0000 7f65d6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:32:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.130+0000 7f65d6ffd700 1 -- 172.21.15.121:0/1777576200 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e004e020 con 0x55a67ffdb640 2024-06-10T00:32:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.130+0000 7f65f77fe700 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67fff8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.130+0000 7f66085b8b80 1 -- 172.21.15.121:0/1777576200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65f00238f0 con 0x55a67ffdb640 2024-06-10T00:32:03.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.130+0000 7f65d6ffd700 1 -- 172.21.15.121:0/1777576200 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f65e0004b20 con 0x55a67ffdb640 2024-06-10T00:32:03.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.131+0000 7f65f7fff700 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffdbf60 0x55a67fff5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:03.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.131+0000 7f65d6ffd700 1 -- 172.21.15.121:0/1777576200 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f65e0032c90 con 0x55a67ffdb640 2024-06-10T00:32:03.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.131+0000 7f66085b8b80 10 client.16113 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:03.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.131+0000 7f66085b8b80 1 -- 172.21.15.121:0/1777576200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a68006bb70 con 0x55a67ffdb640 2024-06-10T00:32:03.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.131+0000 7f65d6ffd700 1 -- 172.21.15.121:0/1777576200 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e00053e0 con 0x55a67ffdb640 2024-06-10T00:32:03.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.131+0000 7f65d6ffd700 1 -- 172.21.15.121:0/1777576200 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f65e0033890 con 0x55a67ffdb640 2024-06-10T00:32:03.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.132+0000 7f66085b8b80 10 client.16113 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:03.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.132+0000 7f66085b8b80 10 client.16113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:32:03.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.132+0000 7f66085b8b80 20 client.16113 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:03.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.132+0000 7f66085b8b80 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a68006c0a0 0x55a68006e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:03.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.132+0000 7f66085b8b80 4 client.16113 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:32:03.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.132+0000 7f66085b8b80 1 -- 172.21.15.121:0/1777576200 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f65f00238f0 con 0x55a68006c0a0 2024-06-10T00:32:03.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.134+0000 7f65f77fe700 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a68006c0a0 0x55a68006e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.138+0000 7f65f77fe700 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a68006c0a0 0x55a68006e4c0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=457e4b3a3701baa5 server_cookie=68f25bf8db207339 in_seq=0 out_seq=0 2024-06-10T00:32:03.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.138+0000 7f65d6ffd700 10 client.16113 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:03.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.141+0000 7f65d6ffd700 1 -- 172.21.15.121:0/1777576200 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x7f65f00238f0 con 0x55a68006c0a0 2024-06-10T00:32:03.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.141+0000 7f65d6ffd700 10 client.16113 handle_command_reply: tid=0 2024-06-10T00:32:03.141 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.141+0000 7f66085b8b80 1 client.16113 shutdown 2024-06-10T00:32:03.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.143+0000 7f66085b8b80 1 -- 172.21.15.121:0/1777576200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdb640 msgr2=0x55a67fff93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:03.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.143+0000 7f66085b8b80 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdb640 0x55a67fff93c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f65e000bcc0 tx=0x7f65e0032eb0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.143+0000 7f66085b8b80 1 -- 172.21.15.121:0/1777576200 shutdown_connections 2024-06-10T00:32:03.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.143+0000 7f66085b8b80 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a68006c0a0 0x55a68006e4c0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.143+0000 7f66085b8b80 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a67ffdbf60 0x55a67fff5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.143+0000 7f65d6ffd700 0 client.16113 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:03.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.143+0000 7f66085b8b80 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a67ffd5f20 0x55a67fff8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.143+0000 7f66085b8b80 1 --2- 172.21.15.121:0/1777576200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a67ffdb640 0x55a67fff93c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.143+0000 7f66085b8b80 1 -- 172.21.15.121:0/1777576200 >> 172.21.15.121:0/1777576200 conn(0x55a67ff00cc0 msgr2=0x55a67ffca510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:03.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.144+0000 7f66085b8b80 1 -- 172.21.15.121:0/1777576200 shutdown_connections 2024-06-10T00:32:03.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.144+0000 7f66085b8b80 1 -- 172.21.15.121:0/1777576200 wait complete. 2024-06-10T00:32:03.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.145+0000 7f66085b8b80 20 client.16113 trim_cache size 0 max 16384 2024-06-10T00:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "scrub active (2271 inodes in the stack)", 2024-06-10T00:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": { 2024-06-10T00:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "7427e957-a8c9-4286-920b-de888b26f65e": { 2024-06-10T00:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/", 2024-06-10T00:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "tag": "7427e957-a8c9-4286-920b-de888b26f65e", 2024-06-10T00:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: "options": "recursive,force" 2024-06-10T00:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:03.146 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:32:03.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.146+0000 7f65fdd35700 1 -- 172.21.15.121:0/4146888759 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f65e40a2f90 msgr2=0x7f65e40a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:03.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.146+0000 7f65fdd35700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f65e40a2f90 0x7f65e40a5450 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009fd0 tx=0x7f65f000e040 comp rx=0 tx=0).stop 2024-06-10T00:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.146+0000 7f65fdd35700 1 -- 172.21.15.121:0/4146888759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f8100a80 msgr2=0x7f65f810a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.146+0000 7f65fdd35700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f8100a80 0x7f65f810a790 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f65e8003a30 tx=0x7f65e80057c0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.147+0000 7f65fdd35700 1 -- 172.21.15.121:0/4146888759 shutdown_connections 2024-06-10T00:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.147+0000 7f65fdd35700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f65e40a2f90 0x7f65e40a5450 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.147+0000 7f65fdd35700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65f81033a0 0x7f65f8105900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.147+0000 7f65fdd35700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f8100a80 0x7f65f810a790 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.147+0000 7f65fdd35700 1 --2- 172.21.15.121:0/4146888759 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f65f8101520 0x7f65f810acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.147+0000 7f65fdd35700 1 -- 172.21.15.121:0/4146888759 >> 172.21.15.121:0/4146888759 conn(0x7f65f8077a00 msgr2=0x7f65f80fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.147+0000 7f65fdd35700 1 -- 172.21.15.121:0/4146888759 shutdown_connections 2024-06-10T00:32:03.148 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:03.147+0000 7f65fdd35700 1 -- 172.21.15.121:0/4146888759 wait complete. 2024-06-10T00:32:03.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:03 smithi121 ceph-mon[32378]: pgmap v125: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-06-10T00:32:03.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:03 smithi135 ceph-mon[36424]: pgmap v125: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-06-10T00:32:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:03 smithi193 ceph-mon[36716]: pgmap v125: 129 pgs: 129 active+clean; 523 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 144 op/s 2024-06-10T00:32:04.494 INFO:tasks.cephfs.filesystem:scrub status for tag:7427e957-a8c9-4286-920b-de888b26f65e - {'path': '/', 'tag': '7427e957-a8c9-4286-920b-de888b26f65e', 'options': 'recursive,force'} 2024-06-10T00:32:05.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:05 smithi121 ceph-mon[32378]: pgmap v126: 129 pgs: 129 active+clean; 583 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.4 MiB/s wr, 463 op/s 2024-06-10T00:32:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:05 smithi135 ceph-mon[36424]: pgmap v126: 129 pgs: 129 active+clean; 583 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.4 MiB/s wr, 463 op/s 2024-06-10T00:32:05.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:05 smithi193 ceph-mon[36716]: pgmap v126: 129 pgs: 129 active+clean; 583 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.4 MiB/s wr, 463 op/s 2024-06-10T00:32:06.203 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 18.2 2024-06-10T00:32:06.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:32:07.295 INFO:tasks.workunit.client.0.smithi121.stdout: 171 1620036 1273 1112893 1406 520432 658 2024-06-10T00:32:07.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:07 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:07.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:07 smithi135 ceph-mon[36424]: pgmap v127: 129 pgs: 129 active+clean; 583 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 407 op/s 2024-06-10T00:32:07.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:07 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:07 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:07 smithi193 ceph-mon[36716]: pgmap v127: 129 pgs: 129 active+clean; 583 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 407 op/s 2024-06-10T00:32:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:07 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:07.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:07 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:07.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:07 smithi121 ceph-mon[32378]: pgmap v127: 129 pgs: 129 active+clean; 583 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 407 op/s 2024-06-10T00:32:07.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:07 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:08.072 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:32:09.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:09 smithi135 ceph-mon[36424]: pgmap v128: 129 pgs: 129 active+clean; 583 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 406 op/s 2024-06-10T00:32:09.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:09 smithi193 ceph-mon[36716]: pgmap v128: 129 pgs: 129 active+clean; 583 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 406 op/s 2024-06-10T00:32:09.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:09 smithi121 ceph-mon[32378]: pgmap v128: 129 pgs: 129 active+clean; 583 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.9 MiB/s wr, 406 op/s 2024-06-10T00:32:11.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:11 smithi121 ceph-mon[32378]: pgmap v129: 129 pgs: 129 active+clean; 684 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 16 MiB/s wr, 804 op/s 2024-06-10T00:32:11.407 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:11 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:11.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.497+0000 7fbde97c5700 1 -- 172.21.15.121:0/441690672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde410ef30 msgr2=0x7fbde4111320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:11.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.497+0000 7fbde97c5700 1 --2- 172.21.15.121:0/441690672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde410ef30 0x7fbde4111320 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbddc00b0a0 tx=0x7fbddc00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:32:11.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.497+0000 7fbde97c5700 1 -- 172.21.15.121:0/441690672 shutdown_connections 2024-06-10T00:32:11.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.497+0000 7fbde97c5700 1 --2- 172.21.15.121:0/441690672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde410ef30 0x7fbde4111320 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:11.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.497+0000 7fbde97c5700 1 --2- 172.21.15.121:0/441690672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde406aa30 0x7fbde406ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:11.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.497+0000 7fbde97c5700 1 --2- 172.21.15.121:0/441690672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde406b350 0x7fbde410e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:11.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.498+0000 7fbde97c5700 1 -- 172.21.15.121:0/441690672 >> 172.21.15.121:0/441690672 conn(0x7fbde4078300 msgr2=0x7fbde4078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:11.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.498+0000 7fbde97c5700 1 -- 172.21.15.121:0/441690672 shutdown_connections 2024-06-10T00:32:11.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.498+0000 7fbde97c5700 1 -- 172.21.15.121:0/441690672 wait complete. 2024-06-10T00:32:11.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.499+0000 7fbde97c5700 1 Processor -- start 2024-06-10T00:32:11.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.499+0000 7fbde97c5700 1 -- start start 2024-06-10T00:32:11.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.499+0000 7fbde97c5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde406aa30 0x7fbde419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:11.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.499+0000 7fbde97c5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde406b350 0x7fbde419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:11.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.499+0000 7fbde97c5700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde410ef30 0x7fbde419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:11.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.499+0000 7fbde97c5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbde4114990 con 0x7fbde406b350 2024-06-10T00:32:11.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.499+0000 7fbde97c5700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fbde4114810 con 0x7fbde410ef30 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.499+0000 7fbde97c5700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbde4114b10 con 0x7fbde406aa30 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde406b350 0x7fbde419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde27fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde406b350 0x7fbde419b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53850/0 (socket says 172.21.15.121:53850) 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde27fc700 1 -- 172.21.15.121:0/2181607372 learned_addr learned my addr 172.21.15.121:0/2181607372 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde2ffd700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde406aa30 0x7fbde419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde37fe700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde410ef30 0x7fbde419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde27fc700 1 -- 172.21.15.121:0/2181607372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde406aa30 msgr2=0x7fbde419ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde27fc700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde406aa30 0x7fbde419ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde27fc700 1 -- 172.21.15.121:0/2181607372 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde410ef30 msgr2=0x7fbde419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde27fc700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde410ef30 0x7fbde419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde27fc700 1 -- 172.21.15.121:0/2181607372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde419fdf0 con 0x7fbde406b350 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde2ffd700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde406aa30 0x7fbde419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:11.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde27fc700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde406b350 0x7fbde419b320 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800e9a0 tx=0x7fbdd800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:11.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbdcbfff700 1 -- 172.21.15.121:0/2181607372 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd800ca10 con 0x7fbde406b350 2024-06-10T00:32:11.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde97c5700 1 -- 172.21.15.121:0/2181607372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde41a00e0 con 0x7fbde406b350 2024-06-10T00:32:11.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbdcbfff700 1 -- 172.21.15.121:0/2181607372 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdd800cbb0 con 0x7fbde406b350 2024-06-10T00:32:11.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbdcbfff700 1 -- 172.21.15.121:0/2181607372 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8018680 con 0x7fbde406b350 2024-06-10T00:32:11.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.500+0000 7fbde97c5700 1 -- 172.21.15.121:0/2181607372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde41a7960 con 0x7fbde406b350 2024-06-10T00:32:11.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.502+0000 7fbde97c5700 1 -- 172.21.15.121:0/2181607372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde4069180 con 0x7fbde406b350 2024-06-10T00:32:11.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.504+0000 7fbdcbfff700 1 -- 172.21.15.121:0/2181607372 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fbdd8010430 con 0x7fbde406b350 2024-06-10T00:32:11.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.504+0000 7fbdcbfff700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdcc0a2d90 0x7fbdcc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:11.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.505+0000 7fbdcbfff700 1 -- 172.21.15.121:0/2181607372 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fbdd80cfdf0 con 0x7fbde406b350 2024-06-10T00:32:11.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.505+0000 7fbde2ffd700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdcc0a2d90 0x7fbdcc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:11.505 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.505+0000 7fbde2ffd700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdcc0a2d90 0x7fbdcc0a5250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4011490 tx=0x7fbdd4009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:11.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:11 smithi135 ceph-mon[36424]: pgmap v129: 129 pgs: 129 active+clean; 684 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 16 MiB/s wr, 804 op/s 2024-06-10T00:32:11.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:11 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:11.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.508+0000 7fbdcbfff700 1 -- 172.21.15.121:0/2181607372 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbdd809c680 con 0x7fbde406b350 2024-06-10T00:32:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:11 smithi193 ceph-mon[36716]: pgmap v129: 129 pgs: 129 active+clean; 684 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 16 MiB/s wr, 804 op/s 2024-06-10T00:32:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:11 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:32:11.840 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:32:11.840 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:30:26.280580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:32:11.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.838+0000 7fbde97c5700 1 -- 172.21.15.121:0/2181607372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbde402dca0 con 0x7fbde406b350 2024-06-10T00:32:11.841 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.839+0000 7fbdcbfff700 1 -- 172.21.15.121:0/2181607372 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11888 (secure 0 0 0) 0x7fbdd809bf80 con 0x7fbde406b350 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.844+0000 7fbde97c5700 1 -- 172.21.15.121:0/2181607372 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdcc0a2d90 msgr2=0x7fbdcc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.844+0000 7fbde97c5700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdcc0a2d90 0x7fbdcc0a5250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4011490 tx=0x7fbdd4009260 comp rx=0 tx=0).stop 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.844+0000 7fbde97c5700 1 -- 172.21.15.121:0/2181607372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde406b350 msgr2=0x7fbde419b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.844+0000 7fbde97c5700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde406b350 0x7fbde419b320 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800e9a0 tx=0x7fbdd800ecb0 comp rx=0 tx=0).stop 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.847+0000 7fbde97c5700 1 -- 172.21.15.121:0/2181607372 shutdown_connections 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.847+0000 7fbde97c5700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fbdcc0a2d90 0x7fbdcc0a5250 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.847+0000 7fbde97c5700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbde410ef30 0x7fbde419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.847+0000 7fbde97c5700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbde406aa30 0x7fbde419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.847+0000 7fbde97c5700 1 --2- 172.21.15.121:0/2181607372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbde406b350 0x7fbde419b320 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.847+0000 7fbde97c5700 1 -- 172.21.15.121:0/2181607372 >> 172.21.15.121:0/2181607372 conn(0x7fbde4078300 msgr2=0x7fbde41102e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.848+0000 7fbde97c5700 1 -- 172.21.15.121:0/2181607372 shutdown_connections 2024-06-10T00:32:11.849 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:11.848+0000 7fbde97c5700 1 -- 172.21.15.121:0/2181607372 wait complete. 2024-06-10T00:32:11.850 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 28 2024-06-10T00:32:12.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:12 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2181607372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:32:12.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:12 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:32:12.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:12 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2181607372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:32:12.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:12 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:32:12.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:12 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2181607372' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:32:12.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:12 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:32:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:13 smithi193 ceph-mon[36716]: pgmap v130: 129 pgs: 129 active+clean; 684 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 14 MiB/s wr, 716 op/s 2024-06-10T00:32:13.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:13 smithi121 ceph-mon[32378]: pgmap v130: 129 pgs: 129 active+clean; 684 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 14 MiB/s wr, 716 op/s 2024-06-10T00:32:13.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:13 smithi135 ceph-mon[36424]: pgmap v130: 129 pgs: 129 active+clean; 684 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 14 MiB/s wr, 716 op/s 2024-06-10T00:32:13.863 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db / --timeout 90 --expiration 138.2 --await-for 45.0' 2024-06-10T00:32:13.863 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db / --timeout 90 --expiration 138.2 --await-for 45.0 2024-06-10T00:32:15.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:15 smithi193 ceph-mon[36716]: pgmap v131: 129 pgs: 129 active+clean; 712 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 17 MiB/s wr, 823 op/s 2024-06-10T00:32:15.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:15 smithi121 ceph-mon[32378]: pgmap v131: 129 pgs: 129 active+clean; 712 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 17 MiB/s wr, 823 op/s 2024-06-10T00:32:15.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:15 smithi135 ceph-mon[36424]: pgmap v131: 129 pgs: 129 active+clean; 712 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 17 MiB/s wr, 823 op/s 2024-06-10T00:32:17.298 INFO:tasks.workunit.client.0.smithi121.stdout: 186 1663251 1066 1151654 1043 532688 1122 2024-06-10T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:17 smithi193 ceph-mon[36716]: pgmap v132: 129 pgs: 129 active+clean; 734 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 13 MiB/s wr, 575 op/s 2024-06-10T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:17 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:32:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:17 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:32:17.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:17 smithi121 ceph-mon[32378]: pgmap v132: 129 pgs: 129 active+clean; 734 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 13 MiB/s wr, 575 op/s 2024-06-10T00:32:17.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:17 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:32:17.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:17 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:32:17.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:17 smithi135 ceph-mon[36424]: pgmap v132: 129 pgs: 129 active+clean; 734 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 13 MiB/s wr, 575 op/s 2024-06-10T00:32:17.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:17 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:32:17.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:17 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:32:18.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:18 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:32:18.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:18 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:32:18.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:18 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:32:19.265 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:32:19.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:19 smithi193 ceph-mon[36716]: pgmap v133: 129 pgs: 129 active+clean; 734 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 13 MiB/s wr, 575 op/s 2024-06-10T00:32:19.692 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:19 smithi121 ceph-mon[32378]: pgmap v133: 129 pgs: 129 active+clean; 734 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 13 MiB/s wr, 575 op/s 2024-06-10T00:32:19.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:19 smithi135 ceph-mon[36424]: pgmap v133: 129 pgs: 129 active+clean; 734 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 13 MiB/s wr, 575 op/s 2024-06-10T00:32:21.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:21 smithi193 ceph-mon[36716]: pgmap v134: 129 pgs: 129 active+clean; 751 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 15 MiB/s wr, 671 op/s 2024-06-10T00:32:21.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:21 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:32:21.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:21 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:32:21.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:21 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:32:21.666 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:21 smithi121 ceph-mon[32378]: pgmap v134: 129 pgs: 129 active+clean; 751 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 15 MiB/s wr, 671 op/s 2024-06-10T00:32:21.666 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:21 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:32:21.666 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:21 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:32:21.666 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:21 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:32:21.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:21 smithi135 ceph-mon[36424]: pgmap v134: 129 pgs: 129 active+clean; 751 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 15 MiB/s wr, 671 op/s 2024-06-10T00:32:21.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:21 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:32:21.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:21 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:32:21.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:21 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:32:22.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.102+0000 7f22790e3700 1 -- 172.21.15.121:0/2214234286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22741132c0 msgr2=0x7f22741156b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:22.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.102+0000 7f22790e3700 1 --2- 172.21.15.121:0/2214234286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22741132c0 0x7f22741156b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f225c01a9a0 tx=0x7f225c01acb0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.105+0000 7f22790e3700 1 -- 172.21.15.121:0/2214234286 shutdown_connections 2024-06-10T00:32:22.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.105+0000 7f22790e3700 1 --2- 172.21.15.121:0/2214234286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22741132c0 0x7f22741156b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.105+0000 7f22790e3700 1 --2- 172.21.15.121:0/2214234286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2274074830 0x7f2274074c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.105+0000 7f22790e3700 1 --2- 172.21.15.121:0/2214234286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2274075150 0x7f2274112d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.105+0000 7f22790e3700 1 -- 172.21.15.121:0/2214234286 >> 172.21.15.121:0/2214234286 conn(0x7f22740fde00 msgr2=0x7f2274100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:22.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.106+0000 7f22790e3700 1 -- 172.21.15.121:0/2214234286 shutdown_connections 2024-06-10T00:32:22.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.106+0000 7f22790e3700 1 -- 172.21.15.121:0/2214234286 wait complete. 2024-06-10T00:32:22.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.107+0000 7f22790e3700 1 Processor -- start 2024-06-10T00:32:22.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.107+0000 7f22790e3700 1 -- start start 2024-06-10T00:32:22.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.108+0000 7f22790e3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2274074830 0x7f2274074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.108+0000 7f22790e3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2274075150 0x7f2274071210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.108+0000 7f22790e3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22741132c0 0x7f2274071750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.108+0000 7f22790e3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2274108e70 con 0x7f2274075150 2024-06-10T00:32:22.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.108+0000 7f22790e3700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2274108cf0 con 0x7f22741132c0 2024-06-10T00:32:22.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.108+0000 7f22790e3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2274108ff0 con 0x7f2274074830 2024-06-10T00:32:22.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.108+0000 7f22737fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2274075150 0x7f2274071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.108+0000 7f22737fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2274075150 0x7f2274071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48030/0 (socket says 172.21.15.121:48030) 2024-06-10T00:32:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.108+0000 7f22737fe700 1 -- 172.21.15.121:0/1239331518 learned_addr learned my addr 172.21.15.121:0/1239331518 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.109+0000 7f22737fe700 1 -- 172.21.15.121:0/1239331518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2274074830 msgr2=0x7f2274074110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:32:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.109+0000 7f22737fe700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2274074830 0x7f2274074110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.109+0000 7f22737fe700 1 -- 172.21.15.121:0/1239331518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22741132c0 msgr2=0x7f2274071750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:32:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.109+0000 7f22737fe700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22741132c0 0x7f2274071750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.109+0000 7f22737fe700 1 -- 172.21.15.121:0/1239331518 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2274072020 con 0x7f2274075150 2024-06-10T00:32:22.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.109+0000 7f22737fe700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2274075150 0x7f2274071210 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f226400dcc0 tx=0x7f2264009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.110+0000 7f22717fa700 1 -- 172.21.15.121:0/1239331518 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2264016070 con 0x7f2274075150 2024-06-10T00:32:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.110+0000 7f22717fa700 1 -- 172.21.15.121:0/1239331518 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2264009b30 con 0x7f2274075150 2024-06-10T00:32:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.110+0000 7f22717fa700 1 -- 172.21.15.121:0/1239331518 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226400f620 con 0x7f2274075150 2024-06-10T00:32:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.110+0000 7f22790e3700 1 -- 172.21.15.121:0/1239331518 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2274112750 con 0x7f2274075150 2024-06-10T00:32:22.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.111+0000 7f22790e3700 1 -- 172.21.15.121:0/1239331518 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2274112c10 con 0x7f2274075150 2024-06-10T00:32:22.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.124+0000 7f2284bc8b80 1 Processor -- start 2024-06-10T00:32:22.125 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.125+0000 7f2284bc8b80 1 -- start start 2024-06-10T00:32:22.126 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.125+0000 7f2284bc8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc6870 0x55af49fc6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.125+0000 7f2284bc8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc7190 0x55af49fd1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.125+0000 7f2284bc8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fd2040 0x55af49fd4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.125+0000 7f2284bc8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55af49eba0f0 con 0x55af49fd2040 2024-06-10T00:32:22.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.125+0000 7f2284bc8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55af49f189e0 con 0x55af49fc7190 2024-06-10T00:32:22.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.125+0000 7f2284bc8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55af49e805f0 con 0x55af49fc6870 2024-06-10T00:32:22.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.126+0000 7f22717fa700 1 -- 172.21.15.121:0/1239331518 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2264019070 con 0x7f2274075150 2024-06-10T00:32:22.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.126+0000 7f22788e2700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc6870 0x55af49fc6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.126+0000 7f22788e2700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc6870 0x55af49fc6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:46620/0 (socket says 172.21.15.121:46620) 2024-06-10T00:32:22.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.126+0000 7f22788e2700 1 -- 172.21.15.121:0/2046349595 learned_addr learned my addr 172.21.15.121:0/2046349595 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:22.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.126+0000 7f2273fff700 1 --2- 172.21.15.121:0/2046349595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc7190 0x55af49fd1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.126+0000 7f22788e2700 1 -- 172.21.15.121:0/2046349595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc7190 msgr2=0x55af49fd1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:22.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.126+0000 7f22788e2700 1 --2- 172.21.15.121:0/2046349595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc7190 0x55af49fd1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.126+0000 7f22788e2700 1 -- 172.21.15.121:0/2046349595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fd2040 msgr2=0x55af49fd4430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-10T00:32:22.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.126+0000 7f22788e2700 1 --2- 172.21.15.121:0/2046349595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fd2040 0x55af49fd4430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.126+0000 7f22788e2700 1 -- 172.21.15.121:0/2046349595 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af49e67a60 con 0x55af49fc6870 2024-06-10T00:32:22.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.126+0000 7f22788e2700 1 --2- 172.21.15.121:0/2046349595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc6870 0x55af49fc6c50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f225c005f50 tx=0x7f225c003810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27d08701b75d7712 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:22.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.127+0000 7f22717fa700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f22600a3060 0x7f22600a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.127+0000 7f226a7fc700 1 -- 172.21.15.121:0/2046349595 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225c0038f0 con 0x55af49fc6870 2024-06-10T00:32:22.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.127+0000 7f226a7fc700 1 -- 172.21.15.121:0/2046349595 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f225c005070 con 0x55af49fc6870 2024-06-10T00:32:22.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.127+0000 7f2273fff700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f22600a3060 0x7f22600a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.128+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2046349595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc6870 msgr2=0x55af49fc6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:22.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.128+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2046349595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc6870 0x55af49fc6c50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f225c005f50 tx=0x7f225c003810 comp rx=0 tx=0).stop 2024-06-10T00:32:22.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.128+0000 7f2273fff700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f22600a3060 0x7f22600a5520 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f226c00f540 tx=0x7f226c00b040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:22.129 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.128+0000 7f22717fa700 1 -- 172.21.15.121:0/1239331518 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f2264010460 con 0x7f2274075150 2024-06-10T00:32:22.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.130+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2046349595 shutdown_connections 2024-06-10T00:32:22.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.130+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2046349595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fd2040 0x55af49fd4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.130+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2046349595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc6870 0x55af49fc6c50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.130+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2046349595 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc7190 0x55af49fd1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.130 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.130+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2046349595 >> 172.21.15.121:0/2046349595 conn(0x55af49ef4210 msgr2=0x55af49fc2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:22.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.135+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2046349595 shutdown_connections 2024-06-10T00:32:22.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.139+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2046349595 wait complete. 2024-06-10T00:32:22.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.139+0000 7f2284bc8b80 1 -- start start 2024-06-10T00:32:22.144 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.139+0000 7f2284bc8b80 1 Processor -- start 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.140+0000 7f2284bc8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc6870 0x55af49fd16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.140+0000 7f2284bc8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 0x55af49fce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.140+0000 7f2284bc8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fd2040 0x55af49fcef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.140+0000 7f2284bc8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55af49eb9150 con 0x55af49fd2040 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.140+0000 7f2284bc8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55af49f149a0 con 0x55af49fc6870 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.140+0000 7f2284bc8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55af49fd4b60 con 0x55af49fc7190 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.140+0000 7f2284bc8b80 10 client.0 _ll_register_callbacks cb 0x55af49eae740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.140+0000 7f2273fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc6870 0x55af49fd16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.141+0000 7f2273fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc6870 0x55af49fd16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:35222/0 (socket says 172.21.15.121:35222) 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.141+0000 7f2273fff700 1 -- 172.21.15.121:0/1641565505 learned_addr learned my addr 172.21.15.121:0/1641565505 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.141+0000 7f22788e2700 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fd2040 0x55af49fcef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.141+0000 7f2273fff700 1 -- 172.21.15.121:0/1641565505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 msgr2=0x55af49fce9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-10T00:32:22.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.141+0000 7f2273fff700 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 0x55af49fce9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.141+0000 7f2273fff700 1 -- 172.21.15.121:0/1641565505 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fd2040 msgr2=0x55af49fcef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.141+0000 7f2273fff700 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fd2040 0x55af49fcef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.141+0000 7f2273fff700 1 -- 172.21.15.121:0/1641565505 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af49ede8e0 con 0x55af49fc6870 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.141+0000 7f22788e2700 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fd2040 0x55af49fcef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.142+0000 7f2273fff700 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc6870 0x55af49fd16c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f226c008a90 tx=0x7f226c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50f29aeb8b69ec8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.142+0000 7f22537fe700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.142+0000 7f22537fe700 1 -- 172.21.15.121:0/1641565505 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c01b070 con 0x55af49fc6870 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.142+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1641565505 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f225c005070 con 0x55af49fc6870 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.142+0000 7f22537fe700 1 -- 172.21.15.121:0/1641565505 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f226c017a60 con 0x55af49fc6870 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.142+0000 7f22537fe700 1 -- 172.21.15.121:0/1641565505 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f226c02f050 con 0x55af49fc6870 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.143+0000 7f2284bc8b80 10 client.25694 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:22.146 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.143+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1641565505 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55af4a0012f0 con 0x55af49fc6870 2024-06-10T00:32:22.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.147+0000 7f22537fe700 1 -- 172.21.15.121:0/1641565505 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f225c005070 con 0x55af49fc6870 2024-06-10T00:32:22.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.147+0000 7f22537fe700 1 -- 172.21.15.121:0/1641565505 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f226c004260 con 0x55af49fc6870 2024-06-10T00:32:22.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.147+0000 7f2284bc8b80 10 client.25694 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:22.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.147+0000 7f2284bc8b80 10 client.25694 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:32:22.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.147+0000 7f2284bc8b80 20 client.25694 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:22.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.147+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55af4a003d90 0x55af4a0061f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.147+0000 7f2284bc8b80 4 client.25694 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:32:22.147 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.147+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1641565505 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af49f25ee0 con 0x55af4a003d90 2024-06-10T00:32:22.150 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.150+0000 7f22737fe700 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55af4a003d90 0x55af4a0061f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.163+0000 7f22737fe700 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55af4a003d90 0x55af4a0061f0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2847343d94513ee4 server_cookie=7af5f8f369805742 in_seq=0 out_seq=0 2024-06-10T00:32:22.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.163+0000 7f22537fe700 10 client.25694 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:22.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.165+0000 7f22537fe700 1 -- 172.21.15.121:0/1641565505 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55af49f25ee0 con 0x55af4a003d90 2024-06-10T00:32:22.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.165+0000 7f22537fe700 10 client.25694 handle_command_reply: tid=0 2024-06-10T00:32:22.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.165+0000 7f2284bc8b80 1 client.25694 shutdown 2024-06-10T00:32:22.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.170+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1641565505 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc6870 msgr2=0x55af49fd16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.170+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc6870 0x55af49fd16c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f226c008a90 tx=0x7f226c004ad0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.170+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1641565505 shutdown_connections 2024-06-10T00:32:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.170+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55af4a003d90 0x55af4a0061f0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.170+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fd2040 0x55af49fcef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.170+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fc6870 0x55af49fd16c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.170+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/1641565505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 0x55af49fce9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.170+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1641565505 >> 172.21.15.121:0/1641565505 conn(0x55af49ef4210 msgr2=0x55af49fc2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:22.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.171+0000 7f22537fe700 0 client.25694 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:22.175 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.174+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1641565505 shutdown_connections 2024-06-10T00:32:22.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.174+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1641565505 wait complete. 2024-06-10T00:32:22.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.175+0000 7f2284bc8b80 20 client.25694 trim_cache size 0 max 16384 2024-06-10T00:32:22.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.230+0000 7f2284bc8b80 1 Processor -- start 2024-06-10T00:32:22.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.231+0000 7f2284bc8b80 1 -- start start 2024-06-10T00:32:22.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.231+0000 7f2284bc8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.231+0000 7f2284bc8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 0x55af49fceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.231+0000 7f2284bc8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fd2040 0x55af4a00bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.231+0000 7f2284bc8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2254011750 con 0x55af49fc6870 2024-06-10T00:32:22.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.231+0000 7f2284bc8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f225400efe0 con 0x55af49fd2040 2024-06-10T00:32:22.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.231+0000 7f2284bc8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f225400ce30 con 0x55af49fc7190 2024-06-10T00:32:22.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.232+0000 7f2273fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.232+0000 7f2273fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48038/0 (socket says 172.21.15.121:48038) 2024-06-10T00:32:22.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.232+0000 7f2273fff700 1 -- 172.21.15.121:0/1570461067 learned_addr learned my addr 172.21.15.121:0/1570461067 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:22.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.233+0000 7f22737fe700 1 --2- 172.21.15.121:0/1570461067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 0x55af49fceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.233+0000 7f22788e2700 1 --2- 172.21.15.121:0/1570461067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fd2040 0x55af4a00bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.233+0000 7f2273fff700 1 -- 172.21.15.121:0/1570461067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 msgr2=0x55af49fceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:22.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.233+0000 7f2273fff700 1 --2- 172.21.15.121:0/1570461067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 0x55af49fceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.233+0000 7f2273fff700 1 -- 172.21.15.121:0/1570461067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fd2040 msgr2=0x55af4a00bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:22.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.233+0000 7f2273fff700 1 --2- 172.21.15.121:0/1570461067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fd2040 0x55af4a00bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.233+0000 7f2273fff700 1 -- 172.21.15.121:0/1570461067 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af4a006560 con 0x55af49fc6870 2024-06-10T00:32:22.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.233+0000 7f22788e2700 1 --2- 172.21.15.121:0/1570461067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fd2040 0x55af4a00bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:32:22.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.233+0000 7f2273fff700 1 --2- 172.21.15.121:0/1570461067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fce980 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f226c0049d0 tx=0x7f226c008e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647cc0266264ac4d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:22.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.234+0000 7f2269ffb700 1 -- 172.21.15.121:0/1570461067 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c037020 con 0x55af49fc6870 2024-06-10T00:32:22.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.234+0000 7f2269ffb700 1 -- 172.21.15.121:0/1570461067 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f226c0094b0 con 0x55af49fc6870 2024-06-10T00:32:22.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.234+0000 7f2269ffb700 1 -- 172.21.15.121:0/1570461067 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c01b040 con 0x55af49fc6870 2024-06-10T00:32:22.237 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.236+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1570461067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 msgr2=0x55af49fce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:22.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.236+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/1570461067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fce980 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f226c0049d0 tx=0x7f226c008e50 comp rx=0 tx=0).stop 2024-06-10T00:32:22.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.237+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1570461067 shutdown_connections 2024-06-10T00:32:22.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.237+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/1570461067 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fd2040 0x55af4a00bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.237+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/1570461067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fce980 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.237+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/1570461067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 0x55af49fceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.238 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.237+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1570461067 >> 172.21.15.121:0/1570461067 conn(0x55af49ef4210 msgr2=0x55af49fce270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:22.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.238+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1570461067 shutdown_connections 2024-06-10T00:32:22.239 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.238+0000 7f2284bc8b80 1 -- 172.21.15.121:0/1570461067 wait complete. 2024-06-10T00:32:22.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.239+0000 7f2284bc8b80 1 -- start start 2024-06-10T00:32:22.240 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.239+0000 7f2284bc8b80 1 Processor -- start 2024-06-10T00:32:22.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.240+0000 7f2284bc8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fda630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.241 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.241+0000 7f2273fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fda630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.241+0000 7f2273fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fda630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48054/0 (socket says 172.21.15.121:48054) 2024-06-10T00:32:22.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.242+0000 7f2273fff700 1 -- 172.21.15.121:0/2606255137 learned_addr learned my addr 172.21.15.121:0/2606255137 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:22.242 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.242+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 0x55af49fcb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.242+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fd2040 0x55af49fcbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.243+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2606255137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f225400d4a0 con 0x55af49fc6870 2024-06-10T00:32:22.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.243+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2606255137 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f225400e910 con 0x55af49fd2040 2024-06-10T00:32:22.244 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.243+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2606255137 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f225400db70 con 0x55af49fc7190 2024-06-10T00:32:22.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.244+0000 7f2273fff700 1 -- 172.21.15.121:0/2606255137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 msgr2=0x55af49fcb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:22.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.244+0000 7f2273fff700 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 0x55af49fcb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.244+0000 7f2273fff700 1 -- 172.21.15.121:0/2606255137 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fd2040 msgr2=0x55af49fcbbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:32:22.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.244+0000 7f2273fff700 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fd2040 0x55af49fcbbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:22.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.244+0000 7f2273fff700 1 -- 172.21.15.121:0/2606255137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af49fcc430 con 0x55af49fc6870 2024-06-10T00:32:22.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.245+0000 7f2273fff700 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fda630 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f226c0177a0 tx=0x7f226c017d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34526b874d10408 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:22.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.247+0000 7f22537fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:32:22.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.247+0000 7f22537fe700 1 -- 172.21.15.121:0/2606255137 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226c040070 con 0x55af49fc6870 2024-06-10T00:32:22.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.248+0000 7f2284bc8b80 10 client.0 _ll_register_callbacks cb 0x55af49eae740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:22.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.248+0000 7f22537fe700 1 -- 172.21.15.121:0/2606255137 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f226c004a90 con 0x55af49fc6870 2024-06-10T00:32:22.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.248+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2606255137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f226c0094b0 con 0x55af49fc6870 2024-06-10T00:32:22.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.249+0000 7f22537fe700 1 -- 172.21.15.121:0/2606255137 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f226c04a050 con 0x55af49fc6870 2024-06-10T00:32:22.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.249+0000 7f2284bc8b80 10 client.16134 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:22.250 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.250+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2606255137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55af4a06e630 con 0x55af49fc6870 2024-06-10T00:32:22.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.251+0000 7f22537fe700 1 -- 172.21.15.121:0/2606255137 <== mon.0 v2:172.21.15.121:3300/0 4 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f226c004c90 con 0x55af49fc6870 2024-06-10T00:32:22.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.251+0000 7f2284bc8b80 10 client.16134 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:22.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.251+0000 7f2284bc8b80 10 client.16134 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:32:22.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.251+0000 7f2284bc8b80 20 client.16134 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:22.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.251+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55af4a06ece0 0x55af4a0711a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:22.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.251+0000 7f2284bc8b80 4 client.16134 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.2, "await_for": 45.0}] 2024-06-10T00:32:22.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.251+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2606255137 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 138.2, "await_for": 45.0}) -- 0x7f226c0094b0 con 0x55af4a06ece0 2024-06-10T00:32:22.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.252+0000 7f22737fe700 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55af4a06ece0 0x55af4a0711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:22.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.253+0000 7f22737fe700 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55af4a06ece0 0x55af4a0711a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3266c56f28bd9615 server_cookie=12883cf392928d8c in_seq=0 out_seq=0 2024-06-10T00:32:22.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:22.253+0000 7f22537fe700 10 client.16134 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:23.560 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:23 smithi135 ceph-mon[36424]: pgmap v135: 129 pgs: 129 active+clean; 751 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.1 MiB/s wr, 273 op/s 2024-06-10T00:32:23.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:23 smithi193 ceph-mon[36716]: pgmap v135: 129 pgs: 129 active+clean; 751 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.1 MiB/s wr, 273 op/s 2024-06-10T00:32:23.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:23 smithi121 ceph-mon[32378]: pgmap v135: 129 pgs: 129 active+clean; 751 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.1 MiB/s wr, 273 op/s 2024-06-10T00:32:25.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:25 smithi193 ceph-mon[36716]: pgmap v136: 129 pgs: 129 active+clean; 755 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 6.6 MiB/s wr, 315 op/s 2024-06-10T00:32:25.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:25 smithi121 ceph-mon[32378]: pgmap v136: 129 pgs: 129 active+clean; 755 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 6.6 MiB/s wr, 315 op/s 2024-06-10T00:32:25.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:25 smithi135 ceph-mon[36424]: pgmap v136: 129 pgs: 129 active+clean; 755 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 6.6 MiB/s wr, 315 op/s 2024-06-10T00:32:27.294 INFO:tasks.workunit.client.0.smithi121.stdout: 197 1223918 685 860201 645 396190 960 2024-06-10T00:32:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:27 smithi193 ceph-mon[36716]: pgmap v137: 129 pgs: 129 active+clean; 761 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.5 MiB/s wr, 241 op/s 2024-06-10T00:32:27.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:27 smithi121 ceph-mon[32378]: pgmap v137: 129 pgs: 129 active+clean; 761 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.5 MiB/s wr, 241 op/s 2024-06-10T00:32:27.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:27 smithi135 ceph-mon[36424]: pgmap v137: 129 pgs: 129 active+clean; 761 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.5 MiB/s wr, 241 op/s 2024-06-10T00:32:29.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:29 smithi193 ceph-mon[36716]: pgmap v138: 129 pgs: 129 active+clean; 761 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 171 op/s 2024-06-10T00:32:29.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:29 smithi121 ceph-mon[32378]: pgmap v138: 129 pgs: 129 active+clean; 761 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 171 op/s 2024-06-10T00:32:29.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:29 smithi135 ceph-mon[36424]: pgmap v138: 129 pgs: 129 active+clean; 761 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 171 op/s 2024-06-10T00:32:31.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:31 smithi193 ceph-mon[36716]: pgmap v139: 129 pgs: 129 active+clean; 787 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.0 MiB/s wr, 555 op/s 2024-06-10T00:32:31.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:31 smithi121 ceph-mon[32378]: pgmap v139: 129 pgs: 129 active+clean; 787 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.0 MiB/s wr, 555 op/s 2024-06-10T00:32:31.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:31 smithi135 ceph-mon[36424]: pgmap v139: 129 pgs: 129 active+clean; 787 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 5.0 MiB/s wr, 555 op/s 2024-06-10T00:32:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:33 smithi193 ceph-mon[36716]: pgmap v140: 129 pgs: 129 active+clean; 787 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 459 op/s 2024-06-10T00:32:33.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:33 smithi121 ceph-mon[32378]: pgmap v140: 129 pgs: 129 active+clean; 787 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 459 op/s 2024-06-10T00:32:33.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:33 smithi135 ceph-mon[36424]: pgmap v140: 129 pgs: 129 active+clean; 787 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.4 MiB/s wr, 459 op/s 2024-06-10T00:32:34.495 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:32:34.736 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:32:35.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:35 smithi121 ceph-mon[32378]: pgmap v141: 129 pgs: 129 active+clean; 813 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.7 MiB/s wr, 631 op/s 2024-06-10T00:32:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:35 smithi193 ceph-mon[36716]: pgmap v141: 129 pgs: 129 active+clean; 813 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.7 MiB/s wr, 631 op/s 2024-06-10T00:32:35.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:35 smithi135 ceph-mon[36424]: pgmap v141: 129 pgs: 129 active+clean; 813 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 5.7 MiB/s wr, 631 op/s 2024-06-10T00:32:37.164 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.163+0000 7f1024bf0700 1 -- 172.21.15.121:0/3066031085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1020107840 msgr2=0x7f102010f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:37.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.163+0000 7f1024bf0700 1 --2- 172.21.15.121:0/3066031085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1020107840 0x7f102010f3c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f101800b0d0 tx=0x7f101800b3e0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.163+0000 7f1024bf0700 1 -- 172.21.15.121:0/3066031085 shutdown_connections 2024-06-10T00:32:37.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.163+0000 7f1024bf0700 1 --2- 172.21.15.121:0/3066031085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1020107840 0x7f102010f3c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.163+0000 7f1024bf0700 1 --2- 172.21.15.121:0/3066031085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1020102740 0x7f1020102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.163+0000 7f1024bf0700 1 --2- 172.21.15.121:0/3066031085 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10201030f0 0x7f1020107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.163+0000 7f1024bf0700 1 -- 172.21.15.121:0/3066031085 >> 172.21.15.121:0/3066031085 conn(0x7f10200795f0 msgr2=0x7f1020079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:37.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.163+0000 7f1024bf0700 1 -- 172.21.15.121:0/3066031085 shutdown_connections 2024-06-10T00:32:37.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.163+0000 7f1024bf0700 1 -- 172.21.15.121:0/3066031085 wait complete. 2024-06-10T00:32:37.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f1024bf0700 1 Processor -- start 2024-06-10T00:32:37.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f1024bf0700 1 -- start start 2024-06-10T00:32:37.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f1024bf0700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1020102740 0x7f1020074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f1024bf0700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10201030f0 0x7f1020074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f1024bf0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1020107840 0x7f1020075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f1024bf0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1020112780 con 0x7f1020107840 2024-06-10T00:32:37.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f1024bf0700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f1020112600 con 0x7f10201030f0 2024-06-10T00:32:37.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f1024bf0700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1020112900 con 0x7f1020102740 2024-06-10T00:32:37.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f101dd9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10201030f0 0x7f1020074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f101dd9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10201030f0 0x7f1020074cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:39068/0 (socket says 172.21.15.121:39068) 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f101ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1020107840 0x7f1020075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f101dd9b700 1 -- 172.21.15.121:0/3761805424 learned_addr learned my addr 172.21.15.121:0/3761805424 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.164+0000 7f101e59c700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1020102740 0x7f1020074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f101dd9b700 1 -- 172.21.15.121:0/3761805424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1020102740 msgr2=0x7f1020074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f101dd9b700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1020102740 0x7f1020074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f101dd9b700 1 -- 172.21.15.121:0/3761805424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1020107840 msgr2=0x7f1020075210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f101dd9b700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1020107840 0x7f1020075210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f101dd9b700 1 -- 172.21.15.121:0/3761805424 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10201a70a0 con 0x7f10201030f0 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f101ed9d700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1020107840 0x7f1020075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f101dd9b700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10201030f0 0x7f1020074cd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f101000e9a0 tx=0x7f101000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:37.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f10177fe700 1 -- 172.21.15.121:0/3761805424 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101000ca10 con 0x7f10201030f0 2024-06-10T00:32:37.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f1024bf0700 1 -- 172.21.15.121:0/3761805424 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10201a7390 con 0x7f10201030f0 2024-06-10T00:32:37.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f10177fe700 1 -- 172.21.15.121:0/3761805424 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f101000cbb0 con 0x7f10201030f0 2024-06-10T00:32:37.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f10177fe700 1 -- 172.21.15.121:0/3761805424 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010018680 con 0x7f10201030f0 2024-06-10T00:32:37.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.165+0000 7f1024bf0700 1 -- 172.21.15.121:0/3761805424 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10201a7920 con 0x7f10201030f0 2024-06-10T00:32:37.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.168+0000 7f10177fe700 1 -- 172.21.15.121:0/3761805424 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1010010430 con 0x7f10201030f0 2024-06-10T00:32:37.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.169+0000 7f10177fe700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f100c0a2d90 0x7f100c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.169+0000 7f10177fe700 1 -- 172.21.15.121:0/3761805424 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f10100cf990 con 0x7f10201030f0 2024-06-10T00:32:37.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.169+0000 7f101e59c700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f100c0a2d90 0x7f100c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.169+0000 7f102f473b80 1 Processor -- start 2024-06-10T00:32:37.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.169+0000 7f101e59c700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f100c0a2d90 0x7f100c0a5250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1008005950 tx=0x7f100800a300 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:37.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.169+0000 7f102f473b80 1 -- start start 2024-06-10T00:32:37.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f102f473b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449c640 0x561f0449ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f102f473b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449cf60 0x561f044969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f102f473b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f04497340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f102f473b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561f043e8810 con 0x561f0449c640 2024-06-10T00:32:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f102f473b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x561f04390720 con 0x561f0449cf60 2024-06-10T00:32:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f102f473b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561f043e47d0 con 0x561f04496f20 2024-06-10T00:32:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449c640 0x561f0449ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449c640 0x561f0449ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36148/0 (socket says 172.21.15.121:36148) 2024-06-10T00:32:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101dd9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449cf60 0x561f044969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101e59c700 1 -- 172.21.15.121:0/2395349097 learned_addr learned my addr 172.21.15.121:0/2395349097 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101ed9d700 1 --2- 172.21.15.121:0/2395349097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f04497340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101dd9b700 1 -- 172.21.15.121:0/2395349097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 msgr2=0x561f04497340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101dd9b700 1 --2- 172.21.15.121:0/2395349097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f04497340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101dd9b700 1 -- 172.21.15.121:0/2395349097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449c640 msgr2=0x561f0449ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101dd9b700 1 --2- 172.21.15.121:0/2395349097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449c640 0x561f0449ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101dd9b700 1 -- 172.21.15.121:0/2395349097 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f04386f50 con 0x561f0449cf60 2024-06-10T00:32:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101e59c700 1 --2- 172.21.15.121:0/2395349097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449c640 0x561f0449ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101ed9d700 1 --2- 172.21.15.121:0/2395349097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f04497340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:32:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.170+0000 7f101dd9b700 1 --2- 172.21.15.121:0/2395349097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449cf60 0x561f044969e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f10100ccd50 tx=0x7f101009bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3649f1a48807c209 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f1014ff9700 1 -- 172.21.15.121:0/2395349097 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10100a4790 con 0x561f0449cf60 2024-06-10T00:32:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f1014ff9700 1 -- 172.21.15.121:0/2395349097 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f101009ce50 con 0x561f0449cf60 2024-06-10T00:32:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f1014ff9700 1 -- 172.21.15.121:0/2395349097 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010010720 con 0x561f0449cf60 2024-06-10T00:32:37.180 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f102f473b80 1 -- 172.21.15.121:0/2395349097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449cf60 msgr2=0x561f044969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f102f473b80 1 --2- 172.21.15.121:0/2395349097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449cf60 0x561f044969e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f10100ccd50 tx=0x7f101009bb20 comp rx=0 tx=0).stop 2024-06-10T00:32:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f102f473b80 1 -- 172.21.15.121:0/2395349097 shutdown_connections 2024-06-10T00:32:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f102f473b80 1 --2- 172.21.15.121:0/2395349097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f04497340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f102f473b80 1 --2- 172.21.15.121:0/2395349097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449c640 0x561f0449ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f102f473b80 1 --2- 172.21.15.121:0/2395349097 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449cf60 0x561f044969e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f102f473b80 1 -- 172.21.15.121:0/2395349097 >> 172.21.15.121:0/2395349097 conn(0x561f043c1cc0 msgr2=0x561f04491850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f102f473b80 1 -- 172.21.15.121:0/2395349097 shutdown_connections 2024-06-10T00:32:37.181 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.171+0000 7f102f473b80 1 -- 172.21.15.121:0/2395349097 wait complete. 2024-06-10T00:32:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f102f473b80 1 -- start start 2024-06-10T00:32:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f102f473b80 1 Processor -- start 2024-06-10T00:32:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f102f473b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f044a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f102f473b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 0x561f044a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f102f473b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044a0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f102f473b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561f044a39d0 con 0x561f0449cf60 2024-06-10T00:32:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f102f473b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x561f04388bf0 con 0x561f0449c640 2024-06-10T00:32:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f102f473b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561f044a3b50 con 0x561f04496f20 2024-06-10T00:32:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f101ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.182 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f102f473b80 10 client.0 _ll_register_callbacks cb 0x561f04382920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f101ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36156/0 (socket says 172.21.15.121:36156) 2024-06-10T00:32:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f101e59c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f044a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f101ed9d700 1 -- 172.21.15.121:0/4072935437 learned_addr learned my addr 172.21.15.121:0/4072935437 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.172+0000 7f101dd9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 0x561f044a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f101e59c700 1 -- 172.21.15.121:0/4072935437 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 msgr2=0x561f044a00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f101e59c700 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 0x561f044a00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f101e59c700 1 -- 172.21.15.121:0/4072935437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 msgr2=0x561f044a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f101e59c700 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f101e59c700 1 -- 172.21.15.121:0/4072935437 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f043378d0 con 0x561f04496f20 2024-06-10T00:32:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f101ed9d700 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:37.183 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f101e59c700 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f044a2de0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f100800f750 tx=0x7f10080047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=199f98bd934b7b36 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f1005ffb700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f1005ffb700 1 -- 172.21.15.121:0/4072935437 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008016070 con 0x561f04496f20 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f102f473b80 1 -- 172.21.15.121:0/4072935437 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f101009ce50 con 0x561f04496f20 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f1005ffb700 1 -- 172.21.15.121:0/4072935437 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f1008004a30 con 0x561f04496f20 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f1005ffb700 1 -- 172.21.15.121:0/4072935437 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1008004c30 con 0x561f04496f20 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f102f473b80 10 client.25618 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f102f473b80 1 -- 172.21.15.121:0/4072935437 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561f044ae490 con 0x561f04496f20 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.173+0000 7f1005ffb700 1 -- 172.21.15.121:0/4072935437 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f101009ce50 con 0x561f04496f20 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.174+0000 7f1005ffb700 1 -- 172.21.15.121:0/4072935437 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1008010a90 con 0x561f04496f20 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.174+0000 7f102f473b80 10 client.25618 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.174+0000 7f102f473b80 10 client.25618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.174+0000 7f102f473b80 20 client.25618 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:37.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.174+0000 7f102f473b80 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561f044b0a70 0x561f044b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.174+0000 7f102f473b80 4 client.25618 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:32:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.174+0000 7f102f473b80 1 -- 172.21.15.121:0/4072935437 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f043f5d10 con 0x561f044b0a70 2024-06-10T00:32:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.174+0000 7f101dd9b700 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561f044b0a70 0x561f044b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.174+0000 7f1005ffb700 10 client.25618 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.174+0000 7f101dd9b700 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561f044b0a70 0x561f044b2e80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddc2e690d6564ce0 server_cookie=a25a906046118f79 in_seq=0 out_seq=0 2024-06-10T00:32:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f1005ffb700 1 -- 172.21.15.121:0/4072935437 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x561f043f5d10 con 0x561f044b0a70 2024-06-10T00:32:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f1005ffb700 10 client.25618 handle_command_reply: tid=0 2024-06-10T00:32:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 1 client.25618 shutdown 2024-06-10T00:32:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 1 -- 172.21.15.121:0/4072935437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 msgr2=0x561f044a2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f044a2de0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f100800f750 tx=0x7f10080047c0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.185 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 1 -- 172.21.15.121:0/4072935437 shutdown_connections 2024-06-10T00:32:37.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561f044b0a70 0x561f044b2e80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f1005ffb700 0 client.25618 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:37.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f044a2de0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 1 --2- 172.21.15.121:0/4072935437 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 0x561f044a00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.186 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 1 -- 172.21.15.121:0/4072935437 >> 172.21.15.121:0/4072935437 conn(0x561f043c1cc0 msgr2=0x561f044959d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:37.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 1 -- 172.21.15.121:0/4072935437 shutdown_connections 2024-06-10T00:32:37.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 1 -- 172.21.15.121:0/4072935437 wait complete. 2024-06-10T00:32:37.187 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.175+0000 7f102f473b80 20 client.25618 trim_cache size 0 max 16384 2024-06-10T00:32:37.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.195+0000 7f102f473b80 1 Processor -- start 2024-06-10T00:32:37.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.195+0000 7f102f473b80 1 -- start start 2024-06-10T00:32:37.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.195+0000 7f102f473b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f04496f20 0x561f044969e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f102f473b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f0449c640 0x561f044acc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f102f473b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449cf60 0x561f0448b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f102f473b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f10201b3100 con 0x561f04496f20 2024-06-10T00:32:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f102f473b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f10201b0990 con 0x561f0449cf60 2024-06-10T00:32:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f102f473b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f10201ae7e0 con 0x561f0449c640 2024-06-10T00:32:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f04496f20 0x561f044969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f04496f20 0x561f044969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36172/0 (socket says 172.21.15.121:36172) 2024-06-10T00:32:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101e59c700 1 -- 172.21.15.121:0/2333049640 learned_addr learned my addr 172.21.15.121:0/2333049640 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101ed9d700 1 --2- 172.21.15.121:0/2333049640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449cf60 0x561f0448b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101dd9b700 1 --2- 172.21.15.121:0/2333049640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f0449c640 0x561f044acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101e59c700 1 -- 172.21.15.121:0/2333049640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f0449c640 msgr2=0x561f044acc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101e59c700 1 --2- 172.21.15.121:0/2333049640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f0449c640 0x561f044acc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101e59c700 1 -- 172.21.15.121:0/2333049640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449cf60 msgr2=0x561f0448b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101e59c700 1 --2- 172.21.15.121:0/2333049640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449cf60 0x561f0448b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101e59c700 1 -- 172.21.15.121:0/2333049640 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f04493080 con 0x561f04496f20 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101dd9b700 1 --2- 172.21.15.121:0/2333049640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f0449c640 0x561f044acc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f101e59c700 1 --2- 172.21.15.121:0/2333049640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f04496f20 0x561f044969e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1008004790 tx=0x7f100800f780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed1691bf1541422 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f1007fff700 1 -- 172.21.15.121:0/2333049640 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008039070 con 0x561f04496f20 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f1007fff700 1 -- 172.21.15.121:0/2333049640 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1008010ba0 con 0x561f04496f20 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.196+0000 7f1007fff700 1 -- 172.21.15.121:0/2333049640 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008004dd0 con 0x561f04496f20 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.197+0000 7f102f473b80 1 -- 172.21.15.121:0/2333049640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f04496f20 msgr2=0x561f044969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.197+0000 7f102f473b80 1 --2- 172.21.15.121:0/2333049640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f04496f20 0x561f044969e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1008004790 tx=0x7f100800f780 comp rx=0 tx=0).stop 2024-06-10T00:32:37.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.197+0000 7f102f473b80 1 -- 172.21.15.121:0/2333049640 shutdown_connections 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.197+0000 7f102f473b80 1 --2- 172.21.15.121:0/2333049640 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449cf60 0x561f0448b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.197+0000 7f102f473b80 1 --2- 172.21.15.121:0/2333049640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f04496f20 0x561f044969e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.197+0000 7f102f473b80 1 --2- 172.21.15.121:0/2333049640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f0449c640 0x561f044acc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.197+0000 7f102f473b80 1 -- 172.21.15.121:0/2333049640 >> 172.21.15.121:0/2333049640 conn(0x561f043c1cc0 msgr2=0x561f0449d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.197+0000 7f102f473b80 1 -- 172.21.15.121:0/2333049640 shutdown_connections 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.197+0000 7f102f473b80 1 -- 172.21.15.121:0/2333049640 wait complete. 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.197+0000 7f102f473b80 1 -- start start 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.197+0000 7f102f473b80 1 Processor -- start 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f102f473b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f044a9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f102f473b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 0x561f044a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f102f473b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044b7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f102f473b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f10201aee50 con 0x561f0449cf60 2024-06-10T00:32:37.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f102f473b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f10201b02c0 con 0x561f0449c640 2024-06-10T00:32:37.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f102f473b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f10201af520 con 0x561f04496f20 2024-06-10T00:32:37.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f102f473b80 10 client.0 _ll_register_callbacks cb 0x561f04382920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:37.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044b7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101dd9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 0x561f044a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101dd9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 0x561f044a96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:39112/0 (socket says 172.21.15.121:39112) 2024-06-10T00:32:37.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044b7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36174/0 (socket says 172.21.15.121:36174) 2024-06-10T00:32:37.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101dd9b700 1 -- 172.21.15.121:0/3521806679 learned_addr learned my addr 172.21.15.121:0/3521806679 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:37.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101e59c700 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f044a9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101dd9b700 1 -- 172.21.15.121:0/3521806679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 msgr2=0x561f044a9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:37.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101dd9b700 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f044a9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101dd9b700 1 -- 172.21.15.121:0/3521806679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 msgr2=0x561f044b7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:37.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101dd9b700 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044b7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101dd9b700 1 -- 172.21.15.121:0/3521806679 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f044b8710 con 0x561f0449c640 2024-06-10T00:32:37.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101ed9d700 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044b7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:37.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f101dd9b700 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 0x561f044a96c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f101000c2d0 tx=0x7f101009bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8b263c8aca37dbf server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:37.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f1005ffb700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:32:37.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f1005ffb700 1 -- 172.21.15.121:0/3521806679 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10100b8070 con 0x561f0449c640 2024-06-10T00:32:37.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f102f473b80 1 -- 172.21.15.121:0/3521806679 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1008010ba0 con 0x561f0449c640 2024-06-10T00:32:37.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.198+0000 7f1005ffb700 1 -- 172.21.15.121:0/3521806679 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f10100ba300 con 0x561f0449c640 2024-06-10T00:32:37.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f1005ffb700 1 -- 172.21.15.121:0/3521806679 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f101009ce50 con 0x561f0449c640 2024-06-10T00:32:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f102f473b80 10 client.25712 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f102f473b80 1 -- 172.21.15.121:0/3521806679 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561f0451b390 con 0x561f0449c640 2024-06-10T00:32:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f1005ffb700 1 -- 172.21.15.121:0/3521806679 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1008010ba0 con 0x561f0449c640 2024-06-10T00:32:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f1005ffb700 1 -- 172.21.15.121:0/3521806679 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f101009bd80 con 0x561f0449c640 2024-06-10T00:32:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f102f473b80 10 client.25712 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f102f473b80 10 client.25712 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:32:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f102f473b80 20 client.25712 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f102f473b80 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561f0451b770 0x561f0451db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f102f473b80 4 client.25712 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:32:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f102f473b80 1 -- 172.21.15.121:0/3521806679 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561f043f0290 con 0x561f0451b770 2024-06-10T00:32:37.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.199+0000 7f101e59c700 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561f0451b770 0x561f0451db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:37.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f1005ffb700 10 client.25712 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:37.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f101e59c700 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561f0451b770 0x561f0451db60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f216886b825be05 server_cookie=a9f079c22dd43395 in_seq=0 out_seq=0 2024-06-10T00:32:37.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f1005ffb700 1 -- 172.21.15.121:0/3521806679 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561f043f0290 con 0x561f0451b770 2024-06-10T00:32:37.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f1005ffb700 10 client.25712 handle_command_reply: tid=0 2024-06-10T00:32:37.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 1 client.25712 shutdown 2024-06-10T00:32:37.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 1 -- 172.21.15.121:0/3521806679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 msgr2=0x561f044a96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:37.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 0x561f044a96c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f101000c2d0 tx=0x7f101009bb20 comp rx=0 tx=0).stop 2024-06-10T00:32:37.209 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 1 -- 172.21.15.121:0/3521806679 shutdown_connections 2024-06-10T00:32:37.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x561f0451b770 0x561f0451db60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f0449cf60 0x561f044b7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f1005ffb700 0 client.25712 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:37.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f04496f20 0x561f044a9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 1 --2- 172.21.15.121:0/3521806679 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561f0449c640 0x561f044a96c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 1 -- 172.21.15.121:0/3521806679 >> 172.21.15.121:0/3521806679 conn(0x561f043c1cc0 msgr2=0x561f0449dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:37.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 1 -- 172.21.15.121:0/3521806679 shutdown_connections 2024-06-10T00:32:37.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 1 -- 172.21.15.121:0/3521806679 wait complete. 2024-06-10T00:32:37.210 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f102f473b80 20 client.25712 trim_cache size 0 max 16384 2024-06-10T00:32:37.211 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:32:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "no active scrubs running", 2024-06-10T00:32:37.211 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": {} 2024-06-10T00:32:37.211 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:32:37.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f1024bf0700 1 -- 172.21.15.121:0/3761805424 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f100c0a2d90 msgr2=0x7f100c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:37.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.200+0000 7f1024bf0700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f100c0a2d90 0x7f100c0a5250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1008005950 tx=0x7f100800a300 comp rx=0 tx=0).stop 2024-06-10T00:32:37.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.201+0000 7f1024bf0700 1 -- 172.21.15.121:0/3761805424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10201030f0 msgr2=0x7f1020074cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:37.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.201+0000 7f1024bf0700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10201030f0 0x7f1020074cd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f101000e9a0 tx=0x7f101000ecb0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.201+0000 7f1024bf0700 1 -- 172.21.15.121:0/3761805424 shutdown_connections 2024-06-10T00:32:37.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.201+0000 7f1024bf0700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f100c0a2d90 0x7f100c0a5250 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.201+0000 7f1024bf0700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1020107840 0x7f1020075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.201+0000 7f1024bf0700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1020102740 0x7f1020074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.212 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.201+0000 7f1024bf0700 1 --2- 172.21.15.121:0/3761805424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10201030f0 0x7f1020074cd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:37.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.201+0000 7f1024bf0700 1 -- 172.21.15.121:0/3761805424 >> 172.21.15.121:0/3761805424 conn(0x7f10200795f0 msgr2=0x7f10201006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:37.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.201+0000 7f1024bf0700 1 -- 172.21.15.121:0/3761805424 shutdown_connections 2024-06-10T00:32:37.213 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:37.201+0000 7f1024bf0700 1 -- 172.21.15.121:0/3761805424 wait complete. 2024-06-10T00:32:37.295 INFO:tasks.workunit.client.0.smithi121.stdout: 197 545 0 403 0 156 0 2024-06-10T00:32:37.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:37 smithi121 ceph-mon[32378]: pgmap v142: 129 pgs: 129 active+clean; 823 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.2 MiB/s wr, 688 op/s 2024-06-10T00:32:37.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:37 smithi193 ceph-mon[36716]: pgmap v142: 129 pgs: 129 active+clean; 823 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.2 MiB/s wr, 688 op/s 2024-06-10T00:32:37.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:37 smithi135 ceph-mon[36424]: pgmap v142: 129 pgs: 129 active+clean; 823 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 6.2 MiB/s wr, 688 op/s 2024-06-10T00:32:38.052 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-10T00:32:38.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:32:38.405 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:32:38.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f22537fe700 1 -- 172.21.15.121:0/2606255137 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f226c0094b0 con 0x55af4a06ece0 2024-06-10T00:32:38.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f22537fe700 10 client.16134 handle_command_reply: tid=0 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 1 client.16134 shutdown 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2606255137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 msgr2=0x55af49fda630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fda630 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f226c0177a0 tx=0x7f226c017d40 comp rx=0 tx=0).stop 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2606255137 shutdown_connections 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55af4a06ece0 0x55af4a0711a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55af49fd2040 0x55af49fcbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55af49fc6870 0x55af49fda630 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f22537fe700 0 client.16134 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 1 --2- 172.21.15.121:0/2606255137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55af49fc7190 0x55af49fcb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2606255137 >> 172.21.15.121:0/2606255137 conn(0x55af49ef4210 msgr2=0x55af49fceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2606255137 shutdown_connections 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 1 -- 172.21.15.121:0/2606255137 wait complete. 2024-06-10T00:32:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.655+0000 7f2284bc8b80 20 client.16134 trim_cache size 0 max 16384 2024-06-10T00:32:38.658 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:32:38.658 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 28, 2024-06-10T00:32:38.658 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:32:38.658 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 10, 2024-06-10T00:32:38.658 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:32:38.658 INFO:teuthology.orchestra.run.smithi121.stdout: "2f9a24d7": { 2024-06-10T00:32:38.658 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 10, 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 138.2, 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:32:38.659 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:32:38.660 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:38.660 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:38.660 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:38.660 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:38.660 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:38.660 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:32:38.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 -- 172.21.15.121:0/1239331518 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f22600a3060 msgr2=0x7f22600a5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f22600a3060 0x7f22600a5520 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f226c00f540 tx=0x7f226c00b040 comp rx=0 tx=0).stop 2024-06-10T00:32:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 -- 172.21.15.121:0/1239331518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2274075150 msgr2=0x7f2274071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2274075150 0x7f2274071210 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f226400dcc0 tx=0x7f2264009520 comp rx=0 tx=0).stop 2024-06-10T00:32:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 -- 172.21.15.121:0/1239331518 shutdown_connections 2024-06-10T00:32:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f22600a3060 0x7f22600a5520 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22741132c0 0x7f2274071750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2274074830 0x7f2274074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 --2- 172.21.15.121:0/1239331518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2274075150 0x7f2274071210 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 -- 172.21.15.121:0/1239331518 >> 172.21.15.121:0/1239331518 conn(0x7f22740fde00 msgr2=0x7f22740fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 -- 172.21.15.121:0/1239331518 shutdown_connections 2024-06-10T00:32:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:38.656+0000 7f22790e3700 1 -- 172.21.15.121:0/1239331518 wait complete. 2024-06-10T00:32:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:39 smithi193 ceph-mon[36716]: pgmap v143: 129 pgs: 129 active+clean; 823 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 655 op/s 2024-06-10T00:32:39.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:39 smithi121 ceph-mon[32378]: pgmap v143: 129 pgs: 129 active+clean; 823 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 655 op/s 2024-06-10T00:32:39.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:39 smithi135 ceph-mon[36424]: pgmap v143: 129 pgs: 129 active+clean; 823 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.6 MiB/s wr, 655 op/s 2024-06-10T00:32:40.340 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '2f9a24d7', quiesce took 34.1 seconds. Will release after: -15.900000000000002 2024-06-10T00:32:40.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:32:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:41 smithi193 ceph-mon[36716]: pgmap v144: 129 pgs: 129 active+clean; 874 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.9 MiB/s wr, 774 op/s 2024-06-10T00:32:41.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:41 smithi121 ceph-mon[32378]: pgmap v144: 129 pgs: 129 active+clean; 874 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.9 MiB/s wr, 774 op/s 2024-06-10T00:32:41.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:41 smithi135 ceph-mon[36424]: pgmap v144: 129 pgs: 129 active+clean; 874 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 9.9 MiB/s wr, 774 op/s 2024-06-10T00:32:42.237 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:32:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:43 smithi193 ceph-mon[36716]: pgmap v145: 129 pgs: 129 active+clean; 874 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.6 MiB/s wr, 390 op/s 2024-06-10T00:32:43.632 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:43 smithi121 ceph-mon[32378]: pgmap v145: 129 pgs: 129 active+clean; 874 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.6 MiB/s wr, 390 op/s 2024-06-10T00:32:43.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.656+0000 7fa2c96ca700 1 -- 172.21.15.121:0/3509831033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2c410eac0 msgr2=0x7fa2c410eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:43.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.656+0000 7fa2c96ca700 1 --2- 172.21.15.121:0/3509831033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2c410eac0 0x7fa2c410eea0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40098f0 tx=0x7fa2b4009c00 comp rx=0 tx=0).stop 2024-06-10T00:32:43.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.657+0000 7fa2c96ca700 1 -- 172.21.15.121:0/3509831033 shutdown_connections 2024-06-10T00:32:43.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.657+0000 7fa2c96ca700 1 --2- 172.21.15.121:0/3509831033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2c4109480 0x7fa2c4115990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:43.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.657+0000 7fa2c96ca700 1 --2- 172.21.15.121:0/3509831033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2c410eac0 0x7fa2c410eea0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:43.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.657+0000 7fa2c96ca700 1 --2- 172.21.15.121:0/3509831033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2c4108ac0 0x7fa2c4108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:43.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.657+0000 7fa2c96ca700 1 -- 172.21.15.121:0/3509831033 >> 172.21.15.121:0/3509831033 conn(0x7fa2c4075c60 msgr2=0x7fa2c41072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:43.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.657+0000 7fa2c96ca700 1 -- 172.21.15.121:0/3509831033 shutdown_connections 2024-06-10T00:32:43.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.657+0000 7fa2c96ca700 1 -- 172.21.15.121:0/3509831033 wait complete. 2024-06-10T00:32:43.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.657+0000 7fa2c96ca700 1 Processor -- start 2024-06-10T00:32:43.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c96ca700 1 -- start start 2024-06-10T00:32:43.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c96ca700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2c4108ac0 0x7fa2c419f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:43.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c96ca700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2c4109480 0x7fa2c419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:43.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c96ca700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2c410eac0 0x7fa2c41a3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:43.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c96ca700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c4118e10 con 0x7fa2c410eac0 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c96ca700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c4118c90 con 0x7fa2c4109480 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c96ca700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa2c4118f90 con 0x7fa2c4108ac0 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c2ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2c4108ac0 0x7fa2c419f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c2ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2c4108ac0 0x7fa2c419f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:50660/0 (socket says 172.21.15.121:50660) 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c2ffd700 1 -- 172.21.15.121:0/1063538536 learned_addr learned my addr 172.21.15.121:0/1063538536 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c27fc700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2c4109480 0x7fa2c419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c37fe700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2c410eac0 0x7fa2c41a3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c2ffd700 1 -- 172.21.15.121:0/1063538536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2c4109480 msgr2=0x7fa2c419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c2ffd700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2c4109480 0x7fa2c419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c2ffd700 1 -- 172.21.15.121:0/1063538536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2c410eac0 msgr2=0x7fa2c41a3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c2ffd700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2c410eac0 0x7fa2c41a3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.658+0000 7fa2c2ffd700 1 -- 172.21.15.121:0/1063538536 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2c41a42d0 con 0x7fa2c4108ac0 2024-06-10T00:32:43.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.659+0000 7fa2c2ffd700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2c4108ac0 0x7fa2c419f2c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009230 tx=0x7fa2b400fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:43.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.659+0000 7fa2abfff700 1 -- 172.21.15.121:0/1063538536 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4005180 con 0x7fa2c4108ac0 2024-06-10T00:32:43.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.659+0000 7fa2c96ca700 1 -- 172.21.15.121:0/1063538536 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2c41a4560 con 0x7fa2c4108ac0 2024-06-10T00:32:43.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.659+0000 7fa2abfff700 1 -- 172.21.15.121:0/1063538536 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2b400b560 con 0x7fa2c4108ac0 2024-06-10T00:32:43.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.659+0000 7fa2abfff700 1 -- 172.21.15.121:0/1063538536 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4003a70 con 0x7fa2c4108ac0 2024-06-10T00:32:43.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.659+0000 7fa2c96ca700 1 -- 172.21.15.121:0/1063538536 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2c41abe40 con 0x7fa2c4108ac0 2024-06-10T00:32:43.662 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.660+0000 7fa2c96ca700 1 -- 172.21.15.121:0/1063538536 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2c410a200 con 0x7fa2c4108ac0 2024-06-10T00:32:43.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.661+0000 7fa2abfff700 1 -- 172.21.15.121:0/1063538536 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa2b4029060 con 0x7fa2c4108ac0 2024-06-10T00:32:43.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.662+0000 7fa2abfff700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa2ac0a2d90 0x7fa2ac0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:43.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.662+0000 7fa2abfff700 1 -- 172.21.15.121:0/1063538536 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fa2b40d0050 con 0x7fa2c4108ac0 2024-06-10T00:32:43.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.662+0000 7fa2c27fc700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa2ac0a2d90 0x7fa2ac0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:43.663 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.663+0000 7fa2c27fc700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa2ac0a2d90 0x7fa2ac0a5250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4077bd0 tx=0x7fa2b8009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:43.668 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.668+0000 7fa2abfff700 1 -- 172.21.15.121:0/1063538536 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa2b4013030 con 0x7fa2c4108ac0 2024-06-10T00:32:43.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:43 smithi135 ceph-mon[36424]: pgmap v145: 129 pgs: 129 active+clean; 874 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.6 MiB/s wr, 390 op/s 2024-06-10T00:32:43.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.953+0000 7fa2c96ca700 1 -- 172.21.15.121:0/1063538536 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2c41a0620 con 0x7fa2c4108ac0 2024-06-10T00:32:43.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.953+0000 7fa2abfff700 1 -- 172.21.15.121:0/1063538536 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11888 (secure 0 0 0) 0x7fa2b40bae20 con 0x7fa2c4108ac0 2024-06-10T00:32:43.955 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:32:43.955 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:30:26.280580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 -- 172.21.15.121:0/1063538536 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa2ac0a2d90 msgr2=0x7fa2ac0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa2ac0a2d90 0x7fa2ac0a5250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4077bd0 tx=0x7fa2b8009260 comp rx=0 tx=0).stop 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 -- 172.21.15.121:0/1063538536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2c4108ac0 msgr2=0x7fa2c419f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2c4108ac0 0x7fa2c419f2c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009230 tx=0x7fa2b400fa70 comp rx=0 tx=0).stop 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 -- 172.21.15.121:0/1063538536 shutdown_connections 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa2ac0a2d90 0x7fa2ac0a5250 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2c410eac0 0x7fa2c41a3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2c4108ac0 0x7fa2c419f2c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 --2- 172.21.15.121:0/1063538536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2c4109480 0x7fa2c419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 -- 172.21.15.121:0/1063538536 >> 172.21.15.121:0/1063538536 conn(0x7fa2c4075c60 msgr2=0x7fa2c4076280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 -- 172.21.15.121:0/1063538536 shutdown_connections 2024-06-10T00:32:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:43.956+0000 7fa2c96ca700 1 -- 172.21.15.121:0/1063538536 wait complete. 2024-06-10T00:32:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 28 2024-06-10T00:32:44.609 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:44 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1063538536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:32:44.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:44 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1063538536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:32:44.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:44 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1063538536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:32:45.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.184+0000 7f40b7f1a700 1 -- 172.21.15.121:0/541020153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b0101440 msgr2=0x7f40b010f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:45.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.184+0000 7f40b7f1a700 1 --2- 172.21.15.121:0/541020153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b0101440 0x7f40b010f330 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f40a800b0a0 tx=0x7f40a800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:32:45.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.184+0000 7f40b7f1a700 1 -- 172.21.15.121:0/541020153 shutdown_connections 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.184+0000 7f40b7f1a700 1 --2- 172.21.15.121:0/541020153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b0101440 0x7f40b010f330 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.184+0000 7f40b7f1a700 1 --2- 172.21.15.121:0/541020153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40b0108510 0x7f40b01088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.184+0000 7f40b7f1a700 1 --2- 172.21.15.121:0/541020153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40b0100a80 0x7f40b0100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.184+0000 7f40b7f1a700 1 -- 172.21.15.121:0/541020153 >> 172.21.15.121:0/541020153 conn(0x7f40b0077a00 msgr2=0x7f40b00fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.185+0000 7f40b7f1a700 1 -- 172.21.15.121:0/541020153 shutdown_connections 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.185+0000 7f40b7f1a700 1 -- 172.21.15.121:0/541020153 wait complete. 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.185+0000 7f40b7f1a700 1 Processor -- start 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.185+0000 7f40b7f1a700 1 -- start start 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b7f1a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40b0100a80 0x7f40b0101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b7f1a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40b0101440 0x7f40b01021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b7f1a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b0108510 0x7f40b01026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b7f1a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f40b0112790 con 0x7f40b0108510 2024-06-10T00:32:45.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b7f1a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f40b0112610 con 0x7f40b0100a80 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b7f1a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f40b0112910 con 0x7f40b0101440 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b64b7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b0108510 0x7f40b01026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b54b5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40b0101440 0x7f40b01021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b5cb6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40b0100a80 0x7f40b0101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b64b7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b0108510 0x7f40b01026e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54530/0 (socket says 172.21.15.121:54530) 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b54b5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40b0101440 0x7f40b01021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:50670/0 (socket says 172.21.15.121:50670) 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b64b7700 1 -- 172.21.15.121:0/3147175835 learned_addr learned my addr 172.21.15.121:0/3147175835 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b64b7700 1 -- 172.21.15.121:0/3147175835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40b0101440 msgr2=0x7f40b01021a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b64b7700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40b0101440 0x7f40b01021a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b64b7700 1 -- 172.21.15.121:0/3147175835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40b0100a80 msgr2=0x7f40b0101c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b64b7700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40b0100a80 0x7f40b0101c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b64b7700 1 -- 172.21.15.121:0/3147175835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b01ab510 con 0x7f40b0108510 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40b64b7700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b0108510 0x7f40b01026e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f40a800b780 tx=0x7f40a8004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:45.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.186+0000 7f40a6ffd700 1 -- 172.21.15.121:0/3147175835 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a8021070 con 0x7f40b0108510 2024-06-10T00:32:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.187+0000 7f40b7f1a700 1 -- 172.21.15.121:0/3147175835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b01ab740 con 0x7f40b0108510 2024-06-10T00:32:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.187+0000 7f40b7f1a700 1 -- 172.21.15.121:0/3147175835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b01abca0 con 0x7f40b0108510 2024-06-10T00:32:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.187+0000 7f40a6ffd700 1 -- 172.21.15.121:0/3147175835 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40a8007b80 con 0x7f40b0108510 2024-06-10T00:32:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.187+0000 7f40a6ffd700 1 -- 172.21.15.121:0/3147175835 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80052a0 con 0x7f40b0108510 2024-06-10T00:32:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.188+0000 7f40b7f1a700 1 -- 172.21.15.121:0/3147175835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40b0069180 con 0x7f40b0108510 2024-06-10T00:32:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.190+0000 7f40a6ffd700 1 -- 172.21.15.121:0/3147175835 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f40a8024070 con 0x7f40b0108510 2024-06-10T00:32:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.190+0000 7f40a6ffd700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f409c0a2e10 0x7f409c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.190+0000 7f40a6ffd700 1 -- 172.21.15.121:0/3147175835 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f40a800f020 con 0x7f40b0108510 2024-06-10T00:32:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.190+0000 7f40b5cb6700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f409c0a2e10 0x7f409c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.191+0000 7f40b5cb6700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f409c0a2e10 0x7f409c0a52d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f40ac009be0 tx=0x7f40ac009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:45.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.194+0000 7f40a6ffd700 1 -- 172.21.15.121:0/3147175835 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f40a802b020 con 0x7f40b0108510 2024-06-10T00:32:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:45 smithi121 ceph-mon[32378]: pgmap v146: 129 pgs: 129 active+clean; 897 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 9.5 MiB/s wr, 558 op/s 2024-06-10T00:32:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.474+0000 7f40b7f1a700 1 -- 172.21.15.121:0/3147175835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40b01abfc0 con 0x7f40b0108510 2024-06-10T00:32:45.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.475+0000 7f40a6ffd700 1 -- 172.21.15.121:0/3147175835 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11888 (secure 0 0 0) 0x7f40a8007d30 con 0x7f40b0108510 2024-06-10T00:32:45.476 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:32:45.476 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:30:26.280580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:32:45.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 -- 172.21.15.121:0/3147175835 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f409c0a2e10 msgr2=0x7f409c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:45.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f409c0a2e10 0x7f409c0a52d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f40ac009be0 tx=0x7f40ac009260 comp rx=0 tx=0).stop 2024-06-10T00:32:45.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 -- 172.21.15.121:0/3147175835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b0108510 msgr2=0x7f40b01026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:45.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b0108510 0x7f40b01026e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f40a800b780 tx=0x7f40a8004d00 comp rx=0 tx=0).stop 2024-06-10T00:32:45.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 -- 172.21.15.121:0/3147175835 shutdown_connections 2024-06-10T00:32:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f409c0a2e10 0x7f409c0a52d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b0108510 0x7f40b01026e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f40b0100a80 0x7f40b0101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 --2- 172.21.15.121:0/3147175835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40b0101440 0x7f40b01021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 -- 172.21.15.121:0/3147175835 >> 172.21.15.121:0/3147175835 conn(0x7f40b0077a00 msgr2=0x7f40b00fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 -- 172.21.15.121:0/3147175835 shutdown_connections 2024-06-10T00:32:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:45.477+0000 7f40b7f1a700 1 -- 172.21.15.121:0/3147175835 wait complete. 2024-06-10T00:32:45.479 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 28 2024-06-10T00:32:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:45 smithi193 ceph-mon[36716]: pgmap v146: 129 pgs: 129 active+clean; 897 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 9.5 MiB/s wr, 558 op/s 2024-06-10T00:32:45.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:45 smithi135 ceph-mon[36424]: pgmap v146: 129 pgs: 129 active+clean; 897 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 9.5 MiB/s wr, 558 op/s 2024-06-10T00:32:46.180 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-10T00:32:46.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:46 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3147175835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:32:46.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:46 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3147175835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:32:46.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:46 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3147175835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:32:46.790 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db --set-id 2f9a24d7 --release --await' 2024-06-10T00:32:46.790 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db --set-id 2f9a24d7 --release --await 2024-06-10T00:32:46.938 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:32:47.295 INFO:tasks.workunit.client.0.smithi121.stdout: 197 83134 0 58214 0 28947 0 2024-06-10T00:32:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:47 smithi193 ceph-mon[36716]: pgmap v147: 129 pgs: 129 active+clean; 901 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.6 MiB/s wr, 442 op/s 2024-06-10T00:32:47.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:47 smithi121 ceph-mon[32378]: pgmap v147: 129 pgs: 129 active+clean; 901 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.6 MiB/s wr, 442 op/s 2024-06-10T00:32:47.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:47 smithi135 ceph-mon[36424]: pgmap v147: 129 pgs: 129 active+clean; 901 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.6 MiB/s wr, 442 op/s 2024-06-10T00:32:48.282 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:32:49.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:49 smithi193 ceph-mon[36716]: pgmap v148: 129 pgs: 129 active+clean; 901 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.7 MiB/s wr, 343 op/s 2024-06-10T00:32:49.696 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:49 smithi121 ceph-mon[32378]: pgmap v148: 129 pgs: 129 active+clean; 901 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.7 MiB/s wr, 343 op/s 2024-06-10T00:32:49.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:49 smithi135 ceph-mon[36424]: pgmap v148: 129 pgs: 129 active+clean; 901 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.7 MiB/s wr, 343 op/s 2024-06-10T00:32:50.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.386+0000 7fddd229a700 1 -- 172.21.15.121:0/3245937625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddcc108500 msgr2=0x7fddcc1088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:50.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.386+0000 7fddd229a700 1 --2- 172.21.15.121:0/3245937625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddcc108500 0x7fddcc1088e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fddbc009990 tx=0x7fddbc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.386+0000 7fddd229a700 1 -- 172.21.15.121:0/3245937625 shutdown_connections 2024-06-10T00:32:50.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.386+0000 7fddd229a700 1 --2- 172.21.15.121:0/3245937625 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc06b580 0x7fddcc10f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.386+0000 7fddd229a700 1 --2- 172.21.15.121:0/3245937625 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddcc108500 0x7fddcc1088e0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.386+0000 7fddd229a700 1 --2- 172.21.15.121:0/3245937625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc06abc0 0x7fddcc06b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.386+0000 7fddd229a700 1 -- 172.21.15.121:0/3245937625 >> 172.21.15.121:0/3245937625 conn(0x7fddcc078340 msgr2=0x7fddcc078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:50.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.386+0000 7fddd229a700 1 -- 172.21.15.121:0/3245937625 shutdown_connections 2024-06-10T00:32:50.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.386+0000 7fddd229a700 1 -- 172.21.15.121:0/3245937625 wait complete. 2024-06-10T00:32:50.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddd229a700 1 Processor -- start 2024-06-10T00:32:50.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddd229a700 1 -- start start 2024-06-10T00:32:50.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddd229a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc06abc0 0x7fddcc198d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddd229a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddcc06b580 0x7fddcc199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddd229a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc108500 0x7fddcc19d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddd229a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1127e0 con 0x7fddcc06b580 2024-06-10T00:32:50.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddd229a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc112660 con 0x7fddcc06abc0 2024-06-10T00:32:50.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddd229a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc112960 con 0x7fddcc108500 2024-06-10T00:32:50.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddcb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddcc06b580 0x7fddcc199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddcb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddcc06b580 0x7fddcc199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47076/0 (socket says 172.21.15.121:47076) 2024-06-10T00:32:50.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddcb7fe700 1 -- 172.21.15.121:0/1350823285 learned_addr learned my addr 172.21.15.121:0/1350823285 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:50.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.387+0000 7fddcbfff700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc06abc0 0x7fddcc198d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddd0837700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc108500 0x7fddcc19d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddcb7fe700 1 -- 172.21.15.121:0/1350823285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc108500 msgr2=0x7fddcc19d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:50.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc108500 0x7fddcc19d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddcb7fe700 1 -- 172.21.15.121:0/1350823285 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc06abc0 msgr2=0x7fddcc198d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:50.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc06abc0 0x7fddcc198d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddcb7fe700 1 -- 172.21.15.121:0/1350823285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddcc19dd20 con 0x7fddcc06b580 2024-06-10T00:32:50.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddd0837700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc108500 0x7fddcc19d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:32:50.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddcc06b580 0x7fddcc199250 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fddc000b530 tx=0x7fddc000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:50.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddc97fa700 1 -- 172.21.15.121:0/1350823285 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc0015020 con 0x7fddcc06b580 2024-06-10T00:32:50.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddc97fa700 1 -- 172.21.15.121:0/1350823285 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddc0011d10 con 0x7fddcc06b580 2024-06-10T00:32:50.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddd229a700 1 -- 172.21.15.121:0/1350823285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddcc19e010 con 0x7fddcc06b580 2024-06-10T00:32:50.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddc97fa700 1 -- 172.21.15.121:0/1350823285 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc000f2e0 con 0x7fddcc06b580 2024-06-10T00:32:50.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.388+0000 7fddd229a700 1 -- 172.21.15.121:0/1350823285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddcc1a5890 con 0x7fddcc06b580 2024-06-10T00:32:50.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.390+0000 7fddc97fa700 1 -- 172.21.15.121:0/1350823285 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fddc0018020 con 0x7fddcc06b580 2024-06-10T00:32:50.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.391+0000 7fddc97fa700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fddb40a2d90 0x7fddb40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.391+0000 7fddcbfff700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fddb40a2d90 0x7fddb40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.391+0000 7fddc97fa700 1 -- 172.21.15.121:0/1350823285 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fddc00cef30 con 0x7fddcc06b580 2024-06-10T00:32:50.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.391+0000 7fddcbfff700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fddb40a2d90 0x7fddb40a5250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0098e0 tx=0x7fddbc005960 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:50.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fdddcb1db80 1 Processor -- start 2024-06-10T00:32:50.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fdddcb1db80 1 -- start start 2024-06-10T00:32:50.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 0x55fc0bbdf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbd8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbd9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fc0bb2a910 con 0x55fc0bbdf560 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55fc0bad2820 con 0x55fc0bbdec40 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fc0bb268d0 con 0x55fc0bbd93d0 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fddcb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbd8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fddcb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbd8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47090/0 (socket says 172.21.15.121:47090) 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fddcb7fe700 1 -- 172.21.15.121:0/3258461629 learned_addr learned my addr 172.21.15.121:0/3258461629 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fddcbfff700 1 --2- 172.21.15.121:0/3258461629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 0x55fc0bbdf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fddd0837700 1 --2- 172.21.15.121:0/3258461629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbd9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fddcb7fe700 1 -- 172.21.15.121:0/3258461629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbd93d0 msgr2=0x55fc0bbd9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/3258461629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbd9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fddcb7fe700 1 -- 172.21.15.121:0/3258461629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 msgr2=0x55fc0bbdf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/3258461629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 0x55fc0bbdf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fddcb7fe700 1 -- 172.21.15.121:0/3258461629 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc0bac9050 con 0x55fc0bbdf560 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.392+0000 7fddcbfff700 1 --2- 172.21.15.121:0/3258461629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 0x55fc0bbdf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.393+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/3258461629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbd8e90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fddc009aed0 tx=0x7fddc009ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975841936913d5ec server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.393+0000 7fddb27fc700 1 -- 172.21.15.121:0/3258461629 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc009f580 con 0x55fc0bbdf560 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.393+0000 7fddb27fc700 1 -- 172.21.15.121:0/3258461629 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddc00a3e40 con 0x55fc0bbdf560 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.393+0000 7fddb27fc700 1 -- 172.21.15.121:0/3258461629 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc00172c0 con 0x55fc0bbdf560 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.393+0000 7fdddcb1db80 1 -- 172.21.15.121:0/3258461629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 msgr2=0x55fc0bbd8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.393+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/3258461629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbd8e90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fddc009aed0 tx=0x7fddc009ac00 comp rx=0 tx=0).stop 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.394+0000 7fdddcb1db80 1 -- 172.21.15.121:0/3258461629 shutdown_connections 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.394+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/3258461629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbd9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.394+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/3258461629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 0x55fc0bbdf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.394+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/3258461629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbd8e90 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.394+0000 7fdddcb1db80 1 -- 172.21.15.121:0/3258461629 >> 172.21.15.121:0/3258461629 conn(0x55fc0bb03dc0 msgr2=0x55fc0bbd4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:50.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.394+0000 7fdddcb1db80 1 -- 172.21.15.121:0/3258461629 shutdown_connections 2024-06-10T00:32:50.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.394+0000 7fdddcb1db80 1 -- 172.21.15.121:0/3258461629 wait complete. 2024-06-10T00:32:50.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.394+0000 7fdddcb1db80 1 -- start start 2024-06-10T00:32:50.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.394+0000 7fdddcb1db80 1 Processor -- start 2024-06-10T00:32:50.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdec40 0x55fc0bbe25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbdf560 0x55fc0bbe2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddcb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdec40 0x55fc0bbe25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fc0bbe5f70 con 0x55fc0bbdec40 2024-06-10T00:32:50.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55fc0bacacf0 con 0x55fc0bbd93d0 2024-06-10T00:32:50.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fc0bbe60f0 con 0x55fc0bbdf560 2024-06-10T00:32:50.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddcb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdec40 0x55fc0bbe25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47104/0 (socket says 172.21.15.121:47104) 2024-06-10T00:32:50.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddcbfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddcb7fe700 1 -- 172.21.15.121:0/2138582869 learned_addr learned my addr 172.21.15.121:0/2138582869 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:50.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddd0837700 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbdf560 0x55fc0bbe2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddcb7fe700 1 -- 172.21.15.121:0/2138582869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbdf560 msgr2=0x55fc0bbe2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbdf560 0x55fc0bbe2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddcb7fe700 1 -- 172.21.15.121:0/2138582869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbd93d0 msgr2=0x55fc0bbe52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddcb7fe700 1 -- 172.21.15.121:0/2138582869 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc0ba799d0 con 0x55fc0bbdec40 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdec40 0x55fc0bbe25d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fddc000b500 tx=0x7fddc009b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37b3d1cb1e8a927 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fdddcb1db80 10 client.0 _ll_register_callbacks cb 0x55fc0bac4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2138582869 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddc00a3e40 con 0x55fc0bbdec40 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddab7fe700 1 -- 172.21.15.121:0/2138582869 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc009f360 con 0x55fc0bbdec40 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.395+0000 7fddab7fe700 1 -- 172.21.15.121:0/2138582869 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fddc00172c0 con 0x55fc0bbdec40 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.396+0000 7fddab7fe700 1 -- 172.21.15.121:0/2138582869 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fddc00174c0 con 0x55fc0bbdec40 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.396+0000 7fdddcb1db80 10 client.16176 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.396+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2138582869 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55fc0bbf0190 con 0x55fc0bbdec40 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.396+0000 7fddab7fe700 1 -- 172.21.15.121:0/2138582869 <== mon.0 v2:172.21.15.121:3300/0 4 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fddc009b430 con 0x55fc0bbdec40 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.396+0000 7fdddcb1db80 10 client.16176 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.396+0000 7fdddcb1db80 10 client.16176 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.396+0000 7fdddcb1db80 20 client.16176 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:50.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.396+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fc0bbf26c0 0x55fc0bbf4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.396+0000 7fdddcb1db80 4 client.16176 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:32:50.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.396+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2138582869 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc0bb37e10 con 0x55fc0bbf26c0 2024-06-10T00:32:50.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.396+0000 7fddcbfff700 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fc0bbf26c0 0x55fc0bbf4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.397+0000 7fddab7fe700 10 client.16176 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:50.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.397+0000 7fddcbfff700 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fc0bbf26c0 0x55fc0bbf4ad0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa34fe9c719f8186 server_cookie=1edc6057b10142bf in_seq=0 out_seq=0 2024-06-10T00:32:50.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.397+0000 7fddab7fe700 1 -- 172.21.15.121:0/2138582869 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55fc0bb37e10 con 0x55fc0bbf26c0 2024-06-10T00:32:50.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.397+0000 7fddab7fe700 10 client.16176 handle_command_reply: tid=0 2024-06-10T00:32:50.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.397+0000 7fdddcb1db80 1 client.16176 shutdown 2024-06-10T00:32:50.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2138582869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdec40 msgr2=0x55fc0bbe25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:50.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdec40 0x55fc0bbe25d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fddc000b500 tx=0x7fddc009b330 comp rx=0 tx=0).stop 2024-06-10T00:32:50.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2138582869 shutdown_connections 2024-06-10T00:32:50.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fc0bbf26c0 0x55fc0bbf4ad0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbdf560 0x55fc0bbe2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fddab7fe700 0 client.16176 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:50.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2138582869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdec40 0x55fc0bbe25d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2138582869 >> 172.21.15.121:0/2138582869 conn(0x55fc0bb03dc0 msgr2=0x55fc0bbd7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:50.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2138582869 shutdown_connections 2024-06-10T00:32:50.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2138582869 wait complete. 2024-06-10T00:32:50.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.398+0000 7fdddcb1db80 20 client.16176 trim_cache size 0 max 16384 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fdddcb1db80 1 Processor -- start 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fdddcb1db80 1 -- start start 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 0x55fc0bbfa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbfa920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1aed20 con 0x55fc0bbdf560 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1ac5b0 con 0x55fc0bbdec40 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1aa400 con 0x55fc0bbd93d0 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddd0837700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbfa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddd0837700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbfa920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47106/0 (socket says 172.21.15.121:47106) 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddd0837700 1 -- 172.21.15.121:0/3793419521 learned_addr learned my addr 172.21.15.121:0/3793419521 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:50.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/3793419521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 0x55fc0bbfa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddcbfff700 1 --2- 172.21.15.121:0/3793419521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddd0837700 1 -- 172.21.15.121:0/3793419521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbd93d0 msgr2=0x55fc0bbe0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddd0837700 1 --2- 172.21.15.121:0/3793419521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddd0837700 1 -- 172.21.15.121:0/3793419521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 msgr2=0x55fc0bbfa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddd0837700 1 --2- 172.21.15.121:0/3793419521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 0x55fc0bbfa3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddd0837700 1 -- 172.21.15.121:0/3793419521 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc0bbce080 con 0x55fc0bbdf560 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddcbfff700 1 --2- 172.21.15.121:0/3793419521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.418+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/3793419521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 0x55fc0bbfa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.419+0000 7fddd0837700 1 --2- 172.21.15.121:0/3793419521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbfa920 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009dc0 tx=0x7fddc4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feddf89a4264c3ba server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.419+0000 7fddb1ffb700 1 -- 172.21.15.121:0/3793419521 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc4019020 con 0x55fc0bbdf560 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.419+0000 7fddb1ffb700 1 -- 172.21.15.121:0/3793419521 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddc40158d0 con 0x55fc0bbdf560 2024-06-10T00:32:50.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.419+0000 7fddb1ffb700 1 -- 172.21.15.121:0/3793419521 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc40147f0 con 0x55fc0bbdf560 2024-06-10T00:32:50.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.419+0000 7fdddcb1db80 1 -- 172.21.15.121:0/3793419521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 msgr2=0x55fc0bbfa920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:50.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.419+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/3793419521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbfa920 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fddc4009dc0 tx=0x7fddc4009040 comp rx=0 tx=0).stop 2024-06-10T00:32:50.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.419+0000 7fdddcb1db80 1 -- 172.21.15.121:0/3793419521 shutdown_connections 2024-06-10T00:32:50.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.419+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/3793419521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbdf560 0x55fc0bbfa920 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.423 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.419+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/3793419521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.419+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/3793419521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdec40 0x55fc0bbfa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.419+0000 7fdddcb1db80 1 -- 172.21.15.121:0/3793419521 >> 172.21.15.121:0/3793419521 conn(0x55fc0bb03dc0 msgr2=0x55fc0bbdf980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fdddcb1db80 1 -- 172.21.15.121:0/3793419521 shutdown_connections 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fdddcb1db80 1 -- 172.21.15.121:0/3793419521 wait complete. 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fdddcb1db80 1 -- start start 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fdddcb1db80 1 Processor -- start 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbdec40 0x55fc0bbe88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fdddcb1db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdf560 0x55fc0bbe8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1aaa70 con 0x55fc0bbd93d0 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1abee0 con 0x55fc0bbdf560 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fdddcb1db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc1ab140 con 0x55fc0bbdec40 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fdddcb1db80 10 client.0 _ll_register_callbacks cb 0x55fc0bac4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:50.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fddcbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fddcbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47120/0 (socket says 172.21.15.121:47120) 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.420+0000 7fddcbfff700 1 -- 172.21.15.121:0/2307417646 learned_addr learned my addr 172.21.15.121:0/2307417646 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddd0837700 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdf560 0x55fc0bbe8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbdec40 0x55fc0bbe88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddcbfff700 1 -- 172.21.15.121:0/2307417646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbdec40 msgr2=0x55fc0bbe88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddcbfff700 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbdec40 0x55fc0bbe88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddcbfff700 1 -- 172.21.15.121:0/2307417646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdf560 msgr2=0x55fc0bbe8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddcbfff700 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdf560 0x55fc0bbe8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddcbfff700 1 -- 172.21.15.121:0/2307417646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc0bc5d3c0 con 0x55fc0bbd93d0 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbdec40 0x55fc0bbe88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddd0837700 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdf560 0x55fc0bbe8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddcbfff700 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe8380 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fddbc000c00 tx=0x7fddbc009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4da82617801454 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:32:50.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddab7fe700 1 -- 172.21.15.121:0/2307417646 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc00bae0 con 0x55fc0bbd93d0 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddab7fe700 1 -- 172.21.15.121:0/2307417646 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fddbc0383c0 con 0x55fc0bbd93d0 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2307417646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fddc40158d0 con 0x55fc0bbd93d0 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddab7fe700 1 -- 172.21.15.121:0/2307417646 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fddbc047050 con 0x55fc0bbd93d0 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fdddcb1db80 10 client.16182 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2307417646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fc0bc5e680 con 0x55fc0bbd93d0 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.421+0000 7fddab7fe700 1 -- 172.21.15.121:0/2307417646 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddc40158d0 con 0x55fc0bbd93d0 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.422+0000 7fddab7fe700 1 -- 172.21.15.121:0/2307417646 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fddbc0378b0 con 0x55fc0bbd93d0 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.422+0000 7fdddcb1db80 10 client.16182 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.422+0000 7fdddcb1db80 10 client.16182 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.422+0000 7fdddcb1db80 20 client.16182 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.422+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fc0bc5ed60 0x55fc0bc611a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.422+0000 7fdddcb1db80 4 client.16182 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "2f9a24d7", "release": true, "await": true}] 2024-06-10T00:32:50.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.422+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2307417646 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "set_id": "2f9a24d7", "release": true, "await": true}) -- 0x55fc0bb32390 con 0x55fc0bc5ed60 2024-06-10T00:32:50.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.422+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fc0bc5ed60 0x55fc0bc611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.422+0000 7fddcb7fe700 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fc0bc5ed60 0x55fc0bc611a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7347b90206693f12 server_cookie=601e32023c53256a in_seq=0 out_seq=0 2024-06-10T00:32:50.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.422+0000 7fddab7fe700 10 client.16182 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:50.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.665+0000 7f74e1925700 1 -- 172.21.15.121:0/4118201143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74dc102ea0 msgr2=0x7f74dc10f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:50.721 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.665+0000 7f74e1925700 1 --2- 172.21.15.121:0/4118201143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74dc102ea0 0x7f74dc10f380 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f74d400b0a0 tx=0x7f74d400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.665+0000 7f74e1925700 1 -- 172.21.15.121:0/4118201143 shutdown_connections 2024-06-10T00:32:50.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.665+0000 7f74e1925700 1 --2- 172.21.15.121:0/4118201143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74dc102ea0 0x7f74dc10f380 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.665+0000 7f74e1925700 1 --2- 172.21.15.121:0/4118201143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc1085f0 0x7f74dc1089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.665+0000 7f74e1925700 1 --2- 172.21.15.121:0/4118201143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74dc1024e0 0x7f74dc102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.665+0000 7f74e1925700 1 -- 172.21.15.121:0/4118201143 >> 172.21.15.121:0/4118201143 conn(0x7f74dc079640 msgr2=0x7f74dc079a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:50.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.665+0000 7f74e1925700 1 -- 172.21.15.121:0/4118201143 shutdown_connections 2024-06-10T00:32:50.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.665+0000 7f74e1925700 1 -- 172.21.15.121:0/4118201143 wait complete. 2024-06-10T00:32:50.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.665+0000 7f74e1925700 1 Processor -- start 2024-06-10T00:32:50.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.665+0000 7f74e1925700 1 -- start start 2024-06-10T00:32:50.722 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74e1925700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74dc1024e0 0x7f74dc19e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74e1925700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74dc102ea0 0x7f74dc19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74e1925700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc1085f0 0x7f74dc1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74e1925700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1127b0 con 0x7f74dc102ea0 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74e1925700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc112630 con 0x7f74dc1024e0 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74e1925700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc112930 con 0x7f74dc1085f0 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74da7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74dc102ea0 0x7f74dc19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74da7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74dc102ea0 0x7f74dc19ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47132/0 (socket says 172.21.15.121:47132) 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74da7fc700 1 -- 172.21.15.121:0/3217033203 learned_addr learned my addr 172.21.15.121:0/3217033203 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74daffd700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74dc1024e0 0x7f74dc19e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74db7fe700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc1085f0 0x7f74dc1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74da7fc700 1 -- 172.21.15.121:0/3217033203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc1085f0 msgr2=0x7f74dc1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74da7fc700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc1085f0 0x7f74dc1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74da7fc700 1 -- 172.21.15.121:0/3217033203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74dc1024e0 msgr2=0x7f74dc19e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74da7fc700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74dc1024e0 0x7f74dc19e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74da7fc700 1 -- 172.21.15.121:0/3217033203 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc199130 con 0x7f74dc102ea0 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74da7fc700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74dc102ea0 0x7f74dc19ed00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00d950 tx=0x7f74cc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74e0923700 1 -- 172.21.15.121:0/3217033203 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc016070 con 0x7f74dc102ea0 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74e1925700 1 -- 172.21.15.121:0/3217033203 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74dc199420 con 0x7f74dc102ea0 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74e0923700 1 -- 172.21.15.121:0/3217033203 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74cc00fd70 con 0x7f74dc102ea0 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.666+0000 7f74e0923700 1 -- 172.21.15.121:0/3217033203 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc00e610 con 0x7f74dc102ea0 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.667+0000 7f74e1925700 1 -- 172.21.15.121:0/3217033203 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74dc1a5690 con 0x7f74dc102ea0 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.669+0000 7f74e0923700 1 -- 172.21.15.121:0/3217033203 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f74cc019070 con 0x7f74dc102ea0 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.669+0000 7f74e0923700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74c80a2d90 0x7f74c80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.670+0000 7f74daffd700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74c80a2d90 0x7f74c80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.670+0000 7f74e0923700 1 -- 172.21.15.121:0/3217033203 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f74cc0cf2f0 con 0x7f74dc102ea0 2024-06-10T00:32:50.724 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.670+0000 7f74daffd700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74c80a2d90 0x7f74c80a5250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f74dc108e10 tx=0x7f74c400b3c0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.670+0000 7f74ec1a8b80 1 Processor -- start 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.670+0000 7f74ec1a8b80 1 -- start start 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.670+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 0x55b9853fc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.670+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fc880 0x55b9853f6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b9852e8f60 con 0x55b9853f67a0 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55b985347850 con 0x55b9853fc880 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b9852af460 con 0x55b9853fbf60 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74db7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74db7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47138/0 (socket says 172.21.15.121:47138) 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74db7fe700 1 -- 172.21.15.121:0/1292875560 learned_addr learned my addr 172.21.15.121:0/1292875560 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74daffd700 1 --2- 172.21.15.121:0/1292875560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 0x55b9853fc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74da7fc700 1 --2- 172.21.15.121:0/1292875560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fc880 0x55b9853f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74db7fe700 1 -- 172.21.15.121:0/1292875560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 msgr2=0x55b9853fc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74db7fe700 1 --2- 172.21.15.121:0/1292875560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 0x55b9853fc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74db7fe700 1 -- 172.21.15.121:0/1292875560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fc880 msgr2=0x55b9853f6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74db7fe700 1 --2- 172.21.15.121:0/1292875560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fc880 0x55b9853f6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74db7fe700 1 -- 172.21.15.121:0/1292875560 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9852968d0 con 0x55b9853f67a0 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74db7fe700 1 --2- 172.21.15.121:0/1292875560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6bc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f74d40090d0 tx=0x7f74d4009100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610cf523af31b28e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74d1ffb700 1 -- 172.21.15.121:0/1292875560 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4020070 con 0x55b9853f67a0 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74d1ffb700 1 -- 172.21.15.121:0/1292875560 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d40074e0 con 0x55b9853f67a0 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.671+0000 7f74d1ffb700 1 -- 172.21.15.121:0/1292875560 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4005260 con 0x55b9853f67a0 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.672+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/1292875560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 msgr2=0x55b9853f6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.672+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/1292875560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6bc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f74d40090d0 tx=0x7f74d4009100 comp rx=0 tx=0).stop 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.672+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/1292875560 shutdown_connections 2024-06-10T00:32:50.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.672+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/1292875560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6bc0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.672+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/1292875560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 0x55b9853fc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.672+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/1292875560 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fc880 0x55b9853f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.672+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/1292875560 >> 172.21.15.121:0/1292875560 conn(0x55b985323080 msgr2=0x55b9853f1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.672+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/1292875560 shutdown_connections 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.672+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/1292875560 wait complete. 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.672+0000 7f74ec1a8b80 1 -- start start 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74ec1a8b80 1 Processor -- start 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b985402580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 0x55b9853ff880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fc880 0x55b9853ffdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b9852e7fc0 con 0x55b9853f67a0 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55b985343810 con 0x55b9853fc880 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b985407610 con 0x55b9853fbf60 2024-06-10T00:32:50.727 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74ec1a8b80 10 client.0 _ll_register_callbacks cb 0x55b9852dd5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b985402580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b985402580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47148/0 (socket says 172.21.15.121:47148) 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74daffd700 1 -- 172.21.15.121:0/3255221850 learned_addr learned my addr 172.21.15.121:0/3255221850 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74da7fc700 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 0x55b9853ff880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74db7fe700 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fc880 0x55b9853ffdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74daffd700 1 -- 172.21.15.121:0/3255221850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 msgr2=0x55b9853ff880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74daffd700 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 0x55b9853ff880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74daffd700 1 -- 172.21.15.121:0/3255221850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fc880 msgr2=0x55b9853ffdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74daffd700 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fc880 0x55b9853ffdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74daffd700 1 -- 172.21.15.121:0/3255221850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b98530d750 con 0x55b9853f67a0 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74daffd700 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b985402580 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f74c4003db0 tx=0x7f74c40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e269a87783c623f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.673+0000 7f74baffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74baffd700 1 -- 172.21.15.121:0/3255221850 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c4003ea0 con 0x55b9853f67a0 2024-06-10T00:32:50.728 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74baffd700 1 -- 172.21.15.121:0/3255221850 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f74c4011ce0 con 0x55b9853f67a0 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/3255221850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74d40074e0 con 0x55b9853f67a0 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74baffd700 1 -- 172.21.15.121:0/3255221850 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f74c4031050 con 0x55b9853f67a0 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74ec1a8b80 10 client.16194 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/3255221850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b985411e00 con 0x55b9853f67a0 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74baffd700 1 -- 172.21.15.121:0/3255221850 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d40074e0 con 0x55b9853f67a0 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74baffd700 1 -- 172.21.15.121:0/3255221850 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f74c401f5f0 con 0x55b9853f67a0 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74ec1a8b80 10 client.16194 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74ec1a8b80 10 client.16194 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74ec1a8b80 20 client.16194 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55b985414440 0x55b985416830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74ec1a8b80 4 client.16194 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.674+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/3255221850 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b985354d50 con 0x55b985414440 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.675+0000 7f74da7fc700 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55b985414440 0x55b985416830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:50.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.675+0000 7f74baffd700 10 client.16194 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:32:50.730 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:50.675+0000 7f74da7fc700 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55b985414440 0x55b985416830 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9366b921ac700aec server_cookie=24f623fe23920aa2 in_seq=0 out_seq=0 2024-06-10T00:32:51.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fddab7fe700 1 -- 172.21.15.121:0/2307417646 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55fc0bb32390 con 0x55fc0bc5ed60 2024-06-10T00:32:51.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fddab7fe700 10 client.16182 handle_command_reply: tid=0 2024-06-10T00:32:51.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fdddcb1db80 1 client.16182 shutdown 2024-06-10T00:32:51.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2307417646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbd93d0 msgr2=0x55fc0bbe8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:51.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe8380 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fddbc000c00 tx=0x7fddbc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2307417646 shutdown_connections 2024-06-10T00:32:51.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fc0bc5ed60 0x55fc0bc611a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fc0bbdf560 0x55fc0bbe8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc0bbd93d0 0x55fc0bbe8380 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fdddcb1db80 1 --2- 172.21.15.121:0/2307417646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc0bbdec40 0x55fc0bbe88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fddab7fe700 0 client.16182 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:51.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2307417646 >> 172.21.15.121:0/2307417646 conn(0x55fc0bb03dc0 msgr2=0x55fc0bbd3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7f74baffd700 1 -- 172.21.15.121:0/3255221850 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b985354d50 con 0x55b985414440 2024-06-10T00:32:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7f74baffd700 10 client.16194 handle_command_reply: tid=0 2024-06-10T00:32:51.256 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7f74ec1a8b80 1 client.16194 shutdown 2024-06-10T00:32:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2307417646 shutdown_connections 2024-06-10T00:32:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.254+0000 7fdddcb1db80 1 -- 172.21.15.121:0/2307417646 wait complete. 2024-06-10T00:32:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7fdddcb1db80 20 client.16182 trim_cache size 0 max 16384 2024-06-10T00:32:51.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/3255221850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 msgr2=0x55b985402580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b985402580 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f74c4003db0 tx=0x7f74c40047c0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/3255221850 shutdown_connections 2024-06-10T00:32:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55b985414440 0x55b985416830 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74baffd700 0 client.16194 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:32:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fc880 0x55b9853ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b985402580 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/3255221850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 0x55b9853ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/3255221850 >> 172.21.15.121:0/3255221850 conn(0x55b985323080 msgr2=0x55b9853f0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:51.258 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 28, 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 12, 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "2f9a24d7": { 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 12, 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 138.2, 2024-06-10T00:32:51.259 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:32:51.260 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:32:51.260 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:32:51.260 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:32:51.260 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:32:51.260 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:32:51.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:51.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:51.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:51.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:51.260 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:32:51.260 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:32:51.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/3255221850 shutdown_connections 2024-06-10T00:32:51.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/3255221850 wait complete. 2024-06-10T00:32:51.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.255+0000 7f74ec1a8b80 20 client.16194 trim_cache size 0 max 16384 2024-06-10T00:32:51.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 -- 172.21.15.121:0/1350823285 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fddb40a2d90 msgr2=0x7fddb40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:51.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fddb40a2d90 0x7fddb40a5250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0098e0 tx=0x7fddbc005960 comp rx=0 tx=0).stop 2024-06-10T00:32:51.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 -- 172.21.15.121:0/1350823285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddcc06b580 msgr2=0x7fddcc199250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:51.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddcc06b580 0x7fddcc199250 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fddc000b530 tx=0x7fddc000b840 comp rx=0 tx=0).stop 2024-06-10T00:32:51.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 -- 172.21.15.121:0/1350823285 shutdown_connections 2024-06-10T00:32:51.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fddb40a2d90 0x7fddb40a5250 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddcc108500 0x7fddcc19d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc06abc0 0x7fddcc198d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 --2- 172.21.15.121:0/1350823285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddcc06b580 0x7fddcc199250 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 -- 172.21.15.121:0/1350823285 >> 172.21.15.121:0/1350823285 conn(0x7fddcc078340 msgr2=0x7fddcc100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:51.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 -- 172.21.15.121:0/1350823285 shutdown_connections 2024-06-10T00:32:51.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.256+0000 7fddd229a700 1 -- 172.21.15.121:0/1350823285 wait complete. 2024-06-10T00:32:51.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.282+0000 7f74ec1a8b80 1 Processor -- start 2024-06-10T00:32:51.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.288+0000 7f74ec1a8b80 1 -- start start 2024-06-10T00:32:51.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.288+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:51.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.288+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fbf60 0x55b9853f0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:51.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.288+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fc880 0x55b9853f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:51.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.288+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1aeab0 con 0x55b9853f67a0 2024-06-10T00:32:51.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.288+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1ac340 con 0x55b9853fbf60 2024-06-10T00:32:51.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.288+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1aa190 con 0x55b9853fc880 2024-06-10T00:32:51.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.290+0000 7f74daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:51.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.290+0000 7f74daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47160/0 (socket says 172.21.15.121:47160) 2024-06-10T00:32:51.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.290+0000 7f74da7fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fbf60 0x55b9853f0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:51.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.290+0000 7f74daffd700 1 -- 172.21.15.121:0/422457588 learned_addr learned my addr 172.21.15.121:0/422457588 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:51.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.291+0000 7f74db7fe700 1 --2- 172.21.15.121:0/422457588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fc880 0x55b9853f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:51.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.295+0000 7f74da7fc700 1 -- 172.21.15.121:0/422457588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fc880 msgr2=0x55b9853f13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:51.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.295+0000 7f74da7fc700 1 --2- 172.21.15.121:0/422457588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fc880 0x55b9853f13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.295+0000 7f74da7fc700 1 -- 172.21.15.121:0/422457588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 msgr2=0x55b9853f6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:51.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.295+0000 7f74da7fc700 1 --2- 172.21.15.121:0/422457588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.295+0000 7f74da7fc700 1 -- 172.21.15.121:0/422457588 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9853fddf0 con 0x55b9853fbf60 2024-06-10T00:32:51.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.295+0000 7f74daffd700 1 --2- 172.21.15.121:0/422457588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:51.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.296+0000 7f74da7fc700 1 --2- 172.21.15.121:0/422457588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fbf60 0x55b9853f0e80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0cc650 tx=0x7f74cc000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a50e842523d3d831 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:51.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.296+0000 7f74d17fa700 1 -- 172.21.15.121:0/422457588 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc020040 con 0x55b9853fbf60 2024-06-10T00:32:51.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.296+0000 7f74d17fa700 1 -- 172.21.15.121:0/422457588 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74cc09f7d0 con 0x55b9853fbf60 2024-06-10T00:32:51.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.296+0000 7f74d17fa700 1 -- 172.21.15.121:0/422457588 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc09b680 con 0x55b9853fbf60 2024-06-10T00:32:51.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.297+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/422457588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fbf60 msgr2=0x55b9853f0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:51.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.297+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/422457588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fbf60 0x55b9853f0e80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0cc650 tx=0x7f74cc000bc0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.297+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/422457588 shutdown_connections 2024-06-10T00:32:51.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.297+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/422457588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fc880 0x55b9853f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.297+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/422457588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853f67a0 0x55b9853f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.297+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/422457588 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853fbf60 0x55b9853f0e80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.297+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/422457588 >> 172.21.15.121:0/422457588 conn(0x55b985323080 msgr2=0x55b9853f6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:51.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.298+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/422457588 shutdown_connections 2024-06-10T00:32:51.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.298+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/422457588 wait complete. 2024-06-10T00:32:51.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.299+0000 7f74ec1a8b80 1 -- start start 2024-06-10T00:32:51.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.299+0000 7f74ec1a8b80 1 Processor -- start 2024-06-10T00:32:51.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.300+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853f67a0 0x55b98540cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:51.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.300+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 0x55b98540d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:51.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.300+0000 7f74ec1a8b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853fc880 0x55b98541b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:51.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.300+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1aa800 con 0x55b9853fc880 2024-06-10T00:32:51.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.300+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1abc70 con 0x55b9853f67a0 2024-06-10T00:32:51.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.300+0000 7f74ec1a8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc1aaed0 con 0x55b9853fbf60 2024-06-10T00:32:51.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.300+0000 7f74ec1a8b80 10 client.0 _ll_register_callbacks cb 0x55b9852dd5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:51.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.301+0000 7f74db7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853fc880 0x55b98541b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:51.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.301+0000 7f74db7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853fc880 0x55b98541b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47176/0 (socket says 172.21.15.121:47176) 2024-06-10T00:32:51.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.301+0000 7f74db7fe700 1 -- 172.21.15.121:0/4050078893 learned_addr learned my addr 172.21.15.121:0/4050078893 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:51.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.301+0000 7f74daffd700 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853f67a0 0x55b98540cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:51.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.302+0000 7f74db7fe700 1 -- 172.21.15.121:0/4050078893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 msgr2=0x55b98540d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:51.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.302+0000 7f74db7fe700 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 0x55b98540d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.302+0000 7f74db7fe700 1 -- 172.21.15.121:0/4050078893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853f67a0 msgr2=0x55b98540cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:51.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.302+0000 7f74db7fe700 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853f67a0 0x55b98540cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.302+0000 7f74db7fe700 1 -- 172.21.15.121:0/4050078893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b98541bc00 con 0x55b9853fc880 2024-06-10T00:32:51.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.302+0000 7f74db7fe700 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853fc880 0x55b98541b500 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f74d400d010 tx=0x7f74d4006250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f932e4e313f594c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:51.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.302+0000 7f74baffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:32:51.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.302+0000 7f74baffd700 1 -- 172.21.15.121:0/4050078893 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4020040 con 0x55b9853fc880 2024-06-10T00:32:51.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.302+0000 7f74baffd700 1 -- 172.21.15.121:0/4050078893 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f74d4007850 con 0x55b9853fc880 2024-06-10T00:32:51.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.303+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/4050078893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74cc09f7d0 con 0x55b9853fc880 2024-06-10T00:32:51.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.303+0000 7f74baffd700 1 -- 172.21.15.121:0/4050078893 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f74d4007a50 con 0x55b9853fc880 2024-06-10T00:32:51.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.303+0000 7f74ec1a8b80 10 client.16200 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:51.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.303+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/4050078893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b985403520 con 0x55b9853fc880 2024-06-10T00:32:51.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.303+0000 7f74baffd700 1 -- 172.21.15.121:0/4050078893 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74cc09f7d0 con 0x55b9853fc880 2024-06-10T00:32:51.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.303+0000 7f74baffd700 1 -- 172.21.15.121:0/4050078893 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f74d4011070 con 0x55b9853fc880 2024-06-10T00:32:51.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.304+0000 7f74ec1a8b80 10 client.16200 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:51.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.304+0000 7f74ec1a8b80 10 client.16200 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:32:51.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.304+0000 7f74ec1a8b80 20 client.16200 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:51.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.304+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55b985414440 0x55b9854058a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:51.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.304+0000 7f74ec1a8b80 4 client.16200 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:32:51.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.304+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/4050078893 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b98534f2d0 con 0x55b985414440 2024-06-10T00:32:51.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.305+0000 7f74daffd700 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55b985414440 0x55b9854058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:51.306 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.305+0000 7f74daffd700 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55b985414440 0x55b9854058a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e0ca1d5c677babf server_cookie=ac0a7bf21877679d in_seq=0 out_seq=0 2024-06-10T00:32:51.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.305+0000 7f74baffd700 10 client.16200 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:32:51.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74baffd700 1 -- 172.21.15.121:0/4050078893 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b98534f2d0 con 0x55b985414440 2024-06-10T00:32:51.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74baffd700 10 client.16200 handle_command_reply: tid=0 2024-06-10T00:32:51.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 1 client.16200 shutdown 2024-06-10T00:32:51.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/4050078893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853fc880 msgr2=0x55b98541b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:51.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853fc880 0x55b98541b500 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f74d400d010 tx=0x7f74d4006250 comp rx=0 tx=0).stop 2024-06-10T00:32:51.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/4050078893 shutdown_connections 2024-06-10T00:32:51.310 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:32:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x55b985414440 0x55b9854058a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b9853fc880 0x55b98541b500 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b9853f67a0 0x55b98540cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 1 --2- 172.21.15.121:0/4050078893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9853fbf60 0x55b98540d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/4050078893 >> 172.21.15.121:0/4050078893 conn(0x55b985323080 msgr2=0x55b9853fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74baffd700 0 client.16200 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:32:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/4050078893 shutdown_connections 2024-06-10T00:32:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 1 -- 172.21.15.121:0/4050078893 wait complete. 2024-06-10T00:32:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.308+0000 7f74ec1a8b80 20 client.16200 trim_cache size 0 max 16384 2024-06-10T00:32:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 -- 172.21.15.121:0/3217033203 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74c80a2d90 msgr2=0x7f74c80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74c80a2d90 0x7f74c80a5250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f74dc108e10 tx=0x7f74c400b3c0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 -- 172.21.15.121:0/3217033203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74dc102ea0 msgr2=0x7f74dc19ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74dc102ea0 0x7f74dc19ed00 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00d950 tx=0x7f74cc00dc60 comp rx=0 tx=0).stop 2024-06-10T00:32:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 -- 172.21.15.121:0/3217033203 shutdown_connections 2024-06-10T00:32:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74c80a2d90 0x7f74c80a5250 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc1085f0 0x7f74dc1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74dc1024e0 0x7f74dc19e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 --2- 172.21.15.121:0/3217033203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74dc102ea0 0x7f74dc19ed00 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 -- 172.21.15.121:0/3217033203 >> 172.21.15.121:0/3217033203 conn(0x7f74dc079640 msgr2=0x7f74dc100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 -- 172.21.15.121:0/3217033203 shutdown_connections 2024-06-10T00:32:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:51.309+0000 7f74e1925700 1 -- 172.21.15.121:0/3217033203 wait complete. 2024-06-10T00:32:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:51 smithi193 ceph-mon[36716]: pgmap v149: 129 pgs: 129 active+clean; 927 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 577 op/s 2024-06-10T00:32:51.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:51 smithi121 ceph-mon[32378]: pgmap v149: 129 pgs: 129 active+clean; 927 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 577 op/s 2024-06-10T00:32:51.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:51 smithi135 ceph-mon[36424]: pgmap v149: 129 pgs: 129 active+clean; 927 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 577 op/s 2024-06-10T00:32:52.433 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '2f9a24d7', total seconds elapsed: 46.2 2024-06-10T00:32:52.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-10T00:32:53.219 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:32:53.621 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:53 smithi121 ceph-mon[32378]: pgmap v150: 129 pgs: 129 active+clean; 927 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-06-10T00:32:53.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:53 smithi193 ceph-mon[36716]: pgmap v150: 129 pgs: 129 active+clean; 927 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-06-10T00:32:53.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:53 smithi135 ceph-mon[36424]: pgmap v150: 129 pgs: 129 active+clean; 927 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 459 op/s 2024-06-10T00:32:54.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.916+0000 7f7c58fe5700 1 -- 172.21.15.121:0/2102057177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c54101d30 msgr2=0x7f7c5410e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:54.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.916+0000 7f7c58fe5700 1 --2- 172.21.15.121:0/2102057177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c54101d30 0x7f7c5410e210 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00b0a0 tx=0x7f7c4c00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.916+0000 7f7c58fe5700 1 -- 172.21.15.121:0/2102057177 shutdown_connections 2024-06-10T00:32:54.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.916+0000 7f7c58fe5700 1 --2- 172.21.15.121:0/2102057177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c54101d30 0x7f7c5410e210 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.916+0000 7f7c58fe5700 1 --2- 172.21.15.121:0/2102057177 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c541073d0 0x7f7c541077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.916+0000 7f7c58fe5700 1 --2- 172.21.15.121:0/2102057177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c54101390 0x7f7c541017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.916+0000 7f7c58fe5700 1 -- 172.21.15.121:0/2102057177 >> 172.21.15.121:0/2102057177 conn(0x7f7c540782c0 msgr2=0x7f7c540786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:54.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.916+0000 7f7c58fe5700 1 -- 172.21.15.121:0/2102057177 shutdown_connections 2024-06-10T00:32:54.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.916+0000 7f7c58fe5700 1 -- 172.21.15.121:0/2102057177 wait complete. 2024-06-10T00:32:54.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.917+0000 7f7c58fe5700 1 Processor -- start 2024-06-10T00:32:54.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.917+0000 7f7c58fe5700 1 -- start start 2024-06-10T00:32:54.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.917+0000 7f7c58fe5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c54101390 0x7f7c5419d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c58fe5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c54101d30 0x7f7c5419dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c58fe5700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c541073d0 0x7f7c54197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c58fe5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7c54111620 con 0x7f7c54101390 2024-06-10T00:32:54.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c58fe5700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f7c541114a0 con 0x7f7c541073d0 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c58fe5700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7c541117a0 con 0x7f7c54101d30 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c5259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c54101390 0x7f7c5419d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c51d9b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c54101d30 0x7f7c5419dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c5259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c54101390 0x7f7c5419d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47202/0 (socket says 172.21.15.121:47202) 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c5259c700 1 -- 172.21.15.121:0/4028915107 learned_addr learned my addr 172.21.15.121:0/4028915107 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c52d9d700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c541073d0 0x7f7c54197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c51d9b700 1 -- 172.21.15.121:0/4028915107 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c541073d0 msgr2=0x7f7c54197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c51d9b700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c541073d0 0x7f7c54197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c51d9b700 1 -- 172.21.15.121:0/4028915107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c54101390 msgr2=0x7f7c5419d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c51d9b700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c54101390 0x7f7c5419d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c51d9b700 1 -- 172.21.15.121:0/4028915107 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c54198060 con 0x7f7c54101d30 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.918+0000 7f7c51d9b700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c54101d30 0x7f7c5419dc30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800e9f0 tx=0x7f7c4800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.919+0000 7f7c437fe700 1 -- 172.21.15.121:0/4028915107 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4800c9c0 con 0x7f7c54101d30 2024-06-10T00:32:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.919+0000 7f7c58fe5700 1 -- 172.21.15.121:0/4028915107 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c54198350 con 0x7f7c54101d30 2024-06-10T00:32:54.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.919+0000 7f7c437fe700 1 -- 172.21.15.121:0/4028915107 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4800cb60 con 0x7f7c54101d30 2024-06-10T00:32:54.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.919+0000 7f7c437fe700 1 -- 172.21.15.121:0/4028915107 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48018680 con 0x7f7c54101d30 2024-06-10T00:32:54.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.919+0000 7f7c58fe5700 1 -- 172.21.15.121:0/4028915107 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c541a45c0 con 0x7f7c54101d30 2024-06-10T00:32:54.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.921+0000 7f7c437fe700 1 -- 172.21.15.121:0/4028915107 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7c48010430 con 0x7f7c54101d30 2024-06-10T00:32:54.921 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.921+0000 7f7c437fe700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7c3c0a2d90 0x7f7c3c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.922+0000 7f7c5259c700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7c3c0a2d90 0x7f7c3c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.922+0000 7f7c437fe700 1 -- 172.21.15.121:0/4028915107 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f7c480cf8f0 con 0x7f7c54101d30 2024-06-10T00:32:54.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.922+0000 7f7c5259c700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7c3c0a2d90 0x7f7c3c0a5250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7c54106be0 tx=0x7f7c44009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:54.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.922+0000 7f7c63868b80 1 Processor -- start 2024-06-10T00:32:54.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c63868b80 1 -- start start 2024-06-10T00:32:54.924 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a39539e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a39543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 0x55b5a3964450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c63868b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b5a38a5810 con 0x55b5a3962060 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c63868b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55b5a384d720 con 0x55b5a3955740 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c63868b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b5a38a17d0 con 0x55b5a3953f20 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c5259c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a39539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c5259c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a39539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:48824/0 (socket says 172.21.15.121:48824) 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c5259c700 1 -- 172.21.15.121:0/4274265322 learned_addr learned my addr 172.21.15.121:0/4274265322 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c52d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 0x55b5a3964450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c51d9b700 1 --2- 172.21.15.121:0/4274265322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a39543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c52d9d700 1 -- 172.21.15.121:0/4274265322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 msgr2=0x55b5a39543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c52d9d700 1 --2- 172.21.15.121:0/4274265322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a39543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.925 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c52d9d700 1 -- 172.21.15.121:0/4274265322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 msgr2=0x55b5a39539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:54.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c52d9d700 1 --2- 172.21.15.121:0/4274265322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a39539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c52d9d700 1 -- 172.21.15.121:0/4274265322 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a3843f50 con 0x55b5a3962060 2024-06-10T00:32:54.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.923+0000 7f7c52d9d700 1 --2- 172.21.15.121:0/4274265322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 0x55b5a3964450 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00b070 tx=0x7f7c4c0092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c72096213c58366 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:54.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.924+0000 7f7c40ff9700 1 -- 172.21.15.121:0/4274265322 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c020070 con 0x55b5a3962060 2024-06-10T00:32:54.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.924+0000 7f7c40ff9700 1 -- 172.21.15.121:0/4274265322 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4c006020 con 0x55b5a3962060 2024-06-10T00:32:54.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.924+0000 7f7c40ff9700 1 -- 172.21.15.121:0/4274265322 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c007b80 con 0x55b5a3962060 2024-06-10T00:32:54.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.924+0000 7f7c63868b80 1 -- 172.21.15.121:0/4274265322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 msgr2=0x55b5a3964450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:54.926 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.924+0000 7f7c63868b80 1 --2- 172.21.15.121:0/4274265322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 0x55b5a3964450 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00b070 tx=0x7f7c4c0092e0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.924+0000 7f7c63868b80 1 -- 172.21.15.121:0/4274265322 shutdown_connections 2024-06-10T00:32:54.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.924+0000 7f7c63868b80 1 --2- 172.21.15.121:0/4274265322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 0x55b5a3964450 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.924+0000 7f7c63868b80 1 --2- 172.21.15.121:0/4274265322 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a39539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.924+0000 7f7c63868b80 1 --2- 172.21.15.121:0/4274265322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a39543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.924+0000 7f7c63868b80 1 -- 172.21.15.121:0/4274265322 >> 172.21.15.121:0/4274265322 conn(0x55b5a387ecc0 msgr2=0x55b5a394e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:54.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c63868b80 1 -- 172.21.15.121:0/4274265322 shutdown_connections 2024-06-10T00:32:54.927 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c63868b80 1 -- 172.21.15.121:0/4274265322 wait complete. 2024-06-10T00:32:54.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c63868b80 1 -- start start 2024-06-10T00:32:54.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c63868b80 1 Processor -- start 2024-06-10T00:32:54.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3953f20 0x55b5a3961890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a395eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3962060 0x55b5a395f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c63868b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b5a3964dd0 con 0x55b5a3953f20 2024-06-10T00:32:54.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c63868b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55b5a3845bf0 con 0x55b5a3955740 2024-06-10T00:32:54.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c63868b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b5a3964f50 con 0x55b5a3962060 2024-06-10T00:32:54.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c63868b80 10 client.0 _ll_register_callbacks cb 0x55b5a383f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:54.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c5259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3953f20 0x55b5a3961890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c51d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a395eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c5259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3953f20 0x55b5a3961890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47230/0 (socket says 172.21.15.121:47230) 2024-06-10T00:32:54.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c5259c700 1 -- 172.21.15.121:0/3719495079 learned_addr learned my addr 172.21.15.121:0/3719495079 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:54.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.925+0000 7f7c52d9d700 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3962060 0x55b5a395f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c51d9b700 1 -- 172.21.15.121:0/3719495079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3962060 msgr2=0x55b5a395f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:54.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c51d9b700 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3962060 0x55b5a395f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c51d9b700 1 -- 172.21.15.121:0/3719495079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3953f20 msgr2=0x55b5a3961890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:54.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c51d9b700 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3953f20 0x55b5a3961890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c51d9b700 1 -- 172.21.15.121:0/3719495079 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a37f48d0 con 0x55b5a3955740 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c5259c700 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3953f20 0x55b5a3961890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c51d9b700 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a395eb90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7c480cccb0 tx=0x7f7c4800cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14803cd98c353a0d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c39ffb700 1 -- 172.21.15.121:0/3719495079 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4809ba80 con 0x55b5a3955740 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c63868b80 1 -- 172.21.15.121:0/3719495079 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c4c006020 con 0x55b5a3955740 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c39ffb700 1 -- 172.21.15.121:0/3719495079 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f7c48010720 con 0x55b5a3955740 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c39ffb700 1 -- 172.21.15.121:0/3719495079 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7c4809fe30 con 0x55b5a3955740 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c63868b80 10 client.25772 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c63868b80 1 -- 172.21.15.121:0/3719495079 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b5a396f7a0 con 0x55b5a3955740 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.926+0000 7f7c39ffb700 1 -- 172.21.15.121:0/3719495079 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4c006020 con 0x55b5a3955740 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.927+0000 7f7c39ffb700 1 -- 172.21.15.121:0/3719495079 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7c4809a940 con 0x55b5a3955740 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.927+0000 7f7c63868b80 10 client.25772 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:54.933 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.927+0000 7f7c63868b80 10 client.25772 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:32:54.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.927+0000 7f7c63868b80 20 client.25772 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:54.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.927+0000 7f7c63868b80 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55b5a3971de0 0x55b5a39741d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.927+0000 7f7c63868b80 4 client.25772 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:32:54.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.927+0000 7f7c63868b80 1 -- 172.21.15.121:0/3719495079 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5a38b2d10 con 0x55b5a3971de0 2024-06-10T00:32:54.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.927+0000 7f7c5259c700 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55b5a3971de0 0x55b5a39741d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.927+0000 7f7c5259c700 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55b5a3971de0 0x55b5a39741d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=836dbf8a26bf4e3f server_cookie=6e582d3c84a4c26b in_seq=0 out_seq=0 2024-06-10T00:32:54.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.927+0000 7f7c39ffb700 10 client.25772 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:32:54.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.928+0000 7f7c39ffb700 1 -- 172.21.15.121:0/3719495079 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b5a38b2d10 con 0x55b5a3971de0 2024-06-10T00:32:54.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.928+0000 7f7c39ffb700 10 client.25772 handle_command_reply: tid=0 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 1 client.25772 shutdown 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 1 -- 172.21.15.121:0/3719495079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 msgr2=0x55b5a395eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a395eb90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7c480cccb0 tx=0x7f7c4800cd80 comp rx=0 tx=0).stop 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 1 -- 172.21.15.121:0/3719495079 shutdown_connections 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55b5a3971de0 0x55b5a39741d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3962060 0x55b5a395f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3953f20 0x55b5a3961890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 1 --2- 172.21.15.121:0/3719495079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a395eb90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 1 -- 172.21.15.121:0/3719495079 >> 172.21.15.121:0/3719495079 conn(0x55b5a387ecc0 msgr2=0x55b5a39509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c39ffb700 0 client.25772 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 1 -- 172.21.15.121:0/3719495079 shutdown_connections 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 1 -- 172.21.15.121:0/3719495079 wait complete. 2024-06-10T00:32:54.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.929+0000 7f7c63868b80 20 client.25772 trim_cache size 0 max 16384 2024-06-10T00:32:54.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.945+0000 7f7c63868b80 1 Processor -- start 2024-06-10T00:32:54.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.945+0000 7f7c63868b80 1 -- start start 2024-06-10T00:32:54.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.945+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a394e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.945+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a39529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.945+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 0x55b5a3952f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c63868b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7c541aea70 con 0x55b5a3962060 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c63868b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f7c541ac300 con 0x55b5a3955740 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c63868b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7c541aa150 con 0x55b5a3953f20 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c52d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 0x55b5a3952f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c5259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a394e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c51d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a39529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c52d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 0x55b5a3952f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47236/0 (socket says 172.21.15.121:47236) 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c52d9d700 1 -- 172.21.15.121:0/2799931273 learned_addr learned my addr 172.21.15.121:0/2799931273 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c5259c700 1 -- 172.21.15.121:0/2799931273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 msgr2=0x55b5a39529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c5259c700 1 --2- 172.21.15.121:0/2799931273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a39529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c5259c700 1 -- 172.21.15.121:0/2799931273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 msgr2=0x55b5a3952f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c5259c700 1 --2- 172.21.15.121:0/2799931273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 0x55b5a3952f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c5259c700 1 -- 172.21.15.121:0/2799931273 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a3955390 con 0x55b5a3953f20 2024-06-10T00:32:54.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c51d9b700 1 --2- 172.21.15.121:0/2799931273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a39529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:54.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c5259c700 1 --2- 172.21.15.121:0/2799931273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a394e940 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c44013010 tx=0x7f7c44014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be6c712e09465376 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:54.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c3bfff700 1 -- 172.21.15.121:0/2799931273 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4400e840 con 0x55b5a3953f20 2024-06-10T00:32:54.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c3bfff700 1 -- 172.21.15.121:0/2799931273 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4401fc70 con 0x55b5a3953f20 2024-06-10T00:32:54.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.946+0000 7f7c3bfff700 1 -- 172.21.15.121:0/2799931273 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4402ab60 con 0x55b5a3953f20 2024-06-10T00:32:54.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.947+0000 7f7c63868b80 1 -- 172.21.15.121:0/2799931273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 msgr2=0x55b5a394e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:54.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.947+0000 7f7c63868b80 1 --2- 172.21.15.121:0/2799931273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a394e940 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c44013010 tx=0x7f7c44014000 comp rx=0 tx=0).stop 2024-06-10T00:32:54.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.947+0000 7f7c63868b80 1 -- 172.21.15.121:0/2799931273 shutdown_connections 2024-06-10T00:32:54.951 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.947+0000 7f7c63868b80 1 --2- 172.21.15.121:0/2799931273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3962060 0x55b5a3952f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.947+0000 7f7c63868b80 1 --2- 172.21.15.121:0/2799931273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a394e940 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.947+0000 7f7c63868b80 1 --2- 172.21.15.121:0/2799931273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3955740 0x55b5a39529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.947+0000 7f7c63868b80 1 -- 172.21.15.121:0/2799931273 >> 172.21.15.121:0/2799931273 conn(0x55b5a387ecc0 msgr2=0x55b5a395eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.947+0000 7f7c63868b80 1 -- 172.21.15.121:0/2799931273 shutdown_connections 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.947+0000 7f7c63868b80 1 -- 172.21.15.121:0/2799931273 wait complete. 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.947+0000 7f7c63868b80 1 -- start start 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.947+0000 7f7c63868b80 1 Processor -- start 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a396a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3955740 0x55b5a396a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c63868b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3962060 0x55b5a395af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c63868b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7c541aa7c0 con 0x55b5a3955740 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c63868b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f7c541abc30 con 0x55b5a3962060 2024-06-10T00:32:54.952 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c63868b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7c541aae90 con 0x55b5a3953f20 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c63868b80 10 client.0 _ll_register_callbacks cb 0x55b5a383f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c51d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3955740 0x55b5a396a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c5259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a396a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c5259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a396a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:50492/0 (socket says 172.21.15.121:50492) 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c52d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3962060 0x55b5a395af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c51d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3955740 0x55b5a396a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47242/0 (socket says 172.21.15.121:47242) 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c5259c700 1 -- 172.21.15.121:0/2989938422 learned_addr learned my addr 172.21.15.121:0/2989938422 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c5259c700 1 -- 172.21.15.121:0/2989938422 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3962060 msgr2=0x55b5a395af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c5259c700 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3962060 0x55b5a395af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c5259c700 1 -- 172.21.15.121:0/2989938422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3955740 msgr2=0x55b5a396a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c5259c700 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3955740 0x55b5a396a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c5259c700 1 -- 172.21.15.121:0/2989938422 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a395b4c0 con 0x55b5a3953f20 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c51d9b700 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3955740 0x55b5a396a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:32:54.953 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c5259c700 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a396a1e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009090 tx=0x7f7c440090c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a052f886cd355c5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.948+0000 7f7c39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c39ffb700 1 -- 172.21.15.121:0/2989938422 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4402a5c0 con 0x55b5a3953f20 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c39ffb700 1 -- 172.21.15.121:0/2989938422 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f7c4401f790 con 0x55b5a3953f20 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c63868b80 1 -- 172.21.15.121:0/2989938422 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c4401fc70 con 0x55b5a3953f20 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c39ffb700 1 -- 172.21.15.121:0/2989938422 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7c4401f990 con 0x55b5a3953f20 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c63868b80 10 client.25678 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c63868b80 1 -- 172.21.15.121:0/2989938422 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b5a39dc0c0 con 0x55b5a3953f20 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c39ffb700 1 -- 172.21.15.121:0/2989938422 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4401fc70 con 0x55b5a3953f20 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c39ffb700 1 -- 172.21.15.121:0/2989938422 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7c4403a070 con 0x55b5a3953f20 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c63868b80 10 client.25678 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c63868b80 10 client.25678 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c63868b80 20 client.25678 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:54.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c63868b80 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55b5a39dc7a0 0x55b5a39deb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c63868b80 4 client.25678 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c63868b80 1 -- 172.21.15.121:0/2989938422 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5a38ad290 con 0x55b5a39dc7a0 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.949+0000 7f7c51d9b700 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55b5a39dc7a0 0x55b5a39deb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c51d9b700 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55b5a39dc7a0 0x55b5a39deb90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=75970ef24a522a76 server_cookie=d28e27629b423270 in_seq=0 out_seq=0 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c39ffb700 10 client.25678 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c39ffb700 1 -- 172.21.15.121:0/2989938422 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5a38ad290 con 0x55b5a39dc7a0 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c39ffb700 10 client.25678 handle_command_reply: tid=0 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 1 client.25678 shutdown 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 1 -- 172.21.15.121:0/2989938422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 msgr2=0x55b5a396a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a396a1e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009090 tx=0x7f7c440090c0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 1 -- 172.21.15.121:0/2989938422 shutdown_connections 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x55b5a39dc7a0 0x55b5a39deb90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5a3962060 0x55b5a395af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5a3953f20 0x55b5a396a1e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 1 --2- 172.21.15.121:0/2989938422 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5a3955740 0x55b5a396a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c39ffb700 0 client.25678 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:32:54.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 1 -- 172.21.15.121:0/2989938422 >> 172.21.15.121:0/2989938422 conn(0x55b5a387ecc0 msgr2=0x55b5a3947e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:54.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 1 -- 172.21.15.121:0/2989938422 shutdown_connections 2024-06-10T00:32:54.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 1 -- 172.21.15.121:0/2989938422 wait complete. 2024-06-10T00:32:54.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.950+0000 7f7c63868b80 20 client.25678 trim_cache size 0 max 16384 2024-06-10T00:32:54.956 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:32:54.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 -- 172.21.15.121:0/4028915107 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7c3c0a2d90 msgr2=0x7f7c3c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:54.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7c3c0a2d90 0x7f7c3c0a5250 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7c54106be0 tx=0x7f7c44009260 comp rx=0 tx=0).stop 2024-06-10T00:32:54.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 -- 172.21.15.121:0/4028915107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c54101d30 msgr2=0x7f7c5419dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:54.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c54101d30 0x7f7c5419dc30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800e9f0 tx=0x7f7c4800ed00 comp rx=0 tx=0).stop 2024-06-10T00:32:54.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 -- 172.21.15.121:0/4028915107 shutdown_connections 2024-06-10T00:32:54.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7c3c0a2d90 0x7f7c3c0a5250 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c541073d0 0x7f7c54197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c54101390 0x7f7c5419d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 --2- 172.21.15.121:0/4028915107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c54101d30 0x7f7c5419dc30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:54.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 -- 172.21.15.121:0/4028915107 >> 172.21.15.121:0/4028915107 conn(0x7f7c540782c0 msgr2=0x7f7c54100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:54.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 -- 172.21.15.121:0/4028915107 shutdown_connections 2024-06-10T00:32:54.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:54.951+0000 7f7c58fe5700 1 -- 172.21.15.121:0/4028915107 wait complete. 2024-06-10T00:32:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:55 smithi193 ceph-mon[36716]: pgmap v151: 129 pgs: 129 active+clean; 933 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.2 MiB/s wr, 500 op/s 2024-06-10T00:32:55.635 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:55 smithi121 ceph-mon[32378]: pgmap v151: 129 pgs: 129 active+clean; 933 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.2 MiB/s wr, 500 op/s 2024-06-10T00:32:55.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-10T00:32:55.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:55 smithi135 ceph-mon[36424]: pgmap v151: 129 pgs: 129 active+clean; 933 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.2 MiB/s wr, 500 op/s 2024-06-10T00:32:55.918 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:32:57.295 INFO:tasks.workunit.client.0.smithi121.stdout: 197 244818 155 174674 165 81809 380 2024-06-10T00:32:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:57 smithi193 ceph-mon[36716]: pgmap v152: 129 pgs: 129 active+clean; 966 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.2 MiB/s wr, 509 op/s 2024-06-10T00:32:57.646 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:57 smithi121 ceph-mon[32378]: pgmap v152: 129 pgs: 129 active+clean; 966 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.2 MiB/s wr, 509 op/s 2024-06-10T00:32:57.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:57 smithi135 ceph-mon[36424]: pgmap v152: 129 pgs: 129 active+clean; 966 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 6.2 MiB/s wr, 509 op/s 2024-06-10T00:32:58.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.507+0000 7feaf3102700 1 -- 172.21.15.121:0/4176114664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaec1096f0 msgr2=0x7feaec109ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:58.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.507+0000 7feaf3102700 1 --2- 172.21.15.121:0/4176114664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaec1096f0 0x7feaec109ad0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feadc009990 tx=0x7feadc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.509+0000 7feaf3102700 1 -- 172.21.15.121:0/4176114664 shutdown_connections 2024-06-10T00:32:58.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.509+0000 7feaf3102700 1 --2- 172.21.15.121:0/4176114664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaec10de50 0x7feaec07a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.509+0000 7feaf3102700 1 --2- 172.21.15.121:0/4176114664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaec1096f0 0x7feaec109ad0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.511 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.509+0000 7feaf3102700 1 --2- 172.21.15.121:0/4176114664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaec10a0a0 0x7feaec10d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.509+0000 7feaf3102700 1 -- 172.21.15.121:0/4176114664 >> 172.21.15.121:0/4176114664 conn(0x7feaec06d600 msgr2=0x7feaec06da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:58.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.509+0000 7feaf3102700 1 -- 172.21.15.121:0/4176114664 shutdown_connections 2024-06-10T00:32:58.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.509+0000 7feaf3102700 1 -- 172.21.15.121:0/4176114664 wait complete. 2024-06-10T00:32:58.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.510+0000 7feaf3102700 1 Processor -- start 2024-06-10T00:32:58.512 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.510+0000 7feaf3102700 1 -- start start 2024-06-10T00:32:58.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.512+0000 7feaf3102700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaec1096f0 0x7feaec07a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.512+0000 7feaf3102700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaec10a0a0 0x7feaec0747e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.512+0000 7feaf3102700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaec10de50 0x7feaec074f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.512+0000 7feaf3102700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feaec07d510 con 0x7feaec10de50 2024-06-10T00:32:58.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.512+0000 7feaf3102700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7feaec07d390 con 0x7feaec1096f0 2024-06-10T00:32:58.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.512+0000 7feaf3102700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feaec07d690 con 0x7feaec10a0a0 2024-06-10T00:32:58.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.515+0000 7feaf169f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaec10de50 0x7feaec074f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.515+0000 7feaf169f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaec10de50 0x7feaec074f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47252/0 (socket says 172.21.15.121:47252) 2024-06-10T00:32:58.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.515+0000 7feaf169f700 1 -- 172.21.15.121:0/2513157922 learned_addr learned my addr 172.21.15.121:0/2513157922 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:58.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.515+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaec1096f0 0x7feaec07a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.515+0000 7feaebfff700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaec10a0a0 0x7feaec0747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.516+0000 7feaebfff700 1 -- 172.21.15.121:0/2513157922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaec1096f0 msgr2=0x7feaec07a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:58.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.516+0000 7feaebfff700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaec1096f0 0x7feaec07a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.516+0000 7feaebfff700 1 -- 172.21.15.121:0/2513157922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaec10de50 msgr2=0x7feaec074f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:58.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.516+0000 7feaebfff700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaec10de50 0x7feaec074f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.516+0000 7feaebfff700 1 -- 172.21.15.121:0/2513157922 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaec137990 con 0x7feaec10a0a0 2024-06-10T00:32:58.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.517+0000 7feaebfff700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaec10a0a0 0x7feaec0747e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feaec06bc00 tx=0x7feae000b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:58.522 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.520+0000 7feae9ffb700 1 -- 172.21.15.121:0/2513157922 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae0015020 con 0x7feaec10a0a0 2024-06-10T00:32:58.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.520+0000 7feae9ffb700 1 -- 172.21.15.121:0/2513157922 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feae0011d10 con 0x7feaec10a0a0 2024-06-10T00:32:58.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.520+0000 7feae9ffb700 1 -- 172.21.15.121:0/2513157922 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae0010e10 con 0x7feaec10a0a0 2024-06-10T00:32:58.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.520+0000 7feaf3102700 1 -- 172.21.15.121:0/2513157922 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaec137c80 con 0x7feaec10a0a0 2024-06-10T00:32:58.523 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.520+0000 7feaf3102700 1 -- 172.21.15.121:0/2513157922 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaec138170 con 0x7feaec10a0a0 2024-06-10T00:32:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.530+0000 7feafd985b80 1 Processor -- start 2024-06-10T00:32:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.531+0000 7feae9ffb700 1 -- 172.21.15.121:0/2513157922 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7feae0018020 con 0x7feaec10a0a0 2024-06-10T00:32:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.533+0000 7feae9ffb700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fead40a2f10 0x7fead40a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.533+0000 7feae9ffb700 1 -- 172.21.15.121:0/2513157922 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7feae00cedd0 con 0x7feaec10a0a0 2024-06-10T00:32:58.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.534+0000 7feafd985b80 1 -- start start 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.534+0000 7feafd985b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5f640 0x55575cd5fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.534+0000 7feafd985b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.534+0000 7feafd985b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd5a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.534+0000 7feafd985b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55575ccab810 con 0x55575cd59f20 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.534+0000 7feafd985b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55575cc53720 con 0x55575cd5ff60 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.534+0000 7feafd985b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55575cca77d0 con 0x55575cd5f640 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.534+0000 7feaebfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.534+0000 7feaebfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:48872/0 (socket says 172.21.15.121:48872) 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.535+0000 7feaebfff700 1 -- 172.21.15.121:0/2959841069 learned_addr learned my addr 172.21.15.121:0/2959841069 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.535+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/2959841069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5f640 0x55575cd5fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.535+0000 7feaf0e9e700 1 -- 172.21.15.121:0/2959841069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 msgr2=0x55575cd599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.535+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/2959841069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.535+0000 7feaf0e9e700 1 -- 172.21.15.121:0/2959841069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 msgr2=0x55575cd5a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:58.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.535+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/2959841069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd5a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.535+0000 7feaf0e9e700 1 -- 172.21.15.121:0/2959841069 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55575cc49f50 con 0x55575cd5f640 2024-06-10T00:32:58.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.535+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/2959841069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5f640 0x55575cd5fa20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feadc0098e0 tx=0x7feadc005c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d983597896c3f5ef server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:58.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.536+0000 7feaf169f700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fead40a2f10 0x7fead40a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.536+0000 7fead2ffd700 1 -- 172.21.15.121:0/2959841069 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feadc003ca0 con 0x55575cd5f640 2024-06-10T00:32:58.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.536+0000 7fead2ffd700 1 -- 172.21.15.121:0/2959841069 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feadc003680 con 0x55575cd5f640 2024-06-10T00:32:58.537 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.536+0000 7fead2ffd700 1 -- 172.21.15.121:0/2959841069 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feadc005550 con 0x55575cd5f640 2024-06-10T00:32:58.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.538+0000 7feafd985b80 1 -- 172.21.15.121:0/2959841069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5f640 msgr2=0x55575cd5fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:58.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.538+0000 7feafd985b80 1 --2- 172.21.15.121:0/2959841069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5f640 0x55575cd5fa20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feadc0098e0 tx=0x7feadc005c60 comp rx=0 tx=0).stop 2024-06-10T00:32:58.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.541+0000 7feaf169f700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fead40a2f10 0x7fead40a53d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feae40070b0 tx=0x7feae4009580 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:58.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.541+0000 7feafd985b80 1 -- 172.21.15.121:0/2959841069 shutdown_connections 2024-06-10T00:32:58.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.541+0000 7feafd985b80 1 --2- 172.21.15.121:0/2959841069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd5a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.541+0000 7feafd985b80 1 --2- 172.21.15.121:0/2959841069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5f640 0x55575cd5fa20 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.541+0000 7feafd985b80 1 --2- 172.21.15.121:0/2959841069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.541+0000 7feafd985b80 1 -- 172.21.15.121:0/2959841069 >> 172.21.15.121:0/2959841069 conn(0x55575cc84cc0 msgr2=0x55575cd54850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:58.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.542+0000 7feafd985b80 1 -- 172.21.15.121:0/2959841069 shutdown_connections 2024-06-10T00:32:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.542+0000 7feafd985b80 1 -- 172.21.15.121:0/2959841069 wait complete. 2024-06-10T00:32:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.543+0000 7feafd985b80 1 -- start start 2024-06-10T00:32:58.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.543+0000 7feafd985b80 1 Processor -- start 2024-06-10T00:32:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.544+0000 7feafd985b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd65d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.544+0000 7feafd985b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5f640 0x55575cd63020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.544+0000 7feafd985b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5ff60 0x55575cd63560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.544+0000 7feafd985b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55575cd66970 con 0x55575cd59f20 2024-06-10T00:32:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.544+0000 7feafd985b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55575cc4bbf0 con 0x55575cd5f640 2024-06-10T00:32:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.544+0000 7feafd985b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55575cd66af0 con 0x55575cd5ff60 2024-06-10T00:32:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.544+0000 7feaf169f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5ff60 0x55575cd63560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.544+0000 7feaf169f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5ff60 0x55575cd63560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:50548/0 (socket says 172.21.15.121:50548) 2024-06-10T00:32:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.544+0000 7feaf169f700 1 -- 172.21.15.121:0/544938461 learned_addr learned my addr 172.21.15.121:0/544938461 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.545+0000 7feafd985b80 10 client.0 _ll_register_callbacks cb 0x55575cc45920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.545+0000 7feaf169f700 1 -- 172.21.15.121:0/544938461 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5f640 msgr2=0x55575cd63020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:32:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.545+0000 7feaf169f700 1 --2- 172.21.15.121:0/544938461 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5f640 0x55575cd63020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.545+0000 7feaf169f700 1 -- 172.21.15.121:0/544938461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 msgr2=0x55575cd65d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.545+0000 7feaf169f700 1 --2- 172.21.15.121:0/544938461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd65d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.545+0000 7feaf169f700 1 -- 172.21.15.121:0/544938461 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55575cbfa8d0 con 0x55575cd5ff60 2024-06-10T00:32:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.545+0000 7feaf169f700 1 --2- 172.21.15.121:0/544938461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5ff60 0x55575cd63560 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feae4009880 tx=0x7feae4004be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36715bdfbec8e9e5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:58.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.545+0000 7feafd985b80 1 -- 172.21.15.121:0/544938461 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feadc003680 con 0x55575cd5ff60 2024-06-10T00:32:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.545+0000 7feacbfff700 10 client.25687 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:32:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.545+0000 7feacbfff700 1 -- 172.21.15.121:0/544938461 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae4027060 con 0x55575cd5ff60 2024-06-10T00:32:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.545+0000 7feacbfff700 1 -- 172.21.15.121:0/544938461 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7feae4007c60 con 0x55575cd5ff60 2024-06-10T00:32:58.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.546+0000 7feacbfff700 1 -- 172.21.15.121:0/544938461 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feae4015ad0 con 0x55575cd5ff60 2024-06-10T00:32:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.549+0000 7feafd985b80 10 client.25687 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:58.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.549+0000 7feafd985b80 1 -- 172.21.15.121:0/544938461 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55575cd71270 con 0x55575cd5ff60 2024-06-10T00:32:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.551+0000 7feacbfff700 1 -- 172.21.15.121:0/544938461 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7feae4015c70 con 0x55575cd5ff60 2024-06-10T00:32:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.551+0000 7feafd985b80 10 client.25687 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.551+0000 7feafd985b80 10 client.25687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:32:58.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.551+0000 7feafd985b80 20 client.25687 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.551+0000 7feafd985b80 1 --2- 172.21.15.121:0/544938461 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55575cd73ba0 0x55575cd76060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.551+0000 7feafd985b80 4 client.25687 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:32:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.551+0000 7feafd985b80 1 -- 172.21.15.121:0/544938461 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7feadc003680 con 0x55575cd73ba0 2024-06-10T00:32:58.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.552+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/544938461 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55575cd73ba0 0x55575cd76060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.560+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/544938461 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55575cd73ba0 0x55575cd76060 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d885003aa593b0b server_cookie=72d4c99294a9f9e1 in_seq=0 out_seq=0 2024-06-10T00:32:58.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.560+0000 7feacbfff700 10 client.25687 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:58.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.566+0000 7feacbfff700 1 -- 172.21.15.121:0/544938461 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x7feadc003680 con 0x55575cd73ba0 2024-06-10T00:32:58.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.566+0000 7feacbfff700 10 client.25687 handle_command_reply: tid=0 2024-06-10T00:32:58.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.566+0000 7feafd985b80 1 client.25687 shutdown 2024-06-10T00:32:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.570+0000 7feafd985b80 1 -- 172.21.15.121:0/544938461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5ff60 msgr2=0x55575cd63560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.570+0000 7feafd985b80 1 --2- 172.21.15.121:0/544938461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5ff60 0x55575cd63560 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7feae4009880 tx=0x7feae4004be0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.570+0000 7feafd985b80 1 -- 172.21.15.121:0/544938461 shutdown_connections 2024-06-10T00:32:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.570+0000 7feafd985b80 1 --2- 172.21.15.121:0/544938461 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55575cd73ba0 0x55575cd76060 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.570+0000 7feafd985b80 1 --2- 172.21.15.121:0/544938461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5ff60 0x55575cd63560 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.570+0000 7feafd985b80 1 --2- 172.21.15.121:0/544938461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd65d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.570+0000 7feafd985b80 1 --2- 172.21.15.121:0/544938461 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5f640 0x55575cd63020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.570+0000 7feafd985b80 1 -- 172.21.15.121:0/544938461 >> 172.21.15.121:0/544938461 conn(0x55575cc84cc0 msgr2=0x55575cd589d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.570+0000 7feacbfff700 0 client.25687 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.570+0000 7feafd985b80 1 -- 172.21.15.121:0/544938461 shutdown_connections 2024-06-10T00:32:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.570+0000 7feafd985b80 1 -- 172.21.15.121:0/544938461 wait complete. 2024-06-10T00:32:58.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.571+0000 7feafd985b80 20 client.25687 trim_cache size 0 max 16384 2024-06-10T00:32:58.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.608+0000 7feafd985b80 1 Processor -- start 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.612+0000 7feafd985b80 1 -- start start 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.612+0000 7feafd985b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.613+0000 7feaf0e9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.613+0000 7feaf0e9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47274/0 (socket says 172.21.15.121:47274) 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.613+0000 7feaf0e9e700 1 -- 172.21.15.121:0/3960641315 learned_addr learned my addr 172.21.15.121:0/3960641315 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.613+0000 7feafd985b80 1 --2- 172.21.15.121:0/3960641315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5f640 0x55575cd5ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.613+0000 7feafd985b80 1 --2- 172.21.15.121:0/3960641315 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd5d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.613+0000 7feafd985b80 1 -- 172.21.15.121:0/3960641315 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feaec1d9580 con 0x55575cd59f20 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.613+0000 7feaebfff700 1 --2- 172.21.15.121:0/3960641315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5f640 0x55575cd5ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.613+0000 7feaf169f700 1 --2- 172.21.15.121:0/3960641315 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd5d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.613+0000 7feafd985b80 1 -- 172.21.15.121:0/3960641315 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7feaec1d6e10 con 0x55575cd5ff60 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.614+0000 7feafd985b80 1 -- 172.21.15.121:0/3960641315 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feaec1d4c60 con 0x55575cd5f640 2024-06-10T00:32:58.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.614+0000 7feaf0e9e700 1 -- 172.21.15.121:0/3960641315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5f640 msgr2=0x55575cd5ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.614+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/3960641315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5f640 0x55575cd5ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.614+0000 7feaf0e9e700 1 -- 172.21.15.121:0/3960641315 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 msgr2=0x55575cd5d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.614+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/3960641315 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd5d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.614+0000 7feaf0e9e700 1 -- 172.21.15.121:0/3960641315 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55575cd56130 con 0x55575cd59f20 2024-06-10T00:32:58.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.615+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/3960641315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd599e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7feadc006010 tx=0x7feadc00b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd354a119689e99 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:58.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.615+0000 7fead27fc700 1 -- 172.21.15.121:0/3960641315 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feadc003680 con 0x55575cd59f20 2024-06-10T00:32:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.615+0000 7fead27fc700 1 -- 172.21.15.121:0/3960641315 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feadc005650 con 0x55575cd59f20 2024-06-10T00:32:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.615+0000 7fead27fc700 1 -- 172.21.15.121:0/3960641315 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feadc00bc70 con 0x55575cd59f20 2024-06-10T00:32:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.617+0000 7feafd985b80 1 -- 172.21.15.121:0/3960641315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 msgr2=0x55575cd599e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.617+0000 7feafd985b80 1 --2- 172.21.15.121:0/3960641315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd599e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7feadc006010 tx=0x7feadc00b560 comp rx=0 tx=0).stop 2024-06-10T00:32:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.617+0000 7feafd985b80 1 -- 172.21.15.121:0/3960641315 shutdown_connections 2024-06-10T00:32:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.617+0000 7feafd985b80 1 --2- 172.21.15.121:0/3960641315 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd5d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.617+0000 7feafd985b80 1 --2- 172.21.15.121:0/3960641315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd59f20 0x55575cd599e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.617+0000 7feafd985b80 1 --2- 172.21.15.121:0/3960641315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd5f640 0x55575cd5ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.617+0000 7feafd985b80 1 -- 172.21.15.121:0/3960641315 >> 172.21.15.121:0/3960641315 conn(0x55575cc84cc0 msgr2=0x55575cd61970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.618+0000 7feafd985b80 1 -- 172.21.15.121:0/3960641315 shutdown_connections 2024-06-10T00:32:58.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.618+0000 7feafd985b80 1 -- 172.21.15.121:0/3960641315 wait complete. 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.618+0000 7feafd985b80 1 -- start start 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.619+0000 7feafd985b80 1 Processor -- start 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.619+0000 7feafd985b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd59f20 0x55575cd6bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.620+0000 7feafd985b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd5f640 0x55575cd6c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.620+0000 7feaebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd5f640 0x55575cd6c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.620+0000 7feaebfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd5f640 0x55575cd6c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47280/0 (socket says 172.21.15.121:47280) 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.620+0000 7feaf0e9e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd59f20 0x55575cd6bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.620+0000 7feaf0e9e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd59f20 0x55575cd6bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:50564/0 (socket says 172.21.15.121:50564) 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.620+0000 7feaebfff700 1 -- 172.21.15.121:0/2681724547 learned_addr learned my addr 172.21.15.121:0/2681724547 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.620+0000 7feafd985b80 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd7a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.621+0000 7feafd985b80 1 -- 172.21.15.121:0/2681724547 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feaec1d52d0 con 0x55575cd5f640 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.621+0000 7feafd985b80 1 -- 172.21.15.121:0/2681724547 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7feaec1d6740 con 0x55575cd5ff60 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.621+0000 7feaf169f700 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd7a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.621+0000 7feafd985b80 1 -- 172.21.15.121:0/2681724547 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feaec1d59a0 con 0x55575cd59f20 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.621+0000 7feaebfff700 1 -- 172.21.15.121:0/2681724547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd59f20 msgr2=0x55575cd6bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.621+0000 7feaebfff700 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd59f20 0x55575cd6bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.621+0000 7feaebfff700 1 -- 172.21.15.121:0/2681724547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 msgr2=0x55575cd7a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.621+0000 7feaebfff700 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd7a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.621+0000 7feaebfff700 1 -- 172.21.15.121:0/2681724547 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55575cd6c930 con 0x55575cd5f640 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.625+0000 7feaebfff700 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd5f640 0x55575cd6c2f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7feae0020000 tx=0x7feae00bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3096dc833fd666 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.626+0000 7feafd985b80 10 client.0 _ll_register_callbacks cb 0x55575cc45920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.626+0000 7feafd985b80 1 -- 172.21.15.121:0/2681724547 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feadc005650 con 0x55575cd5f640 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.626+0000 7feacbfff700 10 client.16227 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.626+0000 7feacbfff700 1 -- 172.21.15.121:0/2681724547 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae009b010 con 0x55575cd5f640 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.626+0000 7feacbfff700 1 -- 172.21.15.121:0/2681724547 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7feae009b270 con 0x55575cd5f640 2024-06-10T00:32:58.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.626+0000 7feacbfff700 1 -- 172.21.15.121:0/2681724547 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feae009f3c0 con 0x55575cd5f640 2024-06-10T00:32:58.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.628+0000 7feafd985b80 10 client.16227 fetch_fsmap learned FSMap version 28 2024-06-10T00:32:58.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.628+0000 7feafd985b80 1 -- 172.21.15.121:0/2681724547 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55575cdddc30 con 0x55575cd5f640 2024-06-10T00:32:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.629+0000 7feacbfff700 1 -- 172.21.15.121:0/2681724547 <== mon.0 v2:172.21.15.121:3300/0 4 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7feae00a37f0 con 0x55575cd5f640 2024-06-10T00:32:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.629+0000 7feafd985b80 10 client.16227 fetch_fsmap finished waiting for FSMap version 28 2024-06-10T00:32:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.629+0000 7feafd985b80 10 client.16227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:32:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.629+0000 7feafd985b80 20 client.16227 populate_metadata read hostname 'smithi121' 2024-06-10T00:32:58.629 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.629+0000 7feafd985b80 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55575cdde350 0x55575cde0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:32:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.629+0000 7feafd985b80 4 client.16227 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:32:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.629+0000 7feafd985b80 1 -- 172.21.15.121:0/2681724547 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7feadc005650 con 0x55575cdde350 2024-06-10T00:32:58.630 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.629+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55575cdde350 0x55575cde0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:32:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.630+0000 7feaf0e9e700 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55575cdde350 0x55575cde0810 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1fa748bfc0c07cf server_cookie=de1659bf22666427 in_seq=0 out_seq=0 2024-06-10T00:32:58.631 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.630+0000 7feacbfff700 10 client.16227 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:58.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.635+0000 7feacbfff700 1 -- 172.21.15.121:0/2681724547 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7feadc005650 con 0x55575cdde350 2024-06-10T00:32:58.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.635+0000 7feacbfff700 10 client.16227 handle_command_reply: tid=0 2024-06-10T00:32:58.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.635+0000 7feafd985b80 1 client.16227 shutdown 2024-06-10T00:32:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.640+0000 7feafd985b80 1 -- 172.21.15.121:0/2681724547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd5f640 msgr2=0x55575cd6c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.640+0000 7feafd985b80 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd5f640 0x55575cd6c2f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7feae0020000 tx=0x7feae00bb040 comp rx=0 tx=0).stop 2024-06-10T00:32:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.640+0000 7feafd985b80 1 -- 172.21.15.121:0/2681724547 shutdown_connections 2024-06-10T00:32:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.640+0000 7feafd985b80 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55575cdde350 0x55575cde0810 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.640+0000 7feafd985b80 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55575cd5ff60 0x55575cd7a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.640+0000 7feafd985b80 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55575cd59f20 0x55575cd6bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.640+0000 7feafd985b80 1 --2- 172.21.15.121:0/2681724547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55575cd5f640 0x55575cd6c2f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.640+0000 7feafd985b80 1 -- 172.21.15.121:0/2681724547 >> 172.21.15.121:0/2681724547 conn(0x55575cc84cc0 msgr2=0x55575cd63540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:58.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.642+0000 7feacbfff700 0 client.16227 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:32:58.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.642+0000 7feafd985b80 1 -- 172.21.15.121:0/2681724547 shutdown_connections 2024-06-10T00:32:58.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.642+0000 7feafd985b80 1 -- 172.21.15.121:0/2681724547 wait complete. 2024-06-10T00:32:58.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.642+0000 7feafd985b80 20 client.16227 trim_cache size 0 max 16384 2024-06-10T00:32:58.645 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:32:58.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 -- 172.21.15.121:0/2513157922 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fead40a2f10 msgr2=0x7fead40a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:58.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fead40a2f10 0x7fead40a53d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7feae40070b0 tx=0x7feae4009580 comp rx=0 tx=0).stop 2024-06-10T00:32:58.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 -- 172.21.15.121:0/2513157922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaec10a0a0 msgr2=0x7feaec0747e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:32:58.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaec10a0a0 0x7feaec0747e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feaec06bc00 tx=0x7feae000b7e0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 -- 172.21.15.121:0/2513157922 shutdown_connections 2024-06-10T00:32:58.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fead40a2f10 0x7fead40a53d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaec10de50 0x7feaec074f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feaec1096f0 0x7feaec07a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 --2- 172.21.15.121:0/2513157922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaec10a0a0 0x7feaec0747e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:32:58.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 -- 172.21.15.121:0/2513157922 >> 172.21.15.121:0/2513157922 conn(0x7feaec06d600 msgr2=0x7feaec07ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:32:58.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 -- 172.21.15.121:0/2513157922 shutdown_connections 2024-06-10T00:32:58.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:32:58.650+0000 7feaf3102700 1 -- 172.21.15.121:0/2513157922 wait complete. 2024-06-10T00:32:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:32:59 smithi193 ceph-mon[36716]: pgmap v153: 129 pgs: 129 active+clean; 966 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 453 op/s 2024-06-10T00:32:59.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:32:59 smithi121 ceph-mon[32378]: pgmap v153: 129 pgs: 129 active+clean; 966 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 453 op/s 2024-06-10T00:32:59.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:32:59 smithi135 ceph-mon[36424]: pgmap v153: 129 pgs: 129 active+clean; 966 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.8 MiB/s wr, 453 op/s 2024-06-10T00:33:00.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:00 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:33:00.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:00 smithi121 ceph-mon[32378]: pgmap v154: 129 pgs: 129 active+clean; 976 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.7 MiB/s wr, 513 op/s 2024-06-10T00:33:00.503 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-10T00:33:00.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-10T00:33:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:00 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:33:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:00 smithi193 ceph-mon[36716]: pgmap v154: 129 pgs: 129 active+clean; 976 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.7 MiB/s wr, 513 op/s 2024-06-10T00:33:00.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:00 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:33:00.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:00 smithi135 ceph-mon[36424]: pgmap v154: 129 pgs: 129 active+clean; 976 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.7 MiB/s wr, 513 op/s 2024-06-10T00:33:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:33:02.875 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:02 smithi121 ceph-mon[32378]: pgmap v155: 129 pgs: 129 active+clean; 976 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 4.4 MiB/s wr, 278 op/s 2024-06-10T00:33:02.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:02 smithi193 ceph-mon[36716]: pgmap v155: 129 pgs: 129 active+clean; 976 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 4.4 MiB/s wr, 278 op/s 2024-06-10T00:33:03.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:02 smithi135 ceph-mon[36424]: pgmap v155: 129 pgs: 129 active+clean; 976 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 4.4 MiB/s wr, 278 op/s 2024-06-10T00:33:03.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.860+0000 7f8f6bb57700 1 -- 172.21.15.121:0/2872528824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f64109e50 msgr2=0x7f8f64115d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:03.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.860+0000 7f8f6bb57700 1 --2- 172.21.15.121:0/2872528824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f64109e50 0x7f8f64115d80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00af90 tx=0x7f8f5c008850 comp rx=0 tx=0).stop 2024-06-10T00:33:03.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.868+0000 7f8f6bb57700 1 -- 172.21.15.121:0/2872528824 shutdown_connections 2024-06-10T00:33:03.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.868+0000 7f8f6bb57700 1 --2- 172.21.15.121:0/2872528824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f64109e50 0x7f8f64115d80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.868+0000 7f8f6bb57700 1 --2- 172.21.15.121:0/2872528824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6410dee0 0x7f8f6410e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.868+0000 7f8f6bb57700 1 --2- 172.21.15.121:0/2872528824 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f64109490 0x7f8f64109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.868+0000 7f8f6bb57700 1 -- 172.21.15.121:0/2872528824 >> 172.21.15.121:0/2872528824 conn(0x7f8f6406d9e0 msgr2=0x7f8f6406ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:03.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.876+0000 7f8f6bb57700 1 -- 172.21.15.121:0/2872528824 shutdown_connections 2024-06-10T00:33:03.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.876+0000 7f8f6bb57700 1 -- 172.21.15.121:0/2872528824 wait complete. 2024-06-10T00:33:03.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.877+0000 7f8f6bb57700 1 Processor -- start 2024-06-10T00:33:03.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.881+0000 7f8f6bb57700 1 -- start start 2024-06-10T00:33:03.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.881+0000 7f8f6bb57700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f64109490 0x7f8f641b0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:03.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.881+0000 7f8f6bb57700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f64109e50 0x7f8f641b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:03.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.882+0000 7f8f6bb57700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f6410dee0 0x7f8f641b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:03.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.882+0000 7f8f6bb57700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8f64118e60 con 0x7f8f64109490 2024-06-10T00:33:03.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.882+0000 7f8f6bb57700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8f64118ce0 con 0x7f8f6410dee0 2024-06-10T00:33:03.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.882+0000 7f8f6bb57700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8f64118fe0 con 0x7f8f64109e50 2024-06-10T00:33:03.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.882+0000 7f8f698f3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f64109490 0x7f8f641b0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:03.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.882+0000 7f8f690f2700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f64109e50 0x7f8f641b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:03.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.882+0000 7f8f698f3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f64109490 0x7f8f641b0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60258/0 (socket says 172.21.15.121:60258) 2024-06-10T00:33:03.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.882+0000 7f8f698f3700 1 -- 172.21.15.121:0/1330759117 learned_addr learned my addr 172.21.15.121:0/1330759117 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:03.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.882+0000 7f8f6a0f4700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f6410dee0 0x7f8f641b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:03.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.883+0000 7f8f698f3700 1 -- 172.21.15.121:0/1330759117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f64109e50 msgr2=0x7f8f641b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:03.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.883+0000 7f8f698f3700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f64109e50 0x7f8f641b0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.883+0000 7f8f698f3700 1 -- 172.21.15.121:0/1330759117 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f6410dee0 msgr2=0x7f8f641b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:03.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.883+0000 7f8f698f3700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f6410dee0 0x7f8f641b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.883+0000 7f8f698f3700 1 -- 172.21.15.121:0/1330759117 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f641b5210 con 0x7f8f64109490 2024-06-10T00:33:03.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.883+0000 7f8f6a0f4700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f6410dee0 0x7f8f641b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:33:03.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.883+0000 7f8f690f2700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f64109e50 0x7f8f641b0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:33:03.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.883+0000 7f8f698f3700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f64109490 0x7f8f641b0200 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f8f5800bb40 tx=0x7f8f5800bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:03.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.885+0000 7f8f56ffd700 1 -- 172.21.15.121:0/1330759117 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f58008960 con 0x7f8f64109490 2024-06-10T00:33:03.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.885+0000 7f8f56ffd700 1 -- 172.21.15.121:0/1330759117 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f58004830 con 0x7f8f64109490 2024-06-10T00:33:03.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.885+0000 7f8f56ffd700 1 -- 172.21.15.121:0/1330759117 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f58005520 con 0x7f8f64109490 2024-06-10T00:33:03.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.885+0000 7f8f6bb57700 1 -- 172.21.15.121:0/1330759117 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f641b5500 con 0x7f8f64109490 2024-06-10T00:33:03.887 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.885+0000 7f8f6bb57700 1 -- 172.21.15.121:0/1330759117 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f641bcd80 con 0x7f8f64109490 2024-06-10T00:33:03.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.893+0000 7f8f56ffd700 1 -- 172.21.15.121:0/1330759117 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8f58016420 con 0x7f8f64109490 2024-06-10T00:33:03.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.904+0000 7f8f763dab80 1 Processor -- start 2024-06-10T00:33:03.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.904+0000 7f8f56ffd700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8f500a2f90 0x7f8f500a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:03.905 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.904+0000 7f8f56ffd700 1 -- 172.21.15.121:0/1330759117 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8f580cfc20 con 0x7f8f64109490 2024-06-10T00:33:03.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.905+0000 7f8f763dab80 1 -- start start 2024-06-10T00:33:03.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.905+0000 7f8f763dab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 0x564545346aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:03.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.906+0000 7f8f6a0f4700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8f500a2f90 0x7f8f500a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:03.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.906+0000 7f8f6a0f4700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8f500a2f90 0x7f8f500a5450 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c008330 tx=0x7f8f5c005c00 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:03.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.906+0000 7f8f763dab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564545346fe0 0x564545340a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:03.912 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.907+0000 7f8f690f2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564545346fe0 0x564545340a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.907+0000 7f8f690f2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564545346fe0 0x564545340a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60268/0 (socket says 172.21.15.121:60268) 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.907+0000 7f8f698f3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 0x564545346aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.907+0000 7f8f698f3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 0x564545346aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:47630/0 (socket says 172.21.15.121:47630) 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.907+0000 7f8f690f2700 1 -- 172.21.15.121:0/2349800177 learned_addr learned my addr 172.21.15.121:0/2349800177 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.907+0000 7f8f763dab80 1 --2- 172.21.15.121:0/2349800177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x5645453413c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.907+0000 7f8f763dab80 1 -- 172.21.15.121:0/2349800177 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564545292890 con 0x564545346fe0 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.907+0000 7f8f763dab80 1 -- 172.21.15.121:0/2349800177 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x56454523a7a0 con 0x5645453466c0 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.907+0000 7f8f763dab80 1 -- 172.21.15.121:0/2349800177 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56454528e850 con 0x564545340fa0 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.908+0000 7f8f690f2700 1 -- 172.21.15.121:0/2349800177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 msgr2=0x5645453413c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.908+0000 7f8f690f2700 1 --2- 172.21.15.121:0/2349800177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x5645453413c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.908+0000 7f8f690f2700 1 -- 172.21.15.121:0/2349800177 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 msgr2=0x564545346aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.908+0000 7f8f690f2700 1 --2- 172.21.15.121:0/2349800177 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 0x564545346aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.908+0000 7f8f690f2700 1 -- 172.21.15.121:0/2349800177 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564545230fd0 con 0x564545346fe0 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.908+0000 7f8f690f2700 1 --2- 172.21.15.121:0/2349800177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564545346fe0 0x564545340a60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000c4f0 tx=0x7f8f6000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af487f26c40c0e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.909+0000 7f8f4ffff700 1 -- 172.21.15.121:0/2349800177 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6001a070 con 0x564545346fe0 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.909+0000 7f8f4ffff700 1 -- 172.21.15.121:0/2349800177 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6000c0d0 con 0x564545346fe0 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.911+0000 7f8f763dab80 1 -- 172.21.15.121:0/2349800177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564545346fe0 msgr2=0x564545340a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.911+0000 7f8f763dab80 1 --2- 172.21.15.121:0/2349800177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564545346fe0 0x564545340a60 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000c4f0 tx=0x7f8f6000c520 comp rx=0 tx=0).stop 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.911+0000 7f8f763dab80 1 -- 172.21.15.121:0/2349800177 shutdown_connections 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.911+0000 7f8f763dab80 1 --2- 172.21.15.121:0/2349800177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x5645453413c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.911+0000 7f8f763dab80 1 --2- 172.21.15.121:0/2349800177 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 0x564545346aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.911+0000 7f8f763dab80 1 --2- 172.21.15.121:0/2349800177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564545346fe0 0x564545340a60 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.911+0000 7f8f763dab80 1 -- 172.21.15.121:0/2349800177 >> 172.21.15.121:0/2349800177 conn(0x56454526bd40 msgr2=0x56454533b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.911+0000 7f8f763dab80 1 -- 172.21.15.121:0/2349800177 shutdown_connections 2024-06-10T00:33:03.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.911+0000 7f8f763dab80 1 -- 172.21.15.121:0/2349800177 wait complete. 2024-06-10T00:33:03.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.912+0000 7f8f763dab80 1 -- start start 2024-06-10T00:33:03.915 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.912+0000 7f8f763dab80 1 Processor -- start 2024-06-10T00:33:03.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.915+0000 7f8f763dab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x56454534cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:03.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.915+0000 7f8f763dab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 0x56454534d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:03.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.915+0000 7f8f698f3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x56454534cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:03.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.915+0000 7f8f690f2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 0x56454534d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:03.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.915+0000 7f8f690f2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 0x56454534d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60270/0 (socket says 172.21.15.121:60270) 2024-06-10T00:33:03.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.915+0000 7f8f698f3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x56454534cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:54116/0 (socket says 172.21.15.121:54116) 2024-06-10T00:33:03.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.915+0000 7f8f690f2700 1 -- 172.21.15.121:0/1126081516 learned_addr learned my addr 172.21.15.121:0/1126081516 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:03.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.915+0000 7f8f763dab80 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564545349f90 0x56454534a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:03.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.915+0000 7f8f763dab80 1 -- 172.21.15.121:0/1126081516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56454534daa0 con 0x5645453466c0 2024-06-10T00:33:03.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.915+0000 7f8f763dab80 1 -- 172.21.15.121:0/1126081516 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x564545232c70 con 0x564545349f90 2024-06-10T00:33:03.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.915+0000 7f8f763dab80 1 -- 172.21.15.121:0/1126081516 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56454534dc20 con 0x564545340fa0 2024-06-10T00:33:03.917 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.916+0000 7f8f6a0f4700 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564545349f90 0x56454534a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:03.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.916+0000 7f8f690f2700 1 -- 172.21.15.121:0/1126081516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 msgr2=0x56454534cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:03.918 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.916+0000 7f8f690f2700 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x56454534cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.916+0000 7f8f690f2700 1 -- 172.21.15.121:0/1126081516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564545349f90 msgr2=0x56454534a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:03.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.916+0000 7f8f690f2700 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564545349f90 0x56454534a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.916+0000 7f8f690f2700 1 -- 172.21.15.121:0/1126081516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645451e1950 con 0x5645453466c0 2024-06-10T00:33:03.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.916+0000 7f8f690f2700 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 0x56454534d070 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000c870 tx=0x7f8f60003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94681319a7ddf5a4 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:03.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.917+0000 7f8f763dab80 10 client.0 _ll_register_callbacks cb 0x56454522c9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:03.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.917+0000 7f8f763dab80 1 -- 172.21.15.121:0/1126081516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56454529fd90 con 0x5645453466c0 2024-06-10T00:33:03.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.917+0000 7f8f4d7fa700 10 client.16239 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:33:03.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.917+0000 7f8f4d7fa700 1 -- 172.21.15.121:0/1126081516 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6001a040 con 0x5645453466c0 2024-06-10T00:33:03.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.917+0000 7f8f4d7fa700 1 -- 172.21.15.121:0/1126081516 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8f60015410 con 0x5645453466c0 2024-06-10T00:33:03.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.918+0000 7f8f4d7fa700 1 -- 172.21.15.121:0/1126081516 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8f6002d050 con 0x5645453466c0 2024-06-10T00:33:03.919 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.918+0000 7f8f763dab80 10 client.16239 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:03.920 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.918+0000 7f8f763dab80 1 -- 172.21.15.121:0/1126081516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({fsmap=30}) -- 0x564545358180 con 0x5645453466c0 2024-06-10T00:33:03.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.918+0000 7f8f4d7fa700 1 -- 172.21.15.121:0/1126081516 <== mon.0 v2:172.21.15.121:3300/0 4 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8f60015610 con 0x5645453466c0 2024-06-10T00:33:03.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.918+0000 7f8f763dab80 10 client.16239 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:03.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.918+0000 7f8f763dab80 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:33:03.922 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.918+0000 7f8f763dab80 20 client.16239 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:03.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.918+0000 7f8f763dab80 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x56454535abe0 0x56454535d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:03.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.918+0000 7f8f763dab80 4 client.16239 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:33:03.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.918+0000 7f8f763dab80 1 -- 172.21.15.121:0/1126081516 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56454529fd90 con 0x56454535abe0 2024-06-10T00:33:03.923 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.920+0000 7f8f698f3700 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x56454535abe0 0x56454535d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:03.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.929+0000 7f8f698f3700 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x56454535abe0 0x56454535d0a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a895aa9b519f27c server_cookie=488910bd9b312047 in_seq=0 out_seq=0 2024-06-10T00:33:03.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.929+0000 7f8f4d7fa700 10 client.16239 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:03.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.936+0000 7f8f4d7fa700 1 -- 172.21.15.121:0/1126081516 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56454529fd90 con 0x56454535abe0 2024-06-10T00:33:03.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.937+0000 7f8f4d7fa700 10 client.16239 handle_command_reply: tid=0 2024-06-10T00:33:03.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.937+0000 7f8f763dab80 1 client.16239 shutdown 2024-06-10T00:33:03.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.937+0000 7f8f763dab80 1 -- 172.21.15.121:0/1126081516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 msgr2=0x56454534d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:03.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.937+0000 7f8f763dab80 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 0x56454534d070 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000c870 tx=0x7f8f60003f90 comp rx=0 tx=0).stop 2024-06-10T00:33:03.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.937+0000 7f8f763dab80 1 -- 172.21.15.121:0/1126081516 shutdown_connections 2024-06-10T00:33:03.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.937+0000 7f8f763dab80 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x56454535abe0 0x56454535d0a0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.937+0000 7f8f763dab80 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564545349f90 0x56454534a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.937+0000 7f8f763dab80 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x56454534cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.937+0000 7f8f763dab80 1 --2- 172.21.15.121:0/1126081516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 0x56454534d070 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:03.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.937+0000 7f8f763dab80 1 -- 172.21.15.121:0/1126081516 >> 172.21.15.121:0/1126081516 conn(0x56454526bd40 msgr2=0x56454533b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:03.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.938+0000 7f8f4d7fa700 0 client.16239 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:03.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.938+0000 7f8f763dab80 1 -- 172.21.15.121:0/1126081516 shutdown_connections 2024-06-10T00:33:03.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.938+0000 7f8f763dab80 1 -- 172.21.15.121:0/1126081516 wait complete. 2024-06-10T00:33:03.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.940+0000 7f8f763dab80 20 client.16239 trim_cache size 0 max 16384 2024-06-10T00:33:03.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.990+0000 7f8f763dab80 1 Processor -- start 2024-06-10T00:33:03.990 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.990+0000 7f8f763dab80 1 -- start start 2024-06-10T00:33:04.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.990+0000 7f8f763dab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564545340fa0 0x564545340a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:04.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.990+0000 7f8f763dab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 0x564545347ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:04.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.990+0000 7f8f763dab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545349f90 0x5645453629b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:04.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.990+0000 7f8f763dab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8f48011750 con 0x564545340fa0 2024-06-10T00:33:04.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.990+0000 7f8f763dab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4800ef80 con 0x5645453466c0 2024-06-10T00:33:04.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.990+0000 7f8f763dab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4800cdd0 con 0x564545349f90 2024-06-10T00:33:04.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.993+0000 7f8f6a0f4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545349f90 0x5645453629b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:04.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.993+0000 7f8f690f2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 0x564545347ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:04.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.993+0000 7f8f690f2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 0x564545347ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:47654/0 (socket says 172.21.15.121:47654) 2024-06-10T00:33:04.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:03.993+0000 7f8f690f2700 1 -- 172.21.15.121:0/2965614002 learned_addr learned my addr 172.21.15.121:0/2965614002 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:04.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.000+0000 7f8f690f2700 1 -- 172.21.15.121:0/2965614002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545349f90 msgr2=0x5645453629b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:04.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.000+0000 7f8f690f2700 1 --2- 172.21.15.121:0/2965614002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545349f90 0x5645453629b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.000+0000 7f8f690f2700 1 -- 172.21.15.121:0/2965614002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564545340fa0 msgr2=0x564545340a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:04.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.000+0000 7f8f690f2700 1 --2- 172.21.15.121:0/2965614002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564545340fa0 0x564545340a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.000+0000 7f8f690f2700 1 -- 172.21.15.121:0/2965614002 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56454534c770 con 0x5645453466c0 2024-06-10T00:33:04.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.000+0000 7f8f690f2700 1 --2- 172.21.15.121:0/2965614002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 0x564545347ab0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8f60000f80 tx=0x7f8f60012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0c25759cae3b4a4 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:04.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.000+0000 7f8f4f7fe700 1 -- 172.21.15.121:0/2965614002 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6001a040 con 0x5645453466c0 2024-06-10T00:33:04.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.001+0000 7f8f4f7fe700 1 -- 172.21.15.121:0/2965614002 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f60024070 con 0x5645453466c0 2024-06-10T00:33:04.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.001+0000 7f8f4f7fe700 1 -- 172.21.15.121:0/2965614002 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f60004a00 con 0x5645453466c0 2024-06-10T00:33:04.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.004+0000 7f8f763dab80 1 -- 172.21.15.121:0/2965614002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 msgr2=0x564545347ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:04.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.004+0000 7f8f763dab80 1 --2- 172.21.15.121:0/2965614002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 0x564545347ab0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8f60000f80 tx=0x7f8f60012000 comp rx=0 tx=0).stop 2024-06-10T00:33:04.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.004+0000 7f8f763dab80 1 -- 172.21.15.121:0/2965614002 shutdown_connections 2024-06-10T00:33:04.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.004+0000 7f8f763dab80 1 --2- 172.21.15.121:0/2965614002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545349f90 0x5645453629b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.004+0000 7f8f763dab80 1 --2- 172.21.15.121:0/2965614002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564545340fa0 0x564545340a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.004+0000 7f8f763dab80 1 --2- 172.21.15.121:0/2965614002 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5645453466c0 0x564545347ab0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.004+0000 7f8f763dab80 1 -- 172.21.15.121:0/2965614002 >> 172.21.15.121:0/2965614002 conn(0x56454526bd40 msgr2=0x564545348740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:04.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.005+0000 7f8f763dab80 1 -- 172.21.15.121:0/2965614002 shutdown_connections 2024-06-10T00:33:04.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.005+0000 7f8f763dab80 1 -- 172.21.15.121:0/2965614002 wait complete. 2024-06-10T00:33:04.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.006+0000 7f8f763dab80 1 -- start start 2024-06-10T00:33:04.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.006+0000 7f8f763dab80 1 Processor -- start 2024-06-10T00:33:04.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.010+0000 7f8f763dab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x564545352f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:04.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.010+0000 7f8f763dab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 0x5645453534d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:04.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.010+0000 7f8f763dab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564545349f90 0x564545361ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:04.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.010+0000 7f8f763dab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4800d440 con 0x5645453466c0 2024-06-10T00:33:04.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.010+0000 7f8f763dab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4800e8b0 con 0x564545349f90 2024-06-10T00:33:04.011 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.010+0000 7f8f763dab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4800db10 con 0x564545340fa0 2024-06-10T00:33:04.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.011+0000 7f8f763dab80 10 client.0 _ll_register_callbacks cb 0x56454522c9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:04.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.018+0000 7f8f6a0f4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564545349f90 0x564545361ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:04.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.019+0000 7f8f690f2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 0x5645453534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:04.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.019+0000 7f8f690f2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 0x5645453534d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60294/0 (socket says 172.21.15.121:60294) 2024-06-10T00:33:04.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.019+0000 7f8f690f2700 1 -- 172.21.15.121:0/504839003 learned_addr learned my addr 172.21.15.121:0/504839003 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:04.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.019+0000 7f8f698f3700 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x564545352f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:04.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.022+0000 7f8f698f3700 1 -- 172.21.15.121:0/504839003 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564545349f90 msgr2=0x564545361ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:04.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.022+0000 7f8f698f3700 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564545349f90 0x564545361ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.022+0000 7f8f698f3700 1 -- 172.21.15.121:0/504839003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 msgr2=0x5645453534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:04.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.022+0000 7f8f698f3700 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 0x5645453534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.022+0000 7f8f698f3700 1 -- 172.21.15.121:0/504839003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564545362030 con 0x564545340fa0 2024-06-10T00:33:04.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.024+0000 7f8f698f3700 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x564545352f90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8f580ccf80 tx=0x7f8f5809b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c016d22834268d16 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:04.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.024+0000 7f8f4d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:33:04.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.025+0000 7f8f4d7fa700 1 -- 172.21.15.121:0/504839003 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f580a49c0 con 0x564545340fa0 2024-06-10T00:33:04.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.025+0000 7f8f763dab80 1 -- 172.21.15.121:0/504839003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f60024070 con 0x564545340fa0 2024-06-10T00:33:04.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.025+0000 7f8f4d7fa700 1 -- 172.21.15.121:0/504839003 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f8f580cd200 con 0x564545340fa0 2024-06-10T00:33:04.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.025+0000 7f8f4d7fa700 1 -- 172.21.15.121:0/504839003 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8f580cd400 con 0x564545340fa0 2024-06-10T00:33:04.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.027+0000 7f8f763dab80 10 client.25708 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:04.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.027+0000 7f8f763dab80 1 -- 172.21.15.121:0/504839003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5645453c5220 con 0x564545340fa0 2024-06-10T00:33:04.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.030+0000 7f8f4d7fa700 1 -- 172.21.15.121:0/504839003 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f60024070 con 0x564545340fa0 2024-06-10T00:33:04.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.030+0000 7f8f4d7fa700 1 -- 172.21.15.121:0/504839003 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f8f580a4530 con 0x564545340fa0 2024-06-10T00:33:04.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.030+0000 7f8f763dab80 10 client.25708 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:04.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.030+0000 7f8f763dab80 10 client.25708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:33:04.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.030+0000 7f8f763dab80 20 client.25708 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:04.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.030+0000 7f8f763dab80 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5645453c59d0 0x5645453c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:04.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.030+0000 7f8f763dab80 4 client.25708 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-10T00:33:04.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.030+0000 7f8f763dab80 1 -- 172.21.15.121:0/504839003 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56454529a310 con 0x5645453c59d0 2024-06-10T00:33:04.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.030+0000 7f8f690f2700 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5645453c59d0 0x5645453c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:04.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.040+0000 7f8f690f2700 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5645453c59d0 0x5645453c7df0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba1634c9a7716c76 server_cookie=53233eca1152e044 in_seq=0 out_seq=0 2024-06-10T00:33:04.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.043+0000 7f8f4d7fa700 10 client.25708 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:04.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.044+0000 7f8f4d7fa700 1 -- 172.21.15.121:0/504839003 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56454529a310 con 0x5645453c59d0 2024-06-10T00:33:04.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.044+0000 7f8f4d7fa700 10 client.25708 handle_command_reply: tid=0 2024-06-10T00:33:04.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.044+0000 7f8f763dab80 1 client.25708 shutdown 2024-06-10T00:33:04.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.047+0000 7f8f763dab80 1 -- 172.21.15.121:0/504839003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 msgr2=0x564545352f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:04.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.047+0000 7f8f763dab80 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x564545352f90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8f580ccf80 tx=0x7f8f5809b7b0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.049+0000 7f8f763dab80 1 -- 172.21.15.121:0/504839003 shutdown_connections 2024-06-10T00:33:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.049+0000 7f8f763dab80 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5645453c59d0 0x5645453c7df0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.049+0000 7f8f763dab80 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564545349f90 0x564545361ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.049+0000 7f8f763dab80 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564545340fa0 0x564545352f90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.049+0000 7f8f763dab80 1 --2- 172.21.15.121:0/504839003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5645453466c0 0x5645453534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.049+0000 7f8f763dab80 1 -- 172.21.15.121:0/504839003 >> 172.21.15.121:0/504839003 conn(0x56454526bd40 msgr2=0x564545334e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:04.051 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.049+0000 7f8f4d7fa700 0 client.25708 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.054+0000 7f8f763dab80 1 -- 172.21.15.121:0/504839003 shutdown_connections 2024-06-10T00:33:04.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.054+0000 7f8f763dab80 1 -- 172.21.15.121:0/504839003 wait complete. 2024-06-10T00:33:04.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.057+0000 7f8f763dab80 20 client.25708 trim_cache size 0 max 16384 2024-06-10T00:33:04.057 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:33:04.057 INFO:teuthology.orchestra.run.smithi121.stdout: "return_code": 0, 2024-06-10T00:33:04.057 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_tag": "52a1e908-ff89-4d0c-9d9d-fa06f7db8344", 2024-06-10T00:33:04.057 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": "asynchronous" 2024-06-10T00:33:04.058 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:33:04.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.058+0000 7f8f6bb57700 1 -- 172.21.15.121:0/1330759117 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8f500a2f90 msgr2=0x7f8f500a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:04.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.058+0000 7f8f6bb57700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8f500a2f90 0x7f8f500a5450 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c008330 tx=0x7f8f5c005c00 comp rx=0 tx=0).stop 2024-06-10T00:33:04.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.058+0000 7f8f6bb57700 1 -- 172.21.15.121:0/1330759117 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f64109490 msgr2=0x7f8f641b0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:04.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.058+0000 7f8f6bb57700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f64109490 0x7f8f641b0200 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f8f5800bb40 tx=0x7f8f5800bf00 comp rx=0 tx=0).stop 2024-06-10T00:33:04.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.058+0000 7f8f6bb57700 1 -- 172.21.15.121:0/1330759117 shutdown_connections 2024-06-10T00:33:04.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.058+0000 7f8f6bb57700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8f500a2f90 0x7f8f500a5450 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.058+0000 7f8f6bb57700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f6410dee0 0x7f8f641b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.058+0000 7f8f6bb57700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f64109490 0x7f8f641b0200 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.058+0000 7f8f6bb57700 1 --2- 172.21.15.121:0/1330759117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f64109e50 0x7f8f641b0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:04.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.058+0000 7f8f6bb57700 1 -- 172.21.15.121:0/1330759117 >> 172.21.15.121:0/1330759117 conn(0x7f8f6406d9e0 msgr2=0x7f8f64071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:04.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.060+0000 7f8f6bb57700 1 -- 172.21.15.121:0/1330759117 shutdown_connections 2024-06-10T00:33:04.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:04.063+0000 7f8f6bb57700 1 -- 172.21.15.121:0/1330759117 wait complete. 2024-06-10T00:33:04.888 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:04 smithi121 ceph-mon[32378]: pgmap v156: 129 pgs: 129 active+clean; 982 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 301 op/s 2024-06-10T00:33:04.888 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:04 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:33:04.888 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:04 smithi121 ceph-mon[32378]: scrub queued for path: / 2024-06-10T00:33:04.889 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:04 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:04.889 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:04 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:04.889 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:04 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:04.889 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:04 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:04 smithi135 ceph-mon[36424]: pgmap v156: 129 pgs: 129 active+clean; 982 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 301 op/s 2024-06-10T00:33:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:04 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:33:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:04 smithi135 ceph-mon[36424]: scrub queued for path: / 2024-06-10T00:33:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:04 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:04 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:04 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:04 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:05.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:04 smithi193 ceph-mon[36716]: pgmap v156: 129 pgs: 129 active+clean; 982 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 301 op/s 2024-06-10T00:33:05.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:04 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:33:05.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:04 smithi193 ceph-mon[36716]: scrub queued for path: / 2024-06-10T00:33:05.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:04 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:05.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:04 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:05.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:04 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:05.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:04 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:05.192 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:33:06.678 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:33:06.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:06 smithi121 ceph-mon[32378]: pgmap v157: 129 pgs: 129 active+clean; 986 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 316 op/s 2024-06-10T00:33:07.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:06 smithi135 ceph-mon[36424]: pgmap v157: 129 pgs: 129 active+clean; 986 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 316 op/s 2024-06-10T00:33:07.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:06 smithi193 ceph-mon[36716]: pgmap v157: 129 pgs: 129 active+clean; 986 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 316 op/s 2024-06-10T00:33:07.295 INFO:tasks.workunit.client.0.smithi121.stdout: 216 1839510 998 1298877 959 603531 575 2024-06-10T00:33:08.946 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:08 smithi121 ceph-mon[32378]: pgmap v158: 129 pgs: 129 active+clean; 986 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 139 op/s 2024-06-10T00:33:09.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:08 smithi135 ceph-mon[36424]: pgmap v158: 129 pgs: 129 active+clean; 986 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 139 op/s 2024-06-10T00:33:09.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:08 smithi193 ceph-mon[36716]: pgmap v158: 129 pgs: 129 active+clean; 986 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 139 op/s 2024-06-10T00:33:09.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.318+0000 7feec321d700 1 -- 172.21.15.121:0/1923572454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feebc110ff0 msgr2=0x7feebc1099d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:09.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.318+0000 7feec321d700 1 --2- 172.21.15.121:0/1923572454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feebc110ff0 0x7feebc1099d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7feeac009990 tx=0x7feeac009ca0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.320+0000 7feec321d700 1 -- 172.21.15.121:0/1923572454 shutdown_connections 2024-06-10T00:33:09.320 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.320+0000 7feec321d700 1 --2- 172.21.15.121:0/1923572454 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feebc11bc70 0x7feebc11e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.320+0000 7feec321d700 1 --2- 172.21.15.121:0/1923572454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feebc110b80 0x7feebc109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.320+0000 7feec321d700 1 --2- 172.21.15.121:0/1923572454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feebc110ff0 0x7feebc1099d0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.320+0000 7feec321d700 1 -- 172.21.15.121:0/1923572454 >> 172.21.15.121:0/1923572454 conn(0x7feebc06d470 msgr2=0x7feebc06d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:09.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.321+0000 7feec321d700 1 -- 172.21.15.121:0/1923572454 shutdown_connections 2024-06-10T00:33:09.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.321+0000 7feec321d700 1 -- 172.21.15.121:0/1923572454 wait complete. 2024-06-10T00:33:09.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.322+0000 7feec321d700 1 Processor -- start 2024-06-10T00:33:09.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.323+0000 7feec321d700 1 -- start start 2024-06-10T00:33:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.324+0000 7feec321d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feebc110b80 0x7feebc119160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.324+0000 7feec321d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feebc110ff0 0x7feebc1196a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.324+0000 7feec321d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feebc11bc70 0x7feebc112490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.324+0000 7feec321d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feebc1216a0 con 0x7feebc110ff0 2024-06-10T00:33:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.324+0000 7feec321d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7feebc121520 con 0x7feebc110b80 2024-06-10T00:33:09.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.324+0000 7feec321d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feebc121820 con 0x7feebc11bc70 2024-06-10T00:33:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.324+0000 7feec221b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feebc110b80 0x7feebc119160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.325+0000 7feec221b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feebc110b80 0x7feebc119160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:56948/0 (socket says 172.21.15.121:56948) 2024-06-10T00:33:09.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.325+0000 7feec221b700 1 -- 172.21.15.121:0/2822024391 learned_addr learned my addr 172.21.15.121:0/2822024391 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:09.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.326+0000 7feec2a1c700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feebc11bc70 0x7feebc112490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.328+0000 7feec221b700 1 -- 172.21.15.121:0/2822024391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feebc11bc70 msgr2=0x7feebc112490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.328+0000 7feec221b700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feebc11bc70 0x7feebc112490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.328+0000 7feec221b700 1 -- 172.21.15.121:0/2822024391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feebc110ff0 msgr2=0x7feebc1196a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.328+0000 7feec221b700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feebc110ff0 0x7feebc1196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.328+0000 7feec221b700 1 -- 172.21.15.121:0/2822024391 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feebc112d90 con 0x7feebc110b80 2024-06-10T00:33:09.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.330+0000 7feec221b700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feebc110b80 0x7feebc119160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feeb800b530 tx=0x7feeb800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.332+0000 7feeb37fe700 1 -- 172.21.15.121:0/2822024391 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8015020 con 0x7feebc110b80 2024-06-10T00:33:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.332+0000 7feeb37fe700 1 -- 172.21.15.121:0/2822024391 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeb8011dd0 con 0x7feebc110b80 2024-06-10T00:33:09.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.332+0000 7feeb37fe700 1 -- 172.21.15.121:0/2822024391 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb800f2e0 con 0x7feebc110b80 2024-06-10T00:33:09.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.337+0000 7feec321d700 1 -- 172.21.15.121:0/2822024391 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feebc113080 con 0x7feebc110b80 2024-06-10T00:33:09.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.337+0000 7feec321d700 1 -- 172.21.15.121:0/2822024391 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feebc116f20 con 0x7feebc110b80 2024-06-10T00:33:09.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.346+0000 7feeb37fe700 1 -- 172.21.15.121:0/2822024391 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7feeb8018020 con 0x7feebc110b80 2024-06-10T00:33:09.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.349+0000 7feeced02b80 1 Processor -- start 2024-06-10T00:33:09.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.349+0000 7feeb37fe700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feea80a2f10 0x7feea80a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.349+0000 7feeb37fe700 1 -- 172.21.15.121:0/2822024391 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7feeb80ceed0 con 0x7feebc110b80 2024-06-10T00:33:09.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.353+0000 7feeced02b80 1 -- start start 2024-06-10T00:33:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.353+0000 7feeced02b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a7ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.353+0000 7feeced02b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7d210 0x55b5c4a76b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.353+0000 7feeced02b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a77080 0x55b5c4a774c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.353+0000 7feeced02b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b5c49c8810 con 0x55b5c4a7d210 2024-06-10T00:33:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.353+0000 7feeced02b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55b5c4970720 con 0x55b5c4a7c8f0 2024-06-10T00:33:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.353+0000 7feeced02b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b5c49c47d0 con 0x55b5c4a77080 2024-06-10T00:33:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.353+0000 7feec221b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a7ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.354+0000 7feec221b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a7ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:56960/0 (socket says 172.21.15.121:56960) 2024-06-10T00:33:09.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.354+0000 7feec221b700 1 -- 172.21.15.121:0/3469889501 learned_addr learned my addr 172.21.15.121:0/3469889501 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:09.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.357+0000 7feec2a1c700 1 --2- 172.21.15.121:0/3469889501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a77080 0x55b5c4a774c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.357+0000 7feec1a1a700 1 --2- 172.21.15.121:0/3469889501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7d210 0x55b5c4a76b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.360+0000 7feec2a1c700 1 -- 172.21.15.121:0/3469889501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7c8f0 msgr2=0x55b5c4a7ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:09.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.360+0000 7feec2a1c700 1 --2- 172.21.15.121:0/3469889501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a7ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.360+0000 7feec2a1c700 1 -- 172.21.15.121:0/3469889501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7d210 msgr2=0x55b5c4a76b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:09.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.360+0000 7feec2a1c700 1 --2- 172.21.15.121:0/3469889501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7d210 0x55b5c4a76b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.360+0000 7feec2a1c700 1 -- 172.21.15.121:0/3469889501 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5c4966f50 con 0x55b5c4a77080 2024-06-10T00:33:09.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.362+0000 7feec2a1c700 1 --2- 172.21.15.121:0/3469889501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a77080 0x55b5c4a774c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feeb400b490 tx=0x7feeb400b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c834ddcf6cccaa55 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:09.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.364+0000 7feeb0d70700 1 -- 172.21.15.121:0/3469889501 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb401b070 con 0x55b5c4a77080 2024-06-10T00:33:09.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.364+0000 7feeb0d70700 1 -- 172.21.15.121:0/3469889501 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeb4008a20 con 0x55b5c4a77080 2024-06-10T00:33:09.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.365+0000 7feeced02b80 1 -- 172.21.15.121:0/3469889501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a77080 msgr2=0x55b5c4a774c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:09.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.365+0000 7feeced02b80 1 --2- 172.21.15.121:0/3469889501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a77080 0x55b5c4a774c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feeb400b490 tx=0x7feeb400b7a0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.365+0000 7feeced02b80 1 -- 172.21.15.121:0/3469889501 shutdown_connections 2024-06-10T00:33:09.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.365+0000 7feeced02b80 1 --2- 172.21.15.121:0/3469889501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a77080 0x55b5c4a774c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.365+0000 7feeced02b80 1 --2- 172.21.15.121:0/3469889501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a7ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.365+0000 7feeced02b80 1 --2- 172.21.15.121:0/3469889501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7d210 0x55b5c4a76b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.365+0000 7feeced02b80 1 -- 172.21.15.121:0/3469889501 >> 172.21.15.121:0/3469889501 conn(0x55b5c49a1cc0 msgr2=0x55b5c4a71af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:09.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.369+0000 7feec2a1c700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feea80a2f10 0x7feea80a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.369+0000 7feeced02b80 1 -- 172.21.15.121:0/3469889501 shutdown_connections 2024-06-10T00:33:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.370+0000 7feeced02b80 1 -- 172.21.15.121:0/3469889501 wait complete. 2024-06-10T00:33:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.371+0000 7feeced02b80 1 -- start start 2024-06-10T00:33:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.371+0000 7feeced02b80 1 Processor -- start 2024-06-10T00:33:09.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.371+0000 7feeced02b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a82cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.371+0000 7feeced02b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7d210 0x55b5c4a7ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.371+0000 7feeced02b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 0x55b5c4a80970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.371+0000 7feeced02b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b5c4a83c20 con 0x55b5c4a7c8f0 2024-06-10T00:33:09.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.371+0000 7feeced02b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55b5c4968bf0 con 0x55b5c4a7d210 2024-06-10T00:33:09.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.371+0000 7feeced02b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b5c4a83da0 con 0x55b5c4a804f0 2024-06-10T00:33:09.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.372+0000 7feeced02b80 10 client.0 _ll_register_callbacks cb 0x55b5c4962920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:09.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.372+0000 7feec2a1c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 0x55b5c4a80970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.372+0000 7feec2a1c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 0x55b5c4a80970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:58618/0 (socket says 172.21.15.121:58618) 2024-06-10T00:33:09.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.372+0000 7feec2a1c700 1 -- 172.21.15.121:0/2780095652 learned_addr learned my addr 172.21.15.121:0/2780095652 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:09.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.373+0000 7feec2a1c700 1 -- 172.21.15.121:0/2780095652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7d210 msgr2=0x55b5c4a7ffb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-10T00:33:09.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.373+0000 7feec2a1c700 1 --2- 172.21.15.121:0/2780095652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7d210 0x55b5c4a7ffb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.373+0000 7feec2a1c700 1 -- 172.21.15.121:0/2780095652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 msgr2=0x55b5c4a82cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:09.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.373+0000 7feec2a1c700 1 --2- 172.21.15.121:0/2780095652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a82cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.373+0000 7feec2a1c700 1 -- 172.21.15.121:0/2780095652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5c49178d0 con 0x55b5c4a804f0 2024-06-10T00:33:09.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.373+0000 7feec2a1c700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feea80a2f10 0x7feea80a53d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feeb400b490 tx=0x7feeb400be80 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:09.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.373+0000 7feec2a1c700 1 --2- 172.21.15.121:0/2780095652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 0x55b5c4a80970 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feeb4016fd0 tx=0x7feeb4025000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b058c37865b3b80 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:09.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.373+0000 7feea5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:33:09.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.374+0000 7feea5ffb700 1 -- 172.21.15.121:0/2780095652 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb401b040 con 0x55b5c4a804f0 2024-06-10T00:33:09.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.374+0000 7feeced02b80 1 -- 172.21.15.121:0/2780095652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b5c49d5d10 con 0x55b5c4a804f0 2024-06-10T00:33:09.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.374+0000 7feea5ffb700 1 -- 172.21.15.121:0/2780095652 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7feeb4022430 con 0x55b5c4a804f0 2024-06-10T00:33:09.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.379+0000 7feea5ffb700 1 -- 172.21.15.121:0/2780095652 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feeb4008e60 con 0x55b5c4a804f0 2024-06-10T00:33:09.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.379+0000 7feeced02b80 10 client.25717 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:09.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.379+0000 7feeced02b80 1 -- 172.21.15.121:0/2780095652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b5c4ab07d0 con 0x55b5c4a804f0 2024-06-10T00:33:09.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.380+0000 7feea5ffb700 1 -- 172.21.15.121:0/2780095652 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b5c49d5d10 con 0x55b5c4a804f0 2024-06-10T00:33:09.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.381+0000 7feea5ffb700 1 -- 172.21.15.121:0/2780095652 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7feeb4021070 con 0x55b5c4a804f0 2024-06-10T00:33:09.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.381+0000 7feeced02b80 10 client.25717 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:09.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.381+0000 7feeced02b80 10 client.25717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:33:09.382 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.381+0000 7feeced02b80 20 client.25717 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:09.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.381+0000 7feeced02b80 1 --2- 172.21.15.121:0/2780095652 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b5c4ab30f0 0x55b5c4ab5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.381+0000 7feeced02b80 4 client.25717 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:33:09.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.381+0000 7feeced02b80 1 -- 172.21.15.121:0/2780095652 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5c49d0290 con 0x55b5c4ab30f0 2024-06-10T00:33:09.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.381+0000 7feec221b700 1 --2- 172.21.15.121:0/2780095652 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b5c4ab30f0 0x55b5c4ab5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.394+0000 7feec221b700 1 --2- 172.21.15.121:0/2780095652 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b5c4ab30f0 0x55b5c4ab5590 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e92e02f472d3b97f server_cookie=9b706acf752c0a0 in_seq=0 out_seq=0 2024-06-10T00:33:09.395 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.394+0000 7feea5ffb700 10 client.25717 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.407+0000 7feea5ffb700 1 -- 172.21.15.121:0/2780095652 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b5c49d0290 con 0x55b5c4ab30f0 2024-06-10T00:33:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.407+0000 7feea5ffb700 10 client.25717 handle_command_reply: tid=0 2024-06-10T00:33:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.407+0000 7feeced02b80 1 client.25717 shutdown 2024-06-10T00:33:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.407+0000 7feeced02b80 1 -- 172.21.15.121:0/2780095652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 msgr2=0x55b5c4a80970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.407+0000 7feeced02b80 1 --2- 172.21.15.121:0/2780095652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 0x55b5c4a80970 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feeb4016fd0 tx=0x7feeb4025000 comp rx=0 tx=0).stop 2024-06-10T00:33:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.408+0000 7feeced02b80 1 -- 172.21.15.121:0/2780095652 shutdown_connections 2024-06-10T00:33:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.408+0000 7feeced02b80 1 --2- 172.21.15.121:0/2780095652 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b5c4ab30f0 0x55b5c4ab5590 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.408+0000 7feeced02b80 1 --2- 172.21.15.121:0/2780095652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 0x55b5c4a80970 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.408+0000 7feeced02b80 1 --2- 172.21.15.121:0/2780095652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a82cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.408+0000 7feeced02b80 1 --2- 172.21.15.121:0/2780095652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7d210 0x55b5c4a7ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.408+0000 7feea5ffb700 0 client.25717 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.408+0000 7feeced02b80 1 -- 172.21.15.121:0/2780095652 >> 172.21.15.121:0/2780095652 conn(0x55b5c49a1cc0 msgr2=0x55b5c4a75b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.408+0000 7feeced02b80 1 -- 172.21.15.121:0/2780095652 shutdown_connections 2024-06-10T00:33:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.408+0000 7feeced02b80 1 -- 172.21.15.121:0/2780095652 wait complete. 2024-06-10T00:33:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.409+0000 7feeced02b80 20 client.25717 trim_cache size 0 max 16384 2024-06-10T00:33:09.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.453+0000 7feeced02b80 1 Processor -- start 2024-06-10T00:33:09.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.454+0000 7feeced02b80 1 -- start start 2024-06-10T00:33:09.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.454+0000 7feeced02b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a792c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.454+0000 7feeced02b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7d210 0x55b5c4a71bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.454+0000 7feeced02b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 0x55b5c4a7c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.454+0000 7feeced02b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feebc1b1580 con 0x55b5c4a7c8f0 2024-06-10T00:33:09.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.454+0000 7feeced02b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7feebc1aedb0 con 0x55b5c4a7d210 2024-06-10T00:33:09.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.454+0000 7feeced02b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feebc1acc00 con 0x55b5c4a804f0 2024-06-10T00:33:09.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.454+0000 7feec2a1c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 0x55b5c4a7c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.455+0000 7feec1a1a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7d210 0x55b5c4a71bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.455+0000 7feec2a1c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 0x55b5c4a7c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:58622/0 (socket says 172.21.15.121:58622) 2024-06-10T00:33:09.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.455+0000 7feec2a1c700 1 -- 172.21.15.121:0/2757361408 learned_addr learned my addr 172.21.15.121:0/2757361408 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.455+0000 7feec221b700 1 --2- 172.21.15.121:0/2757361408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a792c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.456+0000 7feec221b700 1 -- 172.21.15.121:0/2757361408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 msgr2=0x55b5c4a7c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.456+0000 7feec221b700 1 --2- 172.21.15.121:0/2757361408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 0x55b5c4a7c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.456+0000 7feec221b700 1 -- 172.21.15.121:0/2757361408 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7d210 msgr2=0x55b5c4a71bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.456+0000 7feec221b700 1 --2- 172.21.15.121:0/2757361408 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7d210 0x55b5c4a71bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.456+0000 7feec221b700 1 -- 172.21.15.121:0/2757361408 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5c4a835b0 con 0x55b5c4a7c8f0 2024-06-10T00:33:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.457+0000 7feec221b700 1 --2- 172.21.15.121:0/2757361408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a792c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7feeb80975d0 tx=0x7feeb8004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3765c4ac970fdb7c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.458+0000 7feea7fff700 1 -- 172.21.15.121:0/2757361408 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8099470 con 0x55b5c4a7c8f0 2024-06-10T00:33:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.458+0000 7feea7fff700 1 -- 172.21.15.121:0/2757361408 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeb8010380 con 0x55b5c4a7c8f0 2024-06-10T00:33:09.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.458+0000 7feea7fff700 1 -- 172.21.15.121:0/2757361408 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb809b3d0 con 0x55b5c4a7c8f0 2024-06-10T00:33:09.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.460+0000 7feeced02b80 1 -- 172.21.15.121:0/2757361408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 msgr2=0x55b5c4a792c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:09.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.460+0000 7feeced02b80 1 --2- 172.21.15.121:0/2757361408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a792c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7feeb80975d0 tx=0x7feeb8004860 comp rx=0 tx=0).stop 2024-06-10T00:33:09.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.467+0000 7feeced02b80 1 -- 172.21.15.121:0/2757361408 shutdown_connections 2024-06-10T00:33:09.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.467+0000 7feeced02b80 1 --2- 172.21.15.121:0/2757361408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a804f0 0x55b5c4a7c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.467+0000 7feeced02b80 1 --2- 172.21.15.121:0/2757361408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a792c0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.467+0000 7feeced02b80 1 --2- 172.21.15.121:0/2757361408 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a7d210 0x55b5c4a71bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.467+0000 7feeced02b80 1 -- 172.21.15.121:0/2757361408 >> 172.21.15.121:0/2757361408 conn(0x55b5c49a1cc0 msgr2=0x55b5c4a714e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:09.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.468+0000 7feeced02b80 1 -- 172.21.15.121:0/2757361408 shutdown_connections 2024-06-10T00:33:09.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.468+0000 7feeced02b80 1 -- 172.21.15.121:0/2757361408 wait complete. 2024-06-10T00:33:09.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.468+0000 7feeced02b80 1 -- start start 2024-06-10T00:33:09.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.469+0000 7feeced02b80 1 Processor -- start 2024-06-10T00:33:09.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.469+0000 7feeced02b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a89260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.469+0000 7feeced02b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a7d210 0x55b5c4a897a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.470+0000 7feeced02b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a804f0 0x55b5c4aba0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.470+0000 7feeced02b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feebc1ad270 con 0x55b5c4a7c8f0 2024-06-10T00:33:09.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.470+0000 7feeced02b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7feebc1ae6e0 con 0x55b5c4a804f0 2024-06-10T00:33:09.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.470+0000 7feeced02b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feebc1ad940 con 0x55b5c4a7d210 2024-06-10T00:33:09.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.470+0000 7feeced02b80 10 client.0 _ll_register_callbacks cb 0x55b5c4962920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:09.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.470+0000 7feec2a1c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a804f0 0x55b5c4aba0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.470+0000 7feec1a1a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a7d210 0x55b5c4a897a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.470+0000 7feec2a1c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a804f0 0x55b5c4aba0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:56976/0 (socket says 172.21.15.121:56976) 2024-06-10T00:33:09.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.470+0000 7feec2a1c700 1 -- 172.21.15.121:0/3888562924 learned_addr learned my addr 172.21.15.121:0/3888562924 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:09.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.472+0000 7feec221b700 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a89260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.472+0000 7feec1a1a700 1 -- 172.21.15.121:0/3888562924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a804f0 msgr2=0x55b5c4aba0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:09.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.472+0000 7feec1a1a700 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a804f0 0x55b5c4aba0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.472+0000 7feec1a1a700 1 -- 172.21.15.121:0/3888562924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 msgr2=0x55b5c4a89260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:09.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.472+0000 7feec1a1a700 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a89260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.472+0000 7feec1a1a700 1 -- 172.21.15.121:0/3888562924 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5c4aba6a0 con 0x55b5c4a7d210 2024-06-10T00:33:09.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.473+0000 7feec2a1c700 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a804f0 0x55b5c4aba0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:09.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.474+0000 7feec1a1a700 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a7d210 0x55b5c4a897a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feeac0098e0 tx=0x7feeac009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=751c9439fa1e4723 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:09.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.474+0000 7feea5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:33:09.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.474+0000 7feea5ffb700 1 -- 172.21.15.121:0/3888562924 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac0054d0 con 0x55b5c4a7d210 2024-06-10T00:33:09.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.474+0000 7feea5ffb700 1 -- 172.21.15.121:0/3888562924 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7feeac005670 con 0x55b5c4a7d210 2024-06-10T00:33:09.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.474+0000 7feeced02b80 1 -- 172.21.15.121:0/3888562924 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feeb8010380 con 0x55b5c4a7d210 2024-06-10T00:33:09.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.478+0000 7feea5ffb700 1 -- 172.21.15.121:0/3888562924 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feeac003740 con 0x55b5c4a7d210 2024-06-10T00:33:09.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.478+0000 7feeced02b80 10 client.25723 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:09.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.478+0000 7feeced02b80 1 -- 172.21.15.121:0/3888562924 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b5c4b1d760 con 0x55b5c4a7d210 2024-06-10T00:33:09.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.482+0000 7feea5ffb700 1 -- 172.21.15.121:0/3888562924 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feeac028070 con 0x55b5c4a7d210 2024-06-10T00:33:09.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.482+0000 7feea5ffb700 1 -- 172.21.15.121:0/3888562924 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7feeac00f910 con 0x55b5c4a7d210 2024-06-10T00:33:09.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.482+0000 7feeced02b80 10 client.25723 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:09.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.482+0000 7feeced02b80 10 client.25723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:33:09.482 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.482+0000 7feeced02b80 20 client.25723 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:09.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.482+0000 7feeced02b80 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b5c4b1def0 0x55b5c4b20310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:09.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.482+0000 7feeced02b80 4 client.25723 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:33:09.483 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.482+0000 7feeced02b80 1 -- 172.21.15.121:0/3888562924 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7feeb8010380 con 0x55b5c4b1def0 2024-06-10T00:33:09.484 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.483+0000 7feec221b700 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b5c4b1def0 0x55b5c4b20310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:09.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.497+0000 7feec221b700 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b5c4b1def0 0x55b5c4b20310 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79fefb74be635434 server_cookie=11fef0dd28f30afe in_seq=0 out_seq=0 2024-06-10T00:33:09.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.499+0000 7feea5ffb700 10 client.25723 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:09.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.502+0000 7feea5ffb700 1 -- 172.21.15.121:0/3888562924 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+268 (crc 0 0 0) 0x7feeb8010380 con 0x55b5c4b1def0 2024-06-10T00:33:09.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.502+0000 7feea5ffb700 10 client.25723 handle_command_reply: tid=0 2024-06-10T00:33:09.502 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.502+0000 7feeced02b80 1 client.25723 shutdown 2024-06-10T00:33:09.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.503+0000 7feeced02b80 1 -- 172.21.15.121:0/3888562924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a7d210 msgr2=0x55b5c4a897a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:09.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.503+0000 7feeced02b80 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a7d210 0x55b5c4a897a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feeac0098e0 tx=0x7feeac009ee0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.503+0000 7feeced02b80 1 -- 172.21.15.121:0/3888562924 shutdown_connections 2024-06-10T00:33:09.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.503+0000 7feeced02b80 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b5c4b1def0 0x55b5c4b20310 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.503+0000 7feeced02b80 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b5c4a804f0 0x55b5c4aba0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.503+0000 7feeced02b80 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b5c4a7c8f0 0x55b5c4a89260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.503 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.503+0000 7feeced02b80 1 --2- 172.21.15.121:0/3888562924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b5c4a7d210 0x55b5c4a897a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.503+0000 7feeced02b80 1 -- 172.21.15.121:0/3888562924 >> 172.21.15.121:0/3888562924 conn(0x55b5c49a1cc0 msgr2=0x55b5c4a6af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:09.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.503+0000 7feea5ffb700 0 client.25723 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:09.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.504+0000 7feeced02b80 1 -- 172.21.15.121:0/3888562924 shutdown_connections 2024-06-10T00:33:09.504 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.504+0000 7feeced02b80 1 -- 172.21.15.121:0/3888562924 wait complete. 2024-06-10T00:33:09.507 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:33:09.507 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "scrub active (5998 inodes in the stack)", 2024-06-10T00:33:09.507 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": { 2024-06-10T00:33:09.507 INFO:teuthology.orchestra.run.smithi121.stdout: "52a1e908-ff89-4d0c-9d9d-fa06f7db8344": { 2024-06-10T00:33:09.507 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/", 2024-06-10T00:33:09.507 INFO:teuthology.orchestra.run.smithi121.stdout: "tag": "52a1e908-ff89-4d0c-9d9d-fa06f7db8344", 2024-06-10T00:33:09.507 INFO:teuthology.orchestra.run.smithi121.stdout: "options": "recursive,force" 2024-06-10T00:33:09.507 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:33:09.507 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:33:09.507 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:33:09.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.506+0000 7feeced02b80 20 client.25723 trim_cache size 0 max 16384 2024-06-10T00:33:09.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.507+0000 7feec321d700 1 -- 172.21.15.121:0/2822024391 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feea80a2f10 msgr2=0x7feea80a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:09.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.507+0000 7feec321d700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feea80a2f10 0x7feea80a53d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feeb400b490 tx=0x7feeb400be80 comp rx=0 tx=0).stop 2024-06-10T00:33:09.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.507+0000 7feec321d700 1 -- 172.21.15.121:0/2822024391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feebc110b80 msgr2=0x7feebc119160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:09.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.507+0000 7feec321d700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feebc110b80 0x7feebc119160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7feeb800b530 tx=0x7feeb800b840 comp rx=0 tx=0).stop 2024-06-10T00:33:09.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.508+0000 7feec321d700 1 -- 172.21.15.121:0/2822024391 shutdown_connections 2024-06-10T00:33:09.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.508+0000 7feec321d700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feea80a2f10 0x7feea80a53d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.508+0000 7feec321d700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feebc11bc70 0x7feebc112490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.508+0000 7feec321d700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feebc110b80 0x7feebc119160 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.508+0000 7feec321d700 1 --2- 172.21.15.121:0/2822024391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feebc110ff0 0x7feebc1196a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:09.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.508+0000 7feec321d700 1 -- 172.21.15.121:0/2822024391 >> 172.21.15.121:0/2822024391 conn(0x7feebc06d470 msgr2=0x7feebc10e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:09.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.508+0000 7feec321d700 1 -- 172.21.15.121:0/2822024391 shutdown_connections 2024-06-10T00:33:09.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:09.508+0000 7feec321d700 1 -- 172.21.15.121:0/2822024391 wait complete. 2024-06-10T00:33:10.231 INFO:tasks.cephfs.filesystem:scrub status for tag:52a1e908-ff89-4d0c-9d9d-fa06f7db8344 - {'path': '/', 'tag': '52a1e908-ff89-4d0c-9d9d-fa06f7db8344', 'options': 'recursive,force'} 2024-06-10T00:33:10.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:10 smithi121 ceph-mon[32378]: pgmap v159: 129 pgs: 129 active+clean; 1003 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 282 op/s 2024-06-10T00:33:10.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:10 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:11.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:10 smithi135 ceph-mon[36424]: pgmap v159: 129 pgs: 129 active+clean; 1003 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 282 op/s 2024-06-10T00:33:11.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:10 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:11.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:10 smithi193 ceph-mon[36716]: pgmap v159: 129 pgs: 129 active+clean; 1003 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 282 op/s 2024-06-10T00:33:11.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:10 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:11.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:11 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:12.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:11 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:12.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:11 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:12.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:12 smithi121 ceph-mon[32378]: pgmap v160: 129 pgs: 129 active+clean; 1003 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 222 op/s 2024-06-10T00:33:13.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:12 smithi135 ceph-mon[36424]: pgmap v160: 129 pgs: 129 active+clean; 1003 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 222 op/s 2024-06-10T00:33:13.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:12 smithi193 ceph-mon[36716]: pgmap v160: 129 pgs: 129 active+clean; 1003 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 222 op/s 2024-06-10T00:33:14.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:14 smithi121 ceph-mon[32378]: pgmap v161: 129 pgs: 129 active+clean; 1019 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 322 op/s 2024-06-10T00:33:15.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:14 smithi135 ceph-mon[36424]: pgmap v161: 129 pgs: 129 active+clean; 1019 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 322 op/s 2024-06-10T00:33:15.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:14 smithi193 ceph-mon[36716]: pgmap v161: 129 pgs: 129 active+clean; 1019 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 322 op/s 2024-06-10T00:33:16.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:16 smithi121 ceph-mon[32378]: pgmap v162: 129 pgs: 129 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 8.0 MiB/s wr, 555 op/s 2024-06-10T00:33:17.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:16 smithi135 ceph-mon[36424]: pgmap v162: 129 pgs: 129 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 8.0 MiB/s wr, 555 op/s 2024-06-10T00:33:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:16 smithi193 ceph-mon[36716]: pgmap v162: 129 pgs: 129 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 8.0 MiB/s wr, 555 op/s 2024-06-10T00:33:17.295 INFO:tasks.workunit.client.0.smithi121.stdout: 229 1832837 859 1281268 466 567689 739 2024-06-10T00:33:18.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:18 smithi121 ceph-mon[32378]: pgmap v163: 129 pgs: 129 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.4 MiB/s wr, 499 op/s 2024-06-10T00:33:19.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:18 smithi135 ceph-mon[36424]: pgmap v163: 129 pgs: 129 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.4 MiB/s wr, 499 op/s 2024-06-10T00:33:19.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:18 smithi193 ceph-mon[36716]: pgmap v163: 129 pgs: 129 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.4 MiB/s wr, 499 op/s 2024-06-10T00:33:19.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:19.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:19.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:19 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:19 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:20.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:19 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:20.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:33:20.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:20.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: pgmap v164: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 10 MiB/s wr, 741 op/s 2024-06-10T00:33:20.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:20 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:33:21.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: pgmap v164: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 10 MiB/s wr, 741 op/s 2024-06-10T00:33:21.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:20 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:33:21.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: pgmap v164: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 10 MiB/s wr, 741 op/s 2024-06-10T00:33:21.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:20 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:33:21.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:21 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:33:22.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:21 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:33:22.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:21 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:33:22.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:22 smithi121 ceph-mon[32378]: pgmap v165: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.6 MiB/s wr, 598 op/s 2024-06-10T00:33:23.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:22 smithi135 ceph-mon[36424]: pgmap v165: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.6 MiB/s wr, 598 op/s 2024-06-10T00:33:23.032 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:22 smithi193 ceph-mon[36716]: pgmap v165: 129 pgs: 129 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 8.6 MiB/s wr, 598 op/s 2024-06-10T00:33:24.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:24 smithi121 ceph-mon[32378]: pgmap v166: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.4 MiB/s wr, 692 op/s 2024-06-10T00:33:25.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:24 smithi135 ceph-mon[36424]: pgmap v166: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.4 MiB/s wr, 692 op/s 2024-06-10T00:33:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:24 smithi193 ceph-mon[36716]: pgmap v166: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 9.4 MiB/s wr, 692 op/s 2024-06-10T00:33:25.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:25 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:33:25.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:25 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:33:25.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:25 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:33:26.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:25 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:33:26.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:25 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:33:26.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:25 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:33:26.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:25 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:33:26.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:25 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:33:26.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:25 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:33:26.234 INFO:tasks.quiescer.fs.[cephfs]:Total duration (58.9) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-10T00:33:26.234 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.45 2024-06-10T00:33:26.235 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:33:26.757 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:33:27.151 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:26 smithi121 ceph-mon[32378]: pgmap v167: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.5 MiB/s wr, 732 op/s 2024-06-10T00:33:27.152 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:26 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:33:27.152 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:26 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:33:27.152 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:26 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:33:27.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:26 smithi135 ceph-mon[36424]: pgmap v167: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.5 MiB/s wr, 732 op/s 2024-06-10T00:33:27.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:26 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:33:27.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:26 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:33:27.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:26 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:33:27.295 INFO:tasks.workunit.client.0.smithi121.stdout: 241 2158043 928 1486784 622 693098 1690 2024-06-10T00:33:27.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:26 smithi193 ceph-mon[36716]: pgmap v167: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.5 MiB/s wr, 732 op/s 2024-06-10T00:33:27.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:26 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:33:27.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:26 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:33:27.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:26 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:33:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.576+0000 7f64180fe700 1 -- 172.21.15.121:0/2186561617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f641006ab30 msgr2=0x7f641010f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.576+0000 7f64180fe700 1 --2- 172.21.15.121:0/2186561617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f641006ab30 0x7f641010f320 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f640800b0a0 tx=0x7f640800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:33:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.576+0000 7f64180fe700 1 -- 172.21.15.121:0/2186561617 shutdown_connections 2024-06-10T00:33:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.576+0000 7f64180fe700 1 --2- 172.21.15.121:0/2186561617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f641006ab30 0x7f641010f320 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.576+0000 7f64180fe700 1 --2- 172.21.15.121:0/2186561617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6410108500 0x7f64101088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.576+0000 7f64180fe700 1 --2- 172.21.15.121:0/2186561617 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f641006a170 0x7f641006a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.576+0000 7f64180fe700 1 -- 172.21.15.121:0/2186561617 >> 172.21.15.121:0/2186561617 conn(0x7f6410078340 msgr2=0x7f6410078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:28.577 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.576+0000 7f64180fe700 1 -- 172.21.15.121:0/2186561617 shutdown_connections 2024-06-10T00:33:28.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.576+0000 7f64180fe700 1 -- 172.21.15.121:0/2186561617 wait complete. 2024-06-10T00:33:28.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.577+0000 7f64180fe700 1 Processor -- start 2024-06-10T00:33:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.577+0000 7f64180fe700 1 -- start start 2024-06-10T00:33:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.577+0000 7f64180fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f641006a170 0x7f641019e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.577+0000 7f64180fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641006ab30 0x7f641019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.577+0000 7f64180fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410108500 0x7f6410198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f64180fe700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6410112780 con 0x7f6410108500 2024-06-10T00:33:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f64180fe700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6410112600 con 0x7f641006a170 2024-06-10T00:33:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f64180fe700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6410112900 con 0x7f641006ab30 2024-06-10T00:33:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415e9a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f641006a170 0x7f641019e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415699700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641006ab30 0x7f641019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415699700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641006ab30 0x7f641019ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:34370/0 (socket says 172.21.15.121:34370) 2024-06-10T00:33:28.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415e9a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f641006a170 0x7f641019e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:53414/0 (socket says 172.21.15.121:53414) 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415699700 1 -- 172.21.15.121:0/3294461833 learned_addr learned my addr 172.21.15.121:0/3294461833 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f641669b700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410108500 0x7f6410198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415699700 1 -- 172.21.15.121:0/3294461833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f641006a170 msgr2=0x7f641019e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415699700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f641006a170 0x7f641019e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415699700 1 -- 172.21.15.121:0/3294461833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410108500 msgr2=0x7f6410198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415699700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410108500 0x7f6410198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415699700 1 -- 172.21.15.121:0/3294461833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6410199180 con 0x7f641006ab30 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f641669b700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410108500 0x7f6410198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415e9a700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f641006a170 0x7f641019e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.578+0000 7f6415699700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641006ab30 0x7f641019ed50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f640000e9a0 tx=0x7f640000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.579+0000 7f6406ffd700 1 -- 172.21.15.121:0/3294461833 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640000ca10 con 0x7f641006ab30 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.579+0000 7f64180fe700 1 -- 172.21.15.121:0/3294461833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6410199470 con 0x7f641006ab30 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.579+0000 7f6406ffd700 1 -- 172.21.15.121:0/3294461833 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f640000cbb0 con 0x7f641006ab30 2024-06-10T00:33:28.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.579+0000 7f6406ffd700 1 -- 172.21.15.121:0/3294461833 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6400018680 con 0x7f641006ab30 2024-06-10T00:33:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.579+0000 7f64180fe700 1 -- 172.21.15.121:0/3294461833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64101a56e0 con 0x7f641006ab30 2024-06-10T00:33:28.582 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.580+0000 7f64180fe700 1 -- 172.21.15.121:0/3294461833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f641010a9e0 con 0x7f641006ab30 2024-06-10T00:33:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.581+0000 7f6406ffd700 1 -- 172.21.15.121:0/3294461833 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6400010430 con 0x7f641006ab30 2024-06-10T00:33:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.582+0000 7f6406ffd700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f63fc0a2d90 0x7f63fc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.582+0000 7f6415e9a700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f63fc0a2d90 0x7f63fc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.582+0000 7f6406ffd700 1 -- 172.21.15.121:0/3294461833 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f64000cfa20 con 0x7f641006ab30 2024-06-10T00:33:28.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.582+0000 7f6415e9a700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f63fc0a2d90 0x7f63fc0a5250 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6410108000 tx=0x7f640c009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:28.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.586+0000 7f6406ffd700 1 -- 172.21.15.121:0/3294461833 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f640009c2b0 con 0x7f641006ab30 2024-06-10T00:33:28.863 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.863+0000 7f64180fe700 1 -- 172.21.15.121:0/3294461833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6410069180 con 0x7f641006ab30 2024-06-10T00:33:28.864 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.864+0000 7f6406ffd700 1 -- 172.21.15.121:0/3294461833 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11891 (secure 0 0 0) 0x7f640009bbb0 con 0x7f641006ab30 2024-06-10T00:33:28.864 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:33:28.864 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:33:03.460049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.866+0000 7f64180fe700 1 -- 172.21.15.121:0/3294461833 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f63fc0a2d90 msgr2=0x7f63fc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.866+0000 7f64180fe700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f63fc0a2d90 0x7f63fc0a5250 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6410108000 tx=0x7f640c009260 comp rx=0 tx=0).stop 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.866+0000 7f64180fe700 1 -- 172.21.15.121:0/3294461833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641006ab30 msgr2=0x7f641019ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.866+0000 7f64180fe700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641006ab30 0x7f641019ed50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f640000e9a0 tx=0x7f640000ecb0 comp rx=0 tx=0).stop 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.866+0000 7f64180fe700 1 -- 172.21.15.121:0/3294461833 shutdown_connections 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.866+0000 7f64180fe700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f63fc0a2d90 0x7f63fc0a5250 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.866+0000 7f64180fe700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6410108500 0x7f6410198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.866+0000 7f64180fe700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f641006a170 0x7f641019e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.866+0000 7f64180fe700 1 --2- 172.21.15.121:0/3294461833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641006ab30 0x7f641019ed50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.866+0000 7f64180fe700 1 -- 172.21.15.121:0/3294461833 >> 172.21.15.121:0/3294461833 conn(0x7f6410078340 msgr2=0x7f6410100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.867+0000 7f64180fe700 1 -- 172.21.15.121:0/3294461833 shutdown_connections 2024-06-10T00:33:28.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:28.867+0000 7f64180fe700 1 -- 172.21.15.121:0/3294461833 wait complete. 2024-06-10T00:33:28.868 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-06-10T00:33:29.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:28 smithi121 ceph-mon[32378]: pgmap v168: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 475 op/s 2024-06-10T00:33:29.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:28 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3294461833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:33:29.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:28 smithi135 ceph-mon[36424]: pgmap v168: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 475 op/s 2024-06-10T00:33:29.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:28 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3294461833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:33:29.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:28 smithi193 ceph-mon[36716]: pgmap v168: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 475 op/s 2024-06-10T00:33:29.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:28 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3294461833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:33:30.078 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db / --timeout 90 --expiration 149.45 --await-for 45.0' 2024-06-10T00:33:30.079 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db / --timeout 90 --expiration 149.45 --await-for 45.0 2024-06-10T00:33:30.571 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:33:31.014 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:30 smithi121 ceph-mon[32378]: pgmap v169: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.6 MiB/s wr, 677 op/s 2024-06-10T00:33:31.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:30 smithi135 ceph-mon[36424]: pgmap v169: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.6 MiB/s wr, 677 op/s 2024-06-10T00:33:31.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:30 smithi193 ceph-mon[36716]: pgmap v169: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.6 MiB/s wr, 677 op/s 2024-06-10T00:33:32.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.456+0000 7f4afed3d700 1 -- 172.21.15.121:0/481871537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af81015f0 msgr2=0x7f4af81019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:32.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.456+0000 7f4afed3d700 1 --2- 172.21.15.121:0/481871537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af81015f0 0x7f4af81019d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0099c0 tx=0x7f4aec009cd0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.456+0000 7f4afed3d700 1 -- 172.21.15.121:0/481871537 shutdown_connections 2024-06-10T00:33:32.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.456+0000 7f4afed3d700 1 --2- 172.21.15.121:0/481871537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4af81066c0 0x7f4af810e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.456+0000 7f4afed3d700 1 --2- 172.21.15.121:0/481871537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af81015f0 0x7f4af81019d0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.456+0000 7f4afed3d700 1 --2- 172.21.15.121:0/481871537 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4af8101fa0 0x7f4af8105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.456+0000 7f4afed3d700 1 -- 172.21.15.121:0/481871537 >> 172.21.15.121:0/481871537 conn(0x7f4af8078330 msgr2=0x7f4af8078740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:32.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.456+0000 7f4afed3d700 1 -- 172.21.15.121:0/481871537 shutdown_connections 2024-06-10T00:33:32.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.456+0000 7f4afed3d700 1 -- 172.21.15.121:0/481871537 wait complete. 2024-06-10T00:33:32.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4afed3d700 1 Processor -- start 2024-06-10T00:33:32.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4afed3d700 1 -- start start 2024-06-10T00:33:32.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4afed3d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4af81015f0 0x7f4af8199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4afed3d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af8101fa0 0x7f4af8199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4afed3d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4af81066c0 0x7f4af81a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4afed3d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81116d0 con 0x7f4af8101fa0 2024-06-10T00:33:32.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4afed3d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4af8111550 con 0x7f4af81066c0 2024-06-10T00:33:32.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4afed3d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4af8111850 con 0x7f4af81015f0 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4af7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af8101fa0 0x7f4af8199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4af7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af8101fa0 0x7f4af8199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46428/0 (socket says 172.21.15.121:46428) 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4afcad9700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4af81015f0 0x7f4af8199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4af7fff700 1 -- 172.21.15.121:0/3516334919 learned_addr learned my addr 172.21.15.121:0/3516334919 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.457+0000 7f4afd2da700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4af81066c0 0x7f4af81a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4afcad9700 1 -- 172.21.15.121:0/3516334919 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4af81066c0 msgr2=0x7f4af81a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4afcad9700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4af81066c0 0x7f4af81a4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4afcad9700 1 -- 172.21.15.121:0/3516334919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af8101fa0 msgr2=0x7f4af8199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4afcad9700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af8101fa0 0x7f4af8199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4afcad9700 1 -- 172.21.15.121:0/3516334919 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af81a5010 con 0x7f4af81015f0 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4af7fff700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af8101fa0 0x7f4af8199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4afcad9700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4af81015f0 0x7f4af8199a50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0098e0 tx=0x7f4aec00fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4af5ffb700 1 -- 172.21.15.121:0/3516334919 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec027070 con 0x7f4af81015f0 2024-06-10T00:33:32.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4afed3d700 1 -- 172.21.15.121:0/3516334919 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af81a51e0 con 0x7f4af81015f0 2024-06-10T00:33:32.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4af5ffb700 1 -- 172.21.15.121:0/3516334919 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4aec003680 con 0x7f4af81015f0 2024-06-10T00:33:32.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4af5ffb700 1 -- 172.21.15.121:0/3516334919 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec0173f0 con 0x7f4af81015f0 2024-06-10T00:33:32.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.458+0000 7f4afed3d700 1 -- 172.21.15.121:0/3516334919 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af81a55e0 con 0x7f4af81015f0 2024-06-10T00:33:32.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.461+0000 7f4af5ffb700 1 -- 172.21.15.121:0/3516334919 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4aec02a070 con 0x7f4af81015f0 2024-06-10T00:33:32.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.461+0000 7f4af5ffb700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ae80a2e30 0x7f4ae80a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.461+0000 7f4af7fff700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ae80a2e30 0x7f4ae80a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.461+0000 7f4af5ffb700 1 -- 172.21.15.121:0/3516334919 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f4aec0d03f0 con 0x7f4af81015f0 2024-06-10T00:33:32.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4b095c0b80 1 Processor -- start 2024-06-10T00:33:32.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4b095c0b80 1 -- start start 2024-06-10T00:33:32.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4af7fff700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ae80a2e30 0x7f4ae80a52f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4af8107b20 tx=0x7f4ae4007480 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:32.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef5605a0 0x560bef560980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef55a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef55acf0 0x560bef55b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x560bef44d100 con 0x560bef560ec0 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x560bef4ab9f0 con 0x560bef5605a0 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560bef413600 con 0x560bef55acf0 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4af7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef55a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4af7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef55a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46436/0 (socket says 172.21.15.121:46436) 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4af7fff700 1 -- 172.21.15.121:0/2693887677 learned_addr learned my addr 172.21.15.121:0/2693887677 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4afd2da700 1 --2- 172.21.15.121:0/2693887677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef55acf0 0x560bef55b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.462+0000 7f4afcad9700 1 --2- 172.21.15.121:0/2693887677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef5605a0 0x560bef560980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.463+0000 7f4af7fff700 1 -- 172.21.15.121:0/2693887677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef55acf0 msgr2=0x560bef55b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.463+0000 7f4af7fff700 1 --2- 172.21.15.121:0/2693887677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef55acf0 0x560bef55b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.463+0000 7f4af7fff700 1 -- 172.21.15.121:0/2693887677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef5605a0 msgr2=0x560bef560980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.463+0000 7f4af7fff700 1 --2- 172.21.15.121:0/2693887677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef5605a0 0x560bef560980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.463+0000 7f4af7fff700 1 -- 172.21.15.121:0/2693887677 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bef3faa70 con 0x560bef560ec0 2024-06-10T00:33:32.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.463+0000 7f4af7fff700 1 --2- 172.21.15.121:0/2693887677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef55a7b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4ae401b900 tx=0x7f4ae4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11603eaf46125c4 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:32.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.463+0000 7f4ae2ffd700 1 -- 172.21.15.121:0/2693887677 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae402b070 con 0x560bef560ec0 2024-06-10T00:33:32.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.463+0000 7f4ae2ffd700 1 -- 172.21.15.121:0/2693887677 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ae4026b20 con 0x560bef560ec0 2024-06-10T00:33:32.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.463+0000 7f4ae2ffd700 1 -- 172.21.15.121:0/2693887677 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4011cd0 con 0x560bef560ec0 2024-06-10T00:33:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.464+0000 7f4b095c0b80 1 -- 172.21.15.121:0/2693887677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 msgr2=0x560bef55a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:32.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.464+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/2693887677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef55a7b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4ae401b900 tx=0x7f4ae4013000 comp rx=0 tx=0).stop 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.464+0000 7f4b095c0b80 1 -- 172.21.15.121:0/2693887677 shutdown_connections 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.464+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/2693887677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef55acf0 0x560bef55b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.464+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/2693887677 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef5605a0 0x560bef560980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.464+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/2693887677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef55a7b0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.464+0000 7f4b095c0b80 1 -- 172.21.15.121:0/2693887677 >> 172.21.15.121:0/2693887677 conn(0x560bef487220 msgr2=0x560bef555aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.464+0000 7f4b095c0b80 1 -- 172.21.15.121:0/2693887677 shutdown_connections 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.464+0000 7f4b095c0b80 1 -- 172.21.15.121:0/2693887677 wait complete. 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.464+0000 7f4b095c0b80 1 -- start start 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.464+0000 7f4b095c0b80 1 Processor -- start 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef566c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef5605a0 0x560bef563f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef560ec0 0x560bef564470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x560bef44c160 con 0x560bef5605a0 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x560bef4a79b0 con 0x560bef55acf0 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560bef5678a0 con 0x560bef560ec0 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4b095c0b80 10 client.0 _ll_register_callbacks cb 0x560bef441750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4af7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef5605a0 0x560bef563f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4af7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef5605a0 0x560bef563f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46448/0 (socket says 172.21.15.121:46448) 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4af7fff700 1 -- 172.21.15.121:0/1772009153 learned_addr learned my addr 172.21.15.121:0/1772009153 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4afcad9700 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef566c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4afd2da700 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef560ec0 0x560bef564470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4af7fff700 1 -- 172.21.15.121:0/1772009153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef560ec0 msgr2=0x560bef564470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4af7fff700 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef560ec0 0x560bef564470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4af7fff700 1 -- 172.21.15.121:0/1772009153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 msgr2=0x560bef566c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4af7fff700 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef566c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4af7fff700 1 -- 172.21.15.121:0/1772009153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bef4718f0 con 0x560bef5605a0 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4afcad9700 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef566c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4af7fff700 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef5605a0 0x560bef563f30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40118b0 tx=0x7f4ae4027f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aece54941cfa37d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.465+0000 7f4ad7fff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4ad7fff700 1 -- 172.21.15.121:0/1772009153 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae402b040 con 0x560bef5605a0 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4b095c0b80 1 -- 172.21.15.121:0/1772009153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ae4026b20 con 0x560bef5605a0 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4ad7fff700 1 -- 172.21.15.121:0/1772009153 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f4ae4026640 con 0x560bef5605a0 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4ad7fff700 1 -- 172.21.15.121:0/1772009153 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4ae403d050 con 0x560bef5605a0 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4b095c0b80 10 client.16272 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4b095c0b80 1 -- 172.21.15.121:0/1772009153 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560bef572020 con 0x560bef5605a0 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4ad7fff700 1 -- 172.21.15.121:0/1772009153 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ae4026b20 con 0x560bef5605a0 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4ad7fff700 1 -- 172.21.15.121:0/1772009153 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4ae402e070 con 0x560bef5605a0 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4b095c0b80 10 client.16272 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4b095c0b80 10 client.16272 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:33:32.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4b095c0b80 20 client.16272 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x560bef574600 0x560bef5769f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4b095c0b80 4 client.16272 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.466+0000 7f4b095c0b80 1 -- 172.21.15.121:0/1772009153 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bef4b8ef0 con 0x560bef574600 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.467+0000 7f4afcad9700 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x560bef574600 0x560bef5769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.467+0000 7f4afcad9700 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x560bef574600 0x560bef5769f0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87365d016750fd2c server_cookie=d6de852a7a7aaaad in_seq=0 out_seq=0 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.467+0000 7f4ad7fff700 10 client.16272 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4ad7fff700 1 -- 172.21.15.121:0/1772009153 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x560bef4b8ef0 con 0x560bef574600 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4ad7fff700 10 client.16272 handle_command_reply: tid=0 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 1 client.16272 shutdown 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 1 -- 172.21.15.121:0/1772009153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef5605a0 msgr2=0x560bef563f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef5605a0 0x560bef563f30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40118b0 tx=0x7f4ae4027f10 comp rx=0 tx=0).stop 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 1 -- 172.21.15.121:0/1772009153 shutdown_connections 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x560bef574600 0x560bef5769f0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4ad7fff700 0 client.16272 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:32.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef560ec0 0x560bef564470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef566c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/1772009153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef5605a0 0x560bef563f30 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 1 -- 172.21.15.121:0/1772009153 >> 172.21.15.121:0/1772009153 conn(0x560bef487220 msgr2=0x560bef5543c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:32.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 1 -- 172.21.15.121:0/1772009153 shutdown_connections 2024-06-10T00:33:32.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 1 -- 172.21.15.121:0/1772009153 wait complete. 2024-06-10T00:33:32.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.468+0000 7f4b095c0b80 20 client.16272 trim_cache size 0 max 16384 2024-06-10T00:33:32.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4b095c0b80 1 Processor -- start 2024-06-10T00:33:32.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4b095c0b80 1 -- start start 2024-06-10T00:33:32.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef55a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef5605a0 0x560bef562f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef564560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81aeae0 con 0x560bef560ec0 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81ac310 con 0x560bef55acf0 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81aa160 con 0x560bef5605a0 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4afd2da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef564560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4afd2da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef564560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46464/0 (socket says 172.21.15.121:46464) 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4afd2da700 1 -- 172.21.15.121:0/3020733008 learned_addr learned my addr 172.21.15.121:0/3020733008 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4afcad9700 1 --2- 172.21.15.121:0/3020733008 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef55a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.491+0000 7f4af7fff700 1 --2- 172.21.15.121:0/3020733008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef5605a0 0x560bef562f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.492+0000 7f4afd2da700 1 -- 172.21.15.121:0/3020733008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef5605a0 msgr2=0x560bef562f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.492+0000 7f4afd2da700 1 --2- 172.21.15.121:0/3020733008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef5605a0 0x560bef562f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.492+0000 7f4afd2da700 1 -- 172.21.15.121:0/3020733008 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 msgr2=0x560bef55a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:32.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.492+0000 7f4afd2da700 1 --2- 172.21.15.121:0/3020733008 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef55a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.492+0000 7f4afd2da700 1 -- 172.21.15.121:0/3020733008 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bef564e30 con 0x560bef560ec0 2024-06-10T00:33:32.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.492+0000 7f4afd2da700 1 --2- 172.21.15.121:0/3020733008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef564560 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4af00119d0 tx=0x7f4af0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33ac0f3bfbb1582 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:32.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.492+0000 7f4ae27fc700 1 -- 172.21.15.121:0/3020733008 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af0017690 con 0x560bef560ec0 2024-06-10T00:33:32.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.492+0000 7f4ae27fc700 1 -- 172.21.15.121:0/3020733008 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4af0017d10 con 0x560bef560ec0 2024-06-10T00:33:32.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.492+0000 7f4ae27fc700 1 -- 172.21.15.121:0/3020733008 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af001f3e0 con 0x560bef560ec0 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.493+0000 7f4b095c0b80 1 -- 172.21.15.121:0/3020733008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 msgr2=0x560bef564560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.493+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/3020733008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef564560 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4af00119d0 tx=0x7f4af0009920 comp rx=0 tx=0).stop 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.493+0000 7f4b095c0b80 1 -- 172.21.15.121:0/3020733008 shutdown_connections 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.493+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/3020733008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef564560 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.493+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/3020733008 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef55a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.493+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/3020733008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef5605a0 0x560bef562f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.493+0000 7f4b095c0b80 1 -- 172.21.15.121:0/3020733008 >> 172.21.15.121:0/3020733008 conn(0x560bef487220 msgr2=0x560bef5543c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.493+0000 7f4b095c0b80 1 -- 172.21.15.121:0/3020733008 shutdown_connections 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.493+0000 7f4b095c0b80 1 -- 172.21.15.121:0/3020733008 wait complete. 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.493+0000 7f4b095c0b80 1 -- start start 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.493+0000 7f4b095c0b80 1 Processor -- start 2024-06-10T00:33:32.497 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef57bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef5605a0 0x560bef57c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4b095c0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef57c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81aa7d0 con 0x560bef560ec0 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81abc40 con 0x560bef55acf0 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4b095c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4af81aaea0 con 0x560bef5605a0 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4b095c0b80 10 client.0 _ll_register_callbacks cb 0x560bef441750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4afcad9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef57bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4afcad9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef57bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:41548/0 (socket says 172.21.15.121:41548) 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4afcad9700 1 -- 172.21.15.121:0/518359508 learned_addr learned my addr 172.21.15.121:0/518359508 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4afd2da700 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef57c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4af7fff700 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef5605a0 0x560bef57c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4af7fff700 1 -- 172.21.15.121:0/518359508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 msgr2=0x560bef57bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4af7fff700 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef57bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.498 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4af7fff700 1 -- 172.21.15.121:0/518359508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 msgr2=0x560bef57c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4af7fff700 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef57c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4af7fff700 1 -- 172.21.15.121:0/518359508 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bef5de1e0 con 0x560bef5605a0 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4afd2da700 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef57c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.494+0000 7f4af7fff700 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef5605a0 0x560bef57c1f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4005610 tx=0x7f4ae4010e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ed74d8756ea8a05 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4ad7fff700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4ad7fff700 1 -- 172.21.15.121:0/518359508 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae402b040 con 0x560bef5605a0 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4ad7fff700 1 -- 172.21.15.121:0/518359508 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f4ae4003970 con 0x560bef5605a0 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4b095c0b80 1 -- 172.21.15.121:0/518359508 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4af0017d10 con 0x560bef5605a0 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4ad7fff700 1 -- 172.21.15.121:0/518359508 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4ae4011710 con 0x560bef5605a0 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4b095c0b80 10 client.25747 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4b095c0b80 1 -- 172.21.15.121:0/518359508 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560bef5df350 con 0x560bef5605a0 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4ad7fff700 1 -- 172.21.15.121:0/518359508 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4af0017d10 con 0x560bef5605a0 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4ad7fff700 1 -- 172.21.15.121:0/518359508 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f4ae402e070 con 0x560bef5605a0 2024-06-10T00:33:32.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4b095c0b80 10 client.25747 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:32.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4b095c0b80 10 client.25747 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:33:32.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4b095c0b80 20 client.25747 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:32.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x560bef574600 0x560bef5e1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:32.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4b095c0b80 4 client.25747 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.45, "await_for": 45.0}] 2024-06-10T00:33:32.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4b095c0b80 1 -- 172.21.15.121:0/518359508 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.45, "await_for": 45.0}) -- 0x560bef4b3470 con 0x560bef574600 2024-06-10T00:33:32.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.495+0000 7f4afcad9700 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x560bef574600 0x560bef5e1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:32.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.496+0000 7f4afcad9700 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x560bef574600 0x560bef5e1ae0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6cae115a7d721a5 server_cookie=8b6a876dd01a88b3 in_seq=0 out_seq=0 2024-06-10T00:33:32.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:32.496+0000 7f4ad7fff700 10 client.25747 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:32 smithi135 ceph-mon[36424]: pgmap v170: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.1 MiB/s wr, 543 op/s 2024-06-10T00:33:33.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:32 smithi193 ceph-mon[36716]: pgmap v170: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.1 MiB/s wr, 543 op/s 2024-06-10T00:33:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:32 smithi121 ceph-mon[32378]: pgmap v170: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.1 MiB/s wr, 543 op/s 2024-06-10T00:33:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:34 smithi135 ceph-mon[36424]: pgmap v171: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.0 MiB/s wr, 657 op/s 2024-06-10T00:33:35.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:34 smithi193 ceph-mon[36716]: pgmap v171: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.0 MiB/s wr, 657 op/s 2024-06-10T00:33:35.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:34 smithi121 ceph-mon[32378]: pgmap v171: 129 pgs: 129 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.0 MiB/s wr, 657 op/s 2024-06-10T00:33:37.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:36 smithi135 ceph-mon[36424]: pgmap v172: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 11 MiB/s wr, 873 op/s 2024-06-10T00:33:37.295 INFO:tasks.workunit.client.0.smithi121.stdout: 246 21944 297 15508 381 9871 178 2024-06-10T00:33:37.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:36 smithi193 ceph-mon[36716]: pgmap v172: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 11 MiB/s wr, 873 op/s 2024-06-10T00:33:37.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:36 smithi121 ceph-mon[32378]: pgmap v172: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 11 MiB/s wr, 873 op/s 2024-06-10T00:33:39.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:38 smithi135 ceph-mon[36424]: pgmap v173: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 10 MiB/s wr, 733 op/s 2024-06-10T00:33:39.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:38 smithi193 ceph-mon[36716]: pgmap v173: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 10 MiB/s wr, 733 op/s 2024-06-10T00:33:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:38 smithi121 ceph-mon[32378]: pgmap v173: 129 pgs: 129 active+clean; 1.2 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 10 MiB/s wr, 733 op/s 2024-06-10T00:33:40.232 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:33:40.555 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:33:41.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:40 smithi135 ceph-mon[36424]: pgmap v174: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-06-10T00:33:41.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:40 smithi193 ceph-mon[36716]: pgmap v174: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-06-10T00:33:41.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:40 smithi121 ceph-mon[32378]: pgmap v174: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 11 MiB/s wr, 1.04k op/s 2024-06-10T00:33:42.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.505+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/1669435472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a88101d40 msgr2=0x7f7a8810e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:42.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.505+0000 7f7a8dcc8700 1 --2- 172.21.15.121:0/1669435472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a88101d40 0x7f7a8810e230 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000b0a0 tx=0x7f7a8000b3b0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.506+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/1669435472 shutdown_connections 2024-06-10T00:33:42.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.506+0000 7f7a8dcc8700 1 --2- 172.21.15.121:0/1669435472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a88101d40 0x7f7a8810e230 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.506+0000 7f7a8dcc8700 1 --2- 172.21.15.121:0/1669435472 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a881073f0 0x7f7a881077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.506+0000 7f7a8dcc8700 1 --2- 172.21.15.121:0/1669435472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a88101380 0x7f7a88101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.506+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/1669435472 >> 172.21.15.121:0/1669435472 conn(0x7f7a88078300 msgr2=0x7f7a88078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:42.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.506+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/1669435472 shutdown_connections 2024-06-10T00:33:42.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.506+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/1669435472 wait complete. 2024-06-10T00:33:42.541 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.507+0000 7f7a8dcc8700 1 Processor -- start 2024-06-10T00:33:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.507+0000 7f7a8dcc8700 1 -- start start 2024-06-10T00:33:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a8dcc8700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a88101380 0x7f7a8819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a8dcc8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a88101d40 0x7f7a8819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a8dcc8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a881073f0 0x7f7a88198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a8dcc8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a881127c0 con 0x7f7a881073f0 2024-06-10T00:33:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a8dcc8700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f7a88112640 con 0x7f7a88101d40 2024-06-10T00:33:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a8dcc8700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7a88112940 con 0x7f7a88101380 2024-06-10T00:33:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a877fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a88101380 0x7f7a8819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a877fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a88101380 0x7f7a8819e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:48444/0 (socket says 172.21.15.121:48444) 2024-06-10T00:33:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a877fe700 1 -- 172.21.15.121:0/318523692 learned_addr learned my addr 172.21.15.121:0/318523692 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:42.542 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a86ffd700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a88101d40 0x7f7a8819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a87fff700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a881073f0 0x7f7a88198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a877fe700 1 -- 172.21.15.121:0/318523692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a88101d40 msgr2=0x7f7a8819ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a877fe700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a88101d40 0x7f7a8819ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a877fe700 1 -- 172.21.15.121:0/318523692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a881073f0 msgr2=0x7f7a88198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a877fe700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a881073f0 0x7f7a88198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a877fe700 1 -- 172.21.15.121:0/318523692 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a88199020 con 0x7f7a88101380 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.508+0000 7f7a86ffd700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a88101d40 0x7f7a8819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.509+0000 7f7a877fe700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a88101380 0x7f7a8819e6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800e9a0 tx=0x7f7a7800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.509+0000 7f7a84ff9700 1 -- 172.21.15.121:0/318523692 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7800cbd0 con 0x7f7a88101380 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.509+0000 7f7a84ff9700 1 -- 172.21.15.121:0/318523692 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a7800cd70 con 0x7f7a88101380 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.509+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/318523692 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a88199310 con 0x7f7a88101380 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.509+0000 7f7a84ff9700 1 -- 172.21.15.121:0/318523692 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a78018750 con 0x7f7a88101380 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.509+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/318523692 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a881a5520 con 0x7f7a88101380 2024-06-10T00:33:42.543 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.512+0000 7f7a84ff9700 1 -- 172.21.15.121:0/318523692 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f7a780188f0 con 0x7f7a88101380 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.513+0000 7f7a84ff9700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7a700a2e50 0x7f7a700a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.513+0000 7f7a9854bb80 1 Processor -- start 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.513+0000 7f7a84ff9700 1 -- 172.21.15.121:0/318523692 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f7a780cfad0 con 0x7f7a88101380 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a9854bb80 1 -- start start 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a86ffd700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7a700a2e50 0x7f7a700a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e50340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e50880 0x55cee4e4a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e4abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55cee4d3cf60 con 0x55cee4e50880 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55cee4d9b850 con 0x55cee4e4ff60 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cee4d03460 con 0x55cee4e4a7a0 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a877fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e50340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a87fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e4abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.544 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a877fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e50340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:33910/0 (socket says 172.21.15.121:33910) 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a877fe700 1 -- 172.21.15.121:0/3452905682 learned_addr learned my addr 172.21.15.121:0/3452905682 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a877fe700 1 -- 172.21.15.121:0/3452905682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 msgr2=0x55cee4e4abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a877fe700 1 --2- 172.21.15.121:0/3452905682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e4abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a877fe700 1 -- 172.21.15.121:0/3452905682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e50880 msgr2=0x55cee4e4a260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a877fe700 1 --2- 172.21.15.121:0/3452905682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e50880 0x55cee4e4a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.514+0000 7f7a877fe700 1 -- 172.21.15.121:0/3452905682 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cee4cea8d0 con 0x55cee4e4ff60 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.515+0000 7f7a877fe700 1 --2- 172.21.15.121:0/3452905682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e50340 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7a780a4340 tx=0x7f7a7809b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d20f3db204fd5369 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.515+0000 7f7a6dffb700 1 -- 172.21.15.121:0/3452905682 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7809ffb0 con 0x55cee4e4ff60 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.515+0000 7f7a6dffb700 1 -- 172.21.15.121:0/3452905682 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a780a4870 con 0x55cee4e4ff60 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.515+0000 7f7a6dffb700 1 -- 172.21.15.121:0/3452905682 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a780a43e0 con 0x55cee4e4ff60 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a9854bb80 1 -- 172.21.15.121:0/3452905682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 msgr2=0x55cee4e50340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a86ffd700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7a700a2e50 0x7f7a700a5310 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7a88078c40 tx=0x7f7a7c009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:42.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/3452905682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e50340 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7a780a4340 tx=0x7f7a7809b690 comp rx=0 tx=0).stop 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a9854bb80 1 -- 172.21.15.121:0/3452905682 shutdown_connections 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/3452905682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e4abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/3452905682 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e50340 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/3452905682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e50880 0x55cee4e4a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a9854bb80 1 -- 172.21.15.121:0/3452905682 >> 172.21.15.121:0/3452905682 conn(0x55cee4d77080 msgr2=0x55cee4e45220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a9854bb80 1 -- 172.21.15.121:0/3452905682 shutdown_connections 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a9854bb80 1 -- 172.21.15.121:0/3452905682 wait complete. 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a9854bb80 1 -- start start 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.516+0000 7f7a9854bb80 1 Processor -- start 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e56490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4ff60 0x55cee4e539e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 0x55cee4e53f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.546 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55cee4d3bfc0 con 0x55cee4e4ff60 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55cee4d97810 con 0x55cee4e50880 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cee4e57200 con 0x55cee4e4a7a0 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a877fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e56490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a877fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e56490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:48458/0 (socket says 172.21.15.121:48458) 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a9854bb80 10 client.0 _ll_register_callbacks cb 0x55cee4d315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a877fe700 1 -- 172.21.15.121:0/2340445752 learned_addr learned my addr 172.21.15.121:0/2340445752 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a86ffd700 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4ff60 0x55cee4e539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a87fff700 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 0x55cee4e53f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a877fe700 1 -- 172.21.15.121:0/2340445752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 msgr2=0x55cee4e53f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a877fe700 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 0x55cee4e53f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a877fe700 1 -- 172.21.15.121:0/2340445752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4ff60 msgr2=0x55cee4e539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a877fe700 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4ff60 0x55cee4e539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.547 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.517+0000 7f7a877fe700 1 -- 172.21.15.121:0/2340445752 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cee4d61750 con 0x55cee4e4a7a0 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.518+0000 7f7a877fe700 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e56490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7a78010d50 tx=0x7f7a7800eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fabea232081e4bca server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.518+0000 7f7a66ffd700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.518+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2340445752 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a780b8020 con 0x55cee4e4a7a0 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.518+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2340445752 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a780a4870 con 0x55cee4e4a7a0 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.518+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2340445752 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f7a78099f00 con 0x55cee4e4a7a0 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.518+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2340445752 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7a78018df0 con 0x55cee4e4a7a0 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.518+0000 7f7a9854bb80 10 client.25759 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.518+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2340445752 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cee4e6a240 con 0x55cee4e4a7a0 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.518+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2340445752 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a78065070 con 0x55cee4e4a7a0 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.519+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2340445752 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7a78097950 con 0x55cee4e4a7a0 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.519+0000 7f7a9854bb80 10 client.25759 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.519+0000 7f7a9854bb80 10 client.25759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.519+0000 7f7a9854bb80 20 client.25759 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:42.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.519+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cee4e6c710 0x55cee4e6eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.519+0000 7f7a9854bb80 4 client.25759 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.519+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2340445752 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7a780a4870 con 0x55cee4e6c710 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.519+0000 7f7a86ffd700 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cee4e6c710 0x55cee4e6eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.520+0000 7f7a86ffd700 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cee4e6c710 0x55cee4e6eb00 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19824cbd0cf089c server_cookie=f49a22e7946e5a74 in_seq=0 out_seq=0 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.520+0000 7f7a66ffd700 10 client.25759 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.520+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2340445752 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x7f7a780a4870 con 0x55cee4e6c710 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.520+0000 7f7a66ffd700 10 client.25759 handle_command_reply: tid=0 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.520+0000 7f7a9854bb80 1 client.25759 shutdown 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2340445752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 msgr2=0x55cee4e56490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e56490 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7a78010d50 tx=0x7f7a7800eef0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2340445752 shutdown_connections 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cee4e6c710 0x55cee4e6eb00 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a66ffd700 0 client.25759 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:42.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 0x55cee4e53f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e56490 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2340445752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4ff60 0x55cee4e539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2340445752 >> 172.21.15.121:0/2340445752 conn(0x55cee4d77080 msgr2=0x55cee4e44700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2340445752 shutdown_connections 2024-06-10T00:33:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2340445752 wait complete. 2024-06-10T00:33:42.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.521+0000 7f7a9854bb80 20 client.25759 trim_cache size 0 max 16384 2024-06-10T00:33:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.552+0000 7f7a9854bb80 1 Processor -- start 2024-06-10T00:33:42.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.552+0000 7f7a9854bb80 1 -- start start 2024-06-10T00:33:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.552+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e542a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4ff60 0x55cee4e56220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 0x55cee4e44e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a881aeb70 con 0x55cee4e4ff60 2024-06-10T00:33:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f7a881ac3a0 con 0x55cee4e50880 2024-06-10T00:33:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7a881aa1f0 con 0x55cee4e4a7a0 2024-06-10T00:33:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a87fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 0x55cee4e44e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a877fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a87fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 0x55cee4e44e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:33932/0 (socket says 172.21.15.121:33932) 2024-06-10T00:33:42.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a877fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e542a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:48474/0 (socket says 172.21.15.121:48474) 2024-06-10T00:33:42.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a87fff700 1 -- 172.21.15.121:0/2021817892 learned_addr learned my addr 172.21.15.121:0/2021817892 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:42.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a87fff700 1 -- 172.21.15.121:0/2021817892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 msgr2=0x55cee4e542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a87fff700 1 --2- 172.21.15.121:0/2021817892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a87fff700 1 -- 172.21.15.121:0/2021817892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4ff60 msgr2=0x55cee4e56220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a87fff700 1 --2- 172.21.15.121:0/2021817892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4ff60 0x55cee4e56220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.553+0000 7f7a87fff700 1 -- 172.21.15.121:0/2021817892 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cee4e51f50 con 0x55cee4e50880 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.554+0000 7f7a87fff700 1 --2- 172.21.15.121:0/2021817892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 0x55cee4e44e80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009e60 tx=0x7f7a80009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4a1e7c4906a595e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.554+0000 7f7a6d7fa700 1 -- 172.21.15.121:0/2021817892 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80003a00 con 0x55cee4e50880 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.554+0000 7f7a6d7fa700 1 -- 172.21.15.121:0/2021817892 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a80003ba0 con 0x55cee4e50880 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.554+0000 7f7a6d7fa700 1 -- 172.21.15.121:0/2021817892 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80005260 con 0x55cee4e50880 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.555+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2021817892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 msgr2=0x55cee4e44e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.555+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2021817892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 0x55cee4e44e80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009e60 tx=0x7f7a80009520 comp rx=0 tx=0).stop 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.555+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2021817892 shutdown_connections 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.555+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2021817892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e50880 0x55cee4e44e80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.555+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2021817892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e4a7a0 0x55cee4e542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.555+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2021817892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4ff60 0x55cee4e56220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.555+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2021817892 >> 172.21.15.121:0/2021817892 conn(0x55cee4d77080 msgr2=0x55cee4e59cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:42.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.555+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2021817892 shutdown_connections 2024-06-10T00:33:42.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.555+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2021817892 wait complete. 2024-06-10T00:33:42.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.556+0000 7f7a9854bb80 1 -- start start 2024-06-10T00:33:42.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.556+0000 7f7a9854bb80 1 Processor -- start 2024-06-10T00:33:42.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.556+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4a7a0 0x55cee4e66100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.556+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e66640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.556+0000 7f7a9854bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e50880 0x55cee4e66b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.556+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a881aa860 con 0x55cee4e4a7a0 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.556+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f7a881abcd0 con 0x55cee4e4ff60 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.556+0000 7f7a9854bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7a881aaf30 con 0x55cee4e50880 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a86ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e66640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4a7a0 0x55cee4e66100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a86ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e66640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:33938/0 (socket says 172.21.15.121:33938) 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a86ffd700 1 -- 172.21.15.121:0/2733819266 learned_addr learned my addr 172.21.15.121:0/2733819266 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a9854bb80 10 client.0 _ll_register_callbacks cb 0x55cee4d315b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a87fff700 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e50880 0x55cee4e66b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a86ffd700 1 -- 172.21.15.121:0/2733819266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e50880 msgr2=0x55cee4e66b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a86ffd700 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e50880 0x55cee4e66b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a86ffd700 1 -- 172.21.15.121:0/2733819266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4a7a0 msgr2=0x55cee4e66100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:42.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a86ffd700 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4a7a0 0x55cee4e66100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a86ffd700 1 -- 172.21.15.121:0/2733819266 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cee4ed6350 con 0x55cee4e4ff60 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a87fff700 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e50880 0x55cee4e66b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a86ffd700 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e66640 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55cee4e54fb0 tx=0x7f7a7c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4383be8c6e980451 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a66ffd700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2733819266 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c02f060 con 0x55cee4e4ff60 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2733819266 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a80003ba0 con 0x55cee4e4ff60 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.557+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2733819266 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f7a7c003b50 con 0x55cee4e4ff60 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2733819266 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7a7c036050 con 0x55cee4e4ff60 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a9854bb80 10 client.25862 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2733819266 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cee4ed7690 con 0x55cee4e4ff60 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2733819266 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a80003ba0 con 0x55cee4e4ff60 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2733819266 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f7a7c029070 con 0x55cee4e4ff60 2024-06-10T00:33:42.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a9854bb80 10 client.25862 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a9854bb80 10 client.25862 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a9854bb80 20 client.25862 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cee4ed7de0 0x55cee4eda200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a9854bb80 4 client.25862 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2733819266 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cee4da8d50 con 0x55cee4ed7de0 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.558+0000 7f7a877fe700 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cee4ed7de0 0x55cee4eda200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.559+0000 7f7a877fe700 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cee4ed7de0 0x55cee4eda200 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1461f55e0517803 server_cookie=765c6fc083e990cc in_seq=0 out_seq=0 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.559+0000 7f7a66ffd700 10 client.25862 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.559+0000 7f7a66ffd700 1 -- 172.21.15.121:0/2733819266 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cee4da8d50 con 0x55cee4ed7de0 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.559+0000 7f7a66ffd700 10 client.25862 handle_command_reply: tid=0 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.559+0000 7f7a9854bb80 1 client.25862 shutdown 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.559+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2733819266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 msgr2=0x55cee4e66640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.559+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e66640 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55cee4e54fb0 tx=0x7f7a7c004750 comp rx=0 tx=0).stop 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.560+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2733819266 shutdown_connections 2024-06-10T00:33:42.563 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.560+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cee4ed7de0 0x55cee4eda200 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.560+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cee4e50880 0x55cee4e66b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.560+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cee4e4a7a0 0x55cee4e66100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.560+0000 7f7a66ffd700 0 client.25862 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:42.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.560+0000 7f7a9854bb80 1 --2- 172.21.15.121:0/2733819266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cee4e4ff60 0x55cee4e66640 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.560+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2733819266 >> 172.21.15.121:0/2733819266 conn(0x55cee4d77080 msgr2=0x55cee4e59460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:42.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.560+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2733819266 shutdown_connections 2024-06-10T00:33:42.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.560+0000 7f7a9854bb80 1 -- 172.21.15.121:0/2733819266 wait complete. 2024-06-10T00:33:42.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.560+0000 7f7a9854bb80 20 client.25862 trim_cache size 0 max 16384 2024-06-10T00:33:42.564 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:33:42.564 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "no active scrubs running", 2024-06-10T00:33:42.565 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": {} 2024-06-10T00:33:42.565 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/318523692 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7a700a2e50 msgr2=0x7f7a700a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7a700a2e50 0x7f7a700a5310 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7a88078c40 tx=0x7f7a7c009260 comp rx=0 tx=0).stop 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/318523692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a88101380 msgr2=0x7f7a8819e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a88101380 0x7f7a8819e6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800e9a0 tx=0x7f7a7800ecb0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/318523692 shutdown_connections 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f7a700a2e50 0x7f7a700a5310 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a881073f0 0x7f7a88198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a88101380 0x7f7a8819e6c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 --2- 172.21.15.121:0/318523692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7a88101d40 0x7f7a8819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/318523692 >> 172.21.15.121:0/318523692 conn(0x7f7a88078300 msgr2=0x7f7a88108270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/318523692 shutdown_connections 2024-06-10T00:33:42.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:42.561+0000 7f7a8dcc8700 1 -- 172.21.15.121:0/318523692 wait complete. 2024-06-10T00:33:43.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:42 smithi135 ceph-mon[36424]: pgmap v175: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 10 MiB/s wr, 913 op/s 2024-06-10T00:33:43.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:42 smithi193 ceph-mon[36716]: pgmap v175: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 10 MiB/s wr, 913 op/s 2024-06-10T00:33:43.428 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:42 smithi121 ceph-mon[32378]: pgmap v175: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 10 MiB/s wr, 913 op/s 2024-06-10T00:33:43.458 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-10T00:33:43.458 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:33:43.772 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:33:45.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:44 smithi121 ceph-mon[32378]: pgmap v176: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.1 MiB/s wr, 811 op/s 2024-06-10T00:33:45.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:44 smithi135 ceph-mon[36424]: pgmap v176: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.1 MiB/s wr, 811 op/s 2024-06-10T00:33:45.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:44 smithi193 ceph-mon[36716]: pgmap v176: 129 pgs: 129 active+clean; 1.2 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 9.1 MiB/s wr, 811 op/s 2024-06-10T00:33:46.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4ad7fff700 1 -- 172.21.15.121:0/518359508 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x560bef4b3470 con 0x560bef574600 2024-06-10T00:33:46.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4ad7fff700 10 client.25747 handle_command_reply: tid=0 2024-06-10T00:33:46.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 1 client.25747 shutdown 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 1 -- 172.21.15.121:0/518359508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef5605a0 msgr2=0x560bef57c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef5605a0 0x560bef57c1f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4005610 tx=0x7f4ae4010e70 comp rx=0 tx=0).stop 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 1 -- 172.21.15.121:0/518359508 shutdown_connections 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x560bef574600 0x560bef5e1ae0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bef560ec0 0x560bef57c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x560bef55acf0 0x560bef57bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 1 --2- 172.21.15.121:0/518359508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560bef5605a0 0x560bef57c1f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 1 -- 172.21.15.121:0/518359508 >> 172.21.15.121:0/518359508 conn(0x560bef487220 msgr2=0x560bef5543c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4ad7fff700 0 client.25747 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 1 -- 172.21.15.121:0/518359508 shutdown_connections 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 1 -- 172.21.15.121:0/518359508 wait complete. 2024-06-10T00:33:46.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.595+0000 7f4b095c0b80 20 client.25747 trim_cache size 0 max 16384 2024-06-10T00:33:46.597 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:33:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 30, 2024-06-10T00:33:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:33:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 14, 2024-06-10T00:33:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:33:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "5eddb4ce": { 2024-06-10T00:33:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 14, 2024-06-10T00:33:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:33:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:33:46.597 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 149.4, 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:33:46.598 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:33:46.599 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:33:46.599 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:33:46.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 -- 172.21.15.121:0/3516334919 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ae80a2e30 msgr2=0x7f4ae80a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:46.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ae80a2e30 0x7f4ae80a52f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4af8107b20 tx=0x7f4ae4007480 comp rx=0 tx=0).stop 2024-06-10T00:33:46.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 -- 172.21.15.121:0/3516334919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4af81015f0 msgr2=0x7f4af8199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:46.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4af81015f0 0x7f4af8199a50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0098e0 tx=0x7f4aec00fa00 comp rx=0 tx=0).stop 2024-06-10T00:33:46.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 -- 172.21.15.121:0/3516334919 shutdown_connections 2024-06-10T00:33:46.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ae80a2e30 0x7f4ae80a52f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:46.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4af81066c0 0x7f4af81a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:46.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4af81015f0 0x7f4af8199a50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:46.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 --2- 172.21.15.121:0/3516334919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4af8101fa0 0x7f4af8199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:46.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 -- 172.21.15.121:0/3516334919 >> 172.21.15.121:0/3516334919 conn(0x7f4af8078330 msgr2=0x7f4af80ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:46.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 -- 172.21.15.121:0/3516334919 shutdown_connections 2024-06-10T00:33:46.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:46.596+0000 7f4afed3d700 1 -- 172.21.15.121:0/3516334919 wait complete. 2024-06-10T00:33:47.145 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:46 smithi121 ceph-mon[32378]: pgmap v177: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.9 MiB/s wr, 1.16k op/s 2024-06-10T00:33:47.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:46 smithi135 ceph-mon[36424]: pgmap v177: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.9 MiB/s wr, 1.16k op/s 2024-06-10T00:33:47.295 INFO:tasks.workunit.client.0.smithi121.stdout: 246 1232 0 732 0 546 0 2024-06-10T00:33:47.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:46 smithi193 ceph-mon[36716]: pgmap v177: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.9 MiB/s wr, 1.16k op/s 2024-06-10T00:33:47.574 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5eddb4ce', quiesce took 21.3 seconds. Will release after: 8.149999999999999 2024-06-10T00:33:48.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.094+0000 7fb4578ff700 1 -- 172.21.15.121:0/3455489121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb450107400 msgr2=0x7fb4501077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:48.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.094+0000 7fb4578ff700 1 --2- 172.21.15.121:0/3455489121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb450107400 0x7fb4501077e0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb44c009990 tx=0x7fb44c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:33:48.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.094+0000 7fb4578ff700 1 -- 172.21.15.121:0/3455489121 shutdown_connections 2024-06-10T00:33:48.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.094+0000 7fb4578ff700 1 --2- 172.21.15.121:0/3455489121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb450101d50 0x7fb45010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:48.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.094+0000 7fb4578ff700 1 --2- 172.21.15.121:0/3455489121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb450107400 0x7fb4501077e0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:48.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.094+0000 7fb4578ff700 1 --2- 172.21.15.121:0/3455489121 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb450101390 0x7fb450101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:48.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.094+0000 7fb4578ff700 1 -- 172.21.15.121:0/3455489121 >> 172.21.15.121:0/3455489121 conn(0x7fb450078340 msgr2=0x7fb450078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:48.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.094+0000 7fb4578ff700 1 -- 172.21.15.121:0/3455489121 shutdown_connections 2024-06-10T00:33:48.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.094+0000 7fb4578ff700 1 -- 172.21.15.121:0/3455489121 wait complete. 2024-06-10T00:33:48.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb4578ff700 1 Processor -- start 2024-06-10T00:33:48.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb4578ff700 1 -- start start 2024-06-10T00:33:48.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb4578ff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb450101390 0x7fb45019d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:48.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb4578ff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb450101d50 0x7fb45019dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:48.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb4578ff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb450107400 0x7fb450197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:48.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb4578ff700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb450111670 con 0x7fb450101390 2024-06-10T00:33:48.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb4578ff700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fb4501114f0 con 0x7fb450107400 2024-06-10T00:33:48.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb4578ff700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb4501117f0 con 0x7fb450101d50 2024-06-10T00:33:48.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb45569b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb450101390 0x7fb45019d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:48.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb45569b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb450101390 0x7fb45019d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:49520/0 (socket says 172.21.15.121:49520) 2024-06-10T00:33:48.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb45569b700 1 -- 172.21.15.121:0/3805139143 learned_addr learned my addr 172.21.15.121:0/3805139143 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:48.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb454e9a700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb450101d50 0x7fb45019dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:48.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb455e9c700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb450107400 0x7fb450197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:48.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb45569b700 1 -- 172.21.15.121:0/3805139143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb450101d50 msgr2=0x7fb45019dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:48.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb45569b700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb450101d50 0x7fb45019dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:48.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb45569b700 1 -- 172.21.15.121:0/3805139143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb450107400 msgr2=0x7fb450197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:48.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb45569b700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb450107400 0x7fb450197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:48.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb45569b700 1 -- 172.21.15.121:0/3805139143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb450198090 con 0x7fb450101390 2024-06-10T00:33:48.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.095+0000 7fb455e9c700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb450107400 0x7fb450197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:33:48.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.096+0000 7fb45569b700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb450101390 0x7fb45019d770 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb44c000c00 tx=0x7fb44c00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:48.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.096+0000 7fb4467fc700 1 -- 172.21.15.121:0/3805139143 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c027070 con 0x7fb450101390 2024-06-10T00:33:48.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.096+0000 7fb4578ff700 1 -- 172.21.15.121:0/3805139143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb450198320 con 0x7fb450101390 2024-06-10T00:33:48.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.096+0000 7fb4467fc700 1 -- 172.21.15.121:0/3805139143 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb44c003d70 con 0x7fb450101390 2024-06-10T00:33:48.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.096+0000 7fb4467fc700 1 -- 172.21.15.121:0/3805139143 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c00f460 con 0x7fb450101390 2024-06-10T00:33:48.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.096+0000 7fb4578ff700 1 -- 172.21.15.121:0/3805139143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb45006a170 con 0x7fb450101390 2024-06-10T00:33:48.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.097+0000 7fb4578ff700 1 -- 172.21.15.121:0/3805139143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb450102ab0 con 0x7fb450101390 2024-06-10T00:33:48.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.098+0000 7fb4467fc700 1 -- 172.21.15.121:0/3805139143 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb44c02a070 con 0x7fb450101390 2024-06-10T00:33:48.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.099+0000 7fb4467fc700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb43c0a2d90 0x7fb43c0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:48.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.099+0000 7fb454e9a700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb43c0a2d90 0x7fb43c0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:48.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.099+0000 7fb4467fc700 1 -- 172.21.15.121:0/3805139143 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fb44c0d04c0 con 0x7fb450101390 2024-06-10T00:33:48.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.099+0000 7fb454e9a700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb43c0a2d90 0x7fb43c0a5250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb450078940 tx=0x7fb440009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:48.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.104+0000 7fb4467fc700 1 -- 172.21.15.121:0/3805139143 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb44c013070 con 0x7fb450101390 2024-06-10T00:33:48.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.383+0000 7fb4578ff700 1 -- 172.21.15.121:0/3805139143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb450198e60 con 0x7fb450101390 2024-06-10T00:33:48.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.384+0000 7fb4467fc700 1 -- 172.21.15.121:0/3805139143 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11891 (secure 0 0 0) 0x7fb44c09cde0 con 0x7fb450101390 2024-06-10T00:33:48.384 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:33:48.384 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:33:03.460049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:33:48.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 -- 172.21.15.121:0/3805139143 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb43c0a2d90 msgr2=0x7fb43c0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:48.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb43c0a2d90 0x7fb43c0a5250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb450078940 tx=0x7fb440009260 comp rx=0 tx=0).stop 2024-06-10T00:33:48.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 -- 172.21.15.121:0/3805139143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb450101390 msgr2=0x7fb45019d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:48.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb450101390 0x7fb45019d770 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb44c000c00 tx=0x7fb44c00bab0 comp rx=0 tx=0).stop 2024-06-10T00:33:48.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 -- 172.21.15.121:0/3805139143 shutdown_connections 2024-06-10T00:33:48.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb43c0a2d90 0x7fb43c0a5250 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:48.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb450107400 0x7fb450197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:48.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb450101390 0x7fb45019d770 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:48.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 --2- 172.21.15.121:0/3805139143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb450101d50 0x7fb45019dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:48.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 -- 172.21.15.121:0/3805139143 >> 172.21.15.121:0/3805139143 conn(0x7fb450078340 msgr2=0x7fb450100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:48.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 -- 172.21.15.121:0/3805139143 shutdown_connections 2024-06-10T00:33:48.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:48.386+0000 7fb4578ff700 1 -- 172.21.15.121:0/3805139143 wait complete. 2024-06-10T00:33:48.387 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-06-10T00:33:49.071 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-10T00:33:49.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:49 smithi135 ceph-mon[36424]: pgmap v178: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 851 op/s 2024-06-10T00:33:49.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:49 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3805139143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:33:49.340 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:33:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:49 smithi193 ceph-mon[36716]: pgmap v178: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 851 op/s 2024-06-10T00:33:49.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:49 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3805139143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:33:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:49 smithi121 ceph-mon[32378]: pgmap v178: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.7 MiB/s wr, 851 op/s 2024-06-10T00:33:49.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:49 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3805139143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:33:51.070 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:51 smithi121 ceph-mon[32378]: pgmap v179: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 883 op/s 2024-06-10T00:33:51.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.187+0000 7f50cba38700 1 -- 172.21.15.121:0/4161858356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c4108500 msgr2=0x7f50c41088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:51.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.187+0000 7f50cba38700 1 --2- 172.21.15.121:0/4161858356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c4108500 0x7f50c41088e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009990 tx=0x7f50c0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.187+0000 7f50cba38700 1 -- 172.21.15.121:0/4161858356 shutdown_connections 2024-06-10T00:33:51.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.187+0000 7f50cba38700 1 --2- 172.21.15.121:0/4161858356 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50c406ab30 0x7f50c410f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.187+0000 7f50cba38700 1 --2- 172.21.15.121:0/4161858356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c4108500 0x7f50c41088e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.187+0000 7f50cba38700 1 --2- 172.21.15.121:0/4161858356 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50c406a170 0x7f50c406a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.187+0000 7f50cba38700 1 -- 172.21.15.121:0/4161858356 >> 172.21.15.121:0/4161858356 conn(0x7f50c4078340 msgr2=0x7f50c4078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:51.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.187+0000 7f50cba38700 1 -- 172.21.15.121:0/4161858356 shutdown_connections 2024-06-10T00:33:51.189 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.187+0000 7f50cba38700 1 -- 172.21.15.121:0/4161858356 wait complete. 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.188+0000 7f50cba38700 1 Processor -- start 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.188+0000 7f50cba38700 1 -- start start 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.188+0000 7f50cba38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c406a170 0x7f50c419ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.188+0000 7f50cba38700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50c406ab30 0x7f50c419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.188+0000 7f50cba38700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50c4108500 0x7f50c4198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.188+0000 7f50cba38700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41127d0 con 0x7f50c406a170 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.188+0000 7f50cba38700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4112650 con 0x7f50c406ab30 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.188+0000 7f50cba38700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4112950 con 0x7f50c4108500 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.188+0000 7f50c97d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c406a170 0x7f50c419ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.188+0000 7f50c97d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c406a170 0x7f50c419ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52948/0 (socket says 172.21.15.121:52948) 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.188+0000 7f50c97d4700 1 -- 172.21.15.121:0/1988110618 learned_addr learned my addr 172.21.15.121:0/1988110618 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50c8fd3700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50c406ab30 0x7f50c419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50c9fd5700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50c4108500 0x7f50c4198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50c97d4700 1 -- 172.21.15.121:0/1988110618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50c4108500 msgr2=0x7f50c4198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:51.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50c97d4700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50c4108500 0x7f50c4198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50c97d4700 1 -- 172.21.15.121:0/1988110618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50c406ab30 msgr2=0x7f50c419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:51.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50c97d4700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50c406ab30 0x7f50c419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50c97d4700 1 -- 172.21.15.121:0/1988110618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c4199410 con 0x7f50c406a170 2024-06-10T00:33:51.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50c8fd3700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50c406ab30 0x7f50c419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:51.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50c97d4700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c406a170 0x7f50c419ea60 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f50c0005910 tx=0x7f50c000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:51.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50ba7fc700 1 -- 172.21.15.121:0/1988110618 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c0027070 con 0x7f50c406a170 2024-06-10T00:33:51.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50cba38700 1 -- 172.21.15.121:0/1988110618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50c41996a0 con 0x7f50c406a170 2024-06-10T00:33:51.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.189+0000 7f50ba7fc700 1 -- 172.21.15.121:0/1988110618 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50c0004650 con 0x7f50c406a170 2024-06-10T00:33:51.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.190+0000 7f50cba38700 1 -- 172.21.15.121:0/1988110618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50c41a59e0 con 0x7f50c406a170 2024-06-10T00:33:51.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.190+0000 7f50ba7fc700 1 -- 172.21.15.121:0/1988110618 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c000f460 con 0x7f50c406a170 2024-06-10T00:33:51.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.192+0000 7f50ba7fc700 1 -- 172.21.15.121:0/1988110618 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f50c002a070 con 0x7f50c406a170 2024-06-10T00:33:51.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.192+0000 7f50ba7fc700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f50b00a2d90 0x7f50b00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.193+0000 7f50ba7fc700 1 -- 172.21.15.121:0/1988110618 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f50c00d0660 con 0x7f50c406a170 2024-06-10T00:33:51.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.193+0000 7f50c8fd3700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f50b00a2d90 0x7f50b00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.193+0000 7f50c8fd3700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f50b00a2d90 0x7f50b00a5250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f50c419a280 tx=0x7f50b4009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:51.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50d62bbb80 1 Processor -- start 2024-06-10T00:33:51.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50d62bbb80 1 -- start start 2024-06-10T00:33:51.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c17f60 0x557ef2c18340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 0x557ef2c12260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c127a0 0x557ef2c12bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557ef2b04f60 con 0x557ef2c17f60 2024-06-10T00:33:51.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c97d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c17f60 0x557ef2c18340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c8fd3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 0x557ef2c12260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x557ef2b63850 con 0x557ef2c127a0 2024-06-10T00:33:51.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557ef2acb460 con 0x557ef2c18880 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c97d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c17f60 0x557ef2c18340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52950/0 (socket says 172.21.15.121:52950) 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c8fd3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 0x557ef2c12260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:38308/0 (socket says 172.21.15.121:38308) 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c97d4700 1 -- 172.21.15.121:0/3780695550 learned_addr learned my addr 172.21.15.121:0/3780695550 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c9fd5700 1 --2- 172.21.15.121:0/3780695550 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c127a0 0x557ef2c12bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c97d4700 1 -- 172.21.15.121:0/3780695550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 msgr2=0x557ef2c12260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c97d4700 1 --2- 172.21.15.121:0/3780695550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 0x557ef2c12260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c97d4700 1 -- 172.21.15.121:0/3780695550 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c127a0 msgr2=0x557ef2c12bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c97d4700 1 --2- 172.21.15.121:0/3780695550 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c127a0 0x557ef2c12bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c97d4700 1 -- 172.21.15.121:0/3780695550 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ef2ab28d0 con 0x557ef2c17f60 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.194+0000 7f50c8fd3700 1 --2- 172.21.15.121:0/3780695550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 0x557ef2c12260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.195+0000 7f50c97d4700 1 --2- 172.21.15.121:0/3780695550 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c17f60 0x557ef2c18340 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f50c009c580 tx=0x7f50c009c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e3b525b78630c3 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.195+0000 7f50af7fe700 1 -- 172.21.15.121:0/3780695550 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c00cf070 con 0x557ef2c17f60 2024-06-10T00:33:51.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.195+0000 7f50af7fe700 1 -- 172.21.15.121:0/3780695550 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50c009c880 con 0x557ef2c17f60 2024-06-10T00:33:51.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.195+0000 7f50af7fe700 1 -- 172.21.15.121:0/3780695550 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c00227e0 con 0x557ef2c17f60 2024-06-10T00:33:51.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.196+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3780695550 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c17f60 msgr2=0x557ef2c18340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:51.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.196+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/3780695550 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c17f60 0x557ef2c18340 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f50c009c580 tx=0x7f50c009c490 comp rx=0 tx=0).stop 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.196+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3780695550 shutdown_connections 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.196+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/3780695550 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c127a0 0x557ef2c12bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.196+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/3780695550 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c17f60 0x557ef2c18340 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.196+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/3780695550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 0x557ef2c12260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.196+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3780695550 >> 172.21.15.121:0/3780695550 conn(0x557ef2b3f080 msgr2=0x557ef2c0d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.196+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3780695550 shutdown_connections 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.196+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3780695550 wait complete. 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.196+0000 7f50d62bbb80 1 -- start start 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.196+0000 7f50d62bbb80 1 Processor -- start 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c127a0 0x557ef2c25df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c17f60 0x557ef2c26330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c20210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557ef2b03fc0 con 0x557ef2c18880 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x557ef2b5f810 con 0x557ef2c127a0 2024-06-10T00:33:51.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557ef2c1f010 con 0x557ef2c17f60 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50d62bbb80 10 client.0 _ll_register_callbacks cb 0x557ef2af95b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c9fd5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c20210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c9fd5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c20210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52954/0 (socket says 172.21.15.121:52954) 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c9fd5700 1 -- 172.21.15.121:0/3962104156 learned_addr learned my addr 172.21.15.121:0/3962104156 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c97d4700 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c127a0 0x557ef2c25df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c8fd3700 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c17f60 0x557ef2c26330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c9fd5700 1 -- 172.21.15.121:0/3962104156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c17f60 msgr2=0x557ef2c26330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c9fd5700 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c17f60 0x557ef2c26330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c9fd5700 1 -- 172.21.15.121:0/3962104156 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c127a0 msgr2=0x557ef2c25df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c9fd5700 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c127a0 0x557ef2c25df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c9fd5700 1 -- 172.21.15.121:0/3962104156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ef2b29750 con 0x557ef2c18880 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c97d4700 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c127a0 0x557ef2c25df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50c9fd5700 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c20210 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f50bc011f60 tx=0x7f50bc014600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df244748111c452 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.197+0000 7f50acff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:33:51.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50acff9700 1 -- 172.21.15.121:0/3962104156 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc014ba0 con 0x557ef2c18880 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50acff9700 1 -- 172.21.15.121:0/3962104156 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f50bc014d40 con 0x557ef2c18880 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3962104156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50c009c880 con 0x557ef2c18880 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50acff9700 1 -- 172.21.15.121:0/3962104156 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f50bc027050 con 0x557ef2c18880 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50d62bbb80 10 client.16308 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3962104156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557ef2c2dbb0 con 0x557ef2c18880 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50acff9700 1 -- 172.21.15.121:0/3962104156 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50c009c880 con 0x557ef2c18880 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50acff9700 1 -- 172.21.15.121:0/3962104156 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f50bc007500 con 0x557ef2c18880 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50d62bbb80 10 client.16308 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50d62bbb80 10 client.16308 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50d62bbb80 20 client.16308 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x557ef2c301f0 0x557ef2c32600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50d62bbb80 4 client.16308 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3962104156 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ef2b70d50 con 0x557ef2c301f0 2024-06-10T00:33:51.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.198+0000 7f50c97d4700 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x557ef2c301f0 0x557ef2c32600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.199+0000 7f50c97d4700 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x557ef2c301f0 0x557ef2c32600 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8c90297766e67ec5 server_cookie=ff0517c6d2a78c3f in_seq=0 out_seq=0 2024-06-10T00:33:51.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.199+0000 7f50acff9700 10 client.16308 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:33:51.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.200+0000 7f50acff9700 1 -- 172.21.15.121:0/3962104156 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x557ef2b70d50 con 0x557ef2c301f0 2024-06-10T00:33:51.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.200+0000 7f50acff9700 10 client.16308 handle_command_reply: tid=0 2024-06-10T00:33:51.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 1 client.16308 shutdown 2024-06-10T00:33:51.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3962104156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 msgr2=0x557ef2c20210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:51.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c20210 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f50bc011f60 tx=0x7f50bc014600 comp rx=0 tx=0).stop 2024-06-10T00:33:51.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3962104156 shutdown_connections 2024-06-10T00:33:51.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x557ef2c301f0 0x557ef2c32600 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c20210 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c127a0 0x557ef2c25df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/3962104156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c17f60 0x557ef2c26330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3962104156 >> 172.21.15.121:0/3962104156 conn(0x557ef2b3f080 msgr2=0x557ef2c0c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:51.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50acff9700 0 client.16308 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:33:51.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3962104156 shutdown_connections 2024-06-10T00:33:51.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 1 -- 172.21.15.121:0/3962104156 wait complete. 2024-06-10T00:33:51.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.201+0000 7f50d62bbb80 20 client.16308 trim_cache size 0 max 16384 2024-06-10T00:33:51.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50d62bbb80 1 Processor -- start 2024-06-10T00:33:51.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50d62bbb80 1 -- start start 2024-06-10T00:33:51.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c127a0 0x557ef2c14330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c17f60 0x557ef2c20230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c1e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41aebc0 con 0x557ef2c18880 2024-06-10T00:33:51.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41ac3f0 con 0x557ef2c17f60 2024-06-10T00:33:51.218 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41aa260 con 0x557ef2c127a0 2024-06-10T00:33:51.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c9fd5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c1e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c97d4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c127a0 0x557ef2c14330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c8fd3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c17f60 0x557ef2c20230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c97d4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c127a0 0x557ef2c14330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:38324/0 (socket says 172.21.15.121:38324) 2024-06-10T00:33:51.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c9fd5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c1e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52966/0 (socket says 172.21.15.121:52966) 2024-06-10T00:33:51.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c97d4700 1 -- 172.21.15.121:0/2357602524 learned_addr learned my addr 172.21.15.121:0/2357602524 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:51.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c8fd3700 1 -- 172.21.15.121:0/2357602524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c127a0 msgr2=0x557ef2c14330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c8fd3700 1 --2- 172.21.15.121:0/2357602524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c127a0 0x557ef2c14330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c8fd3700 1 -- 172.21.15.121:0/2357602524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 msgr2=0x557ef2c1e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c8fd3700 1 --2- 172.21.15.121:0/2357602524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c1e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c8fd3700 1 -- 172.21.15.121:0/2357602524 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ef2c12dd0 con 0x557ef2c17f60 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c97d4700 1 --2- 172.21.15.121:0/2357602524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c127a0 0x557ef2c14330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.218+0000 7f50c9fd5700 1 --2- 172.21.15.121:0/2357602524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c1e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.219+0000 7f50c8fd3700 1 --2- 172.21.15.121:0/2357602524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c17f60 0x557ef2c20230 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x557ef2c21790 tx=0x7f50b4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d08525db68dfaa54 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.219+0000 7f50aeffd700 1 -- 172.21.15.121:0/2357602524 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4003ae0 con 0x557ef2c17f60 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.219+0000 7f50aeffd700 1 -- 172.21.15.121:0/2357602524 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50b4003c80 con 0x557ef2c17f60 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.219+0000 7f50aeffd700 1 -- 172.21.15.121:0/2357602524 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4028620 con 0x557ef2c17f60 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.220+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2357602524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c17f60 msgr2=0x557ef2c20230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:51.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.220+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/2357602524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c17f60 0x557ef2c20230 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x557ef2c21790 tx=0x7f50b4014000 comp rx=0 tx=0).stop 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.220+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2357602524 shutdown_connections 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.220+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/2357602524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c18880 0x557ef2c1e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.220+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/2357602524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c127a0 0x557ef2c14330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.220+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/2357602524 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c17f60 0x557ef2c20230 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.220+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2357602524 >> 172.21.15.121:0/2357602524 conn(0x557ef2b3f080 msgr2=0x557ef2c11250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.220+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2357602524 shutdown_connections 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.220+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2357602524 wait complete. 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.220+0000 7f50d62bbb80 1 -- start start 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.220+0000 7f50d62bbb80 1 Processor -- start 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c127a0 0x557ef2c151f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c17f60 0x557ef2c15730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50d62bbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 0x557ef2c15c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41aa8b0 con 0x557ef2c127a0 2024-06-10T00:33:51.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41abd20 con 0x557ef2c17f60 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50d62bbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41aaf80 con 0x557ef2c18880 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50d62bbb80 10 client.0 _ll_register_callbacks cb 0x557ef2af95b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50c97d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c127a0 0x557ef2c151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50c97d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c127a0 0x557ef2c151f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52972/0 (socket says 172.21.15.121:52972) 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50c8fd3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c17f60 0x557ef2c15730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50c97d4700 1 -- 172.21.15.121:0/2210052033 learned_addr learned my addr 172.21.15.121:0/2210052033 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50c9fd5700 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 0x557ef2c15c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50c97d4700 1 -- 172.21.15.121:0/2210052033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 msgr2=0x557ef2c15c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50c97d4700 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 0x557ef2c15c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50c97d4700 1 -- 172.21.15.121:0/2210052033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c17f60 msgr2=0x557ef2c15730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50c97d4700 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c17f60 0x557ef2c15730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50c97d4700 1 -- 172.21.15.121:0/2210052033 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ef2c98e90 con 0x557ef2c127a0 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50c97d4700 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c127a0 0x557ef2c151f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f50c009c810 tx=0x7f50c0009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529559a2cc0bad6d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:51.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50acff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50acff9700 1 -- 172.21.15.121:0/2210052033 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c00ba060 con 0x557ef2c127a0 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50acff9700 1 -- 172.21.15.121:0/2210052033 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f50c0022600 con 0x557ef2c127a0 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.221+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2210052033 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50b4003c80 con 0x557ef2c127a0 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50acff9700 1 -- 172.21.15.121:0/2210052033 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f50c00b4070 con 0x557ef2c127a0 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50d62bbb80 10 client.16314 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2210052033 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557ef2c9a100 con 0x557ef2c127a0 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50acff9700 1 -- 172.21.15.121:0/2210052033 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50b4003c80 con 0x557ef2c127a0 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50acff9700 1 -- 172.21.15.121:0/2210052033 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f50c009cae0 con 0x557ef2c127a0 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50d62bbb80 10 client.16314 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50d62bbb80 10 client.16314 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50d62bbb80 20 client.16314 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x557ef2c9a600 0x557ef2c9ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50d62bbb80 4 client.16314 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:33:51.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2210052033 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557ef2b6b2d0 con 0x557ef2c9a600 2024-06-10T00:33:51.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.222+0000 7f50c8fd3700 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x557ef2c9a600 0x557ef2c9ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:51.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.223+0000 7f50acff9700 10 client.16314 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:33:51.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.223+0000 7f50c8fd3700 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x557ef2c9a600 0x557ef2c9ca20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a9320f4e865052ef server_cookie=b2fd5da3642ac24 in_seq=0 out_seq=0 2024-06-10T00:33:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50acff9700 1 -- 172.21.15.121:0/2210052033 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557ef2b6b2d0 con 0x557ef2c9a600 2024-06-10T00:33:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50acff9700 10 client.16314 handle_command_reply: tid=0 2024-06-10T00:33:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 1 client.16314 shutdown 2024-06-10T00:33:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2210052033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c127a0 msgr2=0x557ef2c151f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c127a0 0x557ef2c151f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f50c009c810 tx=0x7f50c0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2210052033 shutdown_connections 2024-06-10T00:33:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x557ef2c9a600 0x557ef2c9ca20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557ef2c18880 0x557ef2c15c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557ef2c127a0 0x557ef2c151f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 1 --2- 172.21.15.121:0/2210052033 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557ef2c17f60 0x557ef2c15730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2210052033 >> 172.21.15.121:0/2210052033 conn(0x557ef2b3f080 msgr2=0x557ef2c140e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:51.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50acff9700 0 client.16314 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:33:51.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2210052033 shutdown_connections 2024-06-10T00:33:51.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 1 -- 172.21.15.121:0/2210052033 wait complete. 2024-06-10T00:33:51.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.224+0000 7f50d62bbb80 20 client.16314 trim_cache size 0 max 16384 2024-06-10T00:33:51.229 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:33:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 -- 172.21.15.121:0/1988110618 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f50b00a2d90 msgr2=0x7f50b00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f50b00a2d90 0x7f50b00a5250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f50c419a280 tx=0x7f50b4009260 comp rx=0 tx=0).stop 2024-06-10T00:33:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 -- 172.21.15.121:0/1988110618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c406a170 msgr2=0x7f50c419ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c406a170 0x7f50c419ea60 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f50c0005910 tx=0x7f50c000bab0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 -- 172.21.15.121:0/1988110618 shutdown_connections 2024-06-10T00:33:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f50b00a2d90 0x7f50b00a5250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50c4108500 0x7f50c4198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50c406a170 0x7f50c419ea60 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 --2- 172.21.15.121:0/1988110618 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50c406ab30 0x7f50c419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 -- 172.21.15.121:0/1988110618 >> 172.21.15.121:0/1988110618 conn(0x7f50c4078340 msgr2=0x7f50c40fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:51.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 -- 172.21.15.121:0/1988110618 shutdown_connections 2024-06-10T00:33:51.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:51.225+0000 7f50cba38700 1 -- 172.21.15.121:0/1988110618 wait complete. 2024-06-10T00:33:51.257 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:51 smithi135 ceph-mon[36424]: pgmap v179: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 883 op/s 2024-06-10T00:33:51.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:51 smithi193 ceph-mon[36716]: pgmap v179: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 883 op/s 2024-06-10T00:33:52.342 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-10T00:33:52.616 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:33:53.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:53 smithi135 ceph-mon[36424]: pgmap v180: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 655 op/s 2024-06-10T00:33:53.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:53 smithi193 ceph-mon[36716]: pgmap v180: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 655 op/s 2024-06-10T00:33:53.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:53 smithi121 ceph-mon[32378]: pgmap v180: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 655 op/s 2024-06-10T00:33:54.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.422+0000 7f66b57db700 1 -- 172.21.15.121:0/4057120984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66b0101440 msgr2=0x7f66b010f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:54.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.422+0000 7f66b57db700 1 --2- 172.21.15.121:0/4057120984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66b0101440 0x7f66b010f330 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f66a800b0a0 tx=0x7f66a800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.422+0000 7f66b57db700 1 -- 172.21.15.121:0/4057120984 shutdown_connections 2024-06-10T00:33:54.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.422+0000 7f66b57db700 1 --2- 172.21.15.121:0/4057120984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66b0101440 0x7f66b010f330 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.422+0000 7f66b57db700 1 --2- 172.21.15.121:0/4057120984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b0108510 0x7f66b01088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.422+0000 7f66b57db700 1 --2- 172.21.15.121:0/4057120984 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66b0100a80 0x7f66b0100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.422+0000 7f66b57db700 1 -- 172.21.15.121:0/4057120984 >> 172.21.15.121:0/4057120984 conn(0x7f66b0077a00 msgr2=0x7f66b00fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:54.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.422+0000 7f66b57db700 1 -- 172.21.15.121:0/4057120984 shutdown_connections 2024-06-10T00:33:54.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.422+0000 7f66b57db700 1 -- 172.21.15.121:0/4057120984 wait complete. 2024-06-10T00:33:54.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66b57db700 1 Processor -- start 2024-06-10T00:33:54.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66b57db700 1 -- start start 2024-06-10T00:33:54.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66b57db700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66b0100a80 0x7f66b019adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66b57db700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b0101440 0x7f66b019b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66b57db700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66b0108510 0x7f66b019f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66b57db700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f66b0112770 con 0x7f66b0100a80 2024-06-10T00:33:54.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66b57db700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f66b01125f0 con 0x7f66b0108510 2024-06-10T00:33:54.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66b57db700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66b01128f0 con 0x7f66b0101440 2024-06-10T00:33:54.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66aeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66b0100a80 0x7f66b019adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66aeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66b0100a80 0x7f66b019adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52980/0 (socket says 172.21.15.121:52980) 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66aeffd700 1 -- 172.21.15.121:0/3087244225 learned_addr learned my addr 172.21.15.121:0/3087244225 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66af7fe700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66b0108510 0x7f66b019f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.423+0000 7f66ae7fc700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b0101440 0x7f66b019b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f66aeffd700 1 -- 172.21.15.121:0/3087244225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b0101440 msgr2=0x7f66b019b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f66aeffd700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b0101440 0x7f66b019b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f66aeffd700 1 -- 172.21.15.121:0/3087244225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66b0108510 msgr2=0x7f66b019f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f66aeffd700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66b0108510 0x7f66b019f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f66aeffd700 1 -- 172.21.15.121:0/3087244225 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b019fd70 con 0x7f66b0100a80 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f66af7fe700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66b0108510 0x7f66b019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f66aeffd700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66b0100a80 0x7f66b019adb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f66a000d950 tx=0x7f66a000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f6697fff700 1 -- 172.21.15.121:0/3087244225 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a0016070 con 0x7f66b0100a80 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f6697fff700 1 -- 172.21.15.121:0/3087244225 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a000fd70 con 0x7f66b0100a80 2024-06-10T00:33:54.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f66b57db700 1 -- 172.21.15.121:0/3087244225 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b01a0060 con 0x7f66b0100a80 2024-06-10T00:33:54.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f6697fff700 1 -- 172.21.15.121:0/3087244225 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a000e610 con 0x7f66b0100a80 2024-06-10T00:33:54.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.424+0000 7f66b57db700 1 -- 172.21.15.121:0/3087244225 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b01a78e0 con 0x7f66b0100a80 2024-06-10T00:33:54.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.426+0000 7f6697fff700 1 -- 172.21.15.121:0/3087244225 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f66a0019070 con 0x7f66b0100a80 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.427+0000 7f6697fff700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f66980a2d90 0x7f66980a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.427+0000 7f6697fff700 1 -- 172.21.15.121:0/3087244225 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f66a00cf230 con 0x7f66b0100a80 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.427+0000 7f66ae7fc700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f66980a2d90 0x7f66980a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66c005eb80 1 Processor -- start 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66ae7fc700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f66980a2d90 0x7f66980a5250 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f66b0103c80 tx=0x7f66a40073d0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66c005eb80 1 -- start start 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16510340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a1650a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a1650abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66c005eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x555a163fcf60 con 0x555a16510880 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66c005eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x555a1645b850 con 0x555a1650a7a0 2024-06-10T00:33:54.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66c005eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555a163c3460 con 0x555a1650ff60 2024-06-10T00:33:54.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66ae7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a1650a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66aeffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16510340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66aeffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16510340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:38364/0 (socket says 172.21.15.121:38364) 2024-06-10T00:33:54.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66aeffd700 1 -- 172.21.15.121:0/1785847508 learned_addr learned my addr 172.21.15.121:0/1785847508 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:54.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66af7fe700 1 --2- 172.21.15.121:0/1785847508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a1650abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66ae7fc700 1 -- 172.21.15.121:0/1785847508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 msgr2=0x555a16510340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:54.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66ae7fc700 1 --2- 172.21.15.121:0/1785847508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16510340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66ae7fc700 1 -- 172.21.15.121:0/1785847508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 msgr2=0x555a1650abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:54.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.428+0000 7f66ae7fc700 1 --2- 172.21.15.121:0/1785847508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a1650abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.429+0000 7f66ae7fc700 1 -- 172.21.15.121:0/1785847508 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a163aa8d0 con 0x555a16510880 2024-06-10T00:33:54.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.429+0000 7f66ae7fc700 1 --2- 172.21.15.121:0/1785847508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a1650a260 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f66a4010760 tx=0x7f66a4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c936f4fb4942a070 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:54.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.429+0000 7f66957fa700 1 -- 172.21.15.121:0/1785847508 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4024070 con 0x555a16510880 2024-06-10T00:33:54.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.429+0000 7f66957fa700 1 -- 172.21.15.121:0/1785847508 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a4015070 con 0x555a16510880 2024-06-10T00:33:54.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.429+0000 7f66957fa700 1 -- 172.21.15.121:0/1785847508 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a40119a0 con 0x555a16510880 2024-06-10T00:33:54.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.430+0000 7f66c005eb80 1 -- 172.21.15.121:0/1785847508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 msgr2=0x555a1650a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:54.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.430+0000 7f66c005eb80 1 --2- 172.21.15.121:0/1785847508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a1650a260 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f66a4010760 tx=0x7f66a4013000 comp rx=0 tx=0).stop 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.430+0000 7f66c005eb80 1 -- 172.21.15.121:0/1785847508 shutdown_connections 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.430+0000 7f66c005eb80 1 --2- 172.21.15.121:0/1785847508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a1650abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.430+0000 7f66c005eb80 1 --2- 172.21.15.121:0/1785847508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16510340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.430+0000 7f66c005eb80 1 --2- 172.21.15.121:0/1785847508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a1650a260 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.430+0000 7f66c005eb80 1 -- 172.21.15.121:0/1785847508 >> 172.21.15.121:0/1785847508 conn(0x555a16437080 msgr2=0x555a16505220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.430+0000 7f66c005eb80 1 -- 172.21.15.121:0/1785847508 shutdown_connections 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.430+0000 7f66c005eb80 1 -- 172.21.15.121:0/1785847508 wait complete. 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.430+0000 7f66c005eb80 1 -- start start 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66c005eb80 1 Processor -- start 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a16516610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16513910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a16513e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66c005eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x555a163fbfc0 con 0x555a16510880 2024-06-10T00:33:54.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66c005eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x555a16457810 con 0x555a1650a7a0 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66c005eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555a16517260 con 0x555a1650ff60 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66c005eb80 10 client.0 _ll_register_callbacks cb 0x555a163f15b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66aeffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a16516610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66ae7fc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16513910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66af7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a16513e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66ae7fc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16513910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:38368/0 (socket says 172.21.15.121:38368) 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66aeffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a16516610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:39344/0 (socket says 172.21.15.121:39344) 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66ae7fc700 1 -- 172.21.15.121:0/2496077232 learned_addr learned my addr 172.21.15.121:0/2496077232 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66af7fe700 1 -- 172.21.15.121:0/2496077232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 msgr2=0x555a16513910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66af7fe700 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16513910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66af7fe700 1 -- 172.21.15.121:0/2496077232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 msgr2=0x555a16516610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66af7fe700 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a16516610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66af7fe700 1 -- 172.21.15.121:0/2496077232 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a16421750 con 0x555a16510880 2024-06-10T00:33:54.437 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.431+0000 7f66aeffd700 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a16516610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f66af7fe700 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a16513e50 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009fd0 tx=0x7f66a8003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e954a4c8c2cdfcee server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f6692ff5700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f6692ff5700 1 -- 172.21.15.121:0/2496077232 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8007a70 con 0x555a16510880 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f6692ff5700 1 -- 172.21.15.121:0/2496077232 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f66a8007c10 con 0x555a16510880 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f66c005eb80 1 -- 172.21.15.121:0/2496077232 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a4015070 con 0x555a16510880 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f6692ff5700 1 -- 172.21.15.121:0/2496077232 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f66a8007e10 con 0x555a16510880 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f66c005eb80 10 client.16326 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f66c005eb80 1 -- 172.21.15.121:0/2496077232 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555a16521a10 con 0x555a16510880 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f6692ff5700 1 -- 172.21.15.121:0/2496077232 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a4015070 con 0x555a16510880 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f6692ff5700 1 -- 172.21.15.121:0/2496077232 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f66a8024070 con 0x555a16510880 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f66c005eb80 10 client.16326 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f66c005eb80 10 client.16326 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:33:54.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.432+0000 7f66c005eb80 20 client.16326 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.433+0000 7f66c005eb80 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x555a16524040 0x555a16526450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.433+0000 7f66c005eb80 4 client.16326 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.433+0000 7f66c005eb80 1 -- 172.21.15.121:0/2496077232 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a16468d50 con 0x555a16524040 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.433+0000 7f66aeffd700 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x555a16524040 0x555a16526450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.433+0000 7f6692ff5700 10 client.16326 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.433+0000 7f66aeffd700 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x555a16524040 0x555a16526450 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c65ad0af4f0d3b89 server_cookie=d70e31d3ed9d505d in_seq=0 out_seq=0 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f6692ff5700 1 -- 172.21.15.121:0/2496077232 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x555a16468d50 con 0x555a16524040 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f6692ff5700 10 client.16326 handle_command_reply: tid=0 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 1 client.16326 shutdown 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 1 -- 172.21.15.121:0/2496077232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 msgr2=0x555a16513e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a16513e50 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009fd0 tx=0x7f66a8003f90 comp rx=0 tx=0).stop 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 1 -- 172.21.15.121:0/2496077232 shutdown_connections 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x555a16524040 0x555a16526450 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a16513e50 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a16516610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 1 --2- 172.21.15.121:0/2496077232 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16513910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 1 -- 172.21.15.121:0/2496077232 >> 172.21.15.121:0/2496077232 conn(0x555a16437080 msgr2=0x555a16504700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:54.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f6692ff5700 0 client.16326 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:33:54.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 1 -- 172.21.15.121:0/2496077232 shutdown_connections 2024-06-10T00:33:54.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 1 -- 172.21.15.121:0/2496077232 wait complete. 2024-06-10T00:33:54.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.434+0000 7f66c005eb80 20 client.16326 trim_cache size 0 max 16384 2024-06-10T00:33:54.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.454+0000 7f66c005eb80 1 Processor -- start 2024-06-10T00:33:54.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.454+0000 7f66c005eb80 1 -- start start 2024-06-10T00:33:54.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.454+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a1650a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.454+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16511430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.454+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a16504e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.454+0000 7f66c005eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f66b01b3020 con 0x555a16510880 2024-06-10T00:33:54.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.454+0000 7f66c005eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f66b01b0850 con 0x555a1650a7a0 2024-06-10T00:33:54.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.454+0000 7f66c005eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66b01ae6a0 con 0x555a1650ff60 2024-06-10T00:33:54.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.454+0000 7f66af7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a16504e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.455+0000 7f66af7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a16504e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53016/0 (socket says 172.21.15.121:53016) 2024-06-10T00:33:54.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.455+0000 7f66af7fe700 1 -- 172.21.15.121:0/3908011038 learned_addr learned my addr 172.21.15.121:0/3908011038 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:54.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.455+0000 7f66ae7fc700 1 --2- 172.21.15.121:0/3908011038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16511430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.455+0000 7f66aeffd700 1 --2- 172.21.15.121:0/3908011038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a1650a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.455+0000 7f66af7fe700 1 -- 172.21.15.121:0/3908011038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 msgr2=0x555a16511430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:54.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.455+0000 7f66af7fe700 1 --2- 172.21.15.121:0/3908011038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16511430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.455+0000 7f66af7fe700 1 -- 172.21.15.121:0/3908011038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 msgr2=0x555a1650a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:54.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.455+0000 7f66af7fe700 1 --2- 172.21.15.121:0/3908011038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a1650a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.455+0000 7f66af7fe700 1 -- 172.21.15.121:0/3908011038 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a16501810 con 0x555a16510880 2024-06-10T00:33:54.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.455+0000 7f66af7fe700 1 --2- 172.21.15.121:0/3908011038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a16504e80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f66a8007630 tx=0x7f66a800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7324bdd119abf5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:54.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.455+0000 7f6694ff9700 1 -- 172.21.15.121:0/3908011038 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8007870 con 0x555a16510880 2024-06-10T00:33:54.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.456+0000 7f6694ff9700 1 -- 172.21.15.121:0/3908011038 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a8007a10 con 0x555a16510880 2024-06-10T00:33:54.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.456+0000 7f6694ff9700 1 -- 172.21.15.121:0/3908011038 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a80052a0 con 0x555a16510880 2024-06-10T00:33:54.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.456+0000 7f66c005eb80 1 -- 172.21.15.121:0/3908011038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 msgr2=0x555a16504e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:54.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.456+0000 7f66c005eb80 1 --2- 172.21.15.121:0/3908011038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a16504e80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f66a8007630 tx=0x7f66a800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.456+0000 7f66c005eb80 1 -- 172.21.15.121:0/3908011038 shutdown_connections 2024-06-10T00:33:54.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.456+0000 7f66c005eb80 1 --2- 172.21.15.121:0/3908011038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a16510880 0x555a16504e80 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.456+0000 7f66c005eb80 1 --2- 172.21.15.121:0/3908011038 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a1650a7a0 0x555a1650a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.456+0000 7f66c005eb80 1 --2- 172.21.15.121:0/3908011038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650ff60 0x555a16511430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.456+0000 7f66c005eb80 1 -- 172.21.15.121:0/3908011038 >> 172.21.15.121:0/3908011038 conn(0x555a16437080 msgr2=0x555a16510d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:54.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.456+0000 7f66c005eb80 1 -- 172.21.15.121:0/3908011038 shutdown_connections 2024-06-10T00:33:54.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.456+0000 7f66c005eb80 1 -- 172.21.15.121:0/3908011038 wait complete. 2024-06-10T00:33:54.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66c005eb80 1 -- start start 2024-06-10T00:33:54.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66c005eb80 1 Processor -- start 2024-06-10T00:33:54.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650a7a0 0x555a1652c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a1650ff60 0x555a1651c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66c005eb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a16510880 0x555a1651cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66c005eb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f66b01aed10 con 0x555a1650ff60 2024-06-10T00:33:54.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66c005eb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f66b01b0180 con 0x555a16510880 2024-06-10T00:33:54.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66c005eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66b01af3e0 con 0x555a1650a7a0 2024-06-10T00:33:54.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66c005eb80 10 client.0 _ll_register_callbacks cb 0x555a163f15b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:54.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66ae7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a1650ff60 0x555a1651c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66ae7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a1650ff60 0x555a1651c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53032/0 (socket says 172.21.15.121:53032) 2024-06-10T00:33:54.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66ae7fc700 1 -- 172.21.15.121:0/425368624 learned_addr learned my addr 172.21.15.121:0/425368624 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:54.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66af7fe700 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a16510880 0x555a1651cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.457+0000 7f66aeffd700 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650a7a0 0x555a1652c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f66af7fe700 1 -- 172.21.15.121:0/425368624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650a7a0 msgr2=0x555a1652c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:54.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f66af7fe700 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650a7a0 0x555a1652c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f66af7fe700 1 -- 172.21.15.121:0/425368624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a1650ff60 msgr2=0x555a1651c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:54.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f66af7fe700 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a1650ff60 0x555a1651c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f66af7fe700 1 -- 172.21.15.121:0/425368624 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a1652ad10 con 0x555a16510880 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f66ae7fc700 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a1650ff60 0x555a1651c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f66af7fe700 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a16510880 0x555a1651cb50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009fd0 tx=0x7f66a8003dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4705bba476af7f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f6692ff5700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f6692ff5700 1 -- 172.21.15.121:0/425368624 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8007870 con 0x555a16510880 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f66c005eb80 1 -- 172.21.15.121:0/425368624 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a8007a10 con 0x555a16510880 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f6692ff5700 1 -- 172.21.15.121:0/425368624 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f66a8005e70 con 0x555a16510880 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f6692ff5700 1 -- 172.21.15.121:0/425368624 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f66a8006070 con 0x555a16510880 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f66c005eb80 10 client.25904 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.458+0000 7f66c005eb80 1 -- 172.21.15.121:0/425368624 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555a1658df80 con 0x555a16510880 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.459+0000 7f6692ff5700 1 -- 172.21.15.121:0/425368624 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66a8007a10 con 0x555a16510880 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.459+0000 7f6692ff5700 1 -- 172.21.15.121:0/425368624 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f66a8024040 con 0x555a16510880 2024-06-10T00:33:54.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.459+0000 7f66c005eb80 10 client.25904 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:54.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.459+0000 7f66c005eb80 10 client.25904 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:33:54.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.459+0000 7f66c005eb80 20 client.25904 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:54.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.459+0000 7f66c005eb80 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x555a1658e5a0 0x555a165909c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:54.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.459+0000 7f66c005eb80 4 client.25904 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:33:54.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.459+0000 7f66c005eb80 1 -- 172.21.15.121:0/425368624 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555a164632d0 con 0x555a1658e5a0 2024-06-10T00:33:54.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.459+0000 7f66aeffd700 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x555a1658e5a0 0x555a165909c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:54.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.460+0000 7f6692ff5700 10 client.25904 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:33:54.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.460+0000 7f66aeffd700 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x555a1658e5a0 0x555a165909c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=efaf88f41bd63a86 server_cookie=39244b0696e2c14a in_seq=0 out_seq=0 2024-06-10T00:33:54.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.460+0000 7f6692ff5700 1 -- 172.21.15.121:0/425368624 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555a164632d0 con 0x555a1658e5a0 2024-06-10T00:33:54.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f6692ff5700 10 client.25904 handle_command_reply: tid=0 2024-06-10T00:33:54.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 1 client.25904 shutdown 2024-06-10T00:33:54.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 1 -- 172.21.15.121:0/425368624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a16510880 msgr2=0x555a1651cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:54.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a16510880 0x555a1651cb50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009fd0 tx=0x7f66a8003dc0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 1 -- 172.21.15.121:0/425368624 shutdown_connections 2024-06-10T00:33:54.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x555a1658e5a0 0x555a165909c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x555a16510880 0x555a1651cb50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555a1650a7a0 0x555a1652c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 1 --2- 172.21.15.121:0/425368624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a1650ff60 0x555a1651c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f6692ff5700 0 client.25904 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:33:54.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 1 -- 172.21.15.121:0/425368624 >> 172.21.15.121:0/425368624 conn(0x555a16437080 msgr2=0x555a1650ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:54.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 1 -- 172.21.15.121:0/425368624 shutdown_connections 2024-06-10T00:33:54.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 1 -- 172.21.15.121:0/425368624 wait complete. 2024-06-10T00:33:54.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.461+0000 7f66c005eb80 20 client.25904 trim_cache size 0 max 16384 2024-06-10T00:33:54.470 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:33:54.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 -- 172.21.15.121:0/3087244225 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f66980a2d90 msgr2=0x7f66980a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:54.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f66980a2d90 0x7f66980a5250 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f66b0103c80 tx=0x7f66a40073d0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 -- 172.21.15.121:0/3087244225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66b0100a80 msgr2=0x7f66b019adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:54.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66b0100a80 0x7f66b019adb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f66a000d950 tx=0x7f66a000dc60 comp rx=0 tx=0).stop 2024-06-10T00:33:54.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 -- 172.21.15.121:0/3087244225 shutdown_connections 2024-06-10T00:33:54.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f66980a2d90 0x7f66980a5250 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f66b0108510 0x7f66b019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66b0100a80 0x7f66b019adb0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 --2- 172.21.15.121:0/3087244225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b0101440 0x7f66b019b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:54.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 -- 172.21.15.121:0/3087244225 >> 172.21.15.121:0/3087244225 conn(0x7f66b0077a00 msgr2=0x7f66b00feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:54.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 -- 172.21.15.121:0/3087244225 shutdown_connections 2024-06-10T00:33:54.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:54.462+0000 7f66b57db700 1 -- 172.21.15.121:0/3087244225 wait complete. 2024-06-10T00:33:55.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:55 smithi135 ceph-mon[36424]: pgmap v181: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 620 op/s 2024-06-10T00:33:55.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:55 smithi193 ceph-mon[36716]: pgmap v181: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 620 op/s 2024-06-10T00:33:55.433 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:55 smithi121 ceph-mon[32378]: pgmap v181: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 620 op/s 2024-06-10T00:33:55.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-10T00:33:55.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:33:55.732 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:33:56.509 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:33:57.164 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:57 smithi121 ceph-mon[32378]: pgmap v182: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 2.7 MiB/s wr, 749 op/s 2024-06-10T00:33:57.295 INFO:tasks.workunit.client.0.smithi121.stdout: 246 48058 0 32456 0 16104 0 2024-06-10T00:33:57.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:57 smithi193 ceph-mon[36716]: pgmap v182: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 2.7 MiB/s wr, 749 op/s 2024-06-10T00:33:57.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:57 smithi135 ceph-mon[36424]: pgmap v182: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 2.7 MiB/s wr, 749 op/s 2024-06-10T00:33:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.241+0000 7f035124c700 1 -- 172.21.15.121:0/2182670093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f034c101390 msgr2=0x7f034c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.241+0000 7f035124c700 1 --2- 172.21.15.121:0/2182670093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f034c101390 0x7f034c101810 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f0340009990 tx=0x7f0340009ca0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.241+0000 7f035124c700 1 -- 172.21.15.121:0/2182670093 shutdown_connections 2024-06-10T00:33:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.241+0000 7f035124c700 1 --2- 172.21.15.121:0/2182670093 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f034c101d50 0x7f034c10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.241+0000 7f035124c700 1 --2- 172.21.15.121:0/2182670093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c107400 0x7f034c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.241+0000 7f035124c700 1 --2- 172.21.15.121:0/2182670093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f034c101390 0x7f034c101810 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.241+0000 7f035124c700 1 -- 172.21.15.121:0/2182670093 >> 172.21.15.121:0/2182670093 conn(0x7f034c078340 msgr2=0x7f034c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.241+0000 7f035124c700 1 -- 172.21.15.121:0/2182670093 shutdown_connections 2024-06-10T00:33:58.243 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.241+0000 7f035124c700 1 -- 172.21.15.121:0/2182670093 wait complete. 2024-06-10T00:33:58.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.242+0000 7f035124c700 1 Processor -- start 2024-06-10T00:33:58.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.242+0000 7f035124c700 1 -- start start 2024-06-10T00:33:58.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f035124c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c101390 0x7f034c103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f035124c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f034c101d50 0x7f034c1035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f035124c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f034c107400 0x7f034c103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f035124c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f034c111670 con 0x7f034c101d50 2024-06-10T00:33:58.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f035124c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f034c1114f0 con 0x7f034c107400 2024-06-10T00:33:58.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f034ad9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c101390 0x7f034c103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.245 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f034b59e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f034c107400 0x7f034c103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f034ad9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c101390 0x7f034c103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:38398/0 (socket says 172.21.15.121:38398) 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f034ad9d700 1 -- 172.21.15.121:0/2862802314 learned_addr learned my addr 172.21.15.121:0/2862802314 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f034a59c700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f034c101d50 0x7f034c1035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f035124c700 1 -- 172.21.15.121:0/2862802314 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f034c1117f0 con 0x7f034c101390 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f034ad9d700 1 -- 172.21.15.121:0/2862802314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f034c107400 msgr2=0x7f034c103af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f034ad9d700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f034c107400 0x7f034c103af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f034ad9d700 1 -- 172.21.15.121:0/2862802314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f034c101d50 msgr2=0x7f034c1035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f034ad9d700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f034c101d50 0x7f034c1035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.243+0000 7f034ad9d700 1 -- 172.21.15.121:0/2862802314 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c1a70f0 con 0x7f034c101390 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.244+0000 7f034ad9d700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c101390 0x7f034c103070 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f033c00d980 tx=0x7f033c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.244+0000 7f0333fff700 1 -- 172.21.15.121:0/2862802314 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c016070 con 0x7f034c101390 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.244+0000 7f0333fff700 1 -- 172.21.15.121:0/2862802314 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f033c004d10 con 0x7f034c101390 2024-06-10T00:33:58.246 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.244+0000 7f0333fff700 1 -- 172.21.15.121:0/2862802314 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c0056c0 con 0x7f034c101390 2024-06-10T00:33:58.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.244+0000 7f035124c700 1 -- 172.21.15.121:0/2862802314 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034c1a73e0 con 0x7f034c101390 2024-06-10T00:33:58.247 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.244+0000 7f035124c700 1 -- 172.21.15.121:0/2862802314 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034c1a7940 con 0x7f034c101390 2024-06-10T00:33:58.248 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.247+0000 7f0333fff700 1 -- 172.21.15.121:0/2862802314 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f033c004750 con 0x7f034c101390 2024-06-10T00:33:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.248+0000 7f0333fff700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f03340a2e10 0x7f03340a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.248+0000 7f035bacfb80 1 Processor -- start 2024-06-10T00:33:58.249 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.248+0000 7f035bacfb80 1 -- start start 2024-06-10T00:33:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.248+0000 7f0333fff700 1 -- 172.21.15.121:0/2862802314 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f033c0cf460 con 0x7f034c101390 2024-06-10T00:33:58.251 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.248+0000 7f034a59c700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f03340a2e10 0x7f03340a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.248+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.248+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f6880 0x55d8349f0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.248+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.248+0000 7f035bacfb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d8348e2f60 con 0x55d8349f6880 2024-06-10T00:33:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.248+0000 7f035bacfb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d834941850 con 0x55d8349f07a0 2024-06-10T00:33:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.248+0000 7f035bacfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d8348a9460 con 0x55d8349f5f60 2024-06-10T00:33:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034b59e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034ad9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034b59e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349f0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:39386/0 (socket says 172.21.15.121:39386) 2024-06-10T00:33:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034b59e700 1 -- 172.21.15.121:0/2361434536 learned_addr learned my addr 172.21.15.121:0/2361434536 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:58.252 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034a59c700 1 --2- 172.21.15.121:0/2361434536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f6880 0x55d8349f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034b59e700 1 -- 172.21.15.121:0/2361434536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 msgr2=0x55d8349f6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034a59c700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f03340a2e10 0x7f03340a52d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f034000b2e0 tx=0x7f034000b610 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034b59e700 1 --2- 172.21.15.121:0/2361434536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034b59e700 1 -- 172.21.15.121:0/2361434536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f6880 msgr2=0x55d8349f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034b59e700 1 --2- 172.21.15.121:0/2361434536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f6880 0x55d8349f0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034b59e700 1 -- 172.21.15.121:0/2361434536 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8348908d0 con 0x55d8349f07a0 2024-06-10T00:33:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f034b59e700 1 --2- 172.21.15.121:0/2361434536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349f0bc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0344011680 tx=0x7f0344009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42588efdfec69a13 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f03317fa700 1 -- 172.21.15.121:0/2361434536 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344019030 con 0x55d8349f07a0 2024-06-10T00:33:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f03317fa700 1 -- 172.21.15.121:0/2361434536 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03440092e0 con 0x55d8349f07a0 2024-06-10T00:33:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.249+0000 7f03317fa700 1 -- 172.21.15.121:0/2361434536 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344007500 con 0x55d8349f07a0 2024-06-10T00:33:58.253 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.250+0000 7f035bacfb80 1 -- 172.21.15.121:0/2361434536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 msgr2=0x55d8349f0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.250+0000 7f035bacfb80 1 --2- 172.21.15.121:0/2361434536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349f0bc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0344011680 tx=0x7f0344009920 comp rx=0 tx=0).stop 2024-06-10T00:33:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.250+0000 7f035bacfb80 1 -- 172.21.15.121:0/2361434536 shutdown_connections 2024-06-10T00:33:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.250+0000 7f035bacfb80 1 --2- 172.21.15.121:0/2361434536 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349f0bc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.250+0000 7f035bacfb80 1 --2- 172.21.15.121:0/2361434536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.250+0000 7f035bacfb80 1 --2- 172.21.15.121:0/2361434536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f6880 0x55d8349f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.250+0000 7f035bacfb80 1 -- 172.21.15.121:0/2361434536 >> 172.21.15.121:0/2361434536 conn(0x55d83491d080 msgr2=0x55d8349eb220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.250+0000 7f035bacfb80 1 -- 172.21.15.121:0/2361434536 shutdown_connections 2024-06-10T00:33:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.250+0000 7f035bacfb80 1 -- 172.21.15.121:0/2361434536 wait complete. 2024-06-10T00:33:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f035bacfb80 1 -- start start 2024-06-10T00:33:58.254 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f035bacfb80 1 Processor -- start 2024-06-10T00:33:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349fc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f6880 0x55d8349f9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f035bacfb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d8348e1fc0 con 0x55d8349f6880 2024-06-10T00:33:58.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f035bacfb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d83493d810 con 0x55d8349f07a0 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f035bacfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d8349fd260 con 0x55d8349f5f60 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f035bacfb80 10 client.0 _ll_register_callbacks cb 0x55d8348d75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f034b59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f6880 0x55d8349f9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f034ad9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349fc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f034ad9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349fc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:39398/0 (socket says 172.21.15.121:39398) 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f034a59c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f034ad9d700 1 -- 172.21.15.121:0/178024477 learned_addr learned my addr 172.21.15.121:0/178024477 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f034b59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f6880 0x55d8349f9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53056/0 (socket says 172.21.15.121:53056) 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f034ad9d700 1 -- 172.21.15.121:0/178024477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 msgr2=0x55d8349f9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f034ad9d700 1 --2- 172.21.15.121:0/178024477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f034ad9d700 1 -- 172.21.15.121:0/178024477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f6880 msgr2=0x55d8349f9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f034ad9d700 1 --2- 172.21.15.121:0/178024477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f6880 0x55d8349f9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.251+0000 7f034ad9d700 1 -- 172.21.15.121:0/178024477 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d834907750 con 0x55d8349f07a0 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f034ad9d700 1 --2- 172.21.15.121:0/178024477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349fc610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f033c097a50 tx=0x7f033c09af50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d586d97bba4d5b1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f032eff5700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f032eff5700 1 -- 172.21.15.121:0/178024477 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c0b4060 con 0x55d8349f07a0 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f035bacfb80 1 -- 172.21.15.121:0/178024477 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03440092e0 con 0x55d8349f07a0 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f032eff5700 1 -- 172.21.15.121:0/178024477 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f033c004a40 con 0x55d8349f07a0 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f032eff5700 1 -- 172.21.15.121:0/178024477 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f033c09bcf0 con 0x55d8349f07a0 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f035bacfb80 10 client.25916 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f035bacfb80 1 -- 172.21.15.121:0/178024477 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d834a07980 con 0x55d8349f07a0 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f032eff5700 1 -- 172.21.15.121:0/178024477 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03440092e0 con 0x55d8349f07a0 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f032eff5700 1 -- 172.21.15.121:0/178024477 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f033c0b6ce0 con 0x55d8349f07a0 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f035bacfb80 10 client.25916 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f035bacfb80 10 client.25916 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:33:58.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f035bacfb80 20 client.25916 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.252+0000 7f035bacfb80 1 --2- 172.21.15.121:0/178024477 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d834a0a3a0 0x55d834a0c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.253+0000 7f035bacfb80 4 client.25916 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.253+0000 7f035bacfb80 1 -- 172.21.15.121:0/178024477 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d83494ed50 con 0x55d834a0a3a0 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.253+0000 7f034a59c700 1 --2- 172.21.15.121:0/178024477 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d834a0a3a0 0x55d834a0c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.253+0000 7f034a59c700 1 --2- 172.21.15.121:0/178024477 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d834a0a3a0 0x55d834a0c800 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0a59f47d065abcb server_cookie=ac7016f7342c570 in_seq=0 out_seq=0 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.253+0000 7f032eff5700 10 client.25916 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f032eff5700 1 -- 172.21.15.121:0/178024477 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d83494ed50 con 0x55d834a0a3a0 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f032eff5700 10 client.25916 handle_command_reply: tid=0 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 1 client.25916 shutdown 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 1 -- 172.21.15.121:0/178024477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 msgr2=0x55d8349fc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 1 --2- 172.21.15.121:0/178024477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349fc610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f033c097a50 tx=0x7f033c09af50 comp rx=0 tx=0).stop 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 1 -- 172.21.15.121:0/178024477 shutdown_connections 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 1 --2- 172.21.15.121:0/178024477 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d834a0a3a0 0x55d834a0c800 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 1 --2- 172.21.15.121:0/178024477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f6880 0x55d8349f9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 1 --2- 172.21.15.121:0/178024477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d8349fc610 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f032eff5700 0 client.25916 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 1 --2- 172.21.15.121:0/178024477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 1 -- 172.21.15.121:0/178024477 >> 172.21.15.121:0/178024477 conn(0x55d83491d080 msgr2=0x55d8349ea700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 1 -- 172.21.15.121:0/178024477 shutdown_connections 2024-06-10T00:33:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 1 -- 172.21.15.121:0/178024477 wait complete. 2024-06-10T00:33:58.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.254+0000 7f035bacfb80 20 client.25916 trim_cache size 0 max 16384 2024-06-10T00:33:58.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.272+0000 7f035bacfb80 1 Processor -- start 2024-06-10T00:33:58.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.272+0000 7f035bacfb80 1 -- start start 2024-06-10T00:33:58.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.272+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f07a0 0x55d8349f0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.272+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.272+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f6880 0x55d8349eae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.272+0000 7f035bacfb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f034c1aed40 con 0x55d8349f07a0 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.272+0000 7f035bacfb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f034c1ac570 con 0x55d8349f6880 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.272+0000 7f035bacfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f034c1aa3c0 con 0x55d8349f5f60 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f07a0 0x55d8349f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034a59c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034ad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f07a0 0x55d8349f0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53058/0 (socket says 172.21.15.121:53058) 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034a59c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:38434/0 (socket says 172.21.15.121:38434) 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034b59e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f6880 0x55d8349eae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034ad9d700 1 -- 172.21.15.121:0/3224960773 learned_addr learned my addr 172.21.15.121:0/3224960773 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034ad9d700 1 -- 172.21.15.121:0/3224960773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 msgr2=0x55d8349f7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:58.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034ad9d700 1 --2- 172.21.15.121:0/3224960773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034ad9d700 1 -- 172.21.15.121:0/3224960773 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f6880 msgr2=0x55d8349eae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:58.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034ad9d700 1 --2- 172.21.15.121:0/3224960773 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f6880 0x55d8349eae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034ad9d700 1 -- 172.21.15.121:0/3224960773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8349fd260 con 0x55d8349f07a0 2024-06-10T00:33:58.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034b59e700 1 --2- 172.21.15.121:0/3224960773 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f6880 0x55d8349eae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:58.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f034ad9d700 1 --2- 172.21.15.121:0/3224960773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f07a0 0x55d8349f0260 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f033c097fc0 tx=0x7f033c09b1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06966648de05c68 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:58.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f0330ff9700 1 -- 172.21.15.121:0/3224960773 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c0b4050 con 0x55d8349f07a0 2024-06-10T00:33:58.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f0330ff9700 1 -- 172.21.15.121:0/3224960773 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f033c09f9a0 con 0x55d8349f07a0 2024-06-10T00:33:58.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.273+0000 7f0330ff9700 1 -- 172.21.15.121:0/3224960773 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c09b5f0 con 0x55d8349f07a0 2024-06-10T00:33:58.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.274+0000 7f035bacfb80 1 -- 172.21.15.121:0/3224960773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f07a0 msgr2=0x55d8349f0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:58.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.274+0000 7f035bacfb80 1 --2- 172.21.15.121:0/3224960773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f07a0 0x55d8349f0260 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f033c097fc0 tx=0x7f033c09b1e0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.274+0000 7f035bacfb80 1 -- 172.21.15.121:0/3224960773 shutdown_connections 2024-06-10T00:33:58.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.274+0000 7f035bacfb80 1 --2- 172.21.15.121:0/3224960773 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f6880 0x55d8349eae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.274+0000 7f035bacfb80 1 --2- 172.21.15.121:0/3224960773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f07a0 0x55d8349f0260 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.274+0000 7f035bacfb80 1 --2- 172.21.15.121:0/3224960773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f5f60 0x55d8349f7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.274+0000 7f035bacfb80 1 -- 172.21.15.121:0/3224960773 >> 172.21.15.121:0/3224960773 conn(0x55d83491d080 msgr2=0x55d8349f6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:58.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.274+0000 7f035bacfb80 1 -- 172.21.15.121:0/3224960773 shutdown_connections 2024-06-10T00:33:58.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.274+0000 7f035bacfb80 1 -- 172.21.15.121:0/3224960773 wait complete. 2024-06-10T00:33:58.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f035bacfb80 1 -- start start 2024-06-10T00:33:58.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f035bacfb80 1 Processor -- start 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d834a12750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f5f60 0x55d834a02520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f035bacfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f6880 0x55d834a02a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f035bacfb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f034c1aaa30 con 0x55d8349f5f60 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f035bacfb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f034c1abea0 con 0x55d8349f07a0 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f035bacfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f034c1ab100 con 0x55d8349f6880 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f035bacfb80 10 client.0 _ll_register_callbacks cb 0x55d8348d75b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f034ad9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d834a12750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f034a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f5f60 0x55d834a02520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f034a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f5f60 0x55d834a02520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53060/0 (socket says 172.21.15.121:53060) 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f034a59c700 1 -- 172.21.15.121:0/2285895652 learned_addr learned my addr 172.21.15.121:0/2285895652 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f034b59e700 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f6880 0x55d834a02a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f034ad9d700 1 -- 172.21.15.121:0/2285895652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f6880 msgr2=0x55d834a02a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f034ad9d700 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f6880 0x55d834a02a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f034ad9d700 1 -- 172.21.15.121:0/2285895652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f5f60 msgr2=0x55d834a02520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.275+0000 7f034ad9d700 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f5f60 0x55d834a02520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.276+0000 7f034ad9d700 1 -- 172.21.15.121:0/2285895652 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d834a031b0 con 0x55d8349f07a0 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.276+0000 7f034a59c700 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f5f60 0x55d834a02520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.276+0000 7f034ad9d700 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d834a12750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f033c007e80 tx=0x7f033c005120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e2835a0eb129cc7 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.276+0000 7f032eff5700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.276+0000 7f032eff5700 1 -- 172.21.15.121:0/2285895652 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c0b4040 con 0x55d8349f07a0 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.276+0000 7f032eff5700 1 -- 172.21.15.121:0/2285895652 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f033c084b30 con 0x55d8349f07a0 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.276+0000 7f035bacfb80 1 -- 172.21.15.121:0/2285895652 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f033c09f9a0 con 0x55d8349f07a0 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.276+0000 7f032eff5700 1 -- 172.21.15.121:0/2285895652 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f033c09af50 con 0x55d8349f07a0 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.276+0000 7f035bacfb80 10 client.25922 fetch_fsmap learned FSMap version 30 2024-06-10T00:33:58.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.276+0000 7f035bacfb80 1 -- 172.21.15.121:0/2285895652 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d834a741f0 con 0x55d8349f07a0 2024-06-10T00:33:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.276+0000 7f032eff5700 1 -- 172.21.15.121:0/2285895652 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f033c09f9a0 con 0x55d8349f07a0 2024-06-10T00:33:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.277+0000 7f032eff5700 1 -- 172.21.15.121:0/2285895652 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f033c09ab60 con 0x55d8349f07a0 2024-06-10T00:33:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.277+0000 7f035bacfb80 10 client.25922 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:33:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.277+0000 7f035bacfb80 10 client.25922 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:33:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.277+0000 7f035bacfb80 20 client.25922 populate_metadata read hostname 'smithi121' 2024-06-10T00:33:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.277+0000 7f035bacfb80 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d834a74660 0x55d834a76a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:33:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.277+0000 7f035bacfb80 4 client.25922 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:33:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.277+0000 7f035bacfb80 1 -- 172.21.15.121:0/2285895652 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d8349492d0 con 0x55d834a74660 2024-06-10T00:33:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.277+0000 7f034a59c700 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d834a74660 0x55d834a76a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:33:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.277+0000 7f032eff5700 10 client.25922 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:58.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.277+0000 7f034a59c700 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d834a74660 0x55d834a76a80 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db5f7c53463c1c26 server_cookie=e6dc300e68410ea2 in_seq=0 out_seq=0 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f032eff5700 1 -- 172.21.15.121:0/2285895652 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d8349492d0 con 0x55d834a74660 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f032eff5700 10 client.25922 handle_command_reply: tid=0 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 1 client.25922 shutdown 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 1 -- 172.21.15.121:0/2285895652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 msgr2=0x55d834a12750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d834a12750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f033c007e80 tx=0x7f033c005120 comp rx=0 tx=0).stop 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 1 -- 172.21.15.121:0/2285895652 shutdown_connections 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d834a74660 0x55d834a76a80 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8349f6880 0x55d834a02a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d8349f07a0 0x55d834a12750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f032eff5700 0 client.25922 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 1 --2- 172.21.15.121:0/2285895652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8349f5f60 0x55d834a02520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 1 -- 172.21.15.121:0/2285895652 >> 172.21.15.121:0/2285895652 conn(0x55d83491d080 msgr2=0x55d8349f4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 1 -- 172.21.15.121:0/2285895652 shutdown_connections 2024-06-10T00:33:58.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 1 -- 172.21.15.121:0/2285895652 wait complete. 2024-06-10T00:33:58.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.278+0000 7f035bacfb80 20 client.25922 trim_cache size 0 max 16384 2024-06-10T00:33:58.285 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:33:58.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 -- 172.21.15.121:0/2862802314 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f03340a2e10 msgr2=0x7f03340a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:58.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f03340a2e10 0x7f03340a52d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f034000b2e0 tx=0x7f034000b610 comp rx=0 tx=0).stop 2024-06-10T00:33:58.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 -- 172.21.15.121:0/2862802314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c101390 msgr2=0x7f034c103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:33:58.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c101390 0x7f034c103070 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f033c00d980 tx=0x7f033c00dc90 comp rx=0 tx=0).stop 2024-06-10T00:33:58.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 -- 172.21.15.121:0/2862802314 shutdown_connections 2024-06-10T00:33:58.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f03340a2e10 0x7f03340a52d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f034c107400 0x7f034c103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f034c101390 0x7f034c103070 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 --2- 172.21.15.121:0/2862802314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f034c101d50 0x7f034c1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:33:58.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 -- 172.21.15.121:0/2862802314 >> 172.21.15.121:0/2862802314 conn(0x7f034c078340 msgr2=0x7f034c1005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:33:58.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 -- 172.21.15.121:0/2862802314 shutdown_connections 2024-06-10T00:33:58.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:33:58.279+0000 7f035124c700 1 -- 172.21.15.121:0/2862802314 wait complete. 2024-06-10T00:33:59.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:33:59 smithi121 ceph-mon[32378]: pgmap v183: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 964 KiB/s wr, 285 op/s 2024-06-10T00:33:59.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:33:59 smithi193 ceph-mon[36716]: pgmap v183: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 964 KiB/s wr, 285 op/s 2024-06-10T00:33:59.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:33:59 smithi135 ceph-mon[36424]: pgmap v183: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 964 KiB/s wr, 285 op/s 2024-06-10T00:34:00.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.196+0000 7f1769af9700 1 -- 172.21.15.121:0/1068674680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f176406a170 msgr2=0x7f176406a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:00.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.196+0000 7f1769af9700 1 --2- 172.21.15.121:0/1068674680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f176406a170 0x7f176406a5f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f1758009990 tx=0x7f1758009ca0 comp rx=0 tx=0).stop 2024-06-10T00:34:00.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.196+0000 7f1769af9700 1 -- 172.21.15.121:0/1068674680 shutdown_connections 2024-06-10T00:34:00.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.196+0000 7f1769af9700 1 --2- 172.21.15.121:0/1068674680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f176406ab30 0x7f176410f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:00.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.196+0000 7f1769af9700 1 --2- 172.21.15.121:0/1068674680 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1764108500 0x7f17641088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:00.198 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.196+0000 7f1769af9700 1 --2- 172.21.15.121:0/1068674680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f176406a170 0x7f176406a5f0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:00.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.196+0000 7f1769af9700 1 -- 172.21.15.121:0/1068674680 >> 172.21.15.121:0/1068674680 conn(0x7f1764078340 msgr2=0x7f1764078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:00.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.197+0000 7f1769af9700 1 -- 172.21.15.121:0/1068674680 shutdown_connections 2024-06-10T00:34:00.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.197+0000 7f1769af9700 1 -- 172.21.15.121:0/1068674680 wait complete. 2024-06-10T00:34:00.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.197+0000 7f1769af9700 1 Processor -- start 2024-06-10T00:34:00.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f1769af9700 1 -- start start 2024-06-10T00:34:00.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f1769af9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f176406a170 0x7f176419e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:00.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f1769af9700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f176406ab30 0x7f176419ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:00.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f1769af9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1764108500 0x7f1764198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f1769af9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1764112760 con 0x7f176406a170 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f1769af9700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f17641125e0 con 0x7f1764108500 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f1769af9700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f17641128e0 con 0x7f176406ab30 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f17637fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f176406a170 0x7f176419e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f1763fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1764108500 0x7f1764198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f17637fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f176406a170 0x7f176419e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50170/0 (socket says 172.21.15.121:50170) 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f17637fe700 1 -- 172.21.15.121:0/2940854516 learned_addr learned my addr 172.21.15.121:0/2940854516 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f1762ffd700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f176406ab30 0x7f176419ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f17637fe700 1 -- 172.21.15.121:0/2940854516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f176406ab30 msgr2=0x7f176419ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f17637fe700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f176406ab30 0x7f176419ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f17637fe700 1 -- 172.21.15.121:0/2940854516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1764108500 msgr2=0x7f1764198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f17637fe700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1764108500 0x7f1764198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f17637fe700 1 -- 172.21.15.121:0/2940854516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17641991c0 con 0x7f176406a170 2024-06-10T00:34:00.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.198+0000 7f17637fe700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f176406a170 0x7f176419e8a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f175400e9a0 tx=0x7f175400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:00.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.199+0000 7f1760ff9700 1 -- 172.21.15.121:0/2940854516 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175400ca10 con 0x7f176406a170 2024-06-10T00:34:00.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.199+0000 7f1769af9700 1 -- 172.21.15.121:0/2940854516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17641994b0 con 0x7f176406a170 2024-06-10T00:34:00.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.199+0000 7f1769af9700 1 -- 172.21.15.121:0/2940854516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17641a5770 con 0x7f176406a170 2024-06-10T00:34:00.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.199+0000 7f1760ff9700 1 -- 172.21.15.121:0/2940854516 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f175400cbb0 con 0x7f176406a170 2024-06-10T00:34:00.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.199+0000 7f1760ff9700 1 -- 172.21.15.121:0/2940854516 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1754018680 con 0x7f176406a170 2024-06-10T00:34:00.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.200+0000 7f1769af9700 1 -- 172.21.15.121:0/2940854516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1764069180 con 0x7f176406a170 2024-06-10T00:34:00.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.201+0000 7f1760ff9700 1 -- 172.21.15.121:0/2940854516 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1754010430 con 0x7f176406a170 2024-06-10T00:34:00.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.202+0000 7f1760ff9700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f174c0a2e10 0x7f174c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:00.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.202+0000 7f1762ffd700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f174c0a2e10 0x7f174c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:00.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.202+0000 7f1760ff9700 1 -- 172.21.15.121:0/2940854516 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f17540cfd70 con 0x7f176406a170 2024-06-10T00:34:00.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.202+0000 7f1762ffd700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f174c0a2e10 0x7f174c0a52d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f175800b2e0 tx=0x7f1758005960 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:00.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.205+0000 7f1760ff9700 1 -- 172.21.15.121:0/2940854516 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f175409c600 con 0x7f176406a170 2024-06-10T00:34:00.685 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.682+0000 7f1769af9700 1 -- 172.21.15.121:0/2940854516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1764199f90 con 0x7f176406a170 2024-06-10T00:34:00.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.684+0000 7f1760ff9700 1 -- 172.21.15.121:0/2940854516 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11891 (secure 0 0 0) 0x7f175409bf00 con 0x7f176406a170 2024-06-10T00:34:00.687 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:34:00.687 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:33:03.460049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:34:00.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.697+0000 7f1769af9700 1 -- 172.21.15.121:0/2940854516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f174c0a2e10 msgr2=0x7f174c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:00.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.697+0000 7f1769af9700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f174c0a2e10 0x7f174c0a52d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f175800b2e0 tx=0x7f1758005960 comp rx=0 tx=0).stop 2024-06-10T00:34:00.699 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.697+0000 7f1769af9700 1 -- 172.21.15.121:0/2940854516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f176406a170 msgr2=0x7f176419e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:00.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.697+0000 7f1769af9700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f176406a170 0x7f176419e8a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f175400e9a0 tx=0x7f175400ecb0 comp rx=0 tx=0).stop 2024-06-10T00:34:00.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.697+0000 7f1769af9700 1 -- 172.21.15.121:0/2940854516 shutdown_connections 2024-06-10T00:34:00.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.697+0000 7f1769af9700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f174c0a2e10 0x7f174c0a52d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:00.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.697+0000 7f1769af9700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1764108500 0x7f1764198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:00.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.697+0000 7f1769af9700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f176406a170 0x7f176419e8a0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:00.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.697+0000 7f1769af9700 1 --2- 172.21.15.121:0/2940854516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f176406ab30 0x7f176419ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:00.700 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.697+0000 7f1769af9700 1 -- 172.21.15.121:0/2940854516 >> 172.21.15.121:0/2940854516 conn(0x7f1764078340 msgr2=0x7f1764100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:00.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.701+0000 7f1769af9700 1 -- 172.21.15.121:0/2940854516 shutdown_connections 2024-06-10T00:34:00.704 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:00.701+0000 7f1769af9700 1 -- 172.21.15.121:0/2940854516 wait complete. 2024-06-10T00:34:00.706 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-06-10T00:34:00.723 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-10T00:34:00.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-10T00:34:01.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:01 smithi193 ceph-mon[36716]: pgmap v184: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.1 MiB/s wr, 422 op/s 2024-06-10T00:34:01.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:01 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2940854516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:01 smithi121 ceph-mon[32378]: pgmap v184: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.1 MiB/s wr, 422 op/s 2024-06-10T00:34:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:01 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2940854516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:01.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:01 smithi135 ceph-mon[36424]: pgmap v184: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 1.1 MiB/s wr, 422 op/s 2024-06-10T00:34:01.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:01 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2940854516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:01.518 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db --set-id 5eddb4ce --release --await' 2024-06-10T00:34:01.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db --set-id 5eddb4ce --release --await 2024-06-10T00:34:01.586 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:03.095 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:03 smithi121 ceph-mon[32378]: pgmap v185: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1017 KiB/s wr, 466 op/s 2024-06-10T00:34:03.321 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.326+0000 7f918d5de700 1 -- 172.21.15.121:0/2424735849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91881084f0 msgr2=0x7f91881088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.326+0000 7f918d5de700 1 --2- 172.21.15.121:0/2424735849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91881084f0 0x7f91881088d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f917c009990 tx=0x7f917c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.326+0000 7f918d5de700 1 -- 172.21.15.121:0/2424735849 shutdown_connections 2024-06-10T00:34:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.326+0000 7f918d5de700 1 --2- 172.21.15.121:0/2424735849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f918806b3c0 0x7f918810f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.326+0000 7f918d5de700 1 --2- 172.21.15.121:0/2424735849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91881084f0 0x7f91881088d0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.326+0000 7f918d5de700 1 --2- 172.21.15.121:0/2424735849 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f918806aa00 0x7f918806ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.326+0000 7f918d5de700 1 -- 172.21.15.121:0/2424735849 >> 172.21.15.121:0/2424735849 conn(0x7f9188078300 msgr2=0x7f9188078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.326+0000 7f918d5de700 1 -- 172.21.15.121:0/2424735849 shutdown_connections 2024-06-10T00:34:03.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.326+0000 7f918d5de700 1 -- 172.21.15.121:0/2424735849 wait complete. 2024-06-10T00:34:03.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.327+0000 7f918d5de700 1 Processor -- start 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.327+0000 7f918d5de700 1 -- start start 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.327+0000 7f918d5de700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f918806aa00 0x7f9188101450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.327+0000 7f918d5de700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f918806b3c0 0x7f9188101990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.327+0000 7f918d5de700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91881084f0 0x7f9188101ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.327+0000 7f918d5de700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9188112750 con 0x7f918806b3c0 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.327+0000 7f918d5de700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f91881125d0 con 0x7f918806aa00 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.327+0000 7f918d5de700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91881128d0 con 0x7f91881084f0 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.327+0000 7f91867fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f918806b3c0 0x7f9188101990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.327+0000 7f91867fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f918806b3c0 0x7f9188101990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50192/0 (socket says 172.21.15.121:50192) 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.327+0000 7f91867fc700 1 -- 172.21.15.121:0/235866487 learned_addr learned my addr 172.21.15.121:0/235866487 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f91877fe700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91881084f0 0x7f9188101ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f9186ffd700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f918806aa00 0x7f9188101450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f91867fc700 1 -- 172.21.15.121:0/235866487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91881084f0 msgr2=0x7f9188101ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f91867fc700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91881084f0 0x7f9188101ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f91867fc700 1 -- 172.21.15.121:0/235866487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f918806aa00 msgr2=0x7f9188101450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f91867fc700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f918806aa00 0x7f9188101450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f91867fc700 1 -- 172.21.15.121:0/235866487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91881a7150 con 0x7f918806b3c0 2024-06-10T00:34:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f91867fc700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f918806b3c0 0x7f9188101990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f917800d980 tx=0x7f917800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f916ffff700 1 -- 172.21.15.121:0/235866487 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9178016070 con 0x7f918806b3c0 2024-06-10T00:34:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f916ffff700 1 -- 172.21.15.121:0/235866487 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9178004500 con 0x7f918806b3c0 2024-06-10T00:34:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f918d5de700 1 -- 172.21.15.121:0/235866487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91881a7440 con 0x7f918806b3c0 2024-06-10T00:34:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f916ffff700 1 -- 172.21.15.121:0/235866487 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9178004d00 con 0x7f918806b3c0 2024-06-10T00:34:03.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.328+0000 7f918d5de700 1 -- 172.21.15.121:0/235866487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91881a79d0 con 0x7f918806b3c0 2024-06-10T00:34:03.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.332+0000 7f916ffff700 1 -- 172.21.15.121:0/235866487 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f917800bbf0 con 0x7f918806b3c0 2024-06-10T00:34:03.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.332+0000 7f916ffff700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f91700a2e10 0x7f91700a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.332+0000 7f9186ffd700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f91700a2e10 0x7f91700a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.332+0000 7f916ffff700 1 -- 172.21.15.121:0/235866487 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f91780cf5b0 con 0x7f918806b3c0 2024-06-10T00:34:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.332+0000 7f9197e61b80 1 Processor -- start 2024-06-10T00:34:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f9197e61b80 1 -- start start 2024-06-10T00:34:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf8ed0 0x55d966cf92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf97f0 0x55d966cf3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966cf3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f9197e61b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d966c3c890 con 0x55d966cf37b0 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f9186ffd700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f91700a2e10 0x7f91700a52d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f917c006010 tx=0x7f917c00b540 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f9197e61b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d966be47a0 con 0x55d966cf8ed0 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f9197e61b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d966c38850 con 0x55d966cf97f0 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f91877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966cf3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f91877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966cf3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50200/0 (socket says 172.21.15.121:50200) 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f91877fe700 1 -- 172.21.15.121:0/415971956 learned_addr learned my addr 172.21.15.121:0/415971956 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f9186ffd700 1 --2- 172.21.15.121:0/415971956 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf8ed0 0x55d966cf92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f91867fc700 1 --2- 172.21.15.121:0/415971956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf97f0 0x55d966cf3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f91877fe700 1 -- 172.21.15.121:0/415971956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf97f0 msgr2=0x55d966cf3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f91877fe700 1 --2- 172.21.15.121:0/415971956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf97f0 0x55d966cf3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f91877fe700 1 -- 172.21.15.121:0/415971956 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf8ed0 msgr2=0x55d966cf92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f91877fe700 1 --2- 172.21.15.121:0/415971956 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf8ed0 0x55d966cf92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f91877fe700 1 -- 172.21.15.121:0/415971956 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d966bdafd0 con 0x55d966cf37b0 2024-06-10T00:34:03.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.333+0000 7f9186ffd700 1 --2- 172.21.15.121:0/415971956 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf8ed0 0x55d966cf92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:34:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.334+0000 7f91877fe700 1 --2- 172.21.15.121:0/415971956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966cf3bd0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f91800084f0 tx=0x7f9180008800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02f1d9069c1ed1f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.334+0000 7f916d7fa700 1 -- 172.21.15.121:0/415971956 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9180003c00 con 0x55d966cf37b0 2024-06-10T00:34:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.334+0000 7f916d7fa700 1 -- 172.21.15.121:0/415971956 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9180003da0 con 0x55d966cf37b0 2024-06-10T00:34:03.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.334+0000 7f916d7fa700 1 -- 172.21.15.121:0/415971956 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91800129c0 con 0x55d966cf37b0 2024-06-10T00:34:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 -- 172.21.15.121:0/415971956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 msgr2=0x55d966cf3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 --2- 172.21.15.121:0/415971956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966cf3bd0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f91800084f0 tx=0x7f9180008800 comp rx=0 tx=0).stop 2024-06-10T00:34:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 -- 172.21.15.121:0/415971956 shutdown_connections 2024-06-10T00:34:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 --2- 172.21.15.121:0/415971956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966cf3bd0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 --2- 172.21.15.121:0/415971956 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf8ed0 0x55d966cf92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 --2- 172.21.15.121:0/415971956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf97f0 0x55d966cf3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 -- 172.21.15.121:0/415971956 >> 172.21.15.121:0/415971956 conn(0x55d966c15d40 msgr2=0x55d966ce58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 -- 172.21.15.121:0/415971956 shutdown_connections 2024-06-10T00:34:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 -- 172.21.15.121:0/415971956 wait complete. 2024-06-10T00:34:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 -- start start 2024-06-10T00:34:03.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 Processor -- start 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf37b0 0x55d966cff540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf8ed0 0x55d966cfc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf97f0 0x55d966cfcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d966d00260 con 0x55d966cf8ed0 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55d966bdcc70 con 0x55d966cf37b0 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.335+0000 7f9197e61b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d966d003e0 con 0x55d966cf97f0 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f9197e61b80 10 client.0 _ll_register_callbacks cb 0x55d966bd69a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f91867fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf8ed0 0x55d966cfc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f91867fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf8ed0 0x55d966cfc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50204/0 (socket says 172.21.15.121:50204) 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f9186ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf37b0 0x55d966cff540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f91867fc700 1 -- 172.21.15.121:0/127671284 learned_addr learned my addr 172.21.15.121:0/127671284 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f91877fe700 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf97f0 0x55d966cfcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f91867fc700 1 -- 172.21.15.121:0/127671284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf97f0 msgr2=0x55d966cfcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f91867fc700 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf97f0 0x55d966cfcd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f91867fc700 1 -- 172.21.15.121:0/127671284 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf37b0 msgr2=0x55d966cff540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f91867fc700 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf37b0 0x55d966cff540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f91867fc700 1 -- 172.21.15.121:0/127671284 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d966b8b950 con 0x55d966cf8ed0 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f91877fe700 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf97f0 0x55d966cfcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f91867fc700 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf8ed0 0x55d966cfc840 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f917809b440 tx=0x7f917809b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c9835352746f36 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.336+0000 7f916aff5700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.337+0000 7f916aff5700 1 -- 172.21.15.121:0/127671284 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91780b4060 con 0x55d966cf8ed0 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.337+0000 7f916aff5700 1 -- 172.21.15.121:0/127671284 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f91780cf1b0 con 0x55d966cf8ed0 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.337+0000 7f9197e61b80 1 -- 172.21.15.121:0/127671284 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9180003da0 con 0x55d966cf8ed0 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.337+0000 7f916aff5700 1 -- 172.21.15.121:0/127671284 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f917809b740 con 0x55d966cf8ed0 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.337+0000 7f9197e61b80 10 client.16368 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.337+0000 7f9197e61b80 1 -- 172.21.15.121:0/127671284 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d966d0abd0 con 0x55d966cf8ed0 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.337+0000 7f916aff5700 1 -- 172.21.15.121:0/127671284 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9180003da0 con 0x55d966cf8ed0 2024-06-10T00:34:03.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.337+0000 7f916aff5700 1 -- 172.21.15.121:0/127671284 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f917809b9a0 con 0x55d966cf8ed0 2024-06-10T00:34:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.337+0000 7f9197e61b80 10 client.16368 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.337+0000 7f9197e61b80 10 client.16368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:34:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.337+0000 7f9197e61b80 20 client.16368 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.338+0000 7f9197e61b80 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d966d0d160 0x55d966d0f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.338+0000 7f9197e61b80 4 client.16368 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:34:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.338+0000 7f9197e61b80 1 -- 172.21.15.121:0/127671284 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d966c49d90 con 0x55d966d0d160 2024-06-10T00:34:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.338+0000 7f9186ffd700 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d966d0d160 0x55d966d0f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.338+0000 7f916aff5700 10 client.16368 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:03.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.338+0000 7f9186ffd700 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d966d0d160 0x55d966d0f550 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2418674eec12ecc server_cookie=2dae9b8f0e13d67e in_seq=0 out_seq=0 2024-06-10T00:34:03.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.339+0000 7f916aff5700 1 -- 172.21.15.121:0/127671284 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55d966c49d90 con 0x55d966d0d160 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.339+0000 7f916aff5700 10 client.16368 handle_command_reply: tid=0 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.339+0000 7f9197e61b80 1 client.16368 shutdown 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.339+0000 7f9197e61b80 1 -- 172.21.15.121:0/127671284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf8ed0 msgr2=0x55d966cfc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.339+0000 7f9197e61b80 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf8ed0 0x55d966cfc840 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f917809b440 tx=0x7f917809b5e0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.340+0000 7f9197e61b80 1 -- 172.21.15.121:0/127671284 shutdown_connections 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.340+0000 7f9197e61b80 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d966d0d160 0x55d966d0f550 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.340+0000 7f9197e61b80 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf97f0 0x55d966cfcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.340+0000 7f916aff5700 0 client.16368 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.340+0000 7f9197e61b80 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf37b0 0x55d966cff540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.340+0000 7f9197e61b80 1 --2- 172.21.15.121:0/127671284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf8ed0 0x55d966cfc840 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.340+0000 7f9197e61b80 1 -- 172.21.15.121:0/127671284 >> 172.21.15.121:0/127671284 conn(0x55d966c15d40 msgr2=0x55d966cf2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.340+0000 7f9197e61b80 1 -- 172.21.15.121:0/127671284 shutdown_connections 2024-06-10T00:34:03.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.340+0000 7f9197e61b80 1 -- 172.21.15.121:0/127671284 wait complete. 2024-06-10T00:34:03.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.340+0000 7f9197e61b80 20 client.16368 trim_cache size 0 max 16384 2024-06-10T00:34:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.361+0000 7f9197e61b80 1 Processor -- start 2024-06-10T00:34:03.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.361+0000 7f9197e61b80 1 -- start start 2024-06-10T00:34:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.361+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf37b0 0x55d966cf3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.361+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf8ed0 0x55d966ce5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.361+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf97f0 0x55d966cdf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.361+0000 7f9197e61b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f91881aec90 con 0x55d966cf97f0 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.361+0000 7f9197e61b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f91881ac4c0 con 0x55d966cf8ed0 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f9197e61b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91881aa310 con 0x55d966cf37b0 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f91877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf97f0 0x55d966cdf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f91877fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf97f0 0x55d966cdf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50220/0 (socket says 172.21.15.121:50220) 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f91877fe700 1 -- 172.21.15.121:0/4136629609 learned_addr learned my addr 172.21.15.121:0/4136629609 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f91867fc700 1 --2- 172.21.15.121:0/4136629609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf8ed0 0x55d966ce5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f9186ffd700 1 --2- 172.21.15.121:0/4136629609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf37b0 0x55d966cf3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f91877fe700 1 -- 172.21.15.121:0/4136629609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf37b0 msgr2=0x55d966cf3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f91877fe700 1 --2- 172.21.15.121:0/4136629609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf37b0 0x55d966cf3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f91877fe700 1 -- 172.21.15.121:0/4136629609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf8ed0 msgr2=0x55d966ce5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f91877fe700 1 --2- 172.21.15.121:0/4136629609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf8ed0 0x55d966ce5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f91877fe700 1 -- 172.21.15.121:0/4136629609 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d966cfd100 con 0x55d966cf97f0 2024-06-10T00:34:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f91877fe700 1 --2- 172.21.15.121:0/4136629609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf97f0 0x55d966cdf710 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9180004a20 tx=0x7f9180007470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488a06161904555d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f916cff9700 1 -- 172.21.15.121:0/4136629609 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9180022020 con 0x55d966cf97f0 2024-06-10T00:34:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f916cff9700 1 -- 172.21.15.121:0/4136629609 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9180004d50 con 0x55d966cf97f0 2024-06-10T00:34:03.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.362+0000 7f916cff9700 1 -- 172.21.15.121:0/4136629609 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918001e700 con 0x55d966cf97f0 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.363+0000 7f9197e61b80 1 -- 172.21.15.121:0/4136629609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf97f0 msgr2=0x55d966cdf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.363+0000 7f9197e61b80 1 --2- 172.21.15.121:0/4136629609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf97f0 0x55d966cdf710 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9180004a20 tx=0x7f9180007470 comp rx=0 tx=0).stop 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.363+0000 7f9197e61b80 1 -- 172.21.15.121:0/4136629609 shutdown_connections 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.363+0000 7f9197e61b80 1 --2- 172.21.15.121:0/4136629609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf97f0 0x55d966cdf710 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.363+0000 7f9197e61b80 1 --2- 172.21.15.121:0/4136629609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf37b0 0x55d966cf3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.363+0000 7f9197e61b80 1 --2- 172.21.15.121:0/4136629609 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf8ed0 0x55d966ce5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.363+0000 7f9197e61b80 1 -- 172.21.15.121:0/4136629609 >> 172.21.15.121:0/4136629609 conn(0x55d966c15d40 msgr2=0x55d966cf79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.363+0000 7f9197e61b80 1 -- 172.21.15.121:0/4136629609 shutdown_connections 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.363+0000 7f9197e61b80 1 -- 172.21.15.121:0/4136629609 wait complete. 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.363+0000 7f9197e61b80 1 -- start start 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9197e61b80 1 Processor -- start 2024-06-10T00:34:03.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966d026a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf8ed0 0x55d966d02be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9197e61b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf97f0 0x55d966d03120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9197e61b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f91881aa980 con 0x55d966cf37b0 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9197e61b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f91881abdf0 con 0x55d966cf97f0 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9197e61b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f91881ab050 con 0x55d966cf8ed0 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9197e61b80 10 client.0 _ll_register_callbacks cb 0x55d966bd69a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9186ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966d026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9186ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966d026a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50222/0 (socket says 172.21.15.121:50222) 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9186ffd700 1 -- 172.21.15.121:0/522146377 learned_addr learned my addr 172.21.15.121:0/522146377 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f91867fc700 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf8ed0 0x55d966d02be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f91877fe700 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf97f0 0x55d966d03120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9186ffd700 1 -- 172.21.15.121:0/522146377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf8ed0 msgr2=0x55d966d02be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:03.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9186ffd700 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf8ed0 0x55d966d02be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9186ffd700 1 -- 172.21.15.121:0/522146377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf97f0 msgr2=0x55d966d03120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9186ffd700 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf97f0 0x55d966d03120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f9186ffd700 1 -- 172.21.15.121:0/522146377 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d966d77f90 con 0x55d966cf37b0 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.364+0000 7f91877fe700 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf97f0 0x55d966d03120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f9186ffd700 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966d026a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f917c009750 tx=0x7f917c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480db0f3d5831164 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f916aff5700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f916aff5700 1 -- 172.21.15.121:0/522146377 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c00fc50 con 0x55d966cf37b0 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f9197e61b80 1 -- 172.21.15.121:0/522146377 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9180004d50 con 0x55d966cf37b0 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f916aff5700 1 -- 172.21.15.121:0/522146377 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f917c00fdf0 con 0x55d966cf37b0 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f916aff5700 1 -- 172.21.15.121:0/522146377 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f917c042050 con 0x55d966cf37b0 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f9197e61b80 10 client.16374 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f9197e61b80 1 -- 172.21.15.121:0/522146377 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d966d79330 con 0x55d966cf37b0 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f916aff5700 1 -- 172.21.15.121:0/522146377 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9180004d50 con 0x55d966cf37b0 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f916aff5700 1 -- 172.21.15.121:0/522146377 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f917c024410 con 0x55d966cf37b0 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f9197e61b80 10 client.16374 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:03.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f9197e61b80 10 client.16374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f9197e61b80 20 client.16374 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f9197e61b80 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d966d79ad0 0x55d966d7bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f9197e61b80 4 client.16374 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.365+0000 7f9197e61b80 1 -- 172.21.15.121:0/522146377 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d966c44310 con 0x55d966d79ad0 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.366+0000 7f91867fc700 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d966d79ad0 0x55d966d7bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.366+0000 7f916aff5700 10 client.16374 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.366+0000 7f91867fc700 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d966d79ad0 0x55d966d7bef0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d25090b9855c2a57 server_cookie=9ffb1f09bf7b8689 in_seq=0 out_seq=0 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.367+0000 7f916aff5700 1 -- 172.21.15.121:0/522146377 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d966c44310 con 0x55d966d79ad0 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.367+0000 7f916aff5700 10 client.16374 handle_command_reply: tid=0 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.367+0000 7f9197e61b80 1 client.16374 shutdown 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f9197e61b80 1 -- 172.21.15.121:0/522146377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 msgr2=0x55d966d026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f9197e61b80 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966d026a0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f917c009750 tx=0x7f917c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f9197e61b80 1 -- 172.21.15.121:0/522146377 shutdown_connections 2024-06-10T00:34:03.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f9197e61b80 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55d966d79ad0 0x55d966d7bef0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f9197e61b80 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55d966cf97f0 0x55d966d03120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f916aff5700 0 client.16374 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f9197e61b80 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d966cf37b0 0x55d966d026a0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f9197e61b80 1 --2- 172.21.15.121:0/522146377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d966cf8ed0 0x55d966d02be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f9197e61b80 1 -- 172.21.15.121:0/522146377 >> 172.21.15.121:0/522146377 conn(0x55d966c15d40 msgr2=0x55d966cfc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f9197e61b80 1 -- 172.21.15.121:0/522146377 shutdown_connections 2024-06-10T00:34:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f9197e61b80 1 -- 172.21.15.121:0/522146377 wait complete. 2024-06-10T00:34:03.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f9197e61b80 20 client.16374 trim_cache size 0 max 16384 2024-06-10T00:34:03.374 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:34:03.374 INFO:teuthology.orchestra.run.smithi121.stdout: "return_code": 0, 2024-06-10T00:34:03.375 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_tag": "38e051c5-0ec8-45cc-b0bf-2191994b1d58", 2024-06-10T00:34:03.375 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": "asynchronous" 2024-06-10T00:34:03.375 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:34:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f918d5de700 1 -- 172.21.15.121:0/235866487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f91700a2e10 msgr2=0x7f91700a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f918d5de700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f91700a2e10 0x7f91700a52d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f917c006010 tx=0x7f917c00b540 comp rx=0 tx=0).stop 2024-06-10T00:34:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f918d5de700 1 -- 172.21.15.121:0/235866487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f918806b3c0 msgr2=0x7f9188101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:03.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f918d5de700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f918806b3c0 0x7f9188101990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f917800d980 tx=0x7f917800dd40 comp rx=0 tx=0).stop 2024-06-10T00:34:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f918d5de700 1 -- 172.21.15.121:0/235866487 shutdown_connections 2024-06-10T00:34:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f918d5de700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f91700a2e10 0x7f91700a52d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f918d5de700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91881084f0 0x7f9188101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f918d5de700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f918806aa00 0x7f9188101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f918d5de700 1 --2- 172.21.15.121:0/235866487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f918806b3c0 0x7f9188101990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.368+0000 7f918d5de700 1 -- 172.21.15.121:0/235866487 >> 172.21.15.121:0/235866487 conn(0x7f9188078300 msgr2=0x7f9188100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.369+0000 7f918d5de700 1 -- 172.21.15.121:0/235866487 shutdown_connections 2024-06-10T00:34:03.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:03.369+0000 7f918d5de700 1 -- 172.21.15.121:0/235866487 wait complete. 2024-06-10T00:34:03.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:03 smithi193 ceph-mon[36716]: pgmap v185: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1017 KiB/s wr, 466 op/s 2024-06-10T00:34:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:03 smithi135 ceph-mon[36424]: pgmap v185: 129 pgs: 129 active+clean; 1.2 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1017 KiB/s wr, 466 op/s 2024-06-10T00:34:04.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:04 smithi193 ceph-mon[36716]: scrub queued for path: / 2024-06-10T00:34:04.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:04 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:04.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:04 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:34:04.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:04 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:34:04.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:04 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:34:04.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:04 smithi121 ceph-mon[32378]: scrub queued for path: / 2024-06-10T00:34:04.418 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:04 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:04.419 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:04 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:34:04.419 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:04 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:34:04.419 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:04 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:34:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:04 smithi135 ceph-mon[36424]: scrub queued for path: / 2024-06-10T00:34:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:04 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:04 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:34:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:04 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:34:04.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:04 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:34:04.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:34:05.099 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:05 smithi121 ceph-mon[32378]: pgmap v186: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 875 KiB/s wr, 411 op/s 2024-06-10T00:34:05.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:05 smithi193 ceph-mon[36716]: pgmap v186: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 875 KiB/s wr, 411 op/s 2024-06-10T00:34:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:05 smithi135 ceph-mon[36424]: pgmap v186: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 875 KiB/s wr, 411 op/s 2024-06-10T00:34:06.019 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:06.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.019+0000 7f68e31f5700 1 -- 172.21.15.121:0/993220351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68dc1084f0 msgr2=0x7f68dc1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:06.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.019+0000 7f68e31f5700 1 --2- 172.21.15.121:0/993220351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68dc1084f0 0x7f68dc1088d0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f68d0009990 tx=0x7f68d0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.019+0000 7f68e31f5700 1 -- 172.21.15.121:0/993220351 shutdown_connections 2024-06-10T00:34:06.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.019+0000 7f68e31f5700 1 --2- 172.21.15.121:0/993220351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68dc06b3c0 0x7f68dc10f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.019+0000 7f68e31f5700 1 --2- 172.21.15.121:0/993220351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68dc1084f0 0x7f68dc1088d0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.019+0000 7f68e31f5700 1 --2- 172.21.15.121:0/993220351 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68dc06aa00 0x7f68dc06ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.019+0000 7f68e31f5700 1 -- 172.21.15.121:0/993220351 >> 172.21.15.121:0/993220351 conn(0x7f68dc078300 msgr2=0x7f68dc078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:06.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.019+0000 7f68e31f5700 1 -- 172.21.15.121:0/993220351 shutdown_connections 2024-06-10T00:34:06.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.019+0000 7f68e31f5700 1 -- 172.21.15.121:0/993220351 wait complete. 2024-06-10T00:34:06.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.019+0000 7f68e31f5700 1 Processor -- start 2024-06-10T00:34:06.021 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.019+0000 7f68e31f5700 1 -- start start 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e31f5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68dc06aa00 0x7f68dc19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e31f5700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68dc06b3c0 0x7f68dc19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e31f5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68dc1084f0 0x7f68dc198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e31f5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc112750 con 0x7f68dc06aa00 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e31f5700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc1125d0 con 0x7f68dc06b3c0 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e31f5700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc1128d0 con 0x7f68dc1084f0 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e0f91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68dc06aa00 0x7f68dc19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e0f91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68dc06aa00 0x7f68dc19ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50242/0 (socket says 172.21.15.121:50242) 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e0f91700 1 -- 172.21.15.121:0/1229248998 learned_addr learned my addr 172.21.15.121:0/1229248998 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68dbfff700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68dc06b3c0 0x7f68dc19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e1792700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68dc1084f0 0x7f68dc198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e0f91700 1 -- 172.21.15.121:0/1229248998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68dc1084f0 msgr2=0x7f68dc198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:06.022 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e0f91700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68dc1084f0 0x7f68dc198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e0f91700 1 -- 172.21.15.121:0/1229248998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68dc06b3c0 msgr2=0x7f68dc19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:06.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e0f91700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68dc06b3c0 0x7f68dc19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e0f91700 1 -- 172.21.15.121:0/1229248998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68dc199410 con 0x7f68dc06aa00 2024-06-10T00:34:06.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.020+0000 7f68e0f91700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68dc06aa00 0x7f68dc19ea60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f68d00038b0 tx=0x7f68d0009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:06.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.021+0000 7f68d9ffb700 1 -- 172.21.15.121:0/1229248998 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0027070 con 0x7f68dc06aa00 2024-06-10T00:34:06.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.021+0000 7f68d9ffb700 1 -- 172.21.15.121:0/1229248998 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68d0004050 con 0x7f68dc06aa00 2024-06-10T00:34:06.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.021+0000 7f68e31f5700 1 -- 172.21.15.121:0/1229248998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68dc1996a0 con 0x7f68dc06aa00 2024-06-10T00:34:06.023 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.021+0000 7f68d9ffb700 1 -- 172.21.15.121:0/1229248998 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0022c00 con 0x7f68dc06aa00 2024-06-10T00:34:06.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.021+0000 7f68e31f5700 1 -- 172.21.15.121:0/1229248998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68dc1a59e0 con 0x7f68dc06aa00 2024-06-10T00:34:06.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.023+0000 7f68d9ffb700 1 -- 172.21.15.121:0/1229248998 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f68d002a070 con 0x7f68dc06aa00 2024-06-10T00:34:06.024 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.024+0000 7f68d9ffb700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68cc0a2e10 0x7f68cc0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.024+0000 7f68dbfff700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68cc0a2e10 0x7f68cc0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.024+0000 7f68d9ffb700 1 -- 172.21.15.121:0/1229248998 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f68d00d0610 con 0x7f68dc06aa00 2024-06-10T00:34:06.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68dbfff700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68cc0a2e10 0x7f68cc0a52d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f68dc19a280 tx=0x7f68c800b410 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:06.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68eda78b80 1 Processor -- start 2024-06-10T00:34:06.025 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68eda78b80 1 -- start start 2024-06-10T00:34:06.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117c4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117c5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68eda78b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5558116b7060 con 0x5558117c4c50 2024-06-10T00:34:06.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68eda78b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x555811715950 con 0x5558117ca500 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68eda78b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55581167d560 con 0x5558117cae20 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68e1792700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68e1792700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117c5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50256/0 (socket says 172.21.15.121:50256) 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68e1792700 1 -- 172.21.15.121:0/1246706516 learned_addr learned my addr 172.21.15.121:0/1246706516 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68dbfff700 1 --2- 172.21.15.121:0/1246706516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68e0f91700 1 --2- 172.21.15.121:0/1246706516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68e1792700 1 -- 172.21.15.121:0/1246706516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 msgr2=0x5558117c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68e1792700 1 --2- 172.21.15.121:0/1246706516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117c4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68e1792700 1 -- 172.21.15.121:0/1246706516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 msgr2=0x5558117ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68e1792700 1 --2- 172.21.15.121:0/1246706516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117ca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.025+0000 7f68e1792700 1 -- 172.21.15.121:0/1246706516 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558116649d0 con 0x5558117c4c50 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.026+0000 7f68dbfff700 1 --2- 172.21.15.121:0/1246706516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:06.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.026+0000 7f68e1792700 1 --2- 172.21.15.121:0/1246706516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117c5070 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f68d400a730 tx=0x7f68d40078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6671885ab5996afc server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.026+0000 7f68c6ffd700 1 -- 172.21.15.121:0/1246706516 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4003a00 con 0x5558117c4c50 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.026+0000 7f68c6ffd700 1 -- 172.21.15.121:0/1246706516 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68d4003ba0 con 0x5558117c4c50 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.026+0000 7f68c6ffd700 1 -- 172.21.15.121:0/1246706516 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4014660 con 0x5558117c4c50 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.027+0000 7f68eda78b80 1 -- 172.21.15.121:0/1246706516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 msgr2=0x5558117c5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.027+0000 7f68eda78b80 1 --2- 172.21.15.121:0/1246706516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117c5070 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f68d400a730 tx=0x7f68d40078d0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.027+0000 7f68eda78b80 1 -- 172.21.15.121:0/1246706516 shutdown_connections 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.027+0000 7f68eda78b80 1 --2- 172.21.15.121:0/1246706516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117c5070 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.027+0000 7f68eda78b80 1 --2- 172.21.15.121:0/1246706516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.027+0000 7f68eda78b80 1 --2- 172.21.15.121:0/1246706516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.027+0000 7f68eda78b80 1 -- 172.21.15.121:0/1246706516 >> 172.21.15.121:0/1246706516 conn(0x5558116f1180 msgr2=0x5558117bfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.027+0000 7f68eda78b80 1 -- 172.21.15.121:0/1246706516 shutdown_connections 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.027+0000 7f68eda78b80 1 -- 172.21.15.121:0/1246706516 wait complete. 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.027+0000 7f68eda78b80 1 -- start start 2024-06-10T00:34:06.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.027+0000 7f68eda78b80 1 Processor -- start 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117c4c50 0x5558117d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117ca500 0x5558117cde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117cae20 0x5558117ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68eda78b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5558116b60c0 con 0x5558117cae20 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68eda78b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x555811711910 con 0x5558117c4c50 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68eda78b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5558117d17a0 con 0x5558117ca500 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68eda78b80 10 client.0 _ll_register_callbacks cb 0x5558116ab6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68e1792700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117cae20 0x5558117ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68e1792700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117cae20 0x5558117ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50268/0 (socket says 172.21.15.121:50268) 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68e1792700 1 -- 172.21.15.121:0/2418824299 learned_addr learned my addr 172.21.15.121:0/2418824299 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68dbfff700 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117ca500 0x5558117cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68e0f91700 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117c4c50 0x5558117d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.032 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68e1792700 1 -- 172.21.15.121:0/2418824299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117ca500 msgr2=0x5558117cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68e1792700 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117ca500 0x5558117cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68e1792700 1 -- 172.21.15.121:0/2418824299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117c4c50 msgr2=0x5558117d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68e1792700 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117c4c50 0x5558117d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68e1792700 1 -- 172.21.15.121:0/2418824299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558116db850 con 0x5558117cae20 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68e0f91700 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117c4c50 0x5558117d0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68e1792700 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117cae20 0x5558117ce340 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f68d4007fa0 tx=0x7f68d400a360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6e0ce0de6a9fab server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68bbfff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.028+0000 7f68bbfff700 1 -- 172.21.15.121:0/2418824299 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4022070 con 0x5558117cae20 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68bbfff700 1 -- 172.21.15.121:0/2418824299 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f68d401e450 con 0x5558117cae20 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68eda78b80 1 -- 172.21.15.121:0/2418824299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68d4003ba0 con 0x5558117cae20 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68bbfff700 1 -- 172.21.15.121:0/2418824299 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f68d4003a00 con 0x5558117cae20 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68eda78b80 10 client.16386 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68eda78b80 1 -- 172.21.15.121:0/2418824299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5558117dbe60 con 0x5558117cae20 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68bbfff700 1 -- 172.21.15.121:0/2418824299 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68d4003ba0 con 0x5558117cae20 2024-06-10T00:34:06.033 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68bbfff700 1 -- 172.21.15.121:0/2418824299 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f68d4014700 con 0x5558117cae20 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68eda78b80 10 client.16386 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68eda78b80 10 client.16386 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68eda78b80 20 client.16386 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68eda78b80 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5558117de3f0 0x5558117e07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68eda78b80 4 client.16386 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68eda78b80 1 -- 172.21.15.121:0/2418824299 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555811722e50 con 0x5558117de3f0 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.029+0000 7f68e0f91700 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5558117de3f0 0x5558117e07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.030+0000 7f68e0f91700 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5558117de3f0 0x5558117e07e0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21b68457247b6c07 server_cookie=9042ccc6e21b660b in_seq=0 out_seq=0 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.030+0000 7f68bbfff700 10 client.16386 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.030+0000 7f68bbfff700 1 -- 172.21.15.121:0/2418824299 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x555811722e50 con 0x5558117de3f0 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.030+0000 7f68bbfff700 10 client.16386 handle_command_reply: tid=0 2024-06-10T00:34:06.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 1 client.16386 shutdown 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 1 -- 172.21.15.121:0/2418824299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117cae20 msgr2=0x5558117ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117cae20 0x5558117ce340 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f68d4007fa0 tx=0x7f68d400a360 comp rx=0 tx=0).stop 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 1 -- 172.21.15.121:0/2418824299 shutdown_connections 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5558117de3f0 0x5558117e07e0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117cae20 0x5558117ce340 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117c4c50 0x5558117d0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68bbfff700 0 client.16386 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 1 --2- 172.21.15.121:0/2418824299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117ca500 0x5558117cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 1 -- 172.21.15.121:0/2418824299 >> 172.21.15.121:0/2418824299 conn(0x5558116f1180 msgr2=0x5558117bebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 1 -- 172.21.15.121:0/2418824299 shutdown_connections 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 1 -- 172.21.15.121:0/2418824299 wait complete. 2024-06-10T00:34:06.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.031+0000 7f68eda78b80 20 client.16386 trim_cache size 0 max 16384 2024-06-10T00:34:06.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.056+0000 7f68eda78b80 1 Processor -- start 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.056+0000 7f68eda78b80 1 -- start start 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.056+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117c4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.056+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.056+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117bf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.056+0000 7f68eda78b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc1aec00 con 0x5558117c4c50 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.056+0000 7f68eda78b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc1ac430 con 0x5558117ca500 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.056+0000 7f68eda78b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc1aa280 con 0x5558117cae20 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68e0f91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68e1792700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117bf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68e0f91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117c4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50270/0 (socket says 172.21.15.121:50270) 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68e0f91700 1 -- 172.21.15.121:0/575224418 learned_addr learned my addr 172.21.15.121:0/575224418 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68dbfff700 1 --2- 172.21.15.121:0/575224418 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68e1792700 1 -- 172.21.15.121:0/575224418 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 msgr2=0x5558117ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:06.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68e1792700 1 --2- 172.21.15.121:0/575224418 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68e1792700 1 -- 172.21.15.121:0/575224418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 msgr2=0x5558117c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68e1792700 1 --2- 172.21.15.121:0/575224418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117c4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68e1792700 1 -- 172.21.15.121:0/575224418 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558117cc370 con 0x5558117cae20 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68e1792700 1 --2- 172.21.15.121:0/575224418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117bf330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f68d4008410 tx=0x7f68d40078d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbea0b6fd5851071 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68c67fc700 1 -- 172.21.15.121:0/575224418 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4022040 con 0x5558117cae20 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68c67fc700 1 -- 172.21.15.121:0/575224418 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68d40263b0 con 0x5558117cae20 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.057+0000 7f68c67fc700 1 -- 172.21.15.121:0/575224418 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4026630 con 0x5558117cae20 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.058+0000 7f68eda78b80 1 -- 172.21.15.121:0/575224418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 msgr2=0x5558117bf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.058+0000 7f68eda78b80 1 --2- 172.21.15.121:0/575224418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117bf330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f68d4008410 tx=0x7f68d40078d0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.058+0000 7f68eda78b80 1 -- 172.21.15.121:0/575224418 shutdown_connections 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.058+0000 7f68eda78b80 1 --2- 172.21.15.121:0/575224418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117bf330 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.058+0000 7f68eda78b80 1 --2- 172.21.15.121:0/575224418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.058+0000 7f68eda78b80 1 --2- 172.21.15.121:0/575224418 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.058+0000 7f68eda78b80 1 -- 172.21.15.121:0/575224418 >> 172.21.15.121:0/575224418 conn(0x5558116f1180 msgr2=0x5558117cb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:06.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.058+0000 7f68eda78b80 1 -- 172.21.15.121:0/575224418 shutdown_connections 2024-06-10T00:34:06.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.058+0000 7f68eda78b80 1 -- 172.21.15.121:0/575224418 wait complete. 2024-06-10T00:34:06.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.058+0000 7f68eda78b80 1 -- start start 2024-06-10T00:34:06.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.058+0000 7f68eda78b80 1 Processor -- start 2024-06-10T00:34:06.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117d7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117e5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68eda78b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117e5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68eda78b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc1aa8f0 con 0x5558117c4c50 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68eda78b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc1abd60 con 0x5558117ca500 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68eda78b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f68dc1aafc0 con 0x5558117cae20 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68e0f91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117d7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68e0f91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117d7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50276/0 (socket says 172.21.15.121:50276) 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68e0f91700 1 -- 172.21.15.121:0/4169844482 learned_addr learned my addr 172.21.15.121:0/4169844482 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68eda78b80 10 client.0 _ll_register_callbacks cb 0x5558116ab6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68e1792700 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117e5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68e0f91700 1 -- 172.21.15.121:0/4169844482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 msgr2=0x5558117e5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68dbfff700 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117e5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68e0f91700 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117e5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68e0f91700 1 -- 172.21.15.121:0/4169844482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 msgr2=0x5558117e5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:06.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68e0f91700 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117e5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68e0f91700 1 -- 172.21.15.121:0/4169844482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558117e6610 con 0x5558117c4c50 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.059+0000 7f68dbfff700 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117e5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68e0f91700 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117d7280 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f68d00cda80 tx=0x7f68d00a3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6770a6760bc03275 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68bbfff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68bbfff700 1 -- 172.21.15.121:0/4169844482 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d00ba060 con 0x5558117c4c50 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68bbfff700 1 -- 172.21.15.121:0/4169844482 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f68d0022470 con 0x5558117c4c50 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68eda78b80 1 -- 172.21.15.121:0/4169844482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68d40263b0 con 0x5558117c4c50 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68bbfff700 1 -- 172.21.15.121:0/4169844482 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f68d009c830 con 0x5558117c4c50 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68eda78b80 10 client.16392 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68eda78b80 1 -- 172.21.15.121:0/4169844482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555811848f50 con 0x5558117c4c50 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68bbfff700 1 -- 172.21.15.121:0/4169844482 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68d40263b0 con 0x5558117c4c50 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68bbfff700 1 -- 172.21.15.121:0/4169844482 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f68d009ca90 con 0x5558117c4c50 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68eda78b80 10 client.16392 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68eda78b80 10 client.16392 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68eda78b80 20 client.16392 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:06.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68eda78b80 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x555811849590 0x55581184b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:06.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68eda78b80 4 client.16392 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5eddb4ce", "release": true, "await": true}] 2024-06-10T00:34:06.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.060+0000 7f68eda78b80 1 -- 172.21.15.121:0/4169844482 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5eddb4ce", "release": true, "await": true}) -- 0x55581171d3d0 con 0x555811849590 2024-06-10T00:34:06.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.061+0000 7f68dbfff700 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x555811849590 0x55581184b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:06.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.061+0000 7f68bbfff700 10 client.16392 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:06.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:06.061+0000 7f68dbfff700 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x555811849590 0x55581184b9d0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6c60fca8f58eee3 server_cookie=abfa418945e2ca32 in_seq=0 out_seq=0 2024-06-10T00:34:06.286 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:06 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:06.286 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:06 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:06.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:06 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:06.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:06 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:06 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:06 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:07.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.042+0000 7f68bbfff700 1 -- 172.21.15.121:0/4169844482 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55581171d3d0 con 0x555811849590 2024-06-10T00:34:07.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.042+0000 7f68bbfff700 10 client.16392 handle_command_reply: tid=0 2024-06-10T00:34:07.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.042+0000 7f68eda78b80 1 client.16392 shutdown 2024-06-10T00:34:07.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68eda78b80 1 -- 172.21.15.121:0/4169844482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 msgr2=0x5558117d7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:07.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68eda78b80 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117d7280 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f68d00cda80 tx=0x7f68d00a3040 comp rx=0 tx=0).stop 2024-06-10T00:34:07.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68eda78b80 1 -- 172.21.15.121:0/4169844482 shutdown_connections 2024-06-10T00:34:07.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68eda78b80 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x555811849590 0x55581184b9d0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:07.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68eda78b80 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558117cae20 0x5558117e5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:07.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68eda78b80 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5558117c4c50 0x5558117d7280 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:07.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68eda78b80 1 --2- 172.21.15.121:0/4169844482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5558117ca500 0x5558117e5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:07.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68bbfff700 0 client.16392 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:07.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68eda78b80 1 -- 172.21.15.121:0/4169844482 >> 172.21.15.121:0/4169844482 conn(0x5558116f1180 msgr2=0x5558117ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:07.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68eda78b80 1 -- 172.21.15.121:0/4169844482 shutdown_connections 2024-06-10T00:34:07.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68eda78b80 1 -- 172.21.15.121:0/4169844482 wait complete. 2024-06-10T00:34:07.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.043+0000 7f68eda78b80 20 client.16392 trim_cache size 0 max 16384 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 30, 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 16, 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: "5eddb4ce": { 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 16, 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:34:07.045 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 149.4, 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:07.046 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:34:07.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 -- 172.21.15.121:0/1229248998 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68cc0a2e10 msgr2=0x7f68cc0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:07.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68cc0a2e10 0x7f68cc0a52d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f68dc19a280 tx=0x7f68c800b410 comp rx=0 tx=0).stop 2024-06-10T00:34:07.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 -- 172.21.15.121:0/1229248998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68dc06aa00 msgr2=0x7f68dc19ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:07.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68dc06aa00 0x7f68dc19ea60 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f68d00038b0 tx=0x7f68d0009f50 comp rx=0 tx=0).stop 2024-06-10T00:34:07.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 -- 172.21.15.121:0/1229248998 shutdown_connections 2024-06-10T00:34:07.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68cc0a2e10 0x7f68cc0a52d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:07.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68dc1084f0 0x7f68dc198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:07.047 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68dc06aa00 0x7f68dc19ea60 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:07.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 --2- 172.21.15.121:0/1229248998 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68dc06b3c0 0x7f68dc19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:07.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 -- 172.21.15.121:0/1229248998 >> 172.21.15.121:0/1229248998 conn(0x7f68dc078300 msgr2=0x7f68dc1000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:07.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 -- 172.21.15.121:0/1229248998 shutdown_connections 2024-06-10T00:34:07.048 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:07.044+0000 7f68e31f5700 1 -- 172.21.15.121:0/1229248998 wait complete. 2024-06-10T00:34:07.295 INFO:tasks.workunit.client.0.smithi121.stdout: 246 373121 3 259509 7 118839 17 2024-06-10T00:34:07.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:07 smithi121 ceph-mon[32378]: pgmap v187: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 1.4 MiB/s wr, 433 op/s 2024-06-10T00:34:07.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:07 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:07.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:07 smithi135 ceph-mon[36424]: pgmap v187: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 1.4 MiB/s wr, 433 op/s 2024-06-10T00:34:07.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:07 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:07 smithi193 ceph-mon[36716]: pgmap v187: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 1.4 MiB/s wr, 433 op/s 2024-06-10T00:34:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:07 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:34:08.884 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5eddb4ce', total seconds elapsed: 42.6 2024-06-10T00:34:08.884 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 29.45 2024-06-10T00:34:08.885 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:34:09.385 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:09 smithi121 ceph-mon[32378]: pgmap v188: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.2 MiB/s wr, 299 op/s 2024-06-10T00:34:09.393 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.393+0000 7fd1f1277700 1 -- 172.21.15.121:0/3712099686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ec101390 msgr2=0x7fd1ec101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:09.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.393+0000 7fd1f1277700 1 --2- 172.21.15.121:0/3712099686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ec101390 0x7fd1ec101810 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009990 tx=0x7fd1e0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.393+0000 7fd1f1277700 1 -- 172.21.15.121:0/3712099686 shutdown_connections 2024-06-10T00:34:09.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.393+0000 7fd1f1277700 1 --2- 172.21.15.121:0/3712099686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1ec101d50 0x7fd1ec10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.393+0000 7fd1f1277700 1 --2- 172.21.15.121:0/3712099686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1ec107400 0x7fd1ec1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.393+0000 7fd1f1277700 1 --2- 172.21.15.121:0/3712099686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ec101390 0x7fd1ec101810 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.393+0000 7fd1f1277700 1 -- 172.21.15.121:0/3712099686 >> 172.21.15.121:0/3712099686 conn(0x7fd1ec078340 msgr2=0x7fd1ec078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:09.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.393+0000 7fd1f1277700 1 -- 172.21.15.121:0/3712099686 shutdown_connections 2024-06-10T00:34:09.394 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.393+0000 7fd1f1277700 1 -- 172.21.15.121:0/3712099686 wait complete. 2024-06-10T00:34:09.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.394+0000 7fd1f1277700 1 Processor -- start 2024-06-10T00:34:09.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.394+0000 7fd1f1277700 1 -- start start 2024-06-10T00:34:09.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1f1277700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1ec101390 0x7fd1ec199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1f1277700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ec101d50 0x7fd1ec199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1f1277700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1ec107400 0x7fd1ec1a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1f1277700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec1116a0 con 0x7fd1ec101d50 2024-06-10T00:34:09.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1f1277700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec111520 con 0x7fd1ec101390 2024-06-10T00:34:09.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1f1277700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec111820 con 0x7fd1ec107400 2024-06-10T00:34:09.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1ec101390 0x7fd1ec199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1ea7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ec101d50 0x7fd1ec199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1ec101390 0x7fd1ec199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:42170/0 (socket says 172.21.15.121:42170) 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1eaffd700 1 -- 172.21.15.121:0/2513471810 learned_addr learned my addr 172.21.15.121:0/2513471810 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1eb7fe700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1ec107400 0x7fd1ec1a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1eaffd700 1 -- 172.21.15.121:0/2513471810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1ec107400 msgr2=0x7fd1ec1a4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1ec107400 0x7fd1ec1a4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1eaffd700 1 -- 172.21.15.121:0/2513471810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ec101d50 msgr2=0x7fd1ec199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ec101d50 0x7fd1ec199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.395+0000 7fd1eaffd700 1 -- 172.21.15.121:0/2513471810 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ec1a5040 con 0x7fd1ec101390 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.396+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1ec101390 0x7fd1ec199a80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00e9a0 tx=0x7fd1dc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.396+0000 7fd1d3fff700 1 -- 172.21.15.121:0/2513471810 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc00ca10 con 0x7fd1ec101390 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.396+0000 7fd1f1277700 1 -- 172.21.15.121:0/2513471810 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1ec1a5270 con 0x7fd1ec101390 2024-06-10T00:34:09.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.396+0000 7fd1d3fff700 1 -- 172.21.15.121:0/2513471810 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1dc00cbb0 con 0x7fd1ec101390 2024-06-10T00:34:09.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.396+0000 7fd1d3fff700 1 -- 172.21.15.121:0/2513471810 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc018680 con 0x7fd1ec101390 2024-06-10T00:34:09.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.396+0000 7fd1f1277700 1 -- 172.21.15.121:0/2513471810 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1ec1a57d0 con 0x7fd1ec101390 2024-06-10T00:34:09.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.398+0000 7fd1d3fff700 1 -- 172.21.15.121:0/2513471810 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd1dc010430 con 0x7fd1ec101390 2024-06-10T00:34:09.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.399+0000 7fd1d3fff700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1d40a2e10 0x7fd1d40a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.399+0000 7fd1d3fff700 1 -- 172.21.15.121:0/2513471810 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fd1dc0cf910 con 0x7fd1ec101390 2024-06-10T00:34:09.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.399+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1d40a2e10 0x7fd1d40a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.400+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1d40a2e10 0x7fd1d40a52d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000b2e0 tx=0x7fd1e0005960 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:09.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.400+0000 7fd1fbafab80 1 Processor -- start 2024-06-10T00:34:09.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.400+0000 7fd1fbafab80 1 -- start start 2024-06-10T00:34:09.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.400+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29802340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.400+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29802880 0x562a297fc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.400+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a297fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.400+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562a296eef60 con 0x562a297fc7a0 2024-06-10T00:34:09.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.400+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x562a2974d850 con 0x562a29802880 2024-06-10T00:34:09.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.400+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562a296b5460 con 0x562a29801f60 2024-06-10T00:34:09.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.400+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29802340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.400+0000 7fd1eb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a297fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29802340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:35128/0 (socket says 172.21.15.121:35128) 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1eaffd700 1 -- 172.21.15.121:0/984130060 learned_addr learned my addr 172.21.15.121:0/984130060 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/984130060 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29802880 0x562a297fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1eaffd700 1 -- 172.21.15.121:0/984130060 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29802880 msgr2=0x562a297fc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/984130060 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29802880 0x562a297fc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1eaffd700 1 -- 172.21.15.121:0/984130060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 msgr2=0x562a297fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/984130060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a297fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1eaffd700 1 -- 172.21.15.121:0/984130060 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a2969c8d0 con 0x562a29801f60 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/984130060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29802340 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc09ba30 tx=0x7fd1dc09baa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31c9bc6b1fbbee4c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1d17fa700 1 -- 172.21.15.121:0/984130060 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc00cd60 con 0x562a29801f60 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1d17fa700 1 -- 172.21.15.121:0/984130060 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1dc09bd00 con 0x562a29801f60 2024-06-10T00:34:09.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.401+0000 7fd1d17fa700 1 -- 172.21.15.121:0/984130060 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc010960 con 0x562a29801f60 2024-06-10T00:34:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.402+0000 7fd1fbafab80 1 -- 172.21.15.121:0/984130060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 msgr2=0x562a29802340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:09.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.402+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/984130060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29802340 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc09ba30 tx=0x7fd1dc09baa0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.402+0000 7fd1fbafab80 1 -- 172.21.15.121:0/984130060 shutdown_connections 2024-06-10T00:34:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.402+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/984130060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a297fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.402+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/984130060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29802340 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.402+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/984130060 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29802880 0x562a297fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.402+0000 7fd1fbafab80 1 -- 172.21.15.121:0/984130060 >> 172.21.15.121:0/984130060 conn(0x562a29729080 msgr2=0x562a297f7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.402+0000 7fd1fbafab80 1 -- 172.21.15.121:0/984130060 shutdown_connections 2024-06-10T00:34:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.402+0000 7fd1fbafab80 1 -- 172.21.15.121:0/984130060 wait complete. 2024-06-10T00:34:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1fbafab80 1 -- start start 2024-06-10T00:34:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1fbafab80 1 Processor -- start 2024-06-10T00:34:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a297fc7a0 0x562a29808630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29805930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a29802880 0x562a29805e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562a296edfc0 con 0x562a29802880 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x562a29749810 con 0x562a297fc7a0 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562a29809200 con 0x562a29801f60 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1ea7fc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29805930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a297fc7a0 0x562a29808630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a297fc7a0 0x562a29808630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:42194/0 (socket says 172.21.15.121:42194) 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1eb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a29802880 0x562a29805e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1eaffd700 1 -- 172.21.15.121:0/266006471 learned_addr learned my addr 172.21.15.121:0/266006471 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1ea7fc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29805930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:35144/0 (socket says 172.21.15.121:35144) 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.403+0000 7fd1fbafab80 10 client.0 _ll_register_callbacks cb 0x562a296e35b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1ea7fc700 1 -- 172.21.15.121:0/266006471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a297fc7a0 msgr2=0x562a29808630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a297fc7a0 0x562a29808630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1ea7fc700 1 -- 172.21.15.121:0/266006471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a29802880 msgr2=0x562a29805e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a29802880 0x562a29805e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1ea7fc700 1 -- 172.21.15.121:0/266006471 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a29713750 con 0x562a29801f60 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a297fc7a0 0x562a29808630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1eb7fe700 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a29802880 0x562a29805e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29805930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00059d0 tx=0x7fd1e0009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87ae5e217294b0e7 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1ceff5700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1ceff5700 1 -- 172.21.15.121:0/266006471 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e000f460 con 0x562a29801f60 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1ceff5700 1 -- 172.21.15.121:0/266006471 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fd1e000f600 con 0x562a29801f60 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1fbafab80 1 -- 172.21.15.121:0/266006471 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1dc09bd00 con 0x562a29801f60 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1ceff5700 1 -- 172.21.15.121:0/266006471 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd1e003c050 con 0x562a29801f60 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.404+0000 7fd1fbafab80 10 client.25873 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.405+0000 7fd1fbafab80 1 -- 172.21.15.121:0/266006471 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562a29813a20 con 0x562a29801f60 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.405+0000 7fd1ceff5700 1 -- 172.21.15.121:0/266006471 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1dc09bd00 con 0x562a29801f60 2024-06-10T00:34:09.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.405+0000 7fd1ceff5700 1 -- 172.21.15.121:0/266006471 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd1e00226f0 con 0x562a29801f60 2024-06-10T00:34:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.405+0000 7fd1fbafab80 10 client.25873 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.405+0000 7fd1fbafab80 10 client.25873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:34:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.405+0000 7fd1fbafab80 20 client.25873 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.405+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562a298160b0 0x562a298184d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.405+0000 7fd1fbafab80 4 client.25873 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:34:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.405+0000 7fd1fbafab80 1 -- 172.21.15.121:0/266006471 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a2975ad50 con 0x562a298160b0 2024-06-10T00:34:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.405+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562a298160b0 0x562a298184d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.406+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562a298160b0 0x562a298184d0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=757df8226c65223c server_cookie=28e9c5b3f3beef4c in_seq=0 out_seq=0 2024-06-10T00:34:09.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.406+0000 7fd1ceff5700 10 client.25873 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:09.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1ceff5700 1 -- 172.21.15.121:0/266006471 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x562a2975ad50 con 0x562a298160b0 2024-06-10T00:34:09.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1ceff5700 10 client.25873 handle_command_reply: tid=0 2024-06-10T00:34:09.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1fbafab80 1 client.25873 shutdown 2024-06-10T00:34:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1fbafab80 1 -- 172.21.15.121:0/266006471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 msgr2=0x562a29805930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29805930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00059d0 tx=0x7fd1e0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1fbafab80 1 -- 172.21.15.121:0/266006471 shutdown_connections 2024-06-10T00:34:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562a298160b0 0x562a298184d0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a29802880 0x562a29805e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a297fc7a0 0x562a29808630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1ceff5700 0 client.25873 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/266006471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29805930 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.407+0000 7fd1fbafab80 1 -- 172.21.15.121:0/266006471 >> 172.21.15.121:0/266006471 conn(0x562a29729080 msgr2=0x562a297f6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:09.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.408+0000 7fd1fbafab80 1 -- 172.21.15.121:0/266006471 shutdown_connections 2024-06-10T00:34:09.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.408+0000 7fd1fbafab80 1 -- 172.21.15.121:0/266006471 wait complete. 2024-06-10T00:34:09.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.408+0000 7fd1fbafab80 20 client.25873 trim_cache size 0 max 16384 2024-06-10T00:34:09.429 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:09.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1fbafab80 1 Processor -- start 2024-06-10T00:34:09.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1fbafab80 1 -- start start 2024-06-10T00:34:09.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a297fc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29805f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29802880 0x562a298034f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec1aec60 con 0x562a297fc7a0 2024-06-10T00:34:09.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec1ac490 con 0x562a29802880 2024-06-10T00:34:09.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec1aa300 con 0x562a29801f60 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a297fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a297fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50782/0 (socket says 172.21.15.121:50782) 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1eaffd700 1 -- 172.21.15.121:0/1009990433 learned_addr learned my addr 172.21.15.121:0/1009990433 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/1009990433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29805f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1eb7fe700 1 --2- 172.21.15.121:0/1009990433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29802880 0x562a298034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1ea7fc700 1 -- 172.21.15.121:0/1009990433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29802880 msgr2=0x562a298034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/1009990433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29802880 0x562a298034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1ea7fc700 1 -- 172.21.15.121:0/1009990433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 msgr2=0x562a297fc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/1009990433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a297fc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1ea7fc700 1 -- 172.21.15.121:0/1009990433 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a29805700 con 0x562a29801f60 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.433+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/1009990433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a297fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.434+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/1009990433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29805f60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0000c00 tx=0x7fd1e000bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daf8778ef1042d6f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:09.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.434+0000 7fd1d0ff9700 1 -- 172.21.15.121:0/1009990433 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e00223f0 con 0x562a29801f60 2024-06-10T00:34:09.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.434+0000 7fd1d0ff9700 1 -- 172.21.15.121:0/1009990433 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1e0033070 con 0x562a29801f60 2024-06-10T00:34:09.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.434+0000 7fd1d0ff9700 1 -- 172.21.15.121:0/1009990433 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e000fd30 con 0x562a29801f60 2024-06-10T00:34:09.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.435+0000 7fd1fbafab80 1 -- 172.21.15.121:0/1009990433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 msgr2=0x562a29805f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.435+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/1009990433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29805f60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0000c00 tx=0x7fd1e000bcf0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.435+0000 7fd1fbafab80 1 -- 172.21.15.121:0/1009990433 shutdown_connections 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.435+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/1009990433 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29802880 0x562a298034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.435+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/1009990433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a297fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.435+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/1009990433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29801f60 0x562a29805f60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.435+0000 7fd1fbafab80 1 -- 172.21.15.121:0/1009990433 >> 172.21.15.121:0/1009990433 conn(0x562a29729080 msgr2=0x562a29804220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.435+0000 7fd1fbafab80 1 -- 172.21.15.121:0/1009990433 shutdown_connections 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.435+0000 7fd1fbafab80 1 -- 172.21.15.121:0/1009990433 wait complete. 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.435+0000 7fd1fbafab80 1 -- start start 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.435+0000 7fd1fbafab80 1 Processor -- start 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a2980e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29801f60 0x562a2980edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1fbafab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29802880 0x562a2981d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec1aa950 con 0x562a297fc7a0 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec1abdc0 con 0x562a29801f60 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1fbafab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ec1ab020 con 0x562a29802880 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1fbafab80 10 client.0 _ll_register_callbacks cb 0x562a296e35b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a2980e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1ea7fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29801f60 0x562a2980edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a2980e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50798/0 (socket says 172.21.15.121:50798) 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1ea7fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29801f60 0x562a2980edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:42222/0 (socket says 172.21.15.121:42222) 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1eaffd700 1 -- 172.21.15.121:0/2639703169 learned_addr learned my addr 172.21.15.121:0/2639703169 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1eb7fe700 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29802880 0x562a2981d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1ea7fc700 1 -- 172.21.15.121:0/2639703169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29802880 msgr2=0x562a2981d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29802880 0x562a2981d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1ea7fc700 1 -- 172.21.15.121:0/2639703169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 msgr2=0x562a2980e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:09.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a2980e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1ea7fc700 1 -- 172.21.15.121:0/2639703169 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a2981dd60 con 0x562a29801f60 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1eb7fe700 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29802880 0x562a2981d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.436+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a2980e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1ea7fc700 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29801f60 0x562a2980edd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x562a29806d60 tx=0x7fd1e0003f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2c59ac86b566e16 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1ceff5700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1ceff5700 1 -- 172.21.15.121:0/2639703169 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e000fbb0 con 0x562a29801f60 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1ceff5700 1 -- 172.21.15.121:0/2639703169 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fd1e000fd50 con 0x562a29801f60 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1fbafab80 1 -- 172.21.15.121:0/2639703169 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1e0033070 con 0x562a29801f60 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1ceff5700 1 -- 172.21.15.121:0/2639703169 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd1e00223f0 con 0x562a29801f60 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1fbafab80 10 client.25979 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1fbafab80 1 -- 172.21.15.121:0/2639703169 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562a298809e0 con 0x562a29801f60 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1ceff5700 1 -- 172.21.15.121:0/2639703169 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1e0033070 con 0x562a29801f60 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1ceff5700 1 -- 172.21.15.121:0/2639703169 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fd1e0048aa0 con 0x562a29801f60 2024-06-10T00:34:09.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1fbafab80 10 client.25979 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:09.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1fbafab80 10 client.25979 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:34:09.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1fbafab80 20 client.25979 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:09.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562a29881000 0x562a29883420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:09.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1fbafab80 4 client.25979 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:34:09.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.437+0000 7fd1fbafab80 1 -- 172.21.15.121:0/2639703169 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562a297552d0 con 0x562a29881000 2024-06-10T00:34:09.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.438+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562a29881000 0x562a29883420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:09.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.438+0000 7fd1eaffd700 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562a29881000 0x562a29883420 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35690e4632acedf7 server_cookie=524b3c2a498a7ecc in_seq=0 out_seq=0 2024-06-10T00:34:09.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.438+0000 7fd1ceff5700 10 client.25979 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:09.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1ceff5700 1 -- 172.21.15.121:0/2639703169 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562a297552d0 con 0x562a29881000 2024-06-10T00:34:09.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1ceff5700 10 client.25979 handle_command_reply: tid=0 2024-06-10T00:34:09.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 1 client.25979 shutdown 2024-06-10T00:34:09.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 1 -- 172.21.15.121:0/2639703169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29801f60 msgr2=0x562a2980edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:09.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29801f60 0x562a2980edd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x562a29806d60 tx=0x7fd1e0003f20 comp rx=0 tx=0).stop 2024-06-10T00:34:09.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 1 -- 172.21.15.121:0/2639703169 shutdown_connections 2024-06-10T00:34:09.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562a29881000 0x562a29883420 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1ceff5700 0 client.25979 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:09.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a29802880 0x562a2981d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.444 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562a297fc7a0 0x562a2980e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 1 --2- 172.21.15.121:0/2639703169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562a29801f60 0x562a2980edd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 1 -- 172.21.15.121:0/2639703169 >> 172.21.15.121:0/2639703169 conn(0x562a29729080 msgr2=0x562a29804ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:09.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 1 -- 172.21.15.121:0/2639703169 shutdown_connections 2024-06-10T00:34:09.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 1 -- 172.21.15.121:0/2639703169 wait complete. 2024-06-10T00:34:09.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.440+0000 7fd1fbafab80 20 client.25979 trim_cache size 0 max 16384 2024-06-10T00:34:09.445 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:34:09.445 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-10T00:34:09.445 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": { 2024-06-10T00:34:09.445 INFO:teuthology.orchestra.run.smithi121.stdout: "38e051c5-0ec8-45cc-b0bf-2191994b1d58": { 2024-06-10T00:34:09.445 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/", 2024-06-10T00:34:09.445 INFO:teuthology.orchestra.run.smithi121.stdout: "tag": "38e051c5-0ec8-45cc-b0bf-2191994b1d58", 2024-06-10T00:34:09.446 INFO:teuthology.orchestra.run.smithi121.stdout: "options": "recursive,force" 2024-06-10T00:34:09.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:09.446 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:09.446 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:34:09.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 -- 172.21.15.121:0/2513471810 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1d40a2e10 msgr2=0x7fd1d40a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:09.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1d40a2e10 0x7fd1d40a52d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000b2e0 tx=0x7fd1e0005960 comp rx=0 tx=0).stop 2024-06-10T00:34:09.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 -- 172.21.15.121:0/2513471810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1ec101390 msgr2=0x7fd1ec199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:09.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1ec101390 0x7fd1ec199a80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00e9a0 tx=0x7fd1dc00ecb0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 -- 172.21.15.121:0/2513471810 shutdown_connections 2024-06-10T00:34:09.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1d40a2e10 0x7fd1d40a52d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1ec107400 0x7fd1ec1a4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1ec101390 0x7fd1ec199a80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 --2- 172.21.15.121:0/2513471810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ec101d50 0x7fd1ec199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:09.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 -- 172.21.15.121:0/2513471810 >> 172.21.15.121:0/2513471810 conn(0x7fd1ec078340 msgr2=0x7fd1ec0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:09.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 -- 172.21.15.121:0/2513471810 shutdown_connections 2024-06-10T00:34:09.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:09.441+0000 7fd1f1277700 1 -- 172.21.15.121:0/2513471810 wait complete. 2024-06-10T00:34:09.507 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:09 smithi135 ceph-mon[36424]: pgmap v188: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.2 MiB/s wr, 299 op/s 2024-06-10T00:34:09.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:09 smithi193 ceph-mon[36716]: pgmap v188: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.2 MiB/s wr, 299 op/s 2024-06-10T00:34:10.501 INFO:tasks.cephfs.filesystem:scrub status for tag:38e051c5-0ec8-45cc-b0bf-2191994b1d58 - {'path': '/', 'tag': '38e051c5-0ec8-45cc-b0bf-2191994b1d58', 'options': 'recursive,force'} 2024-06-10T00:34:11.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:11 smithi121 ceph-mon[32378]: pgmap v189: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.7 MiB/s wr, 316 op/s 2024-06-10T00:34:11.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:11 smithi135 ceph-mon[36424]: pgmap v189: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.7 MiB/s wr, 316 op/s 2024-06-10T00:34:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:11 smithi193 ceph-mon[36716]: pgmap v189: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.7 MiB/s wr, 316 op/s 2024-06-10T00:34:12.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.374+0000 7f351a8b3700 1 -- 172.21.15.121:0/1043511283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f350c0958a0 msgr2=0x7f350c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:12.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.374+0000 7f351a8b3700 1 --2- 172.21.15.121:0/1043511283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f350c0958a0 0x7f350c0047d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f35140517a0 tx=0x7f3514068aa0 comp rx=0 tx=0).stop 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.375+0000 7f351a8b3700 1 -- 172.21.15.121:0/1043511283 shutdown_connections 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.375+0000 7f351a8b3700 1 --2- 172.21.15.121:0/1043511283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f350c0958a0 0x7f350c0047d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.375+0000 7f351a8b3700 1 --2- 172.21.15.121:0/1043511283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c09aca0 0x7f350c09b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.375+0000 7f351a8b3700 1 --2- 172.21.15.121:0/1043511283 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f350c094ee0 0x7f350c095360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.375+0000 7f351a8b3700 1 -- 172.21.15.121:0/1043511283 >> 172.21.15.121:0/1043511283 conn(0x7f350c00a7a0 msgr2=0x7f350c00abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.375+0000 7f351a8b3700 1 -- 172.21.15.121:0/1043511283 shutdown_connections 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.375+0000 7f351a8b3700 1 -- 172.21.15.121:0/1043511283 wait complete. 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.376+0000 7f351a8b3700 1 Processor -- start 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.376+0000 7f351a8b3700 1 -- start start 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.377+0000 7f351a8b3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f350c094ee0 0x7f350c0a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.377+0000 7f351a8b3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f350c0958a0 0x7f350c0a42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:12.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.377+0000 7f351a8b3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c09aca0 0x7f350c0a4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.377+0000 7f351a8b3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f350c0a8260 con 0x7f350c09aca0 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.377+0000 7f351a8b3700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f350c0a80e0 con 0x7f350c0958a0 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.377+0000 7f351a8b3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f350c0a83e0 con 0x7f350c094ee0 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.377+0000 7f3518e50700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c09aca0 0x7f350c0a4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.378+0000 7f3518e50700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c09aca0 0x7f350c0a4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50822/0 (socket says 172.21.15.121:50822) 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.378+0000 7f3518e50700 1 -- 172.21.15.121:0/267718058 learned_addr learned my addr 172.21.15.121:0/267718058 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.378+0000 7f3518e50700 1 -- 172.21.15.121:0/267718058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f350c094ee0 msgr2=0x7f350c0a3da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.378+0000 7f3518e50700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f350c094ee0 0x7f350c0a3da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.378+0000 7f3518e50700 1 -- 172.21.15.121:0/267718058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f350c0958a0 msgr2=0x7f350c0a42e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.378+0000 7f3518e50700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f350c0958a0 0x7f350c0a42e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.378+0000 7f3518e50700 1 -- 172.21.15.121:0/267718058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c14fdf0 con 0x7f350c09aca0 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.379+0000 7f3518e50700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c09aca0 0x7f350c0a4820 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3514065ce0 tx=0x7f3514074ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:12.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.380+0000 7f35117fa700 1 -- 172.21.15.121:0/267718058 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514084030 con 0x7f350c09aca0 2024-06-10T00:34:12.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.380+0000 7f35117fa700 1 -- 172.21.15.121:0/267718058 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f351406d070 con 0x7f350c09aca0 2024-06-10T00:34:12.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.380+0000 7f35117fa700 1 -- 172.21.15.121:0/267718058 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514063030 con 0x7f350c09aca0 2024-06-10T00:34:12.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.380+0000 7f351a8b3700 1 -- 172.21.15.121:0/267718058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350c150080 con 0x7f350c09aca0 2024-06-10T00:34:12.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.380+0000 7f351a8b3700 1 -- 172.21.15.121:0/267718058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f350c1504d0 con 0x7f350c09aca0 2024-06-10T00:34:12.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.382+0000 7f351a8b3700 1 -- 172.21.15.121:0/267718058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f350c096330 con 0x7f350c09aca0 2024-06-10T00:34:12.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.386+0000 7f35117fa700 1 -- 172.21.15.121:0/267718058 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3514087070 con 0x7f350c09aca0 2024-06-10T00:34:12.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.387+0000 7f35117fa700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35040a2ee0 0x7f35040a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:12.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.387+0000 7f35117fa700 1 -- 172.21.15.121:0/267718058 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f3514134920 con 0x7f350c09aca0 2024-06-10T00:34:12.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.388+0000 7f3513fff700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35040a2ee0 0x7f35040a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:12.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.389+0000 7f3513fff700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35040a2ee0 0x7f35040a53a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3508009680 tx=0x7f3508009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:12.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.393+0000 7f35117fa700 1 -- 172.21.15.121:0/267718058 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f35141012c0 con 0x7f350c09aca0 2024-06-10T00:34:12.708 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.708+0000 7f351a8b3700 1 -- 172.21.15.121:0/267718058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f350c150900 con 0x7f350c09aca0 2024-06-10T00:34:12.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.709+0000 7f35117fa700 1 -- 172.21.15.121:0/267718058 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11891 (secure 0 0 0) 0x7f3514100bc0 con 0x7f350c09aca0 2024-06-10T00:34:12.709 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:34:12.709 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:33:03.460049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.711+0000 7f351a8b3700 1 -- 172.21.15.121:0/267718058 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35040a2ee0 msgr2=0x7f35040a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.711+0000 7f351a8b3700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35040a2ee0 0x7f35040a53a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3508009680 tx=0x7f3508009260 comp rx=0 tx=0).stop 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.711+0000 7f351a8b3700 1 -- 172.21.15.121:0/267718058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c09aca0 msgr2=0x7f350c0a4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.711+0000 7f351a8b3700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c09aca0 0x7f350c0a4820 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3514065ce0 tx=0x7f3514074ac0 comp rx=0 tx=0).stop 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.711+0000 7f351a8b3700 1 -- 172.21.15.121:0/267718058 shutdown_connections 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.711+0000 7f351a8b3700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35040a2ee0 0x7f35040a53a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.711+0000 7f351a8b3700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f350c09aca0 0x7f350c0a4820 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.711+0000 7f351a8b3700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f350c094ee0 0x7f350c0a3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.711+0000 7f351a8b3700 1 --2- 172.21.15.121:0/267718058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f350c0958a0 0x7f350c0a42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.711+0000 7f351a8b3700 1 -- 172.21.15.121:0/267718058 >> 172.21.15.121:0/267718058 conn(0x7f350c00a7a0 msgr2=0x7f350c098830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.711+0000 7f351a8b3700 1 -- 172.21.15.121:0/267718058 shutdown_connections 2024-06-10T00:34:12.712 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:12.712+0000 7f351a8b3700 1 -- 172.21.15.121:0/267718058 wait complete. 2024-06-10T00:34:12.713 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-06-10T00:34:13.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:13 smithi121 ceph-mon[32378]: pgmap v190: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-06-10T00:34:13.542 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:13 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/267718058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:13 smithi135 ceph-mon[36424]: pgmap v190: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-06-10T00:34:13.543 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:13 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/267718058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:13.545 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db / --timeout 90 --expiration 149.45 --await-for 45.0' 2024-06-10T00:34:13.545 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db / --timeout 90 --expiration 149.45 --await-for 45.0 2024-06-10T00:34:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:13 smithi193 ceph-mon[36716]: pgmap v190: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 183 op/s 2024-06-10T00:34:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:13 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/267718058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:13.952 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:15.362 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:15 smithi121 ceph-mon[32378]: pgmap v191: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-06-10T00:34:15.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:15 smithi135 ceph-mon[36424]: pgmap v191: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-06-10T00:34:15.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:15 smithi193 ceph-mon[36716]: pgmap v191: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 121 op/s 2024-06-10T00:34:15.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.807+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/3655961306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfc01027a0 msgr2=0x7fcfc0106020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:15.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.807+0000 7fcfc4a8a700 1 --2- 172.21.15.121:0/3655961306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfc01027a0 0x7fcfc0106020 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40099c0 tx=0x7fcfb4009cd0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.807+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/3655961306 shutdown_connections 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.807+0000 7fcfc4a8a700 1 --2- 172.21.15.121:0/3655961306 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfc0106750 0x7fcfc010e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.807+0000 7fcfc4a8a700 1 --2- 172.21.15.121:0/3655961306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfc0101df0 0x7fcfc01021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.807+0000 7fcfc4a8a700 1 --2- 172.21.15.121:0/3655961306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfc01027a0 0x7fcfc0106020 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.807+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/3655961306 >> 172.21.15.121:0/3655961306 conn(0x7fcfc0077a30 msgr2=0x7fcfc00fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.807+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/3655961306 shutdown_connections 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.807+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/3655961306 wait complete. 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.808+0000 7fcfc4a8a700 1 Processor -- start 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.808+0000 7fcfc4a8a700 1 -- start start 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfc4a8a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfc0101df0 0x7fcfc019d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfc4a8a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfc01027a0 0x7fcfc019dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfc4a8a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfc0106750 0x7fcfc0197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfc4a8a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0111680 con 0x7fcfc01027a0 2024-06-10T00:34:15.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfc4a8a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0111500 con 0x7fcfc0101df0 2024-06-10T00:34:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfc4a8a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0111800 con 0x7fcfc0106750 2024-06-10T00:34:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfbe59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfc0101df0 0x7fcfc019d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfbe59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfc0101df0 0x7fcfc019d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:42252/0 (socket says 172.21.15.121:42252) 2024-06-10T00:34:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfbe59c700 1 -- 172.21.15.121:0/2616976478 learned_addr learned my addr 172.21.15.121:0/2616976478 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfbdd9b700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfc01027a0 0x7fcfc019dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfbed9d700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfc0106750 0x7fcfc0197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfbe59c700 1 -- 172.21.15.121:0/2616976478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfc0106750 msgr2=0x7fcfc0197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfc0106750 0x7fcfc0197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfbe59c700 1 -- 172.21.15.121:0/2616976478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfc01027a0 msgr2=0x7fcfc019dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:15.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfc01027a0 0x7fcfc019dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfbe59c700 1 -- 172.21.15.121:0/2616976478 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc0198040 con 0x7fcfc0101df0 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.809+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfc0101df0 0x7fcfc019d690 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000e9c0 tx=0x7fcfb000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.810+0000 7fcfaf7fe700 1 -- 172.21.15.121:0/2616976478 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb000cbd0 con 0x7fcfc0101df0 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.810+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/2616976478 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfc0198330 con 0x7fcfc0101df0 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.810+0000 7fcfaf7fe700 1 -- 172.21.15.121:0/2616976478 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb000ee60 con 0x7fcfc0101df0 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.810+0000 7fcfaf7fe700 1 -- 172.21.15.121:0/2616976478 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb0005630 con 0x7fcfc0101df0 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.810+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/2616976478 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfc006beb0 con 0x7fcfc0101df0 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.813+0000 7fcfaf7fe700 1 -- 172.21.15.121:0/2616976478 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fcfb0004750 con 0x7fcfc0101df0 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.814+0000 7fcfaf7fe700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcfa80a2e30 0x7fcfa80a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.815+0000 7fcfbdd9b700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcfa80a2e30 0x7fcfa80a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.815+0000 7fcfaf7fe700 1 -- 172.21.15.121:0/2616976478 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcfb00e0a90 con 0x7fcfc0101df0 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.815+0000 7fcfbdd9b700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcfa80a2e30 0x7fcfa80a52f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0103e30 tx=0x7fcfb4005fb0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:15.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.815+0000 7fcfcf30db80 1 Processor -- start 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.815+0000 7fcfcf30db80 1 -- start start 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19eef0 0x55fe3c19f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19f810 0x55fe3c199140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c199680 0x55fe3c199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c0e89b0 con 0x55fe3c19f810 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c0908c0 con 0x55fe3c19eef0 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c0e4970 con 0x55fe3c199680 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbdd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19f810 0x55fe3c199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbdd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19f810 0x55fe3c199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50854/0 (socket says 172.21.15.121:50854) 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbdd9b700 1 -- 172.21.15.121:0/2366520501 learned_addr learned my addr 172.21.15.121:0/2366520501 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/2366520501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19eef0 0x55fe3c19f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbed9d700 1 --2- 172.21.15.121:0/2366520501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c199680 0x55fe3c199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:15.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbdd9b700 1 -- 172.21.15.121:0/2366520501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c199680 msgr2=0x55fe3c199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbdd9b700 1 --2- 172.21.15.121:0/2366520501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c199680 0x55fe3c199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbdd9b700 1 -- 172.21.15.121:0/2366520501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19eef0 msgr2=0x55fe3c19f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbdd9b700 1 --2- 172.21.15.121:0/2366520501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19eef0 0x55fe3c19f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbdd9b700 1 -- 172.21.15.121:0/2366520501 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe3c0870f0 con 0x55fe3c19f810 2024-06-10T00:34:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/2366520501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19eef0 0x55fe3c19f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.816+0000 7fcfbdd9b700 1 --2- 172.21.15.121:0/2366520501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19f810 0x55fe3c199140 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4015000 tx=0x7fcfb401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce14f4d94b5d7740 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.817+0000 7fcfacff9700 1 -- 172.21.15.121:0/2366520501 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb4034070 con 0x55fe3c19f810 2024-06-10T00:34:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.817+0000 7fcfacff9700 1 -- 172.21.15.121:0/2366520501 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb400bca0 con 0x55fe3c19f810 2024-06-10T00:34:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.817+0000 7fcfacff9700 1 -- 172.21.15.121:0/2366520501 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb402f7a0 con 0x55fe3c19f810 2024-06-10T00:34:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.818+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2366520501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19f810 msgr2=0x55fe3c199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:15.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.818+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/2366520501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19f810 0x55fe3c199140 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4015000 tx=0x7fcfb401f000 comp rx=0 tx=0).stop 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.818+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2366520501 shutdown_connections 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.818+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/2366520501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c199680 0x55fe3c199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.818+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/2366520501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19eef0 0x55fe3c19f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.818+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/2366520501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19f810 0x55fe3c199140 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.818+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2366520501 >> 172.21.15.121:0/2366520501 conn(0x55fe3c0c1e60 msgr2=0x55fe3c1921c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.818+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2366520501 shutdown_connections 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.818+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2366520501 wait complete. 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.818+0000 7fcfcf30db80 1 -- start start 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.818+0000 7fcfcf30db80 1 Processor -- start 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c1a5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19eef0 0x55fe3c1a2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19f810 0x55fe3c1a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c1a6220 con 0x55fe3c199680 2024-06-10T00:34:15.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c088d90 con 0x55fe3c19f810 2024-06-10T00:34:15.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fe3c1a63a0 con 0x55fe3c19eef0 2024-06-10T00:34:15.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfcf30db80 10 client.0 _ll_register_callbacks cb 0x55fe3c082ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:15.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfbe59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c1a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:15.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfbe59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c1a5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50856/0 (socket says 172.21.15.121:50856) 2024-06-10T00:34:15.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfbe59c700 1 -- 172.21.15.121:0/2947293515 learned_addr learned my addr 172.21.15.121:0/2947293515 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:15.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfbdd9b700 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19eef0 0x55fe3c1a2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:15.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfbed9d700 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19f810 0x55fe3c1a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:15.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfbe59c700 1 -- 172.21.15.121:0/2947293515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19eef0 msgr2=0x55fe3c1a2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:15.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19eef0 0x55fe3c1a2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfbe59c700 1 -- 172.21.15.121:0/2947293515 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19f810 msgr2=0x55fe3c1a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:15.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19f810 0x55fe3c1a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:15.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfbe59c700 1 -- 172.21.15.121:0/2947293515 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe3c037a70 con 0x55fe3c199680 2024-06-10T00:34:15.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c1a5580 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0009fd0 tx=0x7fcfb00ad460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af84121351910a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:15.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.819+0000 7fcfa5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:34:15.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/2947293515 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb00b5890 con 0x55fe3c199680 2024-06-10T00:34:15.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2947293515 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb400bca0 con 0x55fe3c199680 2024-06-10T00:34:15.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/2947293515 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcfb00ace80 con 0x55fe3c199680 2024-06-10T00:34:15.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/2947293515 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcfb00ad080 con 0x55fe3c199680 2024-06-10T00:34:15.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfcf30db80 10 client.16428 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:15.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2947293515 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fe3c196f80 con 0x55fe3c199680 2024-06-10T00:34:15.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/2947293515 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb400bca0 con 0x55fe3c199680 2024-06-10T00:34:15.999 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/2947293515 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcfb0077020 con 0x55fe3c199680 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfcf30db80 10 client.16428 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfcf30db80 10 client.16428 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfcf30db80 20 client.16428 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fe3c198390 0x55fe3c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfcf30db80 4 client.16428 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.820+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2947293515 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe3c0f5eb0 con 0x55fe3c198390 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.821+0000 7fcfbdd9b700 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fe3c198390 0x55fe3c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.821+0000 7fcfa5ffb700 10 client.16428 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.821+0000 7fcfbdd9b700 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fe3c198390 0x55fe3c1987c0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ccaae5a89cf14d8 server_cookie=8031eb8699f6376b in_seq=0 out_seq=0 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/2947293515 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55fe3c0f5eb0 con 0x55fe3c198390 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfa5ffb700 10 client.16428 handle_command_reply: tid=0 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 1 client.16428 shutdown 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2947293515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 msgr2=0x55fe3c1a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:16.000 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c1a5580 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0009fd0 tx=0x7fcfb00ad460 comp rx=0 tx=0).stop 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2947293515 shutdown_connections 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fe3c198390 0x55fe3c1987c0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19f810 0x55fe3c1a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c1a5580 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/2947293515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19eef0 0x55fe3c1a2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2947293515 >> 172.21.15.121:0/2947293515 conn(0x55fe3c0c1e60 msgr2=0x55fe3c195f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfa5ffb700 0 client.16428 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2947293515 shutdown_connections 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 1 -- 172.21.15.121:0/2947293515 wait complete. 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.822+0000 7fcfcf30db80 20 client.16428 trim_cache size 0 max 16384 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfcf30db80 1 Processor -- start 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfcf30db80 1 -- start start 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c191fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:16.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19eef0 0x55fe3c1a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19f810 0x55fe3c1b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc01a62c0 con 0x55fe3c199680 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0073cc0 con 0x55fe3c19f810 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0071b10 con 0x55fe3c19eef0 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfbe59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c191fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfbe59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c191fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50868/0 (socket says 172.21.15.121:50868) 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfbe59c700 1 -- 172.21.15.121:0/3588074011 learned_addr learned my addr 172.21.15.121:0/3588074011 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.845+0000 7fcfbdd9b700 1 --2- 172.21.15.121:0/3588074011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19eef0 0x55fe3c1a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.846+0000 7fcfbed9d700 1 --2- 172.21.15.121:0/3588074011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19f810 0x55fe3c1b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.846+0000 7fcfbe59c700 1 -- 172.21.15.121:0/3588074011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19eef0 msgr2=0x55fe3c1a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.846+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/3588074011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19eef0 0x55fe3c1a2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:16.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.846+0000 7fcfbe59c700 1 -- 172.21.15.121:0/3588074011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19f810 msgr2=0x55fe3c1b8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.846+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/3588074011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19f810 0x55fe3c1b8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.846+0000 7fcfbe59c700 1 -- 172.21.15.121:0/3588074011 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe3c1a51c0 con 0x55fe3c199680 2024-06-10T00:34:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.846+0000 7fcfbed9d700 1 --2- 172.21.15.121:0/3588074011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19f810 0x55fe3c1b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:34:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.846+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/3588074011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c191fe0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000f010 tx=0x7fcfb000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8398e175c5929efe server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.846+0000 7fcfa7fff700 1 -- 172.21.15.121:0/3588074011 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb00b5890 con 0x55fe3c199680 2024-06-10T00:34:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.846+0000 7fcfa7fff700 1 -- 172.21.15.121:0/3588074011 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb0074070 con 0x55fe3c199680 2024-06-10T00:34:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.846+0000 7fcfa7fff700 1 -- 172.21.15.121:0/3588074011 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb00b5400 con 0x55fe3c199680 2024-06-10T00:34:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.847+0000 7fcfcf30db80 1 -- 172.21.15.121:0/3588074011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 msgr2=0x55fe3c191fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.847+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/3588074011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c191fe0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000f010 tx=0x7fcfb000cf90 comp rx=0 tx=0).stop 2024-06-10T00:34:16.003 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.847+0000 7fcfcf30db80 1 -- 172.21.15.121:0/3588074011 shutdown_connections 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.847+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/3588074011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c19f810 0x55fe3c1b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.847+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/3588074011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c199680 0x55fe3c191fe0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.847+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/3588074011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19eef0 0x55fe3c1a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.847+0000 7fcfcf30db80 1 -- 172.21.15.121:0/3588074011 >> 172.21.15.121:0/3588074011 conn(0x55fe3c0c1e60 msgr2=0x55fe3c1918d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.847+0000 7fcfcf30db80 1 -- 172.21.15.121:0/3588074011 shutdown_connections 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.847+0000 7fcfcf30db80 1 -- 172.21.15.121:0/3588074011 wait complete. 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfcf30db80 1 -- start start 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfcf30db80 1 Processor -- start 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c199680 0x55fe3c214fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19eef0 0x55fe3c2154f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfcf30db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19f810 0x55fe3c1b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0072180 con 0x55fe3c19eef0 2024-06-10T00:34:16.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc00735f0 con 0x55fe3c199680 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfcf30db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc0072850 con 0x55fe3c19f810 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfcf30db80 10 client.0 _ll_register_callbacks cb 0x55fe3c082ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfbed9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19f810 0x55fe3c1b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfbed9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19f810 0x55fe3c1b52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:35208/0 (socket says 172.21.15.121:35208) 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfbdd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19eef0 0x55fe3c2154f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfbed9d700 1 -- 172.21.15.121:0/1353064064 learned_addr learned my addr 172.21.15.121:0/1353064064 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.848+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c199680 0x55fe3c214fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfbed9d700 1 -- 172.21.15.121:0/1353064064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c199680 msgr2=0x55fe3c214fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfbed9d700 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c199680 0x55fe3c214fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfbed9d700 1 -- 172.21.15.121:0/1353064064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19eef0 msgr2=0x55fe3c2154f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfbed9d700 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19eef0 0x55fe3c2154f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfbed9d700 1 -- 172.21.15.121:0/1353064064 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe3c1b59b0 con 0x55fe3c19f810 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfbed9d700 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19f810 0x55fe3c1b52b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8016d90 tx=0x7fcfb80077e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e24f485dfd003c7 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:16.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfa5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/1353064064 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8022670 con 0x55fe3c19f810 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/1353064064 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fcfb8022810 con 0x55fe3c19f810 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfcf30db80 1 -- 172.21.15.121:0/1353064064 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfb0074070 con 0x55fe3c19f810 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/1353064064 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcfb8030050 con 0x55fe3c19f810 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfcf30db80 10 client.25900 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfcf30db80 1 -- 172.21.15.121:0/1353064064 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fe3c21c750 con 0x55fe3c19f810 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.849+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/1353064064 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb0074070 con 0x55fe3c19f810 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.850+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/1353064064 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcfb8021a10 con 0x55fe3c19f810 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.850+0000 7fcfcf30db80 10 client.25900 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.850+0000 7fcfcf30db80 10 client.25900 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.850+0000 7fcfcf30db80 20 client.25900 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.850+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fe3c21cfb0 0x55fe3c21f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:16.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.850+0000 7fcfcf30db80 4 client.25900 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.45, "await_for": 45.0}] 2024-06-10T00:34:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.850+0000 7fcfcf30db80 1 -- 172.21.15.121:0/1353064064 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 149.45, "await_for": 45.0}) -- 0x55fe3c0f0430 con 0x55fe3c21cfb0 2024-06-10T00:34:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.850+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fe3c21cfb0 0x55fe3c21f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.850+0000 7fcfa5ffb700 10 client.25900 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:16.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:15.850+0000 7fcfbe59c700 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fe3c21cfb0 0x55fe3c21f3d0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e117adec2edbc33e server_cookie=7514c519f8df07db in_seq=0 out_seq=0 2024-06-10T00:34:17.295 INFO:tasks.workunit.client.0.smithi121.stdout: 258 550369 706 380730 581 182675 866 2024-06-10T00:34:17.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:17 smithi121 ceph-mon[32378]: pgmap v192: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 123 op/s 2024-06-10T00:34:17.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:17 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:34:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:17 smithi135 ceph-mon[36424]: pgmap v192: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 123 op/s 2024-06-10T00:34:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:17 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:34:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:17 smithi193 ceph-mon[36716]: pgmap v192: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 123 op/s 2024-06-10T00:34:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:17 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:34:18.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:18 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:34:18.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:18 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:34:18.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:18 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:34:18.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:18 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:34:18.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:18 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:34:18.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:18 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:34:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:19 smithi135 ceph-mon[36424]: pgmap v193: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 59 op/s 2024-06-10T00:34:19.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:19 smithi193 ceph-mon[36716]: pgmap v193: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 59 op/s 2024-06-10T00:34:19.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:19 smithi121 ceph-mon[32378]: pgmap v193: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 59 op/s 2024-06-10T00:34:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:21 smithi135 ceph-mon[36424]: pgmap v194: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 207 op/s 2024-06-10T00:34:21.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:21 smithi193 ceph-mon[36716]: pgmap v194: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 207 op/s 2024-06-10T00:34:21.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:21 smithi121 ceph-mon[32378]: pgmap v194: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.9 MiB/s wr, 207 op/s 2024-06-10T00:34:23.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:23 smithi135 ceph-mon[36424]: pgmap v195: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 279 op/s 2024-06-10T00:34:23.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:23 smithi193 ceph-mon[36716]: pgmap v195: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 279 op/s 2024-06-10T00:34:23.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:23 smithi121 ceph-mon[32378]: pgmap v195: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.9 MiB/s wr, 279 op/s 2024-06-10T00:34:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:25 smithi135 ceph-mon[36424]: pgmap v196: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 305 op/s 2024-06-10T00:34:25.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:25 smithi193 ceph-mon[36716]: pgmap v196: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 305 op/s 2024-06-10T00:34:25.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:25 smithi121 ceph-mon[32378]: pgmap v196: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 5.5 MiB/s wr, 305 op/s 2024-06-10T00:34:26.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:26 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:34:26.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:26 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:34:26.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:26 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:34:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfa5ffb700 1 -- 172.21.15.121:0/1353064064 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55fe3c0f0430 con 0x55fe3c21cfb0 2024-06-10T00:34:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfa5ffb700 10 client.25900 handle_command_reply: tid=0 2024-06-10T00:34:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 1 client.25900 shutdown 2024-06-10T00:34:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 1 -- 172.21.15.121:0/1353064064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19f810 msgr2=0x55fe3c1b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19f810 0x55fe3c1b52b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8016d90 tx=0x7fcfb80077e0 comp rx=0 tx=0).stop 2024-06-10T00:34:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 1 -- 172.21.15.121:0/1353064064 shutdown_connections 2024-06-10T00:34:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55fe3c21cfb0 0x55fe3c21f3d0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:27.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe3c19f810 0x55fe3c1b52b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55fe3c199680 0x55fe3c214fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfa5ffb700 0 client.25900 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 1 --2- 172.21.15.121:0/1353064064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fe3c19eef0 0x55fe3c2154f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 1 -- 172.21.15.121:0/1353064064 >> 172.21.15.121:0/1353064064 conn(0x55fe3c0c1e60 msgr2=0x55fe3c195f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 1 -- 172.21.15.121:0/1353064064 shutdown_connections 2024-06-10T00:34:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 1 -- 172.21.15.121:0/1353064064 wait complete. 2024-06-10T00:34:27.079 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.077+0000 7fcfcf30db80 20 client.25900 trim_cache size 0 max 16384 2024-06-10T00:34:27.079 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:34:27.079 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 30, 2024-06-10T00:34:27.079 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 18, 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "5095a648": { 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 18, 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 149.4, 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:34:27.080 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:34:27.081 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:34:27.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:27.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:27.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:27.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:27.081 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:27.081 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:34:27.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/2616976478 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcfa80a2e30 msgr2=0x7fcfa80a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcfa80a2e30 0x7fcfa80a52f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0103e30 tx=0x7fcfb4005fb0 comp rx=0 tx=0).stop 2024-06-10T00:34:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/2616976478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfc0101df0 msgr2=0x7fcfc019d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfc0101df0 0x7fcfc019d690 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcfb000e9c0 tx=0x7fcfb000ecd0 comp rx=0 tx=0).stop 2024-06-10T00:34:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/2616976478 shutdown_connections 2024-06-10T00:34:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fcfa80a2e30 0x7fcfa80a52f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfc0106750 0x7fcfc0197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcfc0101df0 0x7fcfc019d690 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 --2- 172.21.15.121:0/2616976478 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfc01027a0 0x7fcfc019dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/2616976478 >> 172.21.15.121:0/2616976478 conn(0x7fcfc0077a30 msgr2=0x7fcfc00fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/2616976478 shutdown_connections 2024-06-10T00:34:27.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:27.078+0000 7fcfc4a8a700 1 -- 172.21.15.121:0/2616976478 wait complete. 2024-06-10T00:34:27.295 INFO:tasks.workunit.client.0.smithi121.stdout: 258 1796 0 783 0 174 0 2024-06-10T00:34:27.393 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:27 smithi121 ceph-mon[32378]: pgmap v197: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.1 MiB/s wr, 385 op/s 2024-06-10T00:34:27.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:27 smithi135 ceph-mon[36424]: pgmap v197: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.1 MiB/s wr, 385 op/s 2024-06-10T00:34:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:27 smithi193 ceph-mon[36716]: pgmap v197: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 6.1 MiB/s wr, 385 op/s 2024-06-10T00:34:27.870 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5095a648', quiesce took 19.0 seconds. Will release after: 10.45 2024-06-10T00:34:29.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:29 smithi135 ceph-mon[36424]: pgmap v198: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 360 op/s 2024-06-10T00:34:29.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:29 smithi193 ceph-mon[36716]: pgmap v198: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 360 op/s 2024-06-10T00:34:29.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:29 smithi121 ceph-mon[32378]: pgmap v198: 129 pgs: 129 active+clean; 1.3 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 360 op/s 2024-06-10T00:34:31.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:31 smithi135 ceph-mon[36424]: pgmap v199: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.3 MiB/s wr, 492 op/s 2024-06-10T00:34:31.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:31 smithi193 ceph-mon[36716]: pgmap v199: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.3 MiB/s wr, 492 op/s 2024-06-10T00:34:31.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:31 smithi121 ceph-mon[32378]: pgmap v199: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.3 MiB/s wr, 492 op/s 2024-06-10T00:34:33.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:33 smithi135 ceph-mon[36424]: pgmap v200: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 433 op/s 2024-06-10T00:34:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:33 smithi193 ceph-mon[36716]: pgmap v200: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 433 op/s 2024-06-10T00:34:33.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:33 smithi121 ceph-mon[32378]: pgmap v200: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 433 op/s 2024-06-10T00:34:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:35 smithi135 ceph-mon[36424]: pgmap v201: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.4 MiB/s wr, 474 op/s 2024-06-10T00:34:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:35 smithi193 ceph-mon[36716]: pgmap v201: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.4 MiB/s wr, 474 op/s 2024-06-10T00:34:35.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:35 smithi121 ceph-mon[32378]: pgmap v201: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.4 MiB/s wr, 474 op/s 2024-06-10T00:34:37.296 INFO:tasks.workunit.client.0.smithi121.stdout: 258 74414 0 51650 0 23315 0 2024-06-10T00:34:37.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:37 smithi135 ceph-mon[36424]: pgmap v202: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 692 op/s 2024-06-10T00:34:37.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:37 smithi193 ceph-mon[36716]: pgmap v202: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 692 op/s 2024-06-10T00:34:37.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:37 smithi121 ceph-mon[32378]: pgmap v202: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 692 op/s 2024-06-10T00:34:38.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:34:38.602 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:39 smithi121 ceph-mon[32378]: pgmap v203: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 599 op/s 2024-06-10T00:34:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:39 smithi135 ceph-mon[36424]: pgmap v203: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 599 op/s 2024-06-10T00:34:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:39 smithi193 ceph-mon[36716]: pgmap v203: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 599 op/s 2024-06-10T00:34:40.502 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:34:40.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.830+0000 7fa9a180d700 1 -- 172.21.15.121:0/2739918824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c101390 msgr2=0x7fa99c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:40.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.830+0000 7fa9a180d700 1 --2- 172.21.15.121:0/2739918824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c101390 0x7fa99c101810 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa990009990 tx=0x7fa990009ca0 comp rx=0 tx=0).stop 2024-06-10T00:34:40.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.830+0000 7fa9a180d700 1 -- 172.21.15.121:0/2739918824 shutdown_connections 2024-06-10T00:34:40.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.830+0000 7fa9a180d700 1 --2- 172.21.15.121:0/2739918824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa99c101d50 0x7fa99c10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:40.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.830+0000 7fa9a180d700 1 --2- 172.21.15.121:0/2739918824 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c107400 0x7fa99c1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:40.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.830+0000 7fa9a180d700 1 --2- 172.21.15.121:0/2739918824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c101390 0x7fa99c101810 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:40.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.830+0000 7fa9a180d700 1 -- 172.21.15.121:0/2739918824 >> 172.21.15.121:0/2739918824 conn(0x7fa99c078340 msgr2=0x7fa99c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:40.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.830+0000 7fa9a180d700 1 -- 172.21.15.121:0/2739918824 shutdown_connections 2024-06-10T00:34:40.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.830+0000 7fa9a180d700 1 -- 172.21.15.121:0/2739918824 wait complete. 2024-06-10T00:34:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa9a180d700 1 Processor -- start 2024-06-10T00:34:40.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa9a180d700 1 -- start start 2024-06-10T00:34:40.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa9a180d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c101390 0x7fa99c19d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:40.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa9a180d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa99c101d50 0x7fa99c19dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:40.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa9a180d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c107400 0x7fa99c197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa9a180d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c111670 con 0x7fa99c107400 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa9a180d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c1114f0 con 0x7fa99c101390 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa9a180d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa99c1117f0 con 0x7fa99c101d50 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa99b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c107400 0x7fa99c197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa99affd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c101390 0x7fa99c19d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa99b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c107400 0x7fa99c197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57330/0 (socket says 172.21.15.121:57330) 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.831+0000 7fa99b7fe700 1 -- 172.21.15.121:0/3189223660 learned_addr learned my addr 172.21.15.121:0/3189223660 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa99a7fc700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa99c101d50 0x7fa99c19dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa99b7fe700 1 -- 172.21.15.121:0/3189223660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa99c101d50 msgr2=0x7fa99c19dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa99b7fe700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa99c101d50 0x7fa99c19dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa99b7fe700 1 -- 172.21.15.121:0/3189223660 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c101390 msgr2=0x7fa99c19d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:40.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa99b7fe700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c101390 0x7fa99c19d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa99b7fe700 1 -- 172.21.15.121:0/3189223660 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99c1980b0 con 0x7fa99c107400 2024-06-10T00:34:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa99affd700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c101390 0x7fa99c19d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa99b7fe700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c107400 0x7fa99c197950 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa99400efb0 tx=0x7fa99400c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa9a080b700 1 -- 172.21.15.121:0/3189223660 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994011670 con 0x7fa99c107400 2024-06-10T00:34:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa9a080b700 1 -- 172.21.15.121:0/3189223660 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa994011cf0 con 0x7fa99c107400 2024-06-10T00:34:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa9a180d700 1 -- 172.21.15.121:0/3189223660 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa99c1983a0 con 0x7fa99c107400 2024-06-10T00:34:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa9a080b700 1 -- 172.21.15.121:0/3189223660 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99401adb0 con 0x7fa99c107400 2024-06-10T00:34:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.832+0000 7fa9a180d700 1 -- 172.21.15.121:0/3189223660 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa99c06a170 con 0x7fa99c107400 2024-06-10T00:34:40.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.833+0000 7fa9a180d700 1 -- 172.21.15.121:0/3189223660 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa99c069180 con 0x7fa99c107400 2024-06-10T00:34:40.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.835+0000 7fa9a080b700 1 -- 172.21.15.121:0/3189223660 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fa994011810 con 0x7fa99c107400 2024-06-10T00:34:40.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.835+0000 7fa9a080b700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9840a2e10 0x7fa9840a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:40.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.836+0000 7fa99affd700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9840a2e10 0x7fa9840a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:40.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.836+0000 7fa9a080b700 1 -- 172.21.15.121:0/3189223660 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fa9940d2110 con 0x7fa99c107400 2024-06-10T00:34:40.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.836+0000 7fa99affd700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9840a2e10 0x7fa9840a52d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa99c198a90 tx=0x7fa98c00b410 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:40.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:40.839+0000 7fa9a080b700 1 -- 172.21.15.121:0/3189223660 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa994016030 con 0x7fa99c107400 2024-06-10T00:34:40.934 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:41.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.106+0000 7fa9a180d700 1 -- 172.21.15.121:0/3189223660 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa99c06ac00 con 0x7fa99c107400 2024-06-10T00:34:41.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.107+0000 7fa9a080b700 1 -- 172.21.15.121:0/3189223660 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11891 (secure 0 0 0) 0x7fa99409ea20 con 0x7fa99c107400 2024-06-10T00:34:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:34:41.107 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:33:03.460049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:34:41.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 -- 172.21.15.121:0/3189223660 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9840a2e10 msgr2=0x7fa9840a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:41.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9840a2e10 0x7fa9840a52d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa99c198a90 tx=0x7fa98c00b410 comp rx=0 tx=0).stop 2024-06-10T00:34:41.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 -- 172.21.15.121:0/3189223660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c107400 msgr2=0x7fa99c197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:41.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c107400 0x7fa99c197950 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fa99400efb0 tx=0x7fa99400c550 comp rx=0 tx=0).stop 2024-06-10T00:34:41.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 -- 172.21.15.121:0/3189223660 shutdown_connections 2024-06-10T00:34:41.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fa9840a2e10 0x7fa9840a52d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:41.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa99c107400 0x7fa99c197950 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:41.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c101390 0x7fa99c19d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:41.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 --2- 172.21.15.121:0/3189223660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa99c101d50 0x7fa99c19dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:41.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 -- 172.21.15.121:0/3189223660 >> 172.21.15.121:0/3189223660 conn(0x7fa99c078340 msgr2=0x7fa99c0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:41.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 -- 172.21.15.121:0/3189223660 shutdown_connections 2024-06-10T00:34:41.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:41.109+0000 7fa9a180d700 1 -- 172.21.15.121:0/3189223660 wait complete. 2024-06-10T00:34:41.111 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-06-10T00:34:41.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:41 smithi121 ceph-mon[32378]: pgmap v204: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.4 MiB/s wr, 762 op/s 2024-06-10T00:34:41.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:41 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3189223660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:41.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:41 smithi135 ceph-mon[36424]: pgmap v204: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.4 MiB/s wr, 762 op/s 2024-06-10T00:34:41.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:41 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3189223660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:41 smithi193 ceph-mon[36716]: pgmap v204: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.4 MiB/s wr, 762 op/s 2024-06-10T00:34:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:41 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3189223660' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:42.374 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db --set-id 5095a648 --release --await' 2024-06-10T00:34:42.374 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db --set-id 5095a648 --release --await 2024-06-10T00:34:43.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:43 smithi135 ceph-mon[36424]: pgmap v205: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 888 KiB/s wr, 716 op/s 2024-06-10T00:34:43.515 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:43 smithi121 ceph-mon[32378]: pgmap v205: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 888 KiB/s wr, 716 op/s 2024-06-10T00:34:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:43 smithi193 ceph-mon[36716]: pgmap v205: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 888 KiB/s wr, 716 op/s 2024-06-10T00:34:44.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.061+0000 7f5e5af89700 1 -- 172.21.15.121:0/976744335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54100a80 msgr2=0x7f5e54100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:44.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.061+0000 7f5e5af89700 1 --2- 172.21.15.121:0/976744335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54100a80 0x7f5e54100f00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009990 tx=0x7f5e44009ca0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 -- 172.21.15.121:0/976744335 shutdown_connections 2024-06-10T00:34:44.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 --2- 172.21.15.121:0/976744335 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e54101440 0x7f5e5410f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 --2- 172.21.15.121:0/976744335 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e541086f0 0x7f5e54108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 --2- 172.21.15.121:0/976744335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54100a80 0x7f5e54100f00 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 -- 172.21.15.121:0/976744335 >> 172.21.15.121:0/976744335 conn(0x7f5e54077a00 msgr2=0x7f5e540fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:44.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 -- 172.21.15.121:0/976744335 shutdown_connections 2024-06-10T00:34:44.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 -- 172.21.15.121:0/976744335 wait complete. 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 Processor -- start 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 -- start start 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e54100a80 0x7f5e5410e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54101440 0x7f5e5410ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e541086f0 0x7f5e541a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5e54112990 con 0x7f5e54101440 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.062+0000 7f5e5af89700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5e54112810 con 0x7f5e54100a80 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e5af89700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e54112b10 con 0x7f5e541086f0 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54101440 0x7f5e5410ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e58d25700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e54100a80 0x7f5e5410e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54101440 0x7f5e5410ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57360/0 (socket says 172.21.15.121:57360) 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e53fff700 1 -- 172.21.15.121:0/3517986088 learned_addr learned my addr 172.21.15.121:0/3517986088 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:44.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e59526700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e541086f0 0x7f5e541a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e53fff700 1 -- 172.21.15.121:0/3517986088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e541086f0 msgr2=0x7f5e541a3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e53fff700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e541086f0 0x7f5e541a3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e53fff700 1 -- 172.21.15.121:0/3517986088 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e54100a80 msgr2=0x7f5e5410e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e53fff700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e54100a80 0x7f5e5410e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e53fff700 1 -- 172.21.15.121:0/3517986088 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e541a4460 con 0x7f5e54101440 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e53fff700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54101440 0x7f5e5410ee00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400bae0 tx=0x7f5e4400bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e51ffb700 1 -- 172.21.15.121:0/3517986088 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44026070 con 0x7f5e54101440 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e51ffb700 1 -- 172.21.15.121:0/3517986088 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e44003d70 con 0x7f5e54101440 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e51ffb700 1 -- 172.21.15.121:0/3517986088 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44021410 con 0x7f5e54101440 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.063+0000 7f5e5af89700 1 -- 172.21.15.121:0/3517986088 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e541a46f0 con 0x7f5e54101440 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.064+0000 7f5e5af89700 1 -- 172.21.15.121:0/3517986088 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e541ac000 con 0x7f5e54101440 2024-06-10T00:34:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.066+0000 7f5e51ffb700 1 -- 172.21.15.121:0/3517986088 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5e44029070 con 0x7f5e54101440 2024-06-10T00:34:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.066+0000 7f5e51ffb700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e400a2d90 0x7f5e400a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.066+0000 7f5e58d25700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e400a2d90 0x7f5e400a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e51ffb700 1 -- 172.21.15.121:0/3517986088 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f5e440d0680 con 0x7f5e54101440 2024-06-10T00:34:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e6580cb80 1 Processor -- start 2024-06-10T00:34:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e58d25700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e400a2d90 0x7f5e400a5250 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009550 tx=0x7f5e4c0091b0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e6580cb80 1 -- start start 2024-06-10T00:34:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407d0640 0x5569407d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 0x5569407ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407caf20 0x5569407cb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55694071c810 con 0x5569407d0f60 2024-06-10T00:34:44.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5569406c4720 con 0x5569407d0640 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5569407187d0 con 0x5569407caf20 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 0x5569407ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 0x5569407ca9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57370/0 (socket says 172.21.15.121:57370) 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e53fff700 1 -- 172.21.15.121:0/1693836197 learned_addr learned my addr 172.21.15.121:0/1693836197 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.068+0000 7f5e58d25700 1 --2- 172.21.15.121:0/1693836197 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407d0640 0x5569407d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.067+0000 7f5e59526700 1 --2- 172.21.15.121:0/1693836197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407caf20 0x5569407cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.068+0000 7f5e53fff700 1 -- 172.21.15.121:0/1693836197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407caf20 msgr2=0x5569407cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.068+0000 7f5e53fff700 1 --2- 172.21.15.121:0/1693836197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407caf20 0x5569407cb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.068+0000 7f5e53fff700 1 -- 172.21.15.121:0/1693836197 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407d0640 msgr2=0x5569407d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.068+0000 7f5e53fff700 1 --2- 172.21.15.121:0/1693836197 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407d0640 0x5569407d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.068+0000 7f5e53fff700 1 -- 172.21.15.121:0/1693836197 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569406baf50 con 0x5569407d0f60 2024-06-10T00:34:44.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.068+0000 7f5e53fff700 1 --2- 172.21.15.121:0/1693836197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 0x5569407ca9e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5e4409c830 tx=0x7f5e4409c200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b973b2f35007204e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.068+0000 7f5e4affd700 1 -- 172.21.15.121:0/1693836197 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4409c8a0 con 0x5569407d0f60 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.068+0000 7f5e4affd700 1 -- 172.21.15.121:0/1693836197 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e440a0c50 con 0x5569407d0f60 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.068+0000 7f5e4affd700 1 -- 172.21.15.121:0/1693836197 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e440d02b0 con 0x5569407d0f60 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 -- 172.21.15.121:0/1693836197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 msgr2=0x5569407ca9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/1693836197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 0x5569407ca9e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5e4409c830 tx=0x7f5e4409c200 comp rx=0 tx=0).stop 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 -- 172.21.15.121:0/1693836197 shutdown_connections 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/1693836197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407caf20 0x5569407cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/1693836197 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407d0640 0x5569407d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/1693836197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 0x5569407ca9e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 -- 172.21.15.121:0/1693836197 >> 172.21.15.121:0/1693836197 conn(0x5569406f5cc0 msgr2=0x5569407c5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 -- 172.21.15.121:0/1693836197 shutdown_connections 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 -- 172.21.15.121:0/1693836197 wait complete. 2024-06-10T00:34:44.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 -- start start 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 Processor -- start 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.069+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407d6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407d4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0f60 0x5569407d45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5569407d76f0 con 0x5569407d0640 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5569406bcbf0 con 0x5569407caf20 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5569407d7870 con 0x5569407d0f60 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e6580cb80 10 client.0 _ll_register_callbacks cb 0x5569406b6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407d4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407d4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57380/0 (socket says 172.21.15.121:57380) 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e53fff700 1 -- 172.21.15.121:0/3868305816 learned_addr learned my addr 172.21.15.121:0/3868305816 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e59526700 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0f60 0x5569407d45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e58d25700 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407d6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e53fff700 1 -- 172.21.15.121:0/3868305816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0f60 msgr2=0x5569407d45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e53fff700 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0f60 0x5569407d45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e53fff700 1 -- 172.21.15.121:0/3868305816 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 msgr2=0x5569407d6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e53fff700 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407d6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e53fff700 1 -- 172.21.15.121:0/3868305816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55694066b8d0 con 0x5569407d0640 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e59526700 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0f60 0x5569407d45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e53fff700 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407d4080 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5e44005560 tx=0x7f5e44005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36fde54bd1bae7ad server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e33fff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e33fff700 1 -- 172.21.15.121:0/3868305816 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4400f6f0 con 0x5569407d0640 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e33fff700 1 -- 172.21.15.121:0/3868305816 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f5e440d02b0 con 0x5569407d0640 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3868305816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e440a0c50 con 0x5569407d0640 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e33fff700 1 -- 172.21.15.121:0/3868305816 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5e440d04b0 con 0x5569407d0640 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e6580cb80 10 client.16452 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:44.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.070+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3868305816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5569407d76f0 con 0x5569407d0640 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.071+0000 7f5e33fff700 1 -- 172.21.15.121:0/3868305816 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e440a0c50 con 0x5569407d0640 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.071+0000 7f5e33fff700 1 -- 172.21.15.121:0/3868305816 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5e4409dbd0 con 0x5569407d0640 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.071+0000 7f5e6580cb80 10 client.16452 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.071+0000 7f5e6580cb80 10 client.16452 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.071+0000 7f5e6580cb80 20 client.16452 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.071+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5569407e44f0 0x5569407e6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.071+0000 7f5e6580cb80 4 client.16452 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.071+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3868305816 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556940729d10 con 0x5569407e44f0 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.071+0000 7f5e58d25700 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5569407e44f0 0x5569407e6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e33fff700 10 client.16452 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e58d25700 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5569407e44f0 0x5569407e6910 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47a9ca0c1979db93 server_cookie=22e150c9e1791dd2 in_seq=0 out_seq=0 2024-06-10T00:34:44.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e33fff700 1 -- 172.21.15.121:0/3868305816 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x556940729d10 con 0x5569407e44f0 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e33fff700 10 client.16452 handle_command_reply: tid=0 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e6580cb80 1 client.16452 shutdown 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3868305816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 msgr2=0x5569407d4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407d4080 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5e44005560 tx=0x7f5e44005c70 comp rx=0 tx=0).stop 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3868305816 shutdown_connections 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5569407e44f0 0x5569407e6910 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0f60 0x5569407d45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407d6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e33fff700 0 client.16452 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/3868305816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407d4080 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.072+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3868305816 >> 172.21.15.121:0/3868305816 conn(0x5569406f5cc0 msgr2=0x5569407c55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.073+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3868305816 shutdown_connections 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.073+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3868305816 wait complete. 2024-06-10T00:34:44.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.073+0000 7f5e6580cb80 20 client.16452 trim_cache size 0 max 16384 2024-06-10T00:34:44.100 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:44.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e6580cb80 1 Processor -- start 2024-06-10T00:34:44.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e6580cb80 1 -- start start 2024-06-10T00:34:44.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407e0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0f60 0x5569407bf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541b74a0 con 0x5569407d0640 2024-06-10T00:34:44.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541b4cd0 con 0x5569407caf20 2024-06-10T00:34:44.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541b2b20 con 0x5569407d0f60 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407e0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e53fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407e0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57392/0 (socket says 172.21.15.121:57392) 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e53fff700 1 -- 172.21.15.121:0/3012400601 learned_addr learned my addr 172.21.15.121:0/3012400601 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e59526700 1 --2- 172.21.15.121:0/3012400601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0f60 0x5569407bf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e58d25700 1 --2- 172.21.15.121:0/3012400601 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e53fff700 1 -- 172.21.15.121:0/3012400601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0f60 msgr2=0x5569407bf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.096+0000 7f5e53fff700 1 --2- 172.21.15.121:0/3012400601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0f60 0x5569407bf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.097+0000 7f5e53fff700 1 -- 172.21.15.121:0/3012400601 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 msgr2=0x5569407ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.097+0000 7f5e53fff700 1 --2- 172.21.15.121:0/3012400601 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.097+0000 7f5e53fff700 1 -- 172.21.15.121:0/3012400601 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569407d74c0 con 0x5569407d0640 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.097+0000 7f5e53fff700 1 --2- 172.21.15.121:0/3012400601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407e0990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5e440ce040 tx=0x7f5e4409adb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837732317f262934 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.097+0000 7f5e4a7fc700 1 -- 172.21.15.121:0/3012400601 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e440a30d0 con 0x5569407d0640 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.097+0000 7f5e4a7fc700 1 -- 172.21.15.121:0/3012400601 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e4409be60 con 0x5569407d0640 2024-06-10T00:34:44.106 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.097+0000 7f5e4a7fc700 1 -- 172.21.15.121:0/3012400601 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4400f6f0 con 0x5569407d0640 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.098+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3012400601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 msgr2=0x5569407e0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.098+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/3012400601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407e0990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5e440ce040 tx=0x7f5e4409adb0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.098+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3012400601 shutdown_connections 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.098+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/3012400601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0f60 0x5569407bf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.098+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/3012400601 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.098+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/3012400601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0640 0x5569407e0990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.098+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3012400601 >> 172.21.15.121:0/3012400601 conn(0x5569406f5cc0 msgr2=0x5569407d1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.098+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3012400601 shutdown_connections 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.098+0000 7f5e6580cb80 1 -- 172.21.15.121:0/3012400601 wait complete. 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.098+0000 7f5e6580cb80 1 -- start start 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.098+0000 7f5e6580cb80 1 Processor -- start 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407ebd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0640 0x5569407ec2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.107 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e6580cb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 0x5569407ec810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e58d25700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407ebd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541b3190 con 0x5569407d0f60 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541b4600 con 0x5569407caf20 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e6580cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e541b3860 con 0x5569407d0640 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e59526700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 0x5569407ec810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e58d25700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407ebd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:50698/0 (socket says 172.21.15.121:50698) 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e58d25700 1 -- 172.21.15.121:0/2551008903 learned_addr learned my addr 172.21.15.121:0/2551008903 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e53fff700 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0640 0x5569407ec2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.099+0000 7f5e6580cb80 10 client.0 _ll_register_callbacks cb 0x5569406b6920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e58d25700 1 -- 172.21.15.121:0/2551008903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0640 msgr2=0x5569407ec2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e58d25700 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0640 0x5569407ec2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e58d25700 1 -- 172.21.15.121:0/2551008903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 msgr2=0x5569407ec810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:44.108 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e58d25700 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 0x5569407ec810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e58d25700 1 -- 172.21.15.121:0/2551008903 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55694084e170 con 0x5569407caf20 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e59526700 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 0x5569407ec810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e58d25700 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407ebd90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c015810 tx=0x7f5e4c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5bd46f9e40c4078 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e33fff700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e33fff700 1 -- 172.21.15.121:0/2551008903 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c01c040 con 0x5569407caf20 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e6580cb80 1 -- 172.21.15.121:0/2551008903 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e4409be60 con 0x5569407caf20 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e33fff700 1 -- 172.21.15.121:0/2551008903 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f5e4c003a00 con 0x5569407caf20 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e33fff700 1 -- 172.21.15.121:0/2551008903 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5e4c038050 con 0x5569407caf20 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e6580cb80 10 client.26021 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.100+0000 7f5e6580cb80 1 -- 172.21.15.121:0/2551008903 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55694084f490 con 0x5569407caf20 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.101+0000 7f5e33fff700 1 -- 172.21.15.121:0/2551008903 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e4409be60 con 0x5569407caf20 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.101+0000 7f5e33fff700 1 -- 172.21.15.121:0/2551008903 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f5e4c0040b0 con 0x5569407caf20 2024-06-10T00:34:44.109 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.101+0000 7f5e6580cb80 10 client.26021 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.101+0000 7f5e6580cb80 10 client.26021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.101+0000 7f5e6580cb80 20 client.26021 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.101+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5569407e44f0 0x556940851c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.101+0000 7f5e6580cb80 4 client.26021 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.101+0000 7f5e6580cb80 1 -- 172.21.15.121:0/2551008903 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556940724290 con 0x5569407e44f0 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.101+0000 7f5e53fff700 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5569407e44f0 0x556940851c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e53fff700 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5569407e44f0 0x556940851c40 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbe778a7e3aa3a35 server_cookie=7e62137638510da3 in_seq=0 out_seq=0 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e33fff700 10 client.26021 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e33fff700 1 -- 172.21.15.121:0/2551008903 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556940724290 con 0x5569407e44f0 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e33fff700 10 client.26021 handle_command_reply: tid=0 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 1 client.26021 shutdown 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 1 -- 172.21.15.121:0/2551008903 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 msgr2=0x5569407ebd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407ebd90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c015810 tx=0x7f5e4c004b40 comp rx=0 tx=0).stop 2024-06-10T00:34:44.110 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 1 -- 172.21.15.121:0/2551008903 shutdown_connections 2024-06-10T00:34:44.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5569407e44f0 0x556940851c40 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5569407d0f60 0x5569407ec810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5569407caf20 0x5569407ebd90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 1 --2- 172.21.15.121:0/2551008903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5569407d0640 0x5569407ec2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e33fff700 0 client.26021 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:44.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 1 -- 172.21.15.121:0/2551008903 >> 172.21.15.121:0/2551008903 conn(0x5569406f5cc0 msgr2=0x5569407d2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:44.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 1 -- 172.21.15.121:0/2551008903 shutdown_connections 2024-06-10T00:34:44.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 1 -- 172.21.15.121:0/2551008903 wait complete. 2024-06-10T00:34:44.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.102+0000 7f5e6580cb80 20 client.26021 trim_cache size 0 max 16384 2024-06-10T00:34:44.111 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:34:44.112 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "no active scrubs running", 2024-06-10T00:34:44.112 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": {} 2024-06-10T00:34:44.112 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:34:44.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 -- 172.21.15.121:0/3517986088 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e400a2d90 msgr2=0x7f5e400a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:44.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e400a2d90 0x7f5e400a5250 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009550 tx=0x7f5e4c0091b0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 -- 172.21.15.121:0/3517986088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54101440 msgr2=0x7f5e5410ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54101440 0x7f5e5410ee00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5e4400bae0 tx=0x7f5e4400bb10 comp rx=0 tx=0).stop 2024-06-10T00:34:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 -- 172.21.15.121:0/3517986088 shutdown_connections 2024-06-10T00:34:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5e400a2d90 0x7f5e400a5250 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e541086f0 0x7f5e541a3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5e54100a80 0x7f5e5410e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 --2- 172.21.15.121:0/3517986088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e54101440 0x7f5e5410ee00 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 -- 172.21.15.121:0/3517986088 >> 172.21.15.121:0/3517986088 conn(0x7f5e54077a00 msgr2=0x7f5e540fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 -- 172.21.15.121:0/3517986088 shutdown_connections 2024-06-10T00:34:44.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:44.103+0000 7f5e5af89700 1 -- 172.21.15.121:0/3517986088 wait complete. 2024-06-10T00:34:44.398 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:44 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:34:44.398 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:44 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:34:44.398 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:44 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:34:44.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:44 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:34:44.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:44 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:34:44.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:44 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:34:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:44 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:34:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:44 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:34:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:44 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:34:45.138 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-10T00:34:45.138 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:34:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:45 smithi121 ceph-mon[32378]: pgmap v206: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 900 KiB/s wr, 632 op/s 2024-06-10T00:34:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:45 smithi135 ceph-mon[36424]: pgmap v206: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 900 KiB/s wr, 632 op/s 2024-06-10T00:34:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:45 smithi193 ceph-mon[36716]: pgmap v206: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 900 KiB/s wr, 632 op/s 2024-06-10T00:34:46.055 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:46.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.778+0000 7f357f326700 1 -- 172.21.15.121:0/3410696839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3578102550 msgr2=0x7f35781029d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:46.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.778+0000 7f357f326700 1 --2- 172.21.15.121:0/3410696839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3578102550 0x7f35781029d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3568009990 tx=0x7f3568009ca0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.778+0000 7f357f326700 1 -- 172.21.15.121:0/3410696839 shutdown_connections 2024-06-10T00:34:46.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.778+0000 7f357f326700 1 --2- 172.21.15.121:0/3410696839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3578102f10 0x7f357810f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.778+0000 7f357f326700 1 --2- 172.21.15.121:0/3410696839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35781084f0 0x7f35781088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.778+0000 7f357f326700 1 --2- 172.21.15.121:0/3410696839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3578102550 0x7f35781029d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.780 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.778+0000 7f357f326700 1 -- 172.21.15.121:0/3410696839 >> 172.21.15.121:0/3410696839 conn(0x7f3578079650 msgr2=0x7f3578079a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.778+0000 7f357f326700 1 -- 172.21.15.121:0/3410696839 shutdown_connections 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.778+0000 7f357f326700 1 -- 172.21.15.121:0/3410696839 wait complete. 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357f326700 1 Processor -- start 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357f326700 1 -- start start 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357f326700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3578102550 0x7f357819ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357f326700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3578102f10 0x7f357819b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357f326700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35781084f0 0x7f357819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357f326700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3578112770 con 0x7f3578102550 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357f326700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f35781125f0 con 0x7f35781084f0 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357f326700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35781128f0 con 0x7f3578102f10 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357d0c2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3578102550 0x7f357819ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357d0c2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3578102550 0x7f357819ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57414/0 (socket says 172.21.15.121:57414) 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357d0c2700 1 -- 172.21.15.121:0/3024826151 learned_addr learned my addr 172.21.15.121:0/3024826151 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357c8c1700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3578102f10 0x7f357819b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.781 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.779+0000 7f357d8c3700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35781084f0 0x7f357819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.780+0000 7f357d0c2700 1 -- 172.21.15.121:0/3024826151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3578102f10 msgr2=0x7f357819b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:46.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.780+0000 7f357d0c2700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3578102f10 0x7f357819b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.780+0000 7f357d0c2700 1 -- 172.21.15.121:0/3024826151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35781084f0 msgr2=0x7f357819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:46.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.780+0000 7f357d0c2700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35781084f0 0x7f357819f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.780+0000 7f357d0c2700 1 -- 172.21.15.121:0/3024826151 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f357819fe50 con 0x7f3578102550 2024-06-10T00:34:46.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.780+0000 7f357d0c2700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3578102550 0x7f357819ae40 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f357400d980 tx=0x7f357400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:46.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.780+0000 7f356e7fc700 1 -- 172.21.15.121:0/3024826151 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3574017070 con 0x7f3578102550 2024-06-10T00:34:46.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.780+0000 7f357f326700 1 -- 172.21.15.121:0/3024826151 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35781a0140 con 0x7f3578102550 2024-06-10T00:34:46.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.780+0000 7f356e7fc700 1 -- 172.21.15.121:0/3024826151 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3574005020 con 0x7f3578102550 2024-06-10T00:34:46.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.780+0000 7f356e7fc700 1 -- 172.21.15.121:0/3024826151 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357400fbf0 con 0x7f3578102550 2024-06-10T00:34:46.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.780+0000 7f357f326700 1 -- 172.21.15.121:0/3024826151 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35781a79c0 con 0x7f3578102550 2024-06-10T00:34:46.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.782+0000 7f356e7fc700 1 -- 172.21.15.121:0/3024826151 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f35740051d0 con 0x7f3578102550 2024-06-10T00:34:46.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.783+0000 7f356e7fc700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35640a2d90 0x7f35640a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.783+0000 7f357c8c1700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35640a2d90 0x7f35640a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.783+0000 7f356e7fc700 1 -- 172.21.15.121:0/3024826151 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f35740cf5d0 con 0x7f3578102550 2024-06-10T00:34:46.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.783+0000 7f357c8c1700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35640a2d90 0x7f35640a5250 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f356800b5c0 tx=0x7f3568005960 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:46.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f3589ba9b80 1 Processor -- start 2024-06-10T00:34:46.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f3589ba9b80 1 -- start start 2024-06-10T00:34:46.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306500 0x5584913068e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x558491300710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x558491301070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5584911f3060 con 0x558491306e20 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x558491251950 con 0x558491300c50 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5584911b9560 con 0x558491306500 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f357c8c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x558491300710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f357c8c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x558491300710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57428/0 (socket says 172.21.15.121:57428) 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f357c8c1700 1 -- 172.21.15.121:0/1329610523 learned_addr learned my addr 172.21.15.121:0/1329610523 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.784+0000 7f357d0c2700 1 --2- 172.21.15.121:0/1329610523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306500 0x5584913068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.785+0000 7f357c8c1700 1 -- 172.21.15.121:0/1329610523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306500 msgr2=0x5584913068e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.785+0000 7f357c8c1700 1 --2- 172.21.15.121:0/1329610523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306500 0x5584913068e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.785+0000 7f357c8c1700 1 -- 172.21.15.121:0/1329610523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 msgr2=0x558491301070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.785+0000 7f357c8c1700 1 --2- 172.21.15.121:0/1329610523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x558491301070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.785+0000 7f357c8c1700 1 -- 172.21.15.121:0/1329610523 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584911a09d0 con 0x558491306e20 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.785+0000 7f357c8c1700 1 --2- 172.21.15.121:0/1329610523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x558491300710 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3568009990 tx=0x7f3568020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b676dcd79179c51 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.785+0000 7f35637fe700 1 -- 172.21.15.121:0/1329610523 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356800f520 con 0x558491306e20 2024-06-10T00:34:46.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.785+0000 7f35637fe700 1 -- 172.21.15.121:0/1329610523 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f356800fba0 con 0x558491306e20 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.785+0000 7f35637fe700 1 -- 172.21.15.121:0/1329610523 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3568024dc0 con 0x558491306e20 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.786+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1329610523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 msgr2=0x558491300710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.786+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/1329610523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x558491300710 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3568009990 tx=0x7f3568020000 comp rx=0 tx=0).stop 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.786+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1329610523 shutdown_connections 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.786+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/1329610523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x558491301070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.786+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/1329610523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306500 0x5584913068e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.786+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/1329610523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x558491300710 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.786+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1329610523 >> 172.21.15.121:0/1329610523 conn(0x55849122d180 msgr2=0x5584912fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.786+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1329610523 shutdown_connections 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.786+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1329610523 wait complete. 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.786+0000 7f3589ba9b80 1 -- start start 2024-06-10T00:34:46.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.786+0000 7f3589ba9b80 1 Processor -- start 2024-06-10T00:34:46.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x55849130cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491309e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306e20 0x55849130a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5584911f20c0 con 0x558491306500 2024-06-10T00:34:46.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55849124d910 con 0x558491300c50 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55849130d740 con 0x558491306e20 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f3589ba9b80 10 client.0 _ll_register_callbacks cb 0x5584911e76b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f357c8c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491309e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f357c8c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491309e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57436/0 (socket says 172.21.15.121:57436) 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f357c8c1700 1 -- 172.21.15.121:0/3967035871 learned_addr learned my addr 172.21.15.121:0/3967035871 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f357d0c2700 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x55849130cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f357d8c3700 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306e20 0x55849130a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f357c8c1700 1 -- 172.21.15.121:0/3967035871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306e20 msgr2=0x55849130a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f357c8c1700 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306e20 0x55849130a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f357c8c1700 1 -- 172.21.15.121:0/3967035871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 msgr2=0x55849130cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f357c8c1700 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x55849130cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f357c8c1700 1 -- 172.21.15.121:0/3967035871 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558491217850 con 0x558491306500 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f357c8c1700 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491309e00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3568031040 tx=0x7f3568024640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1774f1949f6d2c23 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:46.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f3560ff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f3560ff9700 1 -- 172.21.15.121:0/3967035871 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356800f520 con 0x558491306500 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3967035871 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f356800fba0 con 0x558491306500 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.787+0000 7f3560ff9700 1 -- 172.21.15.121:0/3967035871 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f356800f6c0 con 0x558491306500 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f3560ff9700 1 -- 172.21.15.121:0/3967035871 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f356800f8c0 con 0x558491306500 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f3589ba9b80 10 client.16470 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3967035871 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558491317e60 con 0x558491306500 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f3560ff9700 1 -- 172.21.15.121:0/3967035871 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f356800fba0 con 0x558491306500 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f3560ff9700 1 -- 172.21.15.121:0/3967035871 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3568037490 con 0x558491306500 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f3589ba9b80 10 client.16470 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f3589ba9b80 10 client.16470 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f3589ba9b80 20 client.16470 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55849131a330 0x55849131c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f3589ba9b80 4 client.16470 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:34:46.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3967035871 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55849125ee50 con 0x55849131a330 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.788+0000 7f357d0c2700 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55849131a330 0x55849131c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.789+0000 7f357d0c2700 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55849131a330 0x55849131c720 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92bcaee3780b5a52 server_cookie=b135b70936334690 in_seq=0 out_seq=0 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.789+0000 7f3560ff9700 10 client.16470 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.789+0000 7f3560ff9700 1 -- 172.21.15.121:0/3967035871 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55849125ee50 con 0x55849131a330 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.789+0000 7f3560ff9700 10 client.16470 handle_command_reply: tid=0 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 1 client.16470 shutdown 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3967035871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 msgr2=0x558491309e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491309e00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3568031040 tx=0x7f3568024640 comp rx=0 tx=0).stop 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3967035871 shutdown_connections 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55849131a330 0x55849131c720 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306e20 0x55849130a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3560ff9700 0 client.16470 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x55849130cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/3967035871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491309e00 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3967035871 >> 172.21.15.121:0/3967035871 conn(0x55849122d180 msgr2=0x5584912fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:46.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3967035871 shutdown_connections 2024-06-10T00:34:46.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3967035871 wait complete. 2024-06-10T00:34:46.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.790+0000 7f3589ba9b80 20 client.16470 trim_cache size 0 max 16384 2024-06-10T00:34:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.810+0000 7f3589ba9b80 1 Processor -- start 2024-06-10T00:34:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.810+0000 7f3589ba9b80 1 -- start start 2024-06-10T00:34:46.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.810+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x558491316140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.810+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491316680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.810+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306e20 0x5584912fb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.810+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f35781b2fa0 con 0x558491306500 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.810+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f35781b07d0 con 0x558491300c50 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.810+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35781ae670 con 0x558491306e20 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.810+0000 7f357c8c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491316680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.810+0000 7f357c8c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491316680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57446/0 (socket says 172.21.15.121:57446) 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f357c8c1700 1 -- 172.21.15.121:0/3966850906 learned_addr learned my addr 172.21.15.121:0/3966850906 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f357d0c2700 1 --2- 172.21.15.121:0/3966850906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x558491316140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f357d8c3700 1 --2- 172.21.15.121:0/3966850906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306e20 0x5584912fb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f357c8c1700 1 -- 172.21.15.121:0/3966850906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306e20 msgr2=0x5584912fb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f357c8c1700 1 --2- 172.21.15.121:0/3966850906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306e20 0x5584912fb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f357c8c1700 1 -- 172.21.15.121:0/3966850906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 msgr2=0x558491316140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:46.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f357c8c1700 1 --2- 172.21.15.121:0/3966850906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x558491316140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f357c8c1700 1 -- 172.21.15.121:0/3966850906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584912fcd90 con 0x558491306500 2024-06-10T00:34:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f357d0c2700 1 --2- 172.21.15.121:0/3966850906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x558491316140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f357c8c1700 1 --2- 172.21.15.121:0/3966850906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491316680 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f356800ba30 tx=0x7f3568005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b674ceb086244f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f3562ffd700 1 -- 172.21.15.121:0/3966850906 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356800fa60 con 0x558491306500 2024-06-10T00:34:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f3562ffd700 1 -- 172.21.15.121:0/3966850906 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35680246b0 con 0x558491306500 2024-06-10T00:34:46.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.811+0000 7f3562ffd700 1 -- 172.21.15.121:0/3966850906 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3568037400 con 0x558491306500 2024-06-10T00:34:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.812+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3966850906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 msgr2=0x558491316680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.812+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/3966850906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491316680 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f356800ba30 tx=0x7f3568005ce0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.812+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3966850906 shutdown_connections 2024-06-10T00:34:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.812+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/3966850906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306e20 0x5584912fb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.812+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/3966850906 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x558491316140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.812+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/3966850906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306500 0x558491316680 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.812+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3966850906 >> 172.21.15.121:0/3966850906 conn(0x55849122d180 msgr2=0x558491315a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.812+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3966850906 shutdown_connections 2024-06-10T00:34:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.812+0000 7f3589ba9b80 1 -- 172.21.15.121:0/3966850906 wait complete. 2024-06-10T00:34:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.812+0000 7f3589ba9b80 1 -- start start 2024-06-10T00:34:46.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.812+0000 7f3589ba9b80 1 Processor -- start 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x55849131ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306500 0x55849131f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f3589ba9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x55849137dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f35781aece0 con 0x558491306e20 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f35781b0100 con 0x558491300c50 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f3589ba9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35781af360 con 0x558491306500 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f3589ba9b80 10 client.0 _ll_register_callbacks cb 0x5584911e76b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357d8c3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x55849137dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357d8c3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x55849137dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57462/0 (socket says 172.21.15.121:57462) 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357d8c3700 1 -- 172.21.15.121:0/1944263365 learned_addr learned my addr 172.21.15.121:0/1944263365 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357c8c1700 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306500 0x55849131f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357d0c2700 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x55849131ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357d8c3700 1 -- 172.21.15.121:0/1944263365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306500 msgr2=0x55849131f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:46.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357d8c3700 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306500 0x55849131f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357d8c3700 1 -- 172.21.15.121:0/1944263365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 msgr2=0x55849131ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357d8c3700 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x55849131ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357d8c3700 1 -- 172.21.15.121:0/1944263365 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55849137e320 con 0x558491306e20 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357c8c1700 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306500 0x55849131f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f357d8c3700 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x55849137dc40 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f35700079d0 tx=0x7f3570007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f69be6b911afd5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.813+0000 7f3560ff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3560ff9700 1 -- 172.21.15.121:0/1944263365 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570017820 con 0x558491306e20 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3560ff9700 1 -- 172.21.15.121:0/1944263365 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f35700179c0 con 0x558491306e20 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1944263365 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35680246b0 con 0x558491306e20 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3560ff9700 1 -- 172.21.15.121:0/1944263365 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3570017bc0 con 0x558491306e20 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3589ba9b80 10 client.16476 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1944263365 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558491386880 con 0x558491306e20 2024-06-10T00:34:46.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3560ff9700 1 -- 172.21.15.121:0/1944263365 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35680246b0 con 0x558491306e20 2024-06-10T00:34:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3560ff9700 1 -- 172.21.15.121:0/1944263365 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7f3570016530 con 0x558491306e20 2024-06-10T00:34:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3589ba9b80 10 client.16476 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3589ba9b80 10 client.16476 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:34:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3589ba9b80 20 client.16476 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x558491386ff0 0x5584913893e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3589ba9b80 4 client.16476 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5095a648", "release": true, "await": true}] 2024-06-10T00:34:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.814+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1944263365 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5095a648", "release": true, "await": true}) -- 0x5584912593d0 con 0x558491386ff0 2024-06-10T00:34:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.815+0000 7f357d0c2700 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x558491386ff0 0x5584913893e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.815+0000 7f357d0c2700 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x558491386ff0 0x5584913893e0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce70bd545d7c5c2c server_cookie=cdb6796156df5e2f in_seq=0 out_seq=0 2024-06-10T00:34:46.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:46.815+0000 7f3560ff9700 10 client.16476 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:47.296 INFO:tasks.workunit.client.0.smithi121.stdout: 258 143897 0 96871 0 46688 0 2024-06-10T00:34:47.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:47 smithi135 ceph-mon[36424]: pgmap v207: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 607 KiB/s wr, 504 op/s 2024-06-10T00:34:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:47 smithi193 ceph-mon[36716]: pgmap v207: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 607 KiB/s wr, 504 op/s 2024-06-10T00:34:47.675 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:47 smithi121 ceph-mon[32378]: pgmap v207: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 607 KiB/s wr, 504 op/s 2024-06-10T00:34:47.741 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.740+0000 7f3560ff9700 1 -- 172.21.15.121:0/1944263365 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5584912593d0 con 0x558491386ff0 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.740+0000 7f3560ff9700 10 client.16476 handle_command_reply: tid=0 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 1 client.16476 shutdown 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1944263365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 msgr2=0x55849137dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x55849137dc40 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f35700079d0 tx=0x7f3570007a00 comp rx=0 tx=0).stop 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1944263365 shutdown_connections 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x558491386ff0 0x5584913893e0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558491306e20 0x55849137dc40 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x558491300c50 0x55849131ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3560ff9700 0 client.16476 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 1 --2- 172.21.15.121:0/1944263365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558491306500 0x55849131f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1944263365 >> 172.21.15.121:0/1944263365 conn(0x55849122d180 msgr2=0x5584913162c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:47.742 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1944263365 shutdown_connections 2024-06-10T00:34:47.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 1 -- 172.21.15.121:0/1944263365 wait complete. 2024-06-10T00:34:47.743 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.741+0000 7f3589ba9b80 20 client.16476 trim_cache size 0 max 16384 2024-06-10T00:34:47.743 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:34:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 30, 2024-06-10T00:34:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:34:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 20, 2024-06-10T00:34:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:34:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "5095a648": { 2024-06-10T00:34:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 20, 2024-06-10T00:34:47.743 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 149.4, 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:47.744 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:47.745 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:34:47.745 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:34:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 -- 172.21.15.121:0/3024826151 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35640a2d90 msgr2=0x7f35640a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35640a2d90 0x7f35640a5250 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f356800b5c0 tx=0x7f3568005960 comp rx=0 tx=0).stop 2024-06-10T00:34:47.745 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 -- 172.21.15.121:0/3024826151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3578102550 msgr2=0x7f357819ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3578102550 0x7f357819ae40 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f357400d980 tx=0x7f357400dc90 comp rx=0 tx=0).stop 2024-06-10T00:34:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 -- 172.21.15.121:0/3024826151 shutdown_connections 2024-06-10T00:34:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35640a2d90 0x7f35640a5250 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35781084f0 0x7f357819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3578102550 0x7f357819ae40 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 --2- 172.21.15.121:0/3024826151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3578102f10 0x7f357819b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 -- 172.21.15.121:0/3024826151 >> 172.21.15.121:0/3024826151 conn(0x7f3578079650 msgr2=0x7f35781003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 -- 172.21.15.121:0/3024826151 shutdown_connections 2024-06-10T00:34:47.746 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:47.742+0000 7f357f326700 1 -- 172.21.15.121:0/3024826151 wait complete. 2024-06-10T00:34:48.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.426+0000 7f452a4da700 1 -- 172.21.15.121:0/430895306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4524101440 msgr2=0x7f452410f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:48.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.426+0000 7f452a4da700 1 --2- 172.21.15.121:0/430895306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4524101440 0x7f452410f330 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f451c00b0a0 tx=0x7f451c00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:34:48.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.426+0000 7f452a4da700 1 -- 172.21.15.121:0/430895306 shutdown_connections 2024-06-10T00:34:48.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.426+0000 7f452a4da700 1 --2- 172.21.15.121:0/430895306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4524101440 0x7f452410f330 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:48.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.426+0000 7f452a4da700 1 --2- 172.21.15.121:0/430895306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524108510 0x7f45241088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:48.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.426+0000 7f452a4da700 1 --2- 172.21.15.121:0/430895306 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4524100a80 0x7f4524100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:48.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.426+0000 7f452a4da700 1 -- 172.21.15.121:0/430895306 >> 172.21.15.121:0/430895306 conn(0x7f4524077a00 msgr2=0x7f45240fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:48.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.426+0000 7f452a4da700 1 -- 172.21.15.121:0/430895306 shutdown_connections 2024-06-10T00:34:48.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.426+0000 7f452a4da700 1 -- 172.21.15.121:0/430895306 wait complete. 2024-06-10T00:34:48.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f452a4da700 1 Processor -- start 2024-06-10T00:34:48.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f452a4da700 1 -- start start 2024-06-10T00:34:48.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f452a4da700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4524100a80 0x7f4524101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:48.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f452a4da700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524101440 0x7f45241021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:48.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f452a4da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4524108510 0x7f45241026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:48.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f452a4da700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4524112790 con 0x7f4524108510 2024-06-10T00:34:48.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f452a4da700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4524112610 con 0x7f4524100a80 2024-06-10T00:34:48.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f452a4da700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4524112910 con 0x7f4524101440 2024-06-10T00:34:48.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f4523fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4524100a80 0x7f4524101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:48.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f4523fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4524100a80 0x7f4524101c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:50770/0 (socket says 172.21.15.121:50770) 2024-06-10T00:34:48.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f4523fff700 1 -- 172.21.15.121:0/500836533 learned_addr learned my addr 172.21.15.121:0/500836533 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:48.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.427+0000 7f4528a77700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4524108510 0x7f45241026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:48.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f451bfff700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524101440 0x7f45241021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:48.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f4523fff700 1 -- 172.21.15.121:0/500836533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524101440 msgr2=0x7f45241021a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:48.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f4523fff700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524101440 0x7f45241021a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:48.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f4523fff700 1 -- 172.21.15.121:0/500836533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4524108510 msgr2=0x7f45241026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:48.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f4523fff700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4524108510 0x7f45241026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:48.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f4523fff700 1 -- 172.21.15.121:0/500836533 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45241a7190 con 0x7f4524100a80 2024-06-10T00:34:48.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f4523fff700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4524100a80 0x7f4524101c60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f451400e9c0 tx=0x7f451400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:48.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f4521ffb700 1 -- 172.21.15.121:0/500836533 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451400cbd0 con 0x7f4524100a80 2024-06-10T00:34:48.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f4521ffb700 1 -- 172.21.15.121:0/500836533 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f451400ee60 con 0x7f4524100a80 2024-06-10T00:34:48.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f452a4da700 1 -- 172.21.15.121:0/500836533 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45241a7480 con 0x7f4524100a80 2024-06-10T00:34:48.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f4521ffb700 1 -- 172.21.15.121:0/500836533 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4514005630 con 0x7f4524100a80 2024-06-10T00:34:48.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.428+0000 7f452a4da700 1 -- 172.21.15.121:0/500836533 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45241a79c0 con 0x7f4524100a80 2024-06-10T00:34:48.431 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.429+0000 7f452a4da700 1 -- 172.21.15.121:0/500836533 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f452410a9f0 con 0x7f4524100a80 2024-06-10T00:34:48.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.430+0000 7f4521ffb700 1 -- 172.21.15.121:0/500836533 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4514044bb0 con 0x7f4524100a80 2024-06-10T00:34:48.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.431+0000 7f4521ffb700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f45100a2e10 0x7f45100a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:48.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.431+0000 7f451bfff700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f45100a2e10 0x7f45100a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:48.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.431+0000 7f4521ffb700 1 -- 172.21.15.121:0/500836533 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f45140cf920 con 0x7f4524100a80 2024-06-10T00:34:48.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.431+0000 7f451bfff700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f45100a2e10 0x7f45100a52d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4524107c40 tx=0x7f450c009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:48.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.435+0000 7f4521ffb700 1 -- 172.21.15.121:0/500836533 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f451409c1b0 con 0x7f4524100a80 2024-06-10T00:34:48.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.716+0000 7f452a4da700 1 -- 172.21.15.121:0/500836533 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4524069180 con 0x7f4524100a80 2024-06-10T00:34:48.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.717+0000 7f4521ffb700 1 -- 172.21.15.121:0/500836533 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11891 (secure 0 0 0) 0x7f451409bab0 con 0x7f4524100a80 2024-06-10T00:34:48.717 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:34:48.718 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:33:03.460049+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:34:48.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.719+0000 7f452a4da700 1 -- 172.21.15.121:0/500836533 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f45100a2e10 msgr2=0x7f45100a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.719+0000 7f452a4da700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f45100a2e10 0x7f45100a52d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4524107c40 tx=0x7f450c009260 comp rx=0 tx=0).stop 2024-06-10T00:34:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.719+0000 7f452a4da700 1 -- 172.21.15.121:0/500836533 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4524100a80 msgr2=0x7f4524101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.719+0000 7f452a4da700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4524100a80 0x7f4524101c60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f451400e9c0 tx=0x7f451400ecd0 comp rx=0 tx=0).stop 2024-06-10T00:34:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.719+0000 7f452a4da700 1 -- 172.21.15.121:0/500836533 shutdown_connections 2024-06-10T00:34:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.719+0000 7f452a4da700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f45100a2e10 0x7f45100a52d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.719+0000 7f452a4da700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4524108510 0x7f45241026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.719+0000 7f452a4da700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4524100a80 0x7f4524101c60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.719+0000 7f452a4da700 1 --2- 172.21.15.121:0/500836533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4524101440 0x7f45241021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.719+0000 7f452a4da700 1 -- 172.21.15.121:0/500836533 >> 172.21.15.121:0/500836533 conn(0x7f4524077a00 msgr2=0x7f45240ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.720+0000 7f452a4da700 1 -- 172.21.15.121:0/500836533 shutdown_connections 2024-06-10T00:34:48.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:48.720+0000 7f452a4da700 1 -- 172.21.15.121:0/500836533 wait complete. 2024-06-10T00:34:48.721 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 30 2024-06-10T00:34:48.973 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5095a648', total seconds elapsed: 40.1 2024-06-10T00:34:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:49 smithi135 ceph-mon[36424]: pgmap v208: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 400 KiB/s wr, 256 op/s 2024-06-10T00:34:49.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:49 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/500836533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:49.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:49 smithi121 ceph-mon[32378]: pgmap v208: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 400 KiB/s wr, 256 op/s 2024-06-10T00:34:49.625 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:49 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/500836533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:49.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:49 smithi193 ceph-mon[36716]: pgmap v208: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 400 KiB/s wr, 256 op/s 2024-06-10T00:34:49.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:49 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/500836533' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:34:49.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-10T00:34:49.908 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:50.039 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-06-10T00:34:50.039 INFO:tasks.workunit.client.0.smithi121.stdout:Final score for writes: 258 2024-06-10T00:34:50.040 INFO:tasks.workunit.client.0.smithi121.stdout:Final score for reads : 391015 2024-06-10T00:34:50.040 INFO:tasks.workunit.client.0.smithi121.stdout: 2024-06-10T00:34:50.040 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-10T00:34:50.040 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-10T00:34:51.428 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:51 smithi121 ceph-mon[32378]: pgmap v209: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 430 KiB/s wr, 269 op/s 2024-06-10T00:34:51.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:51 smithi135 ceph-mon[36424]: pgmap v209: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 430 KiB/s wr, 269 op/s 2024-06-10T00:34:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:51 smithi193 ceph-mon[36716]: pgmap v209: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 430 KiB/s wr, 269 op/s 2024-06-10T00:34:51.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.797+0000 7ff3ee50d700 1 -- 172.21.15.121:0/1216629516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3e8109390 msgr2=0x7ff3e8115890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:51.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.797+0000 7ff3ee50d700 1 --2- 172.21.15.121:0/1216629516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3e8109390 0x7ff3e8115890 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009f90 tx=0x7ff3e000b330 comp rx=0 tx=0).stop 2024-06-10T00:34:51.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.797+0000 7ff3ee50d700 1 -- 172.21.15.121:0/1216629516 shutdown_connections 2024-06-10T00:34:51.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.797+0000 7ff3ee50d700 1 --2- 172.21.15.121:0/1216629516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3e8109390 0x7ff3e8115890 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.797+0000 7ff3ee50d700 1 --2- 172.21.15.121:0/1216629516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e810e9c0 0x7ff3e810eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.797+0000 7ff3ee50d700 1 --2- 172.21.15.121:0/1216629516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3e81089d0 0x7ff3e8108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.797+0000 7ff3ee50d700 1 -- 172.21.15.121:0/1216629516 >> 172.21.15.121:0/1216629516 conn(0x7ff3e80781f0 msgr2=0x7ff3e81072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:51.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.798+0000 7ff3ee50d700 1 -- 172.21.15.121:0/1216629516 shutdown_connections 2024-06-10T00:34:51.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.798+0000 7ff3ee50d700 1 -- 172.21.15.121:0/1216629516 wait complete. 2024-06-10T00:34:51.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.798+0000 7ff3ee50d700 1 Processor -- start 2024-06-10T00:34:51.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.798+0000 7ff3ee50d700 1 -- start start 2024-06-10T00:34:51.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.799+0000 7ff3ee50d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3e81089d0 0x7ff3e819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.799+0000 7ff3ee50d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3e8109390 0x7ff3e819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.799+0000 7ff3ee50d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e810e9c0 0x7ff3e81a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.799+0000 7ff3ee50d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8118d40 con 0x7ff3e81089d0 2024-06-10T00:34:51.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.799+0000 7ff3ed50b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3e81089d0 0x7ff3e819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.799+0000 7ff3ecd0a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3e8109390 0x7ff3e819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.799+0000 7ff3ecd0a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3e8109390 0x7ff3e819f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:52694/0 (socket says 172.21.15.121:52694) 2024-06-10T00:34:51.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.799+0000 7ff3ecd0a700 1 -- 172.21.15.121:0/2351682512 learned_addr learned my addr 172.21.15.121:0/2351682512 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:51.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.799+0000 7ff3ee50d700 1 -- 172.21.15.121:0/2351682512 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8118bc0 con 0x7ff3e810e9c0 2024-06-10T00:34:51.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.799+0000 7ff3ee50d700 1 -- 172.21.15.121:0/2351682512 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8118ec0 con 0x7ff3e8109390 2024-06-10T00:34:51.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.801+0000 7ff3ed50b700 1 -- 172.21.15.121:0/2351682512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3e8109390 msgr2=0x7ff3e819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:51.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.801+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3e8109390 0x7ff3e819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.801+0000 7ff3ed50b700 1 -- 172.21.15.121:0/2351682512 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e810e9c0 msgr2=0x7ff3e81a39e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-10T00:34:51.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.801+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e810e9c0 0x7ff3e81a39e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.801+0000 7ff3ed50b700 1 -- 172.21.15.121:0/2351682512 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e81a4120 con 0x7ff3e81089d0 2024-06-10T00:34:51.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.801+0000 7ff3ecd0a700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3e8109390 0x7ff3e819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:34:51.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.802+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3e81089d0 0x7ff3e819f110 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7ff3d800e9c0 tx=0x7ff3d800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:51.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.803+0000 7ff3de7fc700 1 -- 172.21.15.121:0/2351682512 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d800cbd0 con 0x7ff3e81089d0 2024-06-10T00:34:51.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.803+0000 7ff3de7fc700 1 -- 172.21.15.121:0/2351682512 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3d8009e50 con 0x7ff3e81089d0 2024-06-10T00:34:51.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.803+0000 7ff3ee50d700 1 -- 172.21.15.121:0/2351682512 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e81a4410 con 0x7ff3e81089d0 2024-06-10T00:34:51.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.803+0000 7ff3de7fc700 1 -- 172.21.15.121:0/2351682512 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d8018680 con 0x7ff3e81089d0 2024-06-10T00:34:51.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.804+0000 7ff3ee50d700 1 -- 172.21.15.121:0/2351682512 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e81abc90 con 0x7ff3e81089d0 2024-06-10T00:34:51.809 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.809+0000 7ff3de7fc700 1 -- 172.21.15.121:0/2351682512 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff3d8010430 con 0x7ff3e81089d0 2024-06-10T00:34:51.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.810+0000 7ff3de7fc700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3d40a2f60 0x7ff3d40a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.811+0000 7ff3de7fc700 1 -- 172.21.15.121:0/2351682512 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff3d80cf780 con 0x7ff3e81089d0 2024-06-10T00:34:51.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.812+0000 7ff3f9ff2b80 1 Processor -- start 2024-06-10T00:34:51.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.812+0000 7ff3ecd0a700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3d40a2f60 0x7ff3d40a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.813+0000 7ff3f9ff2b80 1 -- start start 2024-06-10T00:34:51.813 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.813+0000 7ff3ecd0a700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3d40a2f60 0x7ff3d40a5420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8108440 tx=0x7ff3e400b3a0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:51.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.813+0000 7ff3f9ff2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 0x5557aba07cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.813+0000 7ff3f9ff2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 0x5557aba01b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.814+0000 7ff3ed50b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 0x5557aba07cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.814 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.814+0000 7ff3ed50b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 0x5557aba07cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:50344/0 (socket says 172.21.15.121:50344) 2024-06-10T00:34:51.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.814+0000 7ff3ed50b700 1 -- 172.21.15.121:0/2099750155 learned_addr learned my addr 172.21.15.121:0/2099750155 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:51.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.814+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/2099750155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba02080 0x5557aba024c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.814+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2099750155 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5557ab953810 con 0x5557aba08210 2024-06-10T00:34:51.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.814+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2099750155 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5557ab8fb720 con 0x5557aba078f0 2024-06-10T00:34:51.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.814+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2099750155 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5557ab94f7d0 con 0x5557aba02080 2024-06-10T00:34:51.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.814+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/2099750155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba02080 0x5557aba024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.815 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.814+0000 7ff3ecd0a700 1 --2- 172.21.15.121:0/2099750155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 0x5557aba01b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.815+0000 7ff3ed50b700 1 -- 172.21.15.121:0/2099750155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba02080 msgr2=0x5557aba024c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:51.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.815+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/2099750155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba02080 0x5557aba024c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.815+0000 7ff3ed50b700 1 -- 172.21.15.121:0/2099750155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 msgr2=0x5557aba01b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:51.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.815+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/2099750155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 0x5557aba01b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.815+0000 7ff3ed50b700 1 -- 172.21.15.121:0/2099750155 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557ab8f1f50 con 0x5557aba078f0 2024-06-10T00:34:51.816 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.816+0000 7ff3ecd0a700 1 --2- 172.21.15.121:0/2099750155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 0x5557aba01b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:51.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.817+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/2099750155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 0x5557aba07cd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff3d80a3ff0 tx=0x7ff3d809b290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8a7b7b1537e0a79 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:51.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.817+0000 7ff3d37fe700 1 -- 172.21.15.121:0/2099750155 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d809bfb0 con 0x5557aba078f0 2024-06-10T00:34:51.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.818+0000 7ff3d37fe700 1 -- 172.21.15.121:0/2099750155 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3d809b8b0 con 0x5557aba078f0 2024-06-10T00:34:51.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.818+0000 7ff3d37fe700 1 -- 172.21.15.121:0/2099750155 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d809bfb0 con 0x5557aba078f0 2024-06-10T00:34:51.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.819+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2099750155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 msgr2=0x5557aba07cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:51.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.819+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/2099750155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 0x5557aba07cd0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff3d80a3ff0 tx=0x7ff3d809b290 comp rx=0 tx=0).stop 2024-06-10T00:34:51.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.819+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2099750155 shutdown_connections 2024-06-10T00:34:51.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.819+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/2099750155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba02080 0x5557aba024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.819+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/2099750155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 0x5557aba07cd0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.819+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/2099750155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 0x5557aba01b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.819+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2099750155 >> 172.21.15.121:0/2099750155 conn(0x5557ab92ccc0 msgr2=0x5557ab9fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:51.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.820+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2099750155 shutdown_connections 2024-06-10T00:34:51.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.820+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2099750155 wait complete. 2024-06-10T00:34:51.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.821+0000 7ff3f9ff2b80 1 -- start start 2024-06-10T00:34:51.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.822+0000 7ff3f9ff2b80 1 Processor -- start 2024-06-10T00:34:51.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.822+0000 7ff3f9ff2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba02080 0x5557aba0dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.822+0000 7ff3f9ff2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba078f0 0x5557aba0b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.822+0000 7ff3f9ff2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 0x5557aba0b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.823+0000 7ff3f9ff2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5557aba0eb70 con 0x5557aba078f0 2024-06-10T00:34:51.824 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.823+0000 7ff3f9ff2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5557ab8f3bf0 con 0x5557aba02080 2024-06-10T00:34:51.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.823+0000 7ff3f9ff2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5557aba0ecf0 con 0x5557aba08210 2024-06-10T00:34:51.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.823+0000 7ff3ecd0a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba078f0 0x5557aba0b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.823+0000 7ff3ecd0a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba078f0 0x5557aba0b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42716/0 (socket says 172.21.15.121:42716) 2024-06-10T00:34:51.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.823+0000 7ff3ecd0a700 1 -- 172.21.15.121:0/233235424 learned_addr learned my addr 172.21.15.121:0/233235424 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:51.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.823+0000 7ff3f9ff2b80 10 client.0 _ll_register_callbacks cb 0x5557ab8ed920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:51.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.824+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 0x5557aba0b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.825 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.825+0000 7ff3edd0c700 1 -- 172.21.15.121:0/233235424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba02080 msgr2=0x5557aba0dec0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:34:51.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.825+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba02080 0x5557aba0dec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.825+0000 7ff3edd0c700 1 -- 172.21.15.121:0/233235424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba078f0 msgr2=0x5557aba0b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:51.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.825+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba078f0 0x5557aba0b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.825+0000 7ff3edd0c700 1 -- 172.21.15.121:0/233235424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557ab8a28d0 con 0x5557aba08210 2024-06-10T00:34:51.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.825+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 0x5557aba0b700 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000d010 tx=0x7ff3e000bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4326a4fbaf99aead server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:51.826 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.825+0000 7ff3d0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:34:51.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.828+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/233235424 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0003c00 con 0x5557aba08210 2024-06-10T00:34:51.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.828+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/233235424 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff3e0003da0 con 0x5557aba08210 2024-06-10T00:34:51.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.828+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/233235424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3d809b8b0 con 0x5557aba08210 2024-06-10T00:34:51.828 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.828+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/233235424 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff3e002a640 con 0x5557aba08210 2024-06-10T00:34:51.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.828+0000 7ff3f9ff2b80 10 client.25957 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:51.830 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.829+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/233235424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5557aba2a4a0 con 0x5557aba08210 2024-06-10T00:34:51.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.831+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/233235424 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3d809b8b0 con 0x5557aba08210 2024-06-10T00:34:51.831 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.831+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/233235424 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff3e0011070 con 0x5557aba08210 2024-06-10T00:34:51.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.831+0000 7ff3f9ff2b80 10 client.25957 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:51.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.831+0000 7ff3f9ff2b80 10 client.25957 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:34:51.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.831+0000 7ff3f9ff2b80 20 client.25957 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:51.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.832+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5557aba2ca80 0x5557aba2eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.832+0000 7ff3f9ff2b80 4 client.25957 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:34:51.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.832+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/233235424 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557ab960d10 con 0x5557aba2ca80 2024-06-10T00:34:51.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.832+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5557aba2ca80 0x5557aba2eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.833 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.834+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5557aba2ca80 0x5557aba2eeb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f050b9c6466a0f85 server_cookie=36e08217bf789e87 in_seq=0 out_seq=0 2024-06-10T00:34:51.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.834+0000 7ff3d0ff9700 10 client.25957 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:34:51.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.835+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/233235424 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5557ab960d10 con 0x5557aba2ca80 2024-06-10T00:34:51.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.836+0000 7ff3d0ff9700 10 client.25957 handle_command_reply: tid=0 2024-06-10T00:34:51.836 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.836+0000 7ff3f9ff2b80 1 client.25957 shutdown 2024-06-10T00:34:51.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.836+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/233235424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 msgr2=0x5557aba0b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:51.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.836+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 0x5557aba0b700 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff3e000d010 tx=0x7ff3e000bd20 comp rx=0 tx=0).stop 2024-06-10T00:34:51.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.836+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/233235424 shutdown_connections 2024-06-10T00:34:51.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.837+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5557aba2ca80 0x5557aba2eeb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.837+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 0x5557aba0b700 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.837+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba02080 0x5557aba0dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.837+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/233235424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba078f0 0x5557aba0b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.837+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/233235424 >> 172.21.15.121:0/233235424 conn(0x5557ab92ccc0 msgr2=0x5557aba00b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:51.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.837+0000 7ff3d0ff9700 0 client.25957 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:34:51.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.837+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/233235424 shutdown_connections 2024-06-10T00:34:51.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.837+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/233235424 wait complete. 2024-06-10T00:34:51.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.838+0000 7ff3f9ff2b80 20 client.25957 trim_cache size 0 max 16384 2024-06-10T00:34:51.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.865+0000 7ff3f9ff2b80 1 Processor -- start 2024-06-10T00:34:51.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.866+0000 7ff3f9ff2b80 1 -- start start 2024-06-10T00:34:51.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.866+0000 7ff3f9ff2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba02080 0x5557aba0b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.866+0000 7ff3f9ff2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba078f0 0x5557aba0b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.866 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.866+0000 7ff3f9ff2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 0x5557aba01480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.866+0000 7ff3f9ff2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e81aebe0 con 0x5557aba08210 2024-06-10T00:34:51.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.866+0000 7ff3f9ff2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e81ac410 con 0x5557aba02080 2024-06-10T00:34:51.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.866+0000 7ff3f9ff2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8072ac0 con 0x5557aba078f0 2024-06-10T00:34:51.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.866+0000 7ff3ecd0a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba078f0 0x5557aba0b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.866+0000 7ff3edd0c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 0x5557aba01480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.867 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.866+0000 7ff3ecd0a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba078f0 0x5557aba0b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:52724/0 (socket says 172.21.15.121:52724) 2024-06-10T00:34:51.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.866+0000 7ff3ecd0a700 1 -- 172.21.15.121:0/3685577620 learned_addr learned my addr 172.21.15.121:0/3685577620 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:51.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.867+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/3685577620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba02080 0x5557aba0b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.867+0000 7ff3edd0c700 1 -- 172.21.15.121:0/3685577620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba078f0 msgr2=0x5557aba0b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:51.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.867+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/3685577620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba078f0 0x5557aba0b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.867+0000 7ff3edd0c700 1 -- 172.21.15.121:0/3685577620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba02080 msgr2=0x5557aba0b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:51.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.867+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/3685577620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba02080 0x5557aba0b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.867+0000 7ff3edd0c700 1 -- 172.21.15.121:0/3685577620 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557aba0db00 con 0x5557aba08210 2024-06-10T00:34:51.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.867+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/3685577620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 0x5557aba01480 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff3e002c040 tx=0x7ff3e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555d1b97520d33f server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.868+0000 7ff3d2ffd700 1 -- 172.21.15.121:0/3685577620 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0005e70 con 0x5557aba08210 2024-06-10T00:34:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.868+0000 7ff3d2ffd700 1 -- 172.21.15.121:0/3685577620 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3e0006010 con 0x5557aba08210 2024-06-10T00:34:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.868+0000 7ff3d2ffd700 1 -- 172.21.15.121:0/3685577620 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e002aa40 con 0x5557aba08210 2024-06-10T00:34:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.869+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/3685577620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 msgr2=0x5557aba01480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:51.869 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.869+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/3685577620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 0x5557aba01480 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ff3e002c040 tx=0x7ff3e000b330 comp rx=0 tx=0).stop 2024-06-10T00:34:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.870+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/3685577620 shutdown_connections 2024-06-10T00:34:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.870+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/3685577620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba08210 0x5557aba01480 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.870+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/3685577620 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba02080 0x5557aba0b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.870+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/3685577620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba078f0 0x5557aba0b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.870 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.870+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/3685577620 >> 172.21.15.121:0/3685577620 conn(0x5557ab92ccc0 msgr2=0x5557aba02460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:51.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.872+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/3685577620 shutdown_connections 2024-06-10T00:34:51.872 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.872+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/3685577620 wait complete. 2024-06-10T00:34:51.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.873+0000 7ff3f9ff2b80 1 -- start start 2024-06-10T00:34:51.873 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.873+0000 7ff3f9ff2b80 1 Processor -- start 2024-06-10T00:34:51.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.874+0000 7ff3f9ff2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba02080 0x5557aba34fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.874+0000 7ff3f9ff2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 0x5557aba13f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.874+0000 7ff3f9ff2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 0x5557aba14620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.874+0000 7ff3f9ff2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8073130 con 0x5557aba02080 2024-06-10T00:34:51.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.874+0000 7ff3ecd0a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 0x5557aba13f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.874+0000 7ff3ecd0a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 0x5557aba13f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:50382/0 (socket says 172.21.15.121:50382) 2024-06-10T00:34:51.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.874+0000 7ff3ecd0a700 1 -- 172.21.15.121:0/2077990813 learned_addr learned my addr 172.21.15.121:0/2077990813 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:51.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.874+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 0x5557aba14620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.874+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba02080 0x5557aba34fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.875+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2077990813 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e81abe60 con 0x5557aba078f0 2024-06-10T00:34:51.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.875+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2077990813 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e8073800 con 0x5557aba08210 2024-06-10T00:34:51.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.875+0000 7ff3f9ff2b80 10 client.0 _ll_register_callbacks cb 0x5557ab8ed920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:51.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.875+0000 7ff3edd0c700 1 -- 172.21.15.121:0/2077990813 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 msgr2=0x5557aba13f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:51.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.875+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 0x5557aba13f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.875+0000 7ff3edd0c700 1 -- 172.21.15.121:0/2077990813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba02080 msgr2=0x5557aba34fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:51.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.875+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba02080 0x5557aba34fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.875+0000 7ff3edd0c700 1 -- 172.21.15.121:0/2077990813 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557aba337f0 con 0x5557aba08210 2024-06-10T00:34:51.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.876+0000 7ff3edd0c700 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 0x5557aba14620 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff3e002c040 tx=0x7ff3e0000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=832b4c50929392dc server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:51.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.878+0000 7ff3d0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:34:51.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.878+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/2077990813 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e0005e70 con 0x5557aba08210 2024-06-10T00:34:51.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.878+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/2077990813 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff3e002a9e0 con 0x5557aba08210 2024-06-10T00:34:51.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.879+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2077990813 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3e0006010 con 0x5557aba08210 2024-06-10T00:34:51.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.880+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/2077990813 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff3e003f050 con 0x5557aba08210 2024-06-10T00:34:51.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.880+0000 7ff3f9ff2b80 10 client.25963 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:51.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.880+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2077990813 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5557aba96a80 con 0x5557aba08210 2024-06-10T00:34:51.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.880+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/2077990813 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3e0006010 con 0x5557aba08210 2024-06-10T00:34:51.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.880+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/2077990813 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff3e0031070 con 0x5557aba08210 2024-06-10T00:34:51.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.881+0000 7ff3f9ff2b80 10 client.25963 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:51.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.881+0000 7ff3f9ff2b80 10 client.25963 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:34:51.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.881+0000 7ff3f9ff2b80 20 client.25963 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:51.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.881+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5557aba97130 0x5557aba99550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:51.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.881+0000 7ff3f9ff2b80 4 client.25963 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:34:51.881 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.881+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2077990813 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5557ab95b290 con 0x5557aba97130 2024-06-10T00:34:51.882 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.882+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5557aba97130 0x5557aba99550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:51.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.882+0000 7ff3ed50b700 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5557aba97130 0x5557aba99550 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bf7624264ca8f836 server_cookie=a50d33d37758ec21 in_seq=0 out_seq=0 2024-06-10T00:34:51.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.883+0000 7ff3d0ff9700 10 client.25963 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:34:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.886+0000 7ff3d0ff9700 1 -- 172.21.15.121:0/2077990813 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5557ab95b290 con 0x5557aba97130 2024-06-10T00:34:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.886+0000 7ff3d0ff9700 10 client.25963 handle_command_reply: tid=0 2024-06-10T00:34:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.886+0000 7ff3f9ff2b80 1 client.25963 shutdown 2024-06-10T00:34:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.887+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2077990813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 msgr2=0x5557aba14620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.887+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 0x5557aba14620 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff3e002c040 tx=0x7ff3e0000f40 comp rx=0 tx=0).stop 2024-06-10T00:34:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.887+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2077990813 shutdown_connections 2024-06-10T00:34:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.887+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5557aba97130 0x5557aba99550 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.887+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557aba08210 0x5557aba14620 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.888 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.887+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5557aba02080 0x5557aba34fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.887+0000 7ff3f9ff2b80 1 --2- 172.21.15.121:0/2077990813 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5557aba078f0 0x5557aba13f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.888+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2077990813 >> 172.21.15.121:0/2077990813 conn(0x5557ab92ccc0 msgr2=0x5557aba01420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.888+0000 7ff3d0ff9700 0 client.25963 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:34:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.888+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2077990813 shutdown_connections 2024-06-10T00:34:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.889+0000 7ff3f9ff2b80 1 -- 172.21.15.121:0/2077990813 wait complete. 2024-06-10T00:34:51.889 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.889+0000 7ff3f9ff2b80 20 client.25963 trim_cache size 0 max 16384 2024-06-10T00:34:51.890 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:34:51.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.891+0000 7ff3ee50d700 1 -- 172.21.15.121:0/2351682512 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3d40a2f60 msgr2=0x7ff3d40a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:51.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.891+0000 7ff3ee50d700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3d40a2f60 0x7ff3d40a5420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8108440 tx=0x7ff3e400b3a0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.891+0000 7ff3ee50d700 1 -- 172.21.15.121:0/2351682512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3e81089d0 msgr2=0x7ff3e819f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:51.891 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.891+0000 7ff3ee50d700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3e81089d0 0x7ff3e819f110 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7ff3d800e9c0 tx=0x7ff3d800ed80 comp rx=0 tx=0).stop 2024-06-10T00:34:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.891+0000 7ff3ee50d700 1 -- 172.21.15.121:0/2351682512 shutdown_connections 2024-06-10T00:34:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.891+0000 7ff3ee50d700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff3d40a2f60 0x7ff3d40a5420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.891+0000 7ff3ee50d700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff3e810e9c0 0x7ff3e81a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.891+0000 7ff3ee50d700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3e81089d0 0x7ff3e819f110 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.891+0000 7ff3ee50d700 1 --2- 172.21.15.121:0/2351682512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3e8109390 0x7ff3e819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.892+0000 7ff3ee50d700 1 -- 172.21.15.121:0/2351682512 >> 172.21.15.121:0/2351682512 conn(0x7ff3e80781f0 msgr2=0x7ff3e8078730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.892+0000 7ff3ee50d700 1 -- 172.21.15.121:0/2351682512 shutdown_connections 2024-06-10T00:34:51.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:51.892+0000 7ff3ee50d700 1 -- 172.21.15.121:0/2351682512 wait complete. 2024-06-10T00:34:52.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-10T00:34:52.954 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:53.416 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:53 smithi121 ceph-mon[32378]: pgmap v210: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 218 KiB/s wr, 123 op/s 2024-06-10T00:34:53.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:53 smithi193 ceph-mon[36716]: pgmap v210: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 218 KiB/s wr, 123 op/s 2024-06-10T00:34:53.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:53 smithi135 ceph-mon[36424]: pgmap v210: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 218 KiB/s wr, 123 op/s 2024-06-10T00:34:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.749+0000 7ff7d513a700 1 -- 172.21.15.121:0/935204251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7d01017f0 msgr2=0x7ff7d010f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.749+0000 7ff7d513a700 1 --2- 172.21.15.121:0/935204251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7d01017f0 0x7ff7d010f380 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800b0d0 tx=0x7ff7c800b3e0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.749+0000 7ff7d513a700 1 -- 172.21.15.121:0/935204251 shutdown_connections 2024-06-10T00:34:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.749+0000 7ff7d513a700 1 --2- 172.21.15.121:0/935204251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7d01017f0 0x7ff7d010f380 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.749 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.749+0000 7ff7d513a700 1 --2- 172.21.15.121:0/935204251 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7d0107500 0x7ff7d01078e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.749+0000 7ff7d513a700 1 --2- 172.21.15.121:0/935204251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d0100e30 0x7ff7d01012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.749+0000 7ff7d513a700 1 -- 172.21.15.121:0/935204251 >> 172.21.15.121:0/935204251 conn(0x7ff7d0076730 msgr2=0x7ff7d0076b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.749+0000 7ff7d513a700 1 -- 172.21.15.121:0/935204251 shutdown_connections 2024-06-10T00:34:54.750 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.749+0000 7ff7d513a700 1 -- 172.21.15.121:0/935204251 wait complete. 2024-06-10T00:34:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.750+0000 7ff7d513a700 1 Processor -- start 2024-06-10T00:34:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.750+0000 7ff7d513a700 1 -- start start 2024-06-10T00:34:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.750+0000 7ff7d513a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d0100e30 0x7ff7d019e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.750+0000 7ff7d513a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7d01017f0 0x7ff7d019ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.750+0000 7ff7d513a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7d0107500 0x7ff7d0198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.750+0000 7ff7d513a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01127c0 con 0x7ff7d0107500 2024-06-10T00:34:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.750+0000 7ff7d513a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d0112640 con 0x7ff7d01017f0 2024-06-10T00:34:54.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.750+0000 7ff7d513a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d0112940 con 0x7ff7d0100e30 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7ce59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7d01017f0 0x7ff7d019ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7cf59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7d0107500 0x7ff7d0198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7cf59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7d0107500 0x7ff7d0198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42744/0 (socket says 172.21.15.121:42744) 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7ce59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7d01017f0 0x7ff7d019ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:50392/0 (socket says 172.21.15.121:50392) 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7cf59e700 1 -- 172.21.15.121:0/1801666848 learned_addr learned my addr 172.21.15.121:0/1801666848 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7ced9d700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d0100e30 0x7ff7d019e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7ce59c700 1 -- 172.21.15.121:0/1801666848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d0100e30 msgr2=0x7ff7d019e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7ce59c700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d0100e30 0x7ff7d019e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7ce59c700 1 -- 172.21.15.121:0/1801666848 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7d0107500 msgr2=0x7ff7d0198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7ce59c700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7d0107500 0x7ff7d0198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7ce59c700 1 -- 172.21.15.121:0/1801666848 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d01990e0 con 0x7ff7d01017f0 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7cf59e700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7d0107500 0x7ff7d0198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:54.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7ce59c700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7d01017f0 0x7ff7d019ed00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff7b800e9f0 tx=0x7ff7b800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:54.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7c7fff700 1 -- 172.21.15.121:0/1801666848 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8004220 con 0x7ff7d01017f0 2024-06-10T00:34:54.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7c7fff700 1 -- 172.21.15.121:0/1801666848 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7b8004d10 con 0x7ff7d01017f0 2024-06-10T00:34:54.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.751+0000 7ff7c7fff700 1 -- 172.21.15.121:0/1801666848 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8005020 con 0x7ff7d01017f0 2024-06-10T00:34:54.753 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.752+0000 7ff7d513a700 1 -- 172.21.15.121:0/1801666848 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d01993d0 con 0x7ff7d01017f0 2024-06-10T00:34:54.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.752+0000 7ff7d513a700 1 -- 172.21.15.121:0/1801666848 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d01a5690 con 0x7ff7d01017f0 2024-06-10T00:34:54.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.755+0000 7ff7c7fff700 1 -- 172.21.15.121:0/1801666848 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff7b8004750 con 0x7ff7d01017f0 2024-06-10T00:34:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.756+0000 7ff7c7fff700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff7bc0a2d90 0x7ff7bc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.756+0000 7ff7c7fff700 1 -- 172.21.15.121:0/1801666848 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff7b80cf690 con 0x7ff7d01017f0 2024-06-10T00:34:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.756+0000 7ff7ced9d700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff7bc0a2d90 0x7ff7bc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.756+0000 7ff7df9bdb80 1 Processor -- start 2024-06-10T00:34:54.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.756+0000 7ff7df9bdb80 1 -- start start 2024-06-10T00:34:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1cf60 0x565197f1d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f17260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f177a0 0x565197f17bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7df9bdb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x565197e09f60 con 0x565197f1cf60 2024-06-10T00:34:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7df9bdb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x565197e68850 con 0x565197f177a0 2024-06-10T00:34:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7df9bdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x565197dd0460 con 0x565197f1d880 2024-06-10T00:34:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7ce59c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f17260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7cf59e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f177a0 0x565197f17bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7ce59c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f17260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:52756/0 (socket says 172.21.15.121:52756) 2024-06-10T00:34:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7ce59c700 1 -- 172.21.15.121:0/1766209246 learned_addr learned my addr 172.21.15.121:0/1766209246 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:54.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7ced9d700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff7bc0a2d90 0x7ff7bc0a5250 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0199ac0 tx=0x7ff7c0009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7ced9d700 1 --2- 172.21.15.121:0/1766209246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1cf60 0x565197f1d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7ce59c700 1 -- 172.21.15.121:0/1766209246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f177a0 msgr2=0x565197f17bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7ce59c700 1 --2- 172.21.15.121:0/1766209246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f177a0 0x565197f17bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7ce59c700 1 -- 172.21.15.121:0/1766209246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1cf60 msgr2=0x565197f1d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7ce59c700 1 --2- 172.21.15.121:0/1766209246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1cf60 0x565197f1d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7ce59c700 1 -- 172.21.15.121:0/1766209246 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565197db78d0 con 0x565197f1d880 2024-06-10T00:34:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.757+0000 7ff7ce59c700 1 --2- 172.21.15.121:0/1766209246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f17260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff7b80981f0 tx=0x7ff7b809b180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2bfe807bf38a364 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7c57fa700 1 -- 172.21.15.121:0/1766209246 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b801c040 con 0x565197f1d880 2024-06-10T00:34:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7c57fa700 1 -- 172.21.15.121:0/1766209246 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7b809fbd0 con 0x565197f1d880 2024-06-10T00:34:54.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7c57fa700 1 -- 172.21.15.121:0/1766209246 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b8086070 con 0x565197f1d880 2024-06-10T00:34:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1766209246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 msgr2=0x565197f17260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1766209246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f17260 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff7b80981f0 tx=0x7ff7b809b180 comp rx=0 tx=0).stop 2024-06-10T00:34:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1766209246 shutdown_connections 2024-06-10T00:34:54.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1766209246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f177a0 0x565197f17bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1766209246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1cf60 0x565197f1d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1766209246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f17260 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1766209246 >> 172.21.15.121:0/1766209246 conn(0x565197e44080 msgr2=0x565197f12220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1766209246 shutdown_connections 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.758+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1766209246 wait complete. 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7df9bdb80 1 -- start start 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7df9bdb80 1 Processor -- start 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f177a0 0x565197f23690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 0x565197f20990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f20ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7df9bdb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x565197e08fc0 con 0x565197f177a0 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7df9bdb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x565197e64810 con 0x565197f1cf60 2024-06-10T00:34:54.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7df9bdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x565197f24260 con 0x565197f1d880 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7df9bdb80 10 client.0 _ll_register_callbacks cb 0x565197dfe5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7ced9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f177a0 0x565197f23690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7ce59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 0x565197f20990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7ced9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f177a0 0x565197f23690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42770/0 (socket says 172.21.15.121:42770) 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7ced9d700 1 -- 172.21.15.121:0/259546460 learned_addr learned my addr 172.21.15.121:0/259546460 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.759+0000 7ff7cf59e700 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f20ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7ce59c700 1 -- 172.21.15.121:0/259546460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 msgr2=0x565197f20ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7ce59c700 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f20ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7ce59c700 1 -- 172.21.15.121:0/259546460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f177a0 msgr2=0x565197f23690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7ce59c700 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f177a0 0x565197f23690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7ce59c700 1 -- 172.21.15.121:0/259546460 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565197e2e750 con 0x565197f1cf60 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7ced9d700 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f177a0 0x565197f23690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7ce59c700 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 0x565197f20990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x565197f23f20 tx=0x7ff7b80cca70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df7c7b744d2bc2e8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:54.764 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7b67fc700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7b67fc700 1 -- 172.21.15.121:0/259546460 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b801c070 con 0x565197f1cf60 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7b67fc700 1 -- 172.21.15.121:0/259546460 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff7b80cf2b0 con 0x565197f1cf60 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/259546460 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7b809fbd0 con 0x565197f1cf60 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7b67fc700 1 -- 172.21.15.121:0/259546460 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff7b80cf4b0 con 0x565197f1cf60 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7df9bdb80 10 client.26066 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/259546460 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x565197f2eaa0 con 0x565197f1cf60 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7b67fc700 1 -- 172.21.15.121:0/259546460 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7b809fbd0 con 0x565197f1cf60 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.760+0000 7ff7b67fc700 1 -- 172.21.15.121:0/259546460 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff7b809ac70 con 0x565197f1cf60 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.761+0000 7ff7df9bdb80 10 client.26066 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.761+0000 7ff7df9bdb80 10 client.26066 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.761+0000 7ff7df9bdb80 20 client.26066 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.761+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x565197f30f70 0x565197f33360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.761+0000 7ff7df9bdb80 4 client.26066 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:34:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.761+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/259546460 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565197e75d50 con 0x565197f30f70 2024-06-10T00:34:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.761+0000 7ff7ced9d700 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x565197f30f70 0x565197f33360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.761+0000 7ff7ced9d700 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x565197f30f70 0x565197f33360 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c284838609a7a6d server_cookie=eef16ff49e184168 in_seq=0 out_seq=0 2024-06-10T00:34:54.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.761+0000 7ff7b67fc700 10 client.26066 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7b67fc700 1 -- 172.21.15.121:0/259546460 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x565197e75d50 con 0x565197f30f70 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7b67fc700 10 client.26066 handle_command_reply: tid=0 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7df9bdb80 1 client.26066 shutdown 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/259546460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 msgr2=0x565197f20990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 0x565197f20990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x565197f23f20 tx=0x7ff7b80cca70 comp rx=0 tx=0).stop 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/259546460 shutdown_connections 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x565197f30f70 0x565197f33360 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f20ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f177a0 0x565197f23690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7b67fc700 0 client.26066 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/259546460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 0x565197f20990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/259546460 >> 172.21.15.121:0/259546460 conn(0x565197e44080 msgr2=0x565197f11700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.765+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/259546460 shutdown_connections 2024-06-10T00:34:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.766+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/259546460 wait complete. 2024-06-10T00:34:54.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.766+0000 7ff7df9bdb80 20 client.26066 trim_cache size 0 max 16384 2024-06-10T00:34:54.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.792+0000 7ff7df9bdb80 1 Processor -- start 2024-06-10T00:34:54.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.792+0000 7ff7df9bdb80 1 -- start start 2024-06-10T00:34:54.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f177a0 0x565197f17260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 0x565197f1e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1d880 0x565197f231d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7df9bdb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01b0ea0 con 0x565197f1d880 2024-06-10T00:34:54.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7df9bdb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01ae6d0 con 0x565197f1cf60 2024-06-10T00:34:54.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7df9bdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01ac520 con 0x565197f177a0 2024-06-10T00:34:54.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7ce59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 0x565197f1e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7ce59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 0x565197f1e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:50424/0 (socket says 172.21.15.121:50424) 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7ce59c700 1 -- 172.21.15.121:0/1572886624 learned_addr learned my addr 172.21.15.121:0/1572886624 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7cf59e700 1 --2- 172.21.15.121:0/1572886624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1d880 0x565197f231d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7ced9d700 1 --2- 172.21.15.121:0/1572886624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f177a0 0x565197f17260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7cf59e700 1 -- 172.21.15.121:0/1572886624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f177a0 msgr2=0x565197f17260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.793+0000 7ff7cf59e700 1 --2- 172.21.15.121:0/1572886624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f177a0 0x565197f17260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.794+0000 7ff7cf59e700 1 -- 172.21.15.121:0/1572886624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 msgr2=0x565197f1e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.794+0000 7ff7cf59e700 1 --2- 172.21.15.121:0/1572886624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 0x565197f1e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.794+0000 7ff7cf59e700 1 -- 172.21.15.121:0/1572886624 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565197f20760 con 0x565197f1d880 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.794+0000 7ff7cf59e700 1 --2- 172.21.15.121:0/1572886624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1d880 0x565197f231d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800d010 tx=0x7ff7c8009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319b31d9415f6b8e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.794+0000 7ff7c4ff9700 1 -- 172.21.15.121:0/1572886624 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8009480 con 0x565197f1d880 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.794+0000 7ff7c4ff9700 1 -- 172.21.15.121:0/1572886624 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c800be50 con 0x565197f1d880 2024-06-10T00:34:54.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.794+0000 7ff7c4ff9700 1 -- 172.21.15.121:0/1572886624 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8007540 con 0x565197f1d880 2024-06-10T00:34:54.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.796+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1572886624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1d880 msgr2=0x565197f231d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:54.799 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.796+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1572886624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1d880 0x565197f231d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800d010 tx=0x7ff7c8009390 comp rx=0 tx=0).stop 2024-06-10T00:34:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.798+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1572886624 shutdown_connections 2024-06-10T00:34:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.798+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1572886624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1d880 0x565197f231d0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.798+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1572886624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f177a0 0x565197f17260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.798+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1572886624 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f1cf60 0x565197f1e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.798+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1572886624 >> 172.21.15.121:0/1572886624 conn(0x565197e44080 msgr2=0x565197f1dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.799+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1572886624 shutdown_connections 2024-06-10T00:34:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.799+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1572886624 wait complete. 2024-06-10T00:34:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.800+0000 7ff7df9bdb80 1 -- start start 2024-06-10T00:34:54.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.800+0000 7ff7df9bdb80 1 Processor -- start 2024-06-10T00:34:54.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.800+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f177a0 0x565197f338c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.800+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1cf60 0x565197f33e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.801+0000 7ff7df9bdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f998a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.801+0000 7ff7ced9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f177a0 0x565197f338c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.801+0000 7ff7ced9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f177a0 0x565197f338c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:50426/0 (socket says 172.21.15.121:50426) 2024-06-10T00:34:54.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.801+0000 7ff7ced9d700 1 -- 172.21.15.121:0/1557577562 learned_addr learned my addr 172.21.15.121:0/1557577562 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:54.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.801+0000 7ff7cf59e700 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.801+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1557577562 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01acb90 con 0x565197f1cf60 2024-06-10T00:34:54.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.801+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1557577562 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01ae000 con 0x565197f177a0 2024-06-10T00:34:54.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.801+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1557577562 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01ad260 con 0x565197f1d880 2024-06-10T00:34:54.802 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.801+0000 7ff7df9bdb80 10 client.0 _ll_register_callbacks cb 0x565197dfe5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.802+0000 7ff7cf59e700 1 -- 172.21.15.121:0/1557577562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f177a0 msgr2=0x565197f338c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.802+0000 7ff7cf59e700 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f177a0 0x565197f338c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.802+0000 7ff7cf59e700 1 -- 172.21.15.121:0/1557577562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1cf60 msgr2=0x565197f33e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.802+0000 7ff7cf59e700 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1cf60 0x565197f33e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.802+0000 7ff7cf59e700 1 -- 172.21.15.121:0/1557577562 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565197f99de0 con 0x565197f1d880 2024-06-10T00:34:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.803+0000 7ff7cf59e700 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f998a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8005bd0 tx=0x7ff7c8009df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dddae4ff247a300 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.803+0000 7ff7b67fc700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:34:54.803 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.803+0000 7ff7b67fc700 1 -- 172.21.15.121:0/1557577562 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8033060 con 0x565197f1d880 2024-06-10T00:34:54.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.803+0000 7ff7b67fc700 1 -- 172.21.15.121:0/1557577562 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7ff7c8003c40 con 0x565197f1d880 2024-06-10T00:34:54.804 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.803+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1557577562 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c800be50 con 0x565197f1d880 2024-06-10T00:34:54.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.807+0000 7ff7b67fc700 1 -- 172.21.15.121:0/1557577562 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff7c8003e40 con 0x565197f1d880 2024-06-10T00:34:54.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.807+0000 7ff7df9bdb80 10 client.25981 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:54.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.807+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1557577562 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x565197f9b020 con 0x565197f1d880 2024-06-10T00:34:54.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.807+0000 7ff7b67fc700 1 -- 172.21.15.121:0/1557577562 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7c800be50 con 0x565197f1d880 2024-06-10T00:34:54.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.808+0000 7ff7b67fc700 1 -- 172.21.15.121:0/1557577562 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7ff7c8011070 con 0x565197f1d880 2024-06-10T00:34:54.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.808+0000 7ff7df9bdb80 10 client.25981 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:54.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.808+0000 7ff7df9bdb80 10 client.25981 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:34:54.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.808+0000 7ff7df9bdb80 20 client.25981 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:54.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.808+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x565197f9b520 0x565197f9d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:54.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.808+0000 7ff7df9bdb80 4 client.25981 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:34:54.808 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.808+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1557577562 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565197e702d0 con 0x565197f9b520 2024-06-10T00:34:54.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.810+0000 7ff7ced9d700 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x565197f9b520 0x565197f9d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:54.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.811+0000 7ff7ced9d700 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x565197f9b520 0x565197f9d960 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d1fd910ef48a0339 server_cookie=3cda56e55196159 in_seq=0 out_seq=0 2024-06-10T00:34:54.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.811+0000 7ff7b67fc700 10 client.25981 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:34:54.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.812+0000 7ff7b67fc700 1 -- 172.21.15.121:0/1557577562 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565197e702d0 con 0x565197f9b520 2024-06-10T00:34:54.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.812+0000 7ff7b67fc700 10 client.25981 handle_command_reply: tid=0 2024-06-10T00:34:54.812 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.812+0000 7ff7df9bdb80 1 client.25981 shutdown 2024-06-10T00:34:54.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.817+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1557577562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 msgr2=0x565197f998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:54.817 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.817+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f998a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8005bd0 tx=0x7ff7c8009df0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.817+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1557577562 shutdown_connections 2024-06-10T00:34:54.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.817+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x565197f9b520 0x565197f9d960 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.817+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565197f1d880 0x565197f998a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.817+0000 7ff7b67fc700 0 client.25981 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:34:54.818 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.818+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x565197f177a0 0x565197f338c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.818+0000 7ff7df9bdb80 1 --2- 172.21.15.121:0/1557577562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x565197f1cf60 0x565197f33e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.818+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1557577562 >> 172.21.15.121:0/1557577562 conn(0x565197e44080 msgr2=0x565197f1ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:54.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.818+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1557577562 shutdown_connections 2024-06-10T00:34:54.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.818+0000 7ff7df9bdb80 1 -- 172.21.15.121:0/1557577562 wait complete. 2024-06-10T00:34:54.819 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.819+0000 7ff7df9bdb80 20 client.25981 trim_cache size 0 max 16384 2024-06-10T00:34:54.819 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:34:54.820 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.820+0000 7ff7d513a700 1 -- 172.21.15.121:0/1801666848 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff7bc0a2d90 msgr2=0x7ff7bc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:54.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.820+0000 7ff7d513a700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff7bc0a2d90 0x7ff7bc0a5250 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0199ac0 tx=0x7ff7c0009260 comp rx=0 tx=0).stop 2024-06-10T00:34:54.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.820+0000 7ff7d513a700 1 -- 172.21.15.121:0/1801666848 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7d01017f0 msgr2=0x7ff7d019ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:54.821 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.820+0000 7ff7d513a700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7d01017f0 0x7ff7d019ed00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff7b800e9f0 tx=0x7ff7b800ed00 comp rx=0 tx=0).stop 2024-06-10T00:34:54.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.821+0000 7ff7d513a700 1 -- 172.21.15.121:0/1801666848 shutdown_connections 2024-06-10T00:34:54.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.821+0000 7ff7d513a700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff7bc0a2d90 0x7ff7bc0a5250 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.821+0000 7ff7d513a700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7d0107500 0x7ff7d0198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.821+0000 7ff7d513a700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff7d0100e30 0x7ff7d019e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.821+0000 7ff7d513a700 1 --2- 172.21.15.121:0/1801666848 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff7d01017f0 0x7ff7d019ed00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:54.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.821+0000 7ff7d513a700 1 -- 172.21.15.121:0/1801666848 >> 172.21.15.121:0/1801666848 conn(0x7ff7d0076730 msgr2=0x7ff7d00feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:54.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.821+0000 7ff7d513a700 1 -- 172.21.15.121:0/1801666848 shutdown_connections 2024-06-10T00:34:54.822 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:54.821+0000 7ff7d513a700 1 -- 172.21.15.121:0/1801666848 wait complete. 2024-06-10T00:34:55.559 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:55 smithi135 ceph-mon[36424]: pgmap v211: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 85 op/s 2024-06-10T00:34:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:55 smithi193 ceph-mon[36716]: pgmap v211: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 85 op/s 2024-06-10T00:34:55.650 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:55 smithi121 ceph-mon[32378]: pgmap v211: 129 pgs: 129 active+clean; 1.3 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 85 op/s 2024-06-10T00:34:55.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-10T00:34:55.992 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:34:57.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:57 smithi121 ceph-mon[32378]: pgmap v212: 129 pgs: 129 active+clean; 1.3 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-06-10T00:34:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:57 smithi193 ceph-mon[36716]: pgmap v212: 129 pgs: 129 active+clean; 1.3 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-06-10T00:34:57.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.751+0000 7fdc94f60700 1 -- 172.21.15.121:0/2804904940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc90101390 msgr2=0x7fdc90101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:57.751 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.751+0000 7fdc94f60700 1 --2- 172.21.15.121:0/2804904940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc90101390 0x7fdc90101810 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009990 tx=0x7fdc84009ca0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.751+0000 7fdc94f60700 1 -- 172.21.15.121:0/2804904940 shutdown_connections 2024-06-10T00:34:57.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.751+0000 7fdc94f60700 1 --2- 172.21.15.121:0/2804904940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc90101d50 0x7fdc9010e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.751+0000 7fdc94f60700 1 --2- 172.21.15.121:0/2804904940 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc90107400 0x7fdc901077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.751+0000 7fdc94f60700 1 --2- 172.21.15.121:0/2804904940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc90101390 0x7fdc90101810 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.751+0000 7fdc94f60700 1 -- 172.21.15.121:0/2804904940 >> 172.21.15.121:0/2804904940 conn(0x7fdc90078340 msgr2=0x7fdc90078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:57.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.751+0000 7fdc94f60700 1 -- 172.21.15.121:0/2804904940 shutdown_connections 2024-06-10T00:34:57.752 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.751+0000 7fdc94f60700 1 -- 172.21.15.121:0/2804904940 wait complete. 2024-06-10T00:34:57.754 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc94f60700 1 Processor -- start 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc94f60700 1 -- start start 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc94f60700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc90101390 0x7fdc90103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc94f60700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc90101d50 0x7fdc901035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc94f60700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc90107400 0x7fdc90103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc94f60700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdc90111670 con 0x7fdc90107400 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc94f60700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fdc901114f0 con 0x7fdc90101390 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc94f60700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdc901117f0 con 0x7fdc90101d50 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc8ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc90107400 0x7fdc90103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc8e59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc90101390 0x7fdc90103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc8e59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc90101390 0x7fdc90103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:50448/0 (socket says 172.21.15.121:50448) 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc8ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc90107400 0x7fdc90103af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42806/0 (socket says 172.21.15.121:42806) 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc8e59c700 1 -- 172.21.15.121:0/769176256 learned_addr learned my addr 172.21.15.121:0/769176256 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:57.755 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.752+0000 7fdc8dd9b700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc90101d50 0x7fdc901035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.753+0000 7fdc8ed9d700 1 -- 172.21.15.121:0/769176256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc90101d50 msgr2=0x7fdc901035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.753+0000 7fdc8ed9d700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc90101d50 0x7fdc901035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.753+0000 7fdc8ed9d700 1 -- 172.21.15.121:0/769176256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc90101390 msgr2=0x7fdc90103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.753+0000 7fdc8ed9d700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc90101390 0x7fdc90103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.753+0000 7fdc8ed9d700 1 -- 172.21.15.121:0/769176256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc901a7140 con 0x7fdc90107400 2024-06-10T00:34:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.753+0000 7fdc8dd9b700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc90101d50 0x7fdc901035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.753+0000 7fdc8ed9d700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc90107400 0x7fdc90103af0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fdc88007f10 tx=0x7fdc8800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.753+0000 7fdc7f7fe700 1 -- 172.21.15.121:0/769176256 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc88004db0 con 0x7fdc90107400 2024-06-10T00:34:57.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.753+0000 7fdc94f60700 1 -- 172.21.15.121:0/769176256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc901a7430 con 0x7fdc90107400 2024-06-10T00:34:57.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.753+0000 7fdc94f60700 1 -- 172.21.15.121:0/769176256 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc901a7990 con 0x7fdc90107400 2024-06-10T00:34:57.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.753+0000 7fdc7f7fe700 1 -- 172.21.15.121:0/769176256 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc88003a00 con 0x7fdc90107400 2024-06-10T00:34:57.757 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.754+0000 7fdc7f7fe700 1 -- 172.21.15.121:0/769176256 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc88004680 con 0x7fdc90107400 2024-06-10T00:34:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.756+0000 7fdc7f7fe700 1 -- 172.21.15.121:0/769176256 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fdc8801c020 con 0x7fdc90107400 2024-06-10T00:34:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.757+0000 7fdc7f7fe700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdc780a2e10 0x7fdc780a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.757+0000 7fdc9f7e3b80 1 Processor -- start 2024-06-10T00:34:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.757+0000 7fdc7f7fe700 1 -- 172.21.15.121:0/769176256 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fdc880ced00 con 0x7fdc90107400 2024-06-10T00:34:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.757+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdc780a2e10 0x7fdc780a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.757+0000 7fdc9f7e3b80 1 -- start start 2024-06-10T00:34:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.757+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 0x55cc9188da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 0x55cc918879e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc91888340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55cc917d9810 con 0x55cc9188df60 2024-06-10T00:34:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55cc91781720 con 0x55cc91887f20 2024-06-10T00:34:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cc917d57d0 con 0x55cc9188d640 2024-06-10T00:34:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdc780a2e10 0x7fdc780a52d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009ca0 tx=0x7fdc80009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8e59c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 0x55cc9188da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8e59c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 0x55cc9188da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:52794/0 (socket says 172.21.15.121:52794) 2024-06-10T00:34:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8dd9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 0x55cc918879e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8e59c700 1 -- 172.21.15.121:0/2995506206 learned_addr learned my addr 172.21.15.121:0/2995506206 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8ed9d700 1 --2- 172.21.15.121:0/2995506206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc91888340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8e59c700 1 -- 172.21.15.121:0/2995506206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 msgr2=0x55cc91888340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/2995506206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc91888340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8e59c700 1 -- 172.21.15.121:0/2995506206 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 msgr2=0x55cc918879e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/2995506206 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 0x55cc918879e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8e59c700 1 -- 172.21.15.121:0/2995506206 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc91777f50 con 0x55cc9188d640 2024-06-10T00:34:57.769 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/2995506206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 0x55cc9188da20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000e4b0 tx=0x7fdc80014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6168728b9ad02afc server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc7cff9700 1 -- 172.21.15.121:0/2995506206 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8000ee50 con 0x55cc9188d640 2024-06-10T00:34:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc7cff9700 1 -- 172.21.15.121:0/2995506206 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc8001d8f0 con 0x55cc9188d640 2024-06-10T00:34:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.758+0000 7fdc7cff9700 1 -- 172.21.15.121:0/2995506206 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc80025a40 con 0x55cc9188d640 2024-06-10T00:34:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.759+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/2995506206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 msgr2=0x55cc9188da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.759+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/2995506206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 0x55cc9188da20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdc8000e4b0 tx=0x7fdc80014000 comp rx=0 tx=0).stop 2024-06-10T00:34:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.759+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/2995506206 shutdown_connections 2024-06-10T00:34:57.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.759+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/2995506206 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc91888340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.759+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/2995506206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 0x55cc9188da20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.759+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/2995506206 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 0x55cc918879e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.759+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/2995506206 >> 172.21.15.121:0/2995506206 conn(0x55cc917b2cc0 msgr2=0x55cc91882850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.759+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/2995506206 shutdown_connections 2024-06-10T00:34:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.759+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/2995506206 wait complete. 2024-06-10T00:34:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc9f7e3b80 1 -- start start 2024-06-10T00:34:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc9f7e3b80 1 Processor -- start 2024-06-10T00:34:57.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc91893cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 0x55cc91890fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 0x55cc91891500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55cc918949d0 con 0x55cc9188df60 2024-06-10T00:34:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55cc91779bf0 con 0x55cc91887f20 2024-06-10T00:34:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55cc91894b50 con 0x55cc9188d640 2024-06-10T00:34:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc9f7e3b80 10 client.0 _ll_register_callbacks cb 0x55cc91773920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:57.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc8ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 0x55cc91891500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc8dd9b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 0x55cc91890fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc8ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 0x55cc91891500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42832/0 (socket says 172.21.15.121:42832) 2024-06-10T00:34:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc8ed9d700 1 -- 172.21.15.121:0/1310379556 learned_addr learned my addr 172.21.15.121:0/1310379556 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.760+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc91893cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc8ed9d700 1 -- 172.21.15.121:0/1310379556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 msgr2=0x55cc91890fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc8ed9d700 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 0x55cc91890fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc8ed9d700 1 -- 172.21.15.121:0/1310379556 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 msgr2=0x55cc91893cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc8ed9d700 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc91893cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc8ed9d700 1 -- 172.21.15.121:0/1310379556 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc917288d0 con 0x55cc9188df60 2024-06-10T00:34:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc8ed9d700 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 0x55cc91891500 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fdc8809ae20 tx=0x7fdc8809aa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc652aa0da1e66d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:34:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc75ffb700 1 -- 172.21.15.121:0/1310379556 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc88019040 con 0x55cc9188df60 2024-06-10T00:34:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc75ffb700 1 -- 172.21.15.121:0/1310379556 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fdc8809b0f0 con 0x55cc9188df60 2024-06-10T00:34:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/1310379556 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc8001d8f0 con 0x55cc9188df60 2024-06-10T00:34:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc75ffb700 1 -- 172.21.15.121:0/1310379556 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdc8801c510 con 0x55cc9188df60 2024-06-10T00:34:57.774 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc9f7e3b80 10 client.16527 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/1310379556 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cc9189f330 con 0x55cc9188df60 2024-06-10T00:34:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc75ffb700 1 -- 172.21.15.121:0/1310379556 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc8001d8f0 con 0x55cc9188df60 2024-06-10T00:34:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc75ffb700 1 -- 172.21.15.121:0/1310379556 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fdc880a3670 con 0x55cc9188df60 2024-06-10T00:34:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc9f7e3b80 10 client.16527 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc9f7e3b80 10 client.16527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:34:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.761+0000 7fdc9f7e3b80 20 client.16527 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.762+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cc918a1960 0x55cc918a3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.775 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.762+0000 7fdc9f7e3b80 4 client.16527 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:34:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.762+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/1310379556 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc917e6d10 con 0x55cc918a1960 2024-06-10T00:34:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.762+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cc918a1960 0x55cc918a3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.762+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cc918a1960 0x55cc918a3d50 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae389849d24ca34a server_cookie=58fb45dc4d63691c in_seq=0 out_seq=0 2024-06-10T00:34:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.762+0000 7fdc75ffb700 10 client.16527 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc75ffb700 1 -- 172.21.15.121:0/1310379556 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55cc917e6d10 con 0x55cc918a1960 2024-06-10T00:34:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc75ffb700 10 client.16527 handle_command_reply: tid=0 2024-06-10T00:34:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 1 client.16527 shutdown 2024-06-10T00:34:57.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/1310379556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 msgr2=0x55cc91891500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 0x55cc91891500 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fdc8809ae20 tx=0x7fdc8809aa50 comp rx=0 tx=0).stop 2024-06-10T00:34:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/1310379556 shutdown_connections 2024-06-10T00:34:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cc918a1960 0x55cc918a3d50 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc75ffb700 0 client.16527 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188df60 0x55cc91891500 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc91893cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/1310379556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188d640 0x55cc91890fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/1310379556 >> 172.21.15.121:0/1310379556 conn(0x55cc917b2cc0 msgr2=0x55cc918869d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/1310379556 shutdown_connections 2024-06-10T00:34:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/1310379556 wait complete. 2024-06-10T00:34:57.778 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.763+0000 7fdc9f7e3b80 20 client.16527 trim_cache size 0 max 16384 2024-06-10T00:34:57.779 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:57 smithi135 ceph-mon[36424]: pgmap v212: 129 pgs: 129 active+clean; 1.3 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 159 op/s 2024-06-10T00:34:57.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc9f7e3b80 1 Processor -- start 2024-06-10T00:34:57.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc9f7e3b80 1 -- start start 2024-06-10T00:34:57.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc918879e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188d640 0x55cc9188f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.787 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188df60 0x55cc9188fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdc70011640 con 0x55cc9188d640 2024-06-10T00:34:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7000ee70 con 0x55cc91887f20 2024-06-10T00:34:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7000ccc0 con 0x55cc9188df60 2024-06-10T00:34:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc8e59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc918879e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc8e59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc918879e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:50468/0 (socket says 172.21.15.121:50468) 2024-06-10T00:34:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc8e59c700 1 -- 172.21.15.121:0/2357838136 learned_addr learned my addr 172.21.15.121:0/2357838136 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:57.788 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc8dd9b700 1 --2- 172.21.15.121:0/2357838136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188d640 0x55cc9188f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc8ed9d700 1 --2- 172.21.15.121:0/2357838136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188df60 0x55cc9188fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc8e59c700 1 -- 172.21.15.121:0/2357838136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188df60 msgr2=0x55cc9188fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/2357838136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188df60 0x55cc9188fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc8e59c700 1 -- 172.21.15.121:0/2357838136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188d640 msgr2=0x55cc9188f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/2357838136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188d640 0x55cc9188f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.779+0000 7fdc8e59c700 1 -- 172.21.15.121:0/2357838136 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc91893800 con 0x55cc91887f20 2024-06-10T00:34:57.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.780+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/2357838136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc918879e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdc800096b0 tx=0x7fdc800254d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cde31eaabc5b8076 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.780+0000 7fdc77fff700 1 -- 172.21.15.121:0/2357838136 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc80005040 con 0x55cc91887f20 2024-06-10T00:34:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.780+0000 7fdc77fff700 1 -- 172.21.15.121:0/2357838136 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc800056c0 con 0x55cc91887f20 2024-06-10T00:34:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.780+0000 7fdc77fff700 1 -- 172.21.15.121:0/2357838136 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc80004ad0 con 0x55cc91887f20 2024-06-10T00:34:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.780+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/2357838136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 msgr2=0x55cc918879e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.780+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/2357838136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc918879e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdc800096b0 tx=0x7fdc800254d0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.780+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/2357838136 shutdown_connections 2024-06-10T00:34:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.780+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/2357838136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188df60 0x55cc9188fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.790 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.780+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/2357838136 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc91887f20 0x55cc918879e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.780+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/2357838136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc9188d640 0x55cc9188f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.780+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/2357838136 >> 172.21.15.121:0/2357838136 conn(0x55cc917b2cc0 msgr2=0x55cc91890f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/2357838136 shutdown_connections 2024-06-10T00:34:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/2357838136 wait complete. 2024-06-10T00:34:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc9f7e3b80 1 -- start start 2024-06-10T00:34:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc9f7e3b80 1 Processor -- start 2024-06-10T00:34:57.791 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc91887f20 0x55cc9189a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc9188d640 0x55cc918a8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc9f7e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188df60 0x55cc918a9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7000d330 con 0x55cc91887f20 2024-06-10T00:34:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7000e7a0 con 0x55cc9188d640 2024-06-10T00:34:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc9f7e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7000da00 con 0x55cc9188df60 2024-06-10T00:34:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc9f7e3b80 10 client.0 _ll_register_callbacks cb 0x55cc91773920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:34:57.792 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc8e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc91887f20 0x55cc9189a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc8dd9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc9188d640 0x55cc918a8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc8dd9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc9188d640 0x55cc918a8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:50478/0 (socket says 172.21.15.121:50478) 2024-06-10T00:34:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc8dd9b700 1 -- 172.21.15.121:0/4133489262 learned_addr learned my addr 172.21.15.121:0/4133489262 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:34:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.781+0000 7fdc8ed9d700 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188df60 0x55cc918a9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc8dd9b700 1 -- 172.21.15.121:0/4133489262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188df60 msgr2=0x55cc918a9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc8dd9b700 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188df60 0x55cc918a9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc8dd9b700 1 -- 172.21.15.121:0/4133489262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc91887f20 msgr2=0x55cc9189a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:34:57.793 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc8dd9b700 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc91887f20 0x55cc9189a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc8dd9b700 1 -- 172.21.15.121:0/4133489262 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc918a9820 con 0x55cc9188d640 2024-06-10T00:34:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc8ed9d700 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188df60 0x55cc918a9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc91887f20 0x55cc9189a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:34:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc8dd9b700 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc9188d640 0x55cc918a8cf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009990 tx=0x7fdc8400bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b5aabab0e112f8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:34:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:34:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc75ffb700 1 -- 172.21.15.121:0/4133489262 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc84003a90 con 0x55cc9188d640 2024-06-10T00:34:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc75ffb700 1 -- 172.21.15.121:0/4133489262 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7fdc84003c30 con 0x55cc9188d640 2024-06-10T00:34:57.794 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/4133489262 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cc917e1290 con 0x55cc9188d640 2024-06-10T00:34:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc75ffb700 1 -- 172.21.15.121:0/4133489262 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdc84038050 con 0x55cc9188d640 2024-06-10T00:34:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc9f7e3b80 10 client.26090 fetch_fsmap learned FSMap version 30 2024-06-10T00:34:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/4133489262 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cc9190c400 con 0x55cc9188d640 2024-06-10T00:34:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.782+0000 7fdc75ffb700 1 -- 172.21.15.121:0/4133489262 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55cc917e1290 con 0x55cc9188d640 2024-06-10T00:34:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc75ffb700 1 -- 172.21.15.121:0/4133489262 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fdc8400f4f0 con 0x55cc9188d640 2024-06-10T00:34:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 10 client.26090 fetch_fsmap finished waiting for FSMap version 30 2024-06-10T00:34:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 10 client.26090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:34:57.795 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 20 client.26090 populate_metadata read hostname 'smithi121' 2024-06-10T00:34:57.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cc918a1960 0x55cc9190ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:34:57.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 4 client.26090 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:34:57.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/4133489262 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cc9165b250 con 0x55cc918a1960 2024-06-10T00:34:57.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cc918a1960 0x55cc9190ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:34:57.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc8e59c700 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cc918a1960 0x55cc9190ec10 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58c3215e027c20d1 server_cookie=65395b651c5aca45 in_seq=0 out_seq=0 2024-06-10T00:34:57.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc75ffb700 10 client.26090 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:57.796 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc75ffb700 1 -- 172.21.15.121:0/4133489262 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cc9165b250 con 0x55cc918a1960 2024-06-10T00:34:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc75ffb700 10 client.26090 handle_command_reply: tid=0 2024-06-10T00:34:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 1 client.26090 shutdown 2024-06-10T00:34:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/4133489262 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc9188d640 msgr2=0x55cc918a8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc9188d640 0x55cc918a8cf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdc84009990 tx=0x7fdc8400bb20 comp rx=0 tx=0).stop 2024-06-10T00:34:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/4133489262 shutdown_connections 2024-06-10T00:34:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55cc918a1960 0x55cc9190ec10 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc9188df60 0x55cc918a9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.797 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc91887f20 0x55cc9189a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.783+0000 7fdc9f7e3b80 1 --2- 172.21.15.121:0/4133489262 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55cc9188d640 0x55cc918a8cf0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/4133489262 >> 172.21.15.121:0/4133489262 conn(0x55cc917b2cc0 msgr2=0x55cc91890f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc75ffb700 0 client.26090 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:34:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/4133489262 shutdown_connections 2024-06-10T00:34:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc9f7e3b80 1 -- 172.21.15.121:0/4133489262 wait complete. 2024-06-10T00:34:57.798 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc9f7e3b80 20 client.26090 trim_cache size 0 max 16384 2024-06-10T00:34:57.799 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:34:57.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 -- 172.21.15.121:0/769176256 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdc780a2e10 msgr2=0x7fdc780a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:57.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdc780a2e10 0x7fdc780a52d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009ca0 tx=0x7fdc80009260 comp rx=0 tx=0).stop 2024-06-10T00:34:57.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 -- 172.21.15.121:0/769176256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc90107400 msgr2=0x7fdc90103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:34:57.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc90107400 0x7fdc90103af0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fdc88007f10 tx=0x7fdc8800d3b0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 -- 172.21.15.121:0/769176256 shutdown_connections 2024-06-10T00:34:57.800 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdc780a2e10 0x7fdc780a52d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc90107400 0x7fdc90103af0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdc90101390 0x7fdc90103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 --2- 172.21.15.121:0/769176256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc90101d50 0x7fdc901035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:34:57.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 -- 172.21.15.121:0/769176256 >> 172.21.15.121:0/769176256 conn(0x7fdc90078340 msgr2=0x7fdc90100670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:34:57.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 -- 172.21.15.121:0/769176256 shutdown_connections 2024-06-10T00:34:57.801 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:34:57.784+0000 7fdc94f60700 1 -- 172.21.15.121:0/769176256 wait complete. 2024-06-10T00:34:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:59 smithi193 ceph-mon[36716]: pgmap v213: 129 pgs: 129 active+clean; 1.3 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 157 op/s 2024-06-10T00:34:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:34:59 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:34:59.665 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-10T00:34:59.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-10T00:34:59.690 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:59 smithi121 ceph-mon[32378]: pgmap v213: 129 pgs: 129 active+clean; 1.3 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 157 op/s 2024-06-10T00:34:59.690 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:34:59 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:34:59.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:59 smithi135 ceph-mon[36424]: pgmap v213: 129 pgs: 129 active+clean; 1.3 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 157 op/s 2024-06-10T00:34:59.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:34:59 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:34:59.921 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:35:01.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:01 smithi121 ceph-mon[32378]: pgmap v214: 129 pgs: 129 active+clean; 1.3 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 237 op/s 2024-06-10T00:35:01.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:01 smithi193 ceph-mon[36716]: pgmap v214: 129 pgs: 129 active+clean; 1.3 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 237 op/s 2024-06-10T00:35:01.757 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:01 smithi135 ceph-mon[36424]: pgmap v214: 129 pgs: 129 active+clean; 1.3 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 237 op/s 2024-06-10T00:35:02.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.086+0000 7f359fd8f700 1 -- 172.21.15.121:0/2185731531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359806ab60 msgr2=0x7f359806afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:02.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.086+0000 7f359fd8f700 1 --2- 172.21.15.121:0/2185731531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359806ab60 0x7f359806afe0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3588009990 tx=0x7f3588009ca0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.086+0000 7f359fd8f700 1 -- 172.21.15.121:0/2185731531 shutdown_connections 2024-06-10T00:35:02.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.086+0000 7f359fd8f700 1 --2- 172.21.15.121:0/2185731531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f359806b520 0x7f359810f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.086+0000 7f359fd8f700 1 --2- 172.21.15.121:0/2185731531 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598108530 0x7f3598108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.086+0000 7f359fd8f700 1 --2- 172.21.15.121:0/2185731531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359806ab60 0x7f359806afe0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.086+0000 7f359fd8f700 1 -- 172.21.15.121:0/2185731531 >> 172.21.15.121:0/2185731531 conn(0x7f35980783d0 msgr2=0x7f35980787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:02.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.086+0000 7f359fd8f700 1 -- 172.21.15.121:0/2185731531 shutdown_connections 2024-06-10T00:35:02.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.086+0000 7f359fd8f700 1 -- 172.21.15.121:0/2185731531 wait complete. 2024-06-10T00:35:02.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.086+0000 7f359fd8f700 1 Processor -- start 2024-06-10T00:35:02.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359fd8f700 1 -- start start 2024-06-10T00:35:02.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359fd8f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f359806ab60 0x7f3598101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359fd8f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359806b520 0x7f35981019d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359fd8f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598108530 0x7f3598101f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359fd8f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f35981127b0 con 0x7f359806b520 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359fd8f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f3598112630 con 0x7f3598108530 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359fd8f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3598112930 con 0x7f359806ab60 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359d32a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359806b520 0x7f35981019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359db2b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f359806ab60 0x7f3598101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359d32a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359806b520 0x7f35981019d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54354/0 (socket says 172.21.15.121:54354) 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359d32a700 1 -- 172.21.15.121:0/862085227 learned_addr learned my addr 172.21.15.121:0/862085227 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359e32c700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598108530 0x7f3598101f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359db2b700 1 -- 172.21.15.121:0/862085227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598108530 msgr2=0x7f3598101f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359db2b700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598108530 0x7f3598101f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359db2b700 1 -- 172.21.15.121:0/862085227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359806b520 msgr2=0x7f35981019d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359db2b700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359806b520 0x7f35981019d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.090 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359db2b700 1 -- 172.21.15.121:0/862085227 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35981a71d0 con 0x7f359806ab60 2024-06-10T00:35:02.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359d32a700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359806b520 0x7f35981019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:02.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.087+0000 7f359db2b700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f359806ab60 0x7f3598101490 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f359400e9f0 tx=0x7f359400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:02.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.088+0000 7f358effd700 1 -- 172.21.15.121:0/862085227 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359400c9c0 con 0x7f359806ab60 2024-06-10T00:35:02.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.088+0000 7f359fd8f700 1 -- 172.21.15.121:0/862085227 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35981a74c0 con 0x7f359806ab60 2024-06-10T00:35:02.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.088+0000 7f358effd700 1 -- 172.21.15.121:0/862085227 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f359400cb60 con 0x7f359806ab60 2024-06-10T00:35:02.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.088+0000 7f358effd700 1 -- 172.21.15.121:0/862085227 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594018680 con 0x7f359806ab60 2024-06-10T00:35:02.091 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.088+0000 7f359fd8f700 1 -- 172.21.15.121:0/862085227 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35981a7a00 con 0x7f359806ab60 2024-06-10T00:35:02.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.091+0000 7f358effd700 1 -- 172.21.15.121:0/862085227 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3594010430 con 0x7f359806ab60 2024-06-10T00:35:02.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.091+0000 7f358effd700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35840a2d90 0x7f35840a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.091+0000 7f359d32a700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35840a2d90 0x7f35840a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.091+0000 7f358effd700 1 -- 172.21.15.121:0/862085227 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f35940cf870 con 0x7f359806ab60 2024-06-10T00:35:02.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f35aa612b80 1 Processor -- start 2024-06-10T00:35:02.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f359d32a700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35840a2d90 0x7f35840a5250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3598078a90 tx=0x7f3588009590 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:02.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f35aa612b80 1 -- start start 2024-06-10T00:35:02.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2a93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f35aa612b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b0bd1fa890 con 0x55b0bd2aefe0 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f35aa612b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55b0bd1a27a0 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f35aa612b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b0bd1f6850 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f359d32a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f359db2b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f359d32a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2a8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54370/0 (socket says 172.21.15.121:54370) 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f359d32a700 1 -- 172.21.15.121:0/3895667234 learned_addr learned my addr 172.21.15.121:0/3895667234 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.092+0000 7f359e32c700 1 --2- 172.21.15.121:0/3895667234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2a93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.093+0000 7f359db2b700 1 -- 172.21.15.121:0/3895667234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 msgr2=0x55b0bd2a93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.093+0000 7f359db2b700 1 --2- 172.21.15.121:0/3895667234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2a93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.093+0000 7f359db2b700 1 -- 172.21.15.121:0/3895667234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2aefe0 msgr2=0x55b0bd2a8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:02.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.093+0000 7f359db2b700 1 --2- 172.21.15.121:0/3895667234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2a8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.093+0000 7f359db2b700 1 -- 172.21.15.121:0/3895667234 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0bd198fd0 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.093+0000 7f359e32c700 1 --2- 172.21.15.121:0/3895667234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2a93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.093+0000 7f359db2b700 1 --2- 172.21.15.121:0/3895667234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2aeaa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f35940ccc30 tx=0x7f3594097dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd0d40d5ee061590 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.093+0000 7f3583fff700 1 -- 172.21.15.121:0/3895667234 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359409ba00 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.093+0000 7f3583fff700 1 -- 172.21.15.121:0/3895667234 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f359409fdb0 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.093+0000 7f3583fff700 1 -- 172.21.15.121:0/3895667234 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35940a41e0 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.094+0000 7f35aa612b80 1 -- 172.21.15.121:0/3895667234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 msgr2=0x55b0bd2aeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.094+0000 7f35aa612b80 1 --2- 172.21.15.121:0/3895667234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2aeaa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f35940ccc30 tx=0x7f3594097dd0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.094+0000 7f35aa612b80 1 -- 172.21.15.121:0/3895667234 shutdown_connections 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.094+0000 7f35aa612b80 1 --2- 172.21.15.121:0/3895667234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2a93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.094+0000 7f35aa612b80 1 --2- 172.21.15.121:0/3895667234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2aeaa0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.094+0000 7f35aa612b80 1 --2- 172.21.15.121:0/3895667234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2a8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.099 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.094+0000 7f35aa612b80 1 -- 172.21.15.121:0/3895667234 >> 172.21.15.121:0/3895667234 conn(0x55b0bd1d3d40 msgr2=0x55b0bd2a38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.094+0000 7f35aa612b80 1 -- 172.21.15.121:0/3895667234 shutdown_connections 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.094+0000 7f35aa612b80 1 -- 172.21.15.121:0/3895667234 wait complete. 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.094+0000 7f35aa612b80 1 -- start start 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.094+0000 7f35aa612b80 1 Processor -- start 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2b2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f35aa612b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b0bd2b5a50 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f35aa612b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55b0bd19ac70 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f35aa612b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b0bd2b5bd0 con 0x55b0bd2aefe0 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f35aa612b80 10 client.0 _ll_register_callbacks cb 0x55b0bd1949a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f359db2b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.100 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f359db2b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2b4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54374/0 (socket says 172.21.15.121:54374) 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f359db2b700 1 -- 172.21.15.121:0/380371044 learned_addr learned my addr 172.21.15.121:0/380371044 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f359d32a700 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f359e32c700 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f359db2b700 1 -- 172.21.15.121:0/380371044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2aefe0 msgr2=0x55b0bd2b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f359db2b700 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f359db2b700 1 -- 172.21.15.121:0/380371044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 msgr2=0x55b0bd2b2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f359db2b700 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2b2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f359db2b700 1 -- 172.21.15.121:0/380371044 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0bd149950 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f359db2b700 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2b4e30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f35940cfb90 tx=0x7f359409b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7791465166fbcaf3 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f35817fa700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f35817fa700 1 -- 172.21.15.121:0/380371044 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35940a4670 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f35817fa700 1 -- 172.21.15.121:0/380371044 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f35940a43f0 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.095+0000 7f35aa612b80 1 -- 172.21.15.121:0/380371044 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f359409fdb0 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f35817fa700 1 -- 172.21.15.121:0/380371044 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f359409ba00 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f35aa612b80 10 client.16545 fetch_fsmap learned FSMap version 31 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f35aa612b80 1 -- 172.21.15.121:0/380371044 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b0bd2c0410 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f35817fa700 1 -- 172.21.15.121:0/380371044 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f359409fdb0 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f35817fa700 1 -- 172.21.15.121:0/380371044 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f3594097750 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f35aa612b80 10 client.16545 fetch_fsmap finished waiting for FSMap version 31 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f35aa612b80 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f35aa612b80 20 client.16545 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f35aa612b80 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b0bd2c28e0 0x55b0bd2c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f35aa612b80 4 client.16545 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f35aa612b80 1 -- 172.21.15.121:0/380371044 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0bd207d90 con 0x55b0bd2c28e0 2024-06-10T00:35:02.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.096+0000 7f359d32a700 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b0bd2c28e0 0x55b0bd2c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.097+0000 7f35817fa700 10 client.16545 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:02.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.097+0000 7f359d32a700 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b0bd2c28e0 0x55b0bd2c4cd0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4b117553a24fa49 server_cookie=5a34b0af89a6445b in_seq=0 out_seq=0 2024-06-10T00:35:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35817fa700 1 -- 172.21.15.121:0/380371044 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55b0bd207d90 con 0x55b0bd2c28e0 2024-06-10T00:35:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35817fa700 10 client.16545 handle_command_reply: tid=0 2024-06-10T00:35:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 1 client.16545 shutdown 2024-06-10T00:35:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 1 -- 172.21.15.121:0/380371044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 msgr2=0x55b0bd2b4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2b4e30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f35940cfb90 tx=0x7f359409b900 comp rx=0 tx=0).stop 2024-06-10T00:35:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 1 -- 172.21.15.121:0/380371044 shutdown_connections 2024-06-10T00:35:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b0bd2c28e0 0x55b0bd2c4cd0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35817fa700 0 client.16545 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2b4e30 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 1 --2- 172.21.15.121:0/380371044 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2b2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 1 -- 172.21.15.121:0/380371044 >> 172.21.15.121:0/380371044 conn(0x55b0bd1d3d40 msgr2=0x55b0bd2a7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:02.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 1 -- 172.21.15.121:0/380371044 shutdown_connections 2024-06-10T00:35:02.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 1 -- 172.21.15.121:0/380371044 wait complete. 2024-06-10T00:35:02.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.098+0000 7f35aa612b80 20 client.16545 trim_cache size 0 max 16384 2024-06-10T00:35:02.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f35aa612b80 1 Processor -- start 2024-06-10T00:35:02.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f35aa612b80 1 -- start start 2024-06-10T00:35:02.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd29d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.121 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2afc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f35aa612b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f35981b3110 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f35aa612b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f35981b0910 con 0x55b0bd2aefe0 2024-06-10T00:35:02.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f35aa612b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35981ae760 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f359d32a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd29d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f359d32a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd29d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54378/0 (socket says 172.21.15.121:54378) 2024-06-10T00:35:02.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f359d32a700 1 -- 172.21.15.121:0/1024896349 learned_addr learned my addr 172.21.15.121:0/1024896349 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:02.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f359e32c700 1 --2- 172.21.15.121:0/1024896349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.119+0000 7f359db2b700 1 --2- 172.21.15.121:0/1024896349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f359d32a700 1 -- 172.21.15.121:0/1024896349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 msgr2=0x55b0bd2a8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:02.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f359d32a700 1 --2- 172.21.15.121:0/1024896349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2a8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.122 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f359d32a700 1 -- 172.21.15.121:0/1024896349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2aefe0 msgr2=0x55b0bd2afc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f359d32a700 1 --2- 172.21.15.121:0/1024896349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2afc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f359d32a700 1 -- 172.21.15.121:0/1024896349 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0bd2b5a50 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f359db2b700 1 --2- 172.21.15.121:0/1024896349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2a8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f359d32a700 1 --2- 172.21.15.121:0/1024896349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd29d740 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f3588005850 tx=0x7f3588023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2add1e840a890d6c server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f35837fe700 1 -- 172.21.15.121:0/1024896349 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588004690 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f35837fe700 1 -- 172.21.15.121:0/1024896349 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3588004d10 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f35837fe700 1 -- 172.21.15.121:0/1024896349 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3588034420 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f35aa612b80 1 -- 172.21.15.121:0/1024896349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2ae6c0 msgr2=0x55b0bd29d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.120+0000 7f35aa612b80 1 --2- 172.21.15.121:0/1024896349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd29d740 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f3588005850 tx=0x7f3588023000 comp rx=0 tx=0).stop 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 -- 172.21.15.121:0/1024896349 shutdown_connections 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 --2- 172.21.15.121:0/1024896349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2afc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 --2- 172.21.15.121:0/1024896349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2a8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.123 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 --2- 172.21.15.121:0/1024896349 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd29d740 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 -- 172.21.15.121:0/1024896349 >> 172.21.15.121:0/1024896349 conn(0x55b0bd1d3d40 msgr2=0x55b0bd2ad220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:02.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 -- 172.21.15.121:0/1024896349 shutdown_connections 2024-06-10T00:35:02.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 -- 172.21.15.121:0/1024896349 wait complete. 2024-06-10T00:35:02.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 -- start start 2024-06-10T00:35:02.124 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 Processor -- start 2024-06-10T00:35:02.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2c8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2c6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f35981aedd0 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f35981b0240 con 0x55b0bd2ae6c0 2024-06-10T00:35:02.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.121+0000 7f35aa612b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35981af4a0 con 0x55b0bd2aefe0 2024-06-10T00:35:02.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f35aa612b80 10 client.0 _ll_register_callbacks cb 0x55b0bd1949a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:02.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f359db2b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.127 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f359db2b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54384/0 (socket says 172.21.15.121:54384) 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f359db2b700 1 -- 172.21.15.121:0/434243360 learned_addr learned my addr 172.21.15.121:0/434243360 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f359d32a700 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f359e32c700 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f359db2b700 1 -- 172.21.15.121:0/434243360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2aefe0 msgr2=0x55b0bd2c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f359db2b700 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f359db2b700 1 -- 172.21.15.121:0/434243360 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 msgr2=0x55b0bd2c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f359db2b700 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f359db2b700 1 -- 172.21.15.121:0/434243360 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0bd2c71c0 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f359db2b700 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2c8290 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f359400e9c0 tx=0x7f359400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f88a82d6b913dd5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f35817fa700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f35817fa700 1 -- 172.21.15.121:0/434243360 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359409fdb0 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.128 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f35817fa700 1 -- 172.21.15.121:0/434243360 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9437+0+0 (secure 0 0 0) 0x7f359409be70 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f35aa612b80 1 -- 172.21.15.121:0/434243360 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3588004d10 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f35817fa700 1 -- 172.21.15.121:0/434243360 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f359400cd10 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f35aa612b80 10 client.16551 fetch_fsmap learned FSMap version 31 2024-06-10T00:35:02.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.122+0000 7f35aa612b80 1 -- 172.21.15.121:0/434243360 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b0bd32e0e0 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.123+0000 7f35817fa700 1 -- 172.21.15.121:0/434243360 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3588004d10 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.132 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.123+0000 7f35817fa700 1 -- 172.21.15.121:0/434243360 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 31) ==== 8366+0+0 (secure 0 0 0) 0x7f359409aee0 con 0x55b0bd2a8fa0 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.123+0000 7f35aa612b80 10 client.16551 fetch_fsmap finished waiting for FSMap version 31 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.123+0000 7f35aa612b80 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.123+0000 7f35aa612b80 20 client.16551 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.123+0000 7f35aa612b80 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b0bd32e600 0x55b0bd330a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.123+0000 7f35aa612b80 4 client.16551 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.123+0000 7f35aa612b80 1 -- 172.21.15.121:0/434243360 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b0bd202310 con 0x55b0bd32e600 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.123+0000 7f359d32a700 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b0bd32e600 0x55b0bd330a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.123+0000 7f359d32a700 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b0bd32e600 0x55b0bd330a20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=553dc4333f426f6c server_cookie=611923fd6eb089f9 in_seq=0 out_seq=0 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.123+0000 7f35817fa700 10 client.16551 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.124+0000 7f35817fa700 1 -- 172.21.15.121:0/434243360 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b0bd202310 con 0x55b0bd32e600 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.124+0000 7f35817fa700 10 client.16551 handle_command_reply: tid=0 2024-06-10T00:35:02.133 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.124+0000 7f35aa612b80 1 client.16551 shutdown 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.124+0000 7f35aa612b80 1 -- 172.21.15.121:0/434243360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 msgr2=0x55b0bd2c8290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.124+0000 7f35aa612b80 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2c8290 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f359400e9c0 tx=0x7f359400cef0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.124+0000 7f35aa612b80 1 -- 172.21.15.121:0/434243360 shutdown_connections 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.124+0000 7f35aa612b80 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55b0bd32e600 0x55b0bd330a20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.124+0000 7f35aa612b80 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b0bd2aefe0 0x55b0bd2c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.124+0000 7f35aa612b80 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b0bd2a8fa0 0x55b0bd2c8290 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.124+0000 7f35aa612b80 1 --2- 172.21.15.121:0/434243360 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55b0bd2ae6c0 0x55b0bd2c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.124+0000 7f35aa612b80 1 -- 172.21.15.121:0/434243360 >> 172.21.15.121:0/434243360 conn(0x55b0bd1d3d40 msgr2=0x55b0bd2afad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f35817fa700 0 client.16551 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f35aa612b80 1 -- 172.21.15.121:0/434243360 shutdown_connections 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f35aa612b80 1 -- 172.21.15.121:0/434243360 wait complete. 2024-06-10T00:35:02.134 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f35aa612b80 20 client.16551 trim_cache size 0 max 16384 2024-06-10T00:35:02.135 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:35:02.135 INFO:teuthology.orchestra.run.smithi121.stdout: "return_code": 0, 2024-06-10T00:35:02.135 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_tag": "3ba0f9de-d1ff-43e3-b1cf-66fe15decd0f", 2024-06-10T00:35:02.135 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": "asynchronous" 2024-06-10T00:35:02.135 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:35:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 -- 172.21.15.121:0/862085227 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35840a2d90 msgr2=0x7f35840a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35840a2d90 0x7f35840a5250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3598078a90 tx=0x7f3588009590 comp rx=0 tx=0).stop 2024-06-10T00:35:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 -- 172.21.15.121:0/862085227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f359806ab60 msgr2=0x7f3598101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f359806ab60 0x7f3598101490 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f359400e9f0 tx=0x7f359400ed00 comp rx=0 tx=0).stop 2024-06-10T00:35:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 -- 172.21.15.121:0/862085227 shutdown_connections 2024-06-10T00:35:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35840a2d90 0x7f35840a5250 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3598108530 0x7f3598101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f359806ab60 0x7f3598101490 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 --2- 172.21.15.121:0/862085227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359806b520 0x7f35981019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 -- 172.21.15.121:0/862085227 >> 172.21.15.121:0/862085227 conn(0x7f35980783d0 msgr2=0x7f3598100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 -- 172.21.15.121:0/862085227 shutdown_connections 2024-06-10T00:35:02.137 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:02.125+0000 7f359fd8f700 1 -- 172.21.15.121:0/862085227 wait complete. 2024-06-10T00:35:03.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:35:03.341 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:35:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:03 smithi193 ceph-mon[36716]: pgmap v215: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 274 op/s 2024-06-10T00:35:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:03 smithi193 ceph-mon[36716]: scrub queued for path: / 2024-06-10T00:35:03.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:03 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:03.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:03 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:35:03.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:03 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:35:03.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:03 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:35:03.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:03 smithi193 ceph-mon[36716]: osdmap e102: 12 total, 12 up, 12 in 2024-06-10T00:35:03.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:03 smithi121 ceph-mon[32378]: pgmap v215: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 274 op/s 2024-06-10T00:35:03.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:03 smithi121 ceph-mon[32378]: scrub queued for path: / 2024-06-10T00:35:03.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:03 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:03.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:03 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:35:03.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:03 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:35:03.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:03 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:35:03.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:03 smithi121 ceph-mon[32378]: osdmap e102: 12 total, 12 up, 12 in 2024-06-10T00:35:03.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:03 smithi135 ceph-mon[36424]: pgmap v215: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.1 MiB/s wr, 274 op/s 2024-06-10T00:35:03.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:03 smithi135 ceph-mon[36424]: scrub queued for path: / 2024-06-10T00:35:03.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:03 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:03.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:03 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:35:03.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:03 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:35:03.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:03 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:35:03.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:03 smithi135 ceph-mon[36424]: osdmap e102: 12 total, 12 up, 12 in 2024-06-10T00:35:04.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:04 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:35:04.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:04 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:35:04.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:04 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:35:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:05 smithi193 ceph-mon[36716]: pgmap v217: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 309 op/s 2024-06-10T00:35:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:05 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:05.637 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:05 smithi121 ceph-mon[32378]: pgmap v217: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 309 op/s 2024-06-10T00:35:05.637 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:05 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:05.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:05 smithi135 ceph-mon[36424]: pgmap v217: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 309 op/s 2024-06-10T00:35:05.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:05 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:05.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.893+0000 7feb81abe700 1 -- 172.21.15.121:0/3058354781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb7c107400 msgr2=0x7feb7c1077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:05.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.893+0000 7feb81abe700 1 --2- 172.21.15.121:0/3058354781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb7c107400 0x7feb7c1077e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7feb6c009990 tx=0x7feb6c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.893+0000 7feb81abe700 1 -- 172.21.15.121:0/3058354781 shutdown_connections 2024-06-10T00:35:05.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.893+0000 7feb81abe700 1 --2- 172.21.15.121:0/3058354781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb7c101d50 0x7feb7c10e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.894 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.893+0000 7feb81abe700 1 --2- 172.21.15.121:0/3058354781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb7c107400 0x7feb7c1077e0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.893+0000 7feb81abe700 1 --2- 172.21.15.121:0/3058354781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb7c101390 0x7feb7c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.893+0000 7feb81abe700 1 -- 172.21.15.121:0/3058354781 >> 172.21.15.121:0/3058354781 conn(0x7feb7c078340 msgr2=0x7feb7c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:05.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.893+0000 7feb81abe700 1 -- 172.21.15.121:0/3058354781 shutdown_connections 2024-06-10T00:35:05.895 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.894+0000 7feb81abe700 1 -- 172.21.15.121:0/3058354781 wait complete. 2024-06-10T00:35:05.896 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.894+0000 7feb81abe700 1 Processor -- start 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb81abe700 1 -- start start 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb81abe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb7c101390 0x7feb7c19d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb81abe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb7c101d50 0x7feb7c19dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb81abe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb7c107400 0x7feb7c1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb81abe700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1116f0 con 0x7feb7c107400 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb81abe700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c111570 con 0x7feb7c101d50 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb81abe700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c111870 con 0x7feb7c101390 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb7affd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb7c101d50 0x7feb7c19dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb7affd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb7c101d50 0x7feb7c19dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:33348/0 (socket says 172.21.15.121:33348) 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb7bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb7c107400 0x7feb7c1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb7affd700 1 -- 172.21.15.121:0/275247410 learned_addr learned my addr 172.21.15.121:0/275247410 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.895+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb7c101390 0x7feb7c19d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.897 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.896+0000 7feb7affd700 1 -- 172.21.15.121:0/275247410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb7c101390 msgr2=0x7feb7c19d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.896+0000 7feb7affd700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb7c101390 0x7feb7c19d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.896+0000 7feb7affd700 1 -- 172.21.15.121:0/275247410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb7c107400 msgr2=0x7feb7c1979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.896+0000 7feb7affd700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb7c107400 0x7feb7c1979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.896+0000 7feb7affd700 1 -- 172.21.15.121:0/275247410 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7c198130 con 0x7feb7c101d50 2024-06-10T00:35:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.896+0000 7feb7affd700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb7c101d50 0x7feb7c19dd50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feb6400e9c0 tx=0x7feb6400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.896+0000 7feb78ff9700 1 -- 172.21.15.121:0/275247410 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6400ee60 con 0x7feb7c101d50 2024-06-10T00:35:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.896+0000 7feb81abe700 1 -- 172.21.15.121:0/275247410 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7c198420 con 0x7feb7c101d50 2024-06-10T00:35:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.896+0000 7feb78ff9700 1 -- 172.21.15.121:0/275247410 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb64004830 con 0x7feb7c101d50 2024-06-10T00:35:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.896+0000 7feb78ff9700 1 -- 172.21.15.121:0/275247410 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64005690 con 0x7feb7c101d50 2024-06-10T00:35:05.898 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.896+0000 7feb81abe700 1 -- 172.21.15.121:0/275247410 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb7c06a170 con 0x7feb7c101d50 2024-06-10T00:35:05.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.898+0000 7feb78ff9700 1 -- 172.21.15.121:0/275247410 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7feb64004d50 con 0x7feb7c101d50 2024-06-10T00:35:05.899 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.899+0000 7feb78ff9700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feb680a2d90 0x7feb680a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.899+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feb680a2d90 0x7feb680a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.899+0000 7feb78ff9700 1 -- 172.21.15.121:0/275247410 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9613+0+0 (secure 0 0 0) 0x7feb640cf1d0 con 0x7feb7c101d50 2024-06-10T00:35:05.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feb680a2d90 0x7feb680a5250 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feb7c198b10 tx=0x7feb6c01a040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:05.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb8c341b80 1 Processor -- start 2024-06-10T00:35:05.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb8c341b80 1 -- start start 2024-06-10T00:35:05.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 0x5626777d6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 0x5626777d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d0f20 0x5626777d1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb8c341b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562677722810 con 0x5626777d6f60 2024-06-10T00:35:05.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb8c341b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5626776ca720 con 0x5626777d6640 2024-06-10T00:35:05.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb8c341b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56267771e7d0 con 0x5626777d0f20 2024-06-10T00:35:05.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb7b7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 0x5626777d6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb7affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 0x5626777d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb7bfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d0f20 0x5626777d1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb7b7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 0x5626777d6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:33364/0 (socket says 172.21.15.121:33364) 2024-06-10T00:35:05.901 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb7b7fe700 1 -- 172.21.15.121:0/3136137076 learned_addr learned my addr 172.21.15.121:0/3136137076 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:05.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb7affd700 1 -- 172.21.15.121:0/3136137076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d0f20 msgr2=0x5626777d1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:05.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.900+0000 7feb7affd700 1 --2- 172.21.15.121:0/3136137076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d0f20 0x5626777d1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.901+0000 7feb7affd700 1 -- 172.21.15.121:0/3136137076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 msgr2=0x5626777d6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:05.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.901+0000 7feb7affd700 1 --2- 172.21.15.121:0/3136137076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 0x5626777d6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.901+0000 7feb7affd700 1 -- 172.21.15.121:0/3136137076 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626776c0f50 con 0x5626777d6f60 2024-06-10T00:35:05.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.901+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/3136137076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 0x5626777d6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:05.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.901+0000 7feb7affd700 1 --2- 172.21.15.121:0/3136137076 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 0x5626777d09e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7feb640cc4e0 tx=0x7feb6409b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec94a9e675edd4f0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:05.906 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.901+0000 7feb71ffb700 1 -- 172.21.15.121:0/3136137076 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6409f660 con 0x5626777d6f60 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.901+0000 7feb71ffb700 1 -- 172.21.15.121:0/3136137076 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb640a3f20 con 0x5626777d6f60 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.901+0000 7feb71ffb700 1 -- 172.21.15.121:0/3136137076 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb640bb340 con 0x5626777d6f60 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.901+0000 7feb8c341b80 1 -- 172.21.15.121:0/3136137076 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 msgr2=0x5626777d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.901+0000 7feb8c341b80 1 --2- 172.21.15.121:0/3136137076 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 0x5626777d09e0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7feb640cc4e0 tx=0x7feb6409b1b0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 -- 172.21.15.121:0/3136137076 shutdown_connections 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 --2- 172.21.15.121:0/3136137076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d0f20 0x5626777d1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 --2- 172.21.15.121:0/3136137076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 0x5626777d6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 --2- 172.21.15.121:0/3136137076 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 0x5626777d09e0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 -- 172.21.15.121:0/3136137076 >> 172.21.15.121:0/3136137076 conn(0x5626776fbcc0 msgr2=0x5626777cb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 -- 172.21.15.121:0/3136137076 shutdown_connections 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 -- 172.21.15.121:0/3136137076 wait complete. 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 -- start start 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 Processor -- start 2024-06-10T00:35:05.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 0x5626777dcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6640 0x5626777da060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6f60 0x5626777da5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5626777dd910 con 0x5626777d6640 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5626776c2bf0 con 0x5626777d0f20 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.902+0000 7feb8c341b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5626777dda90 con 0x5626777d6f60 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb8c341b80 10 client.0 _ll_register_callbacks cb 0x5626776bc920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6640 0x5626777da060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6640 0x5626777da060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54418/0 (socket says 172.21.15.121:54418) 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7b7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 0x5626777dcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7affd700 1 -- 172.21.15.121:0/3203945629 learned_addr learned my addr 172.21.15.121:0/3203945629 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7bfff700 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6f60 0x5626777da5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7affd700 1 -- 172.21.15.121:0/3203945629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6f60 msgr2=0x5626777da5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:05.908 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7affd700 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6f60 0x5626777da5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7affd700 1 -- 172.21.15.121:0/3203945629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 msgr2=0x5626777dcd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7affd700 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 0x5626777dcd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7affd700 1 -- 172.21.15.121:0/3203945629 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626776718d0 con 0x5626777d6640 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 0x5626777dcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb7affd700 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6640 0x5626777da060 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7feb640cc590 tx=0x7feb640097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2ce8ddc0276653 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb5affd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb5affd700 1 -- 172.21.15.121:0/3203945629 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6409f660 con 0x5626777d6640 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb5affd700 1 -- 172.21.15.121:0/3203945629 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9613+0+0 (secure 0 0 0) 0x7feb6409b720 con 0x5626777d6640 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb8c341b80 1 -- 172.21.15.121:0/3203945629 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb640a3f20 con 0x5626777d6640 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb5affd700 1 -- 172.21.15.121:0/3203945629 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feb640b4280 con 0x5626777d6640 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb8c341b80 10 client.16563 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.903+0000 7feb8c341b80 1 -- 172.21.15.121:0/3203945629 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5626777dd910 con 0x5626777d6640 2024-06-10T00:35:05.909 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.904+0000 7feb5affd700 1 -- 172.21.15.121:0/3203945629 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb640a3f20 con 0x5626777d6640 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.904+0000 7feb5affd700 1 -- 172.21.15.121:0/3203945629 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7feb6409a700 con 0x5626777d6640 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.904+0000 7feb8c341b80 10 client.16563 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.904+0000 7feb8c341b80 10 client.16563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.904+0000 7feb8c341b80 20 client.16563 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.904+0000 7feb8c341b80 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5626777ea5f0 0x5626777ec9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.904+0000 7feb8c341b80 4 client.16563 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.904+0000 7feb8c341b80 1 -- 172.21.15.121:0/3203945629 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56267772fd10 con 0x5626777ea5f0 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.904+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5626777ea5f0 0x5626777ec9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.904+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5626777ea5f0 0x5626777ec9e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f51bdef09d275093 server_cookie=bd227e2dd28cd27e in_seq=0 out_seq=0 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.905+0000 7feb5affd700 10 client.16563 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.905+0000 7feb5affd700 1 -- 172.21.15.121:0/3203945629 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x56267772fd10 con 0x5626777ea5f0 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.905+0000 7feb5affd700 10 client.16563 handle_command_reply: tid=0 2024-06-10T00:35:05.910 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.905+0000 7feb8c341b80 1 client.16563 shutdown 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.905+0000 7feb8c341b80 1 -- 172.21.15.121:0/3203945629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6640 msgr2=0x5626777da060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.905+0000 7feb8c341b80 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6640 0x5626777da060 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7feb640cc590 tx=0x7feb640097b0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.905+0000 7feb8c341b80 1 -- 172.21.15.121:0/3203945629 shutdown_connections 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.905+0000 7feb8c341b80 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5626777ea5f0 0x5626777ec9e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.905+0000 7feb8c341b80 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6f60 0x5626777da5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.906+0000 7feb8c341b80 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 0x5626777dcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.906+0000 7feb8c341b80 1 --2- 172.21.15.121:0/3203945629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6640 0x5626777da060 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.906+0000 7feb8c341b80 1 -- 172.21.15.121:0/3203945629 >> 172.21.15.121:0/3203945629 conn(0x5626776fbcc0 msgr2=0x5626777cb5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.906+0000 7feb5affd700 0 client.16563 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.906+0000 7feb8c341b80 1 -- 172.21.15.121:0/3203945629 shutdown_connections 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.906+0000 7feb8c341b80 1 -- 172.21.15.121:0/3203945629 wait complete. 2024-06-10T00:35:05.911 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.906+0000 7feb8c341b80 20 client.16563 trim_cache size 0 max 16384 2024-06-10T00:35:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb8c341b80 1 Processor -- start 2024-06-10T00:35:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb8c341b80 1 -- start start 2024-06-10T00:35:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d0f20 0x5626777e6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 0x5626777d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6f60 0x5626777d4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb8c341b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1aef80 con 0x5626777d0f20 2024-06-10T00:35:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb8c341b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1ac750 con 0x5626777d6640 2024-06-10T00:35:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb8c341b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1aa5c0 con 0x5626777d6f60 2024-06-10T00:35:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb7b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d0f20 0x5626777e6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.928 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb7b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d0f20 0x5626777e6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54426/0 (socket says 172.21.15.121:54426) 2024-06-10T00:35:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb7b7fe700 1 -- 172.21.15.121:0/2410300266 learned_addr learned my addr 172.21.15.121:0/2410300266 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb7bfff700 1 --2- 172.21.15.121:0/2410300266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6f60 0x5626777d4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb7affd700 1 --2- 172.21.15.121:0/2410300266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 0x5626777d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb7b7fe700 1 -- 172.21.15.121:0/2410300266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6f60 msgr2=0x5626777d4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/2410300266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6f60 0x5626777d4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb7b7fe700 1 -- 172.21.15.121:0/2410300266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 msgr2=0x5626777d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/2410300266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 0x5626777d09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb7b7fe700 1 -- 172.21.15.121:0/2410300266 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626777d16c0 con 0x5626777d0f20 2024-06-10T00:35:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.927+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/2410300266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d0f20 0x5626777e6be0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7feb6c000c00 tx=0x7feb6c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d112e5309ac17877 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb717fa700 1 -- 172.21.15.121:0/2410300266 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c037020 con 0x5626777d0f20 2024-06-10T00:35:05.929 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb717fa700 1 -- 172.21.15.121:0/2410300266 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6c003d60 con 0x5626777d0f20 2024-06-10T00:35:05.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb717fa700 1 -- 172.21.15.121:0/2410300266 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c033450 con 0x5626777d0f20 2024-06-10T00:35:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb8c341b80 1 -- 172.21.15.121:0/2410300266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d0f20 msgr2=0x5626777e6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb8c341b80 1 --2- 172.21.15.121:0/2410300266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d0f20 0x5626777e6be0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7feb6c000c00 tx=0x7feb6c01f000 comp rx=0 tx=0).stop 2024-06-10T00:35:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb8c341b80 1 -- 172.21.15.121:0/2410300266 shutdown_connections 2024-06-10T00:35:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb8c341b80 1 --2- 172.21.15.121:0/2410300266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6f60 0x5626777d4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb8c341b80 1 --2- 172.21.15.121:0/2410300266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d0f20 0x5626777e6be0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb8c341b80 1 --2- 172.21.15.121:0/2410300266 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d6640 0x5626777d09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb8c341b80 1 -- 172.21.15.121:0/2410300266 >> 172.21.15.121:0/2410300266 conn(0x5626776fbcc0 msgr2=0x5626777d7400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb8c341b80 1 -- 172.21.15.121:0/2410300266 shutdown_connections 2024-06-10T00:35:05.934 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.928+0000 7feb8c341b80 1 -- 172.21.15.121:0/2410300266 wait complete. 2024-06-10T00:35:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb8c341b80 1 -- start start 2024-06-10T00:35:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb8c341b80 1 Processor -- start 2024-06-10T00:35:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 0x5626777e3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6640 0x5626777f1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb8c341b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 0x5626777f2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb8c341b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1aac10 con 0x5626777d6f60 2024-06-10T00:35:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb8c341b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1ac080 con 0x5626777d0f20 2024-06-10T00:35:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb8c341b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c1ab2e0 con 0x5626777d6640 2024-06-10T00:35:05.935 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb8c341b80 10 client.0 _ll_register_callbacks cb 0x5626776bc920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb7bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 0x5626777f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb7b7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 0x5626777e3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb7bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 0x5626777f2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54438/0 (socket says 172.21.15.121:54438) 2024-06-10T00:35:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb7affd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6640 0x5626777f1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.936 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.929+0000 7feb7bfff700 1 -- 172.21.15.121:0/1693220079 learned_addr learned my addr 172.21.15.121:0/1693220079 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb7b7fe700 1 -- 172.21.15.121:0/1693220079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6640 msgr2=0x5626777f1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6640 0x5626777f1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb7b7fe700 1 -- 172.21.15.121:0/1693220079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 msgr2=0x5626777f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 0x5626777f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb7b7fe700 1 -- 172.21.15.121:0/1693220079 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562677853fe0 con 0x5626777d0f20 2024-06-10T00:35:05.937 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb7bfff700 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 0x5626777f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:05.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb7b7fe700 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 0x5626777e3650 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0098e0 tx=0x7feb6c0058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4faf526fc013d3a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:05.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb5affd700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:35:05.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb5affd700 1 -- 172.21.15.121:0/1693220079 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c037050 con 0x5626777d0f20 2024-06-10T00:35:05.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb8c341b80 1 -- 172.21.15.121:0/1693220079 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb6c003d60 con 0x5626777d0f20 2024-06-10T00:35:05.938 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb5affd700 1 -- 172.21.15.121:0/1693220079 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9613+0+0 (secure 0 0 0) 0x7feb6c00f660 con 0x5626777d0f20 2024-06-10T00:35:05.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb5affd700 1 -- 172.21.15.121:0/1693220079 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feb6c003680 con 0x5626777d0f20 2024-06-10T00:35:05.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb8c341b80 10 client.26126 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:05.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb8c341b80 1 -- 172.21.15.121:0/1693220079 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5626778551d0 con 0x5626777d0f20 2024-06-10T00:35:05.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.930+0000 7feb5affd700 1 -- 172.21.15.121:0/1693220079 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6c003d60 con 0x5626777d0f20 2024-06-10T00:35:05.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb5affd700 1 -- 172.21.15.121:0/1693220079 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7feb6c045b70 con 0x5626777d0f20 2024-06-10T00:35:05.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb8c341b80 10 client.26126 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:05.939 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb8c341b80 10 client.26126 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:35:05.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb8c341b80 20 client.26126 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:05.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb8c341b80 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5626778557b0 0x562677857bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:05.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb8c341b80 4 client.26126 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:35:05.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb8c341b80 1 -- 172.21.15.121:0/1693220079 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56267772a290 con 0x5626778557b0 2024-06-10T00:35:05.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb7affd700 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5626778557b0 0x562677857bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:05.940 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb7affd700 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5626778557b0 0x562677857bd0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d28422caf803581b server_cookie=43add5bd84a10d4c in_seq=0 out_seq=0 2024-06-10T00:35:05.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb5affd700 10 client.26126 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:05.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb5affd700 1 -- 172.21.15.121:0/1693220079 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56267772a290 con 0x5626778557b0 2024-06-10T00:35:05.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb5affd700 10 client.26126 handle_command_reply: tid=0 2024-06-10T00:35:05.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.931+0000 7feb8c341b80 1 client.26126 shutdown 2024-06-10T00:35:05.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb8c341b80 1 -- 172.21.15.121:0/1693220079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 msgr2=0x5626777e3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:05.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb8c341b80 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 0x5626777e3650 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0098e0 tx=0x7feb6c0058e0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.941 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb8c341b80 1 -- 172.21.15.121:0/1693220079 shutdown_connections 2024-06-10T00:35:05.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb8c341b80 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5626778557b0 0x562677857bd0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb8c341b80 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5626777d6f60 0x5626777f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb8c341b80 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5626777d0f20 0x5626777e3650 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb5affd700 0 client.26126 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:05.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb8c341b80 1 --2- 172.21.15.121:0/1693220079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5626777d6640 0x5626777f1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.942 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb8c341b80 1 -- 172.21.15.121:0/1693220079 >> 172.21.15.121:0/1693220079 conn(0x5626776fbcc0 msgr2=0x5626777d7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb8c341b80 1 -- 172.21.15.121:0/1693220079 shutdown_connections 2024-06-10T00:35:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb8c341b80 1 -- 172.21.15.121:0/1693220079 wait complete. 2024-06-10T00:35:05.943 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb8c341b80 20 client.26126 trim_cache size 0 max 16384 2024-06-10T00:35:05.943 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:35:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-10T00:35:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": { 2024-06-10T00:35:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "3ba0f9de-d1ff-43e3-b1cf-66fe15decd0f": { 2024-06-10T00:35:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/", 2024-06-10T00:35:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "tag": "3ba0f9de-d1ff-43e3-b1cf-66fe15decd0f", 2024-06-10T00:35:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: "options": "recursive,force" 2024-06-10T00:35:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:35:05.944 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:35:05.945 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:35:05.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb81abe700 1 -- 172.21.15.121:0/275247410 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feb680a2d90 msgr2=0x7feb680a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:05.946 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb81abe700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feb680a2d90 0x7feb680a5250 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7feb7c198b10 tx=0x7feb6c01a040 comp rx=0 tx=0).stop 2024-06-10T00:35:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb81abe700 1 -- 172.21.15.121:0/275247410 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb7c101d50 msgr2=0x7feb7c19dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb81abe700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb7c101d50 0x7feb7c19dd50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7feb6400e9c0 tx=0x7feb6400ecd0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb81abe700 1 -- 172.21.15.121:0/275247410 shutdown_connections 2024-06-10T00:35:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb81abe700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7feb680a2d90 0x7feb680a5250 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb81abe700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb7c107400 0x7feb7c1979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb81abe700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feb7c101390 0x7feb7c19d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb81abe700 1 --2- 172.21.15.121:0/275247410 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7feb7c101d50 0x7feb7c19dd50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.932+0000 7feb81abe700 1 -- 172.21.15.121:0/275247410 >> 172.21.15.121:0/275247410 conn(0x7feb7c078340 msgr2=0x7feb7c0fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:05.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.933+0000 7feb81abe700 1 -- 172.21.15.121:0/275247410 shutdown_connections 2024-06-10T00:35:05.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:05.933+0000 7feb81abe700 1 -- 172.21.15.121:0/275247410 wait complete. 2024-06-10T00:35:06.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:06 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:06.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:06 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:06.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:06 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:07.335 INFO:tasks.cephfs.filesystem:scrub status for tag:3ba0f9de-d1ff-43e3-b1cf-66fe15decd0f - {'path': '/', 'tag': '3ba0f9de-d1ff-43e3-b1cf-66fe15decd0f', 'options': 'recursive,force'} 2024-06-10T00:35:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:07 smithi193 ceph-mon[36716]: pgmap v218: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 347 op/s 2024-06-10T00:35:07.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:07 smithi121 ceph-mon[32378]: pgmap v218: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 347 op/s 2024-06-10T00:35:07.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:07 smithi135 ceph-mon[36424]: pgmap v218: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 347 op/s 2024-06-10T00:35:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:09 smithi193 ceph-mon[36716]: pgmap v219: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 347 op/s 2024-06-10T00:35:09.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:09 smithi121 ceph-mon[32378]: pgmap v219: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 347 op/s 2024-06-10T00:35:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:09 smithi135 ceph-mon[36424]: pgmap v219: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 347 op/s 2024-06-10T00:35:10.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:10 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:10.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:10 smithi121 ceph-mon[32378]: osdmap e103: 12 total, 12 up, 12 in 2024-06-10T00:35:10.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:10 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:10.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:10 smithi193 ceph-mon[36716]: osdmap e103: 12 total, 12 up, 12 in 2024-06-10T00:35:10.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:10 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:10.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:10 smithi135 ceph-mon[36424]: osdmap e103: 12 total, 12 up, 12 in 2024-06-10T00:35:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:11 smithi193 ceph-mon[36716]: pgmap v220: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-06-10T00:35:11.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:11 smithi121 ceph-mon[32378]: pgmap v220: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-06-10T00:35:11.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:11 smithi135 ceph-mon[36424]: pgmap v220: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-06-10T00:35:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:13 smithi193 ceph-mon[36716]: pgmap v222: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 308 op/s 2024-06-10T00:35:13.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:13 smithi121 ceph-mon[32378]: pgmap v222: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 308 op/s 2024-06-10T00:35:13.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:13 smithi135 ceph-mon[36424]: pgmap v222: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.2 MiB/s wr, 308 op/s 2024-06-10T00:35:15.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:15 smithi193 ceph-mon[36716]: pgmap v223: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 282 op/s 2024-06-10T00:35:15.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:15 smithi121 ceph-mon[32378]: pgmap v223: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 282 op/s 2024-06-10T00:35:15.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:15 smithi135 ceph-mon[36424]: pgmap v223: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 282 op/s 2024-06-10T00:35:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:17 smithi193 ceph-mon[36716]: pgmap v224: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-06-10T00:35:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:17 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:35:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:17 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:35:17.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:17 smithi121 ceph-mon[32378]: pgmap v224: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-06-10T00:35:17.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:17 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:35:17.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:17 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:35:17.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:17 smithi135 ceph-mon[36424]: pgmap v224: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-06-10T00:35:17.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:17 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:35:17.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:17 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:35:18.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:18 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:35:18.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:18 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:35:18.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:18 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:35:19.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:19 smithi193 ceph-mon[36716]: pgmap v225: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-06-10T00:35:19.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:19 smithi121 ceph-mon[32378]: pgmap v225: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-06-10T00:35:19.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:19 smithi135 ceph-mon[36424]: pgmap v225: 129 pgs: 129 active+clean; 1.4 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 212 op/s 2024-06-10T00:35:21.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:21 smithi193 ceph-mon[36716]: pgmap v226: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 253 op/s 2024-06-10T00:35:21.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:21 smithi121 ceph-mon[32378]: pgmap v226: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 253 op/s 2024-06-10T00:35:21.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:21 smithi135 ceph-mon[36424]: pgmap v226: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.9 MiB/s wr, 253 op/s 2024-06-10T00:35:23.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:23 smithi193 ceph-mon[36716]: pgmap v227: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 223 op/s 2024-06-10T00:35:23.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:23 smithi121 ceph-mon[32378]: pgmap v227: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 223 op/s 2024-06-10T00:35:23.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:23 smithi135 ceph-mon[36424]: pgmap v227: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 223 op/s 2024-06-10T00:35:25.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:25 smithi193 ceph-mon[36716]: pgmap v228: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-06-10T00:35:25.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:25 smithi121 ceph-mon[32378]: pgmap v228: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-06-10T00:35:25.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:25 smithi135 ceph-mon[36424]: pgmap v228: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 219 op/s 2024-06-10T00:35:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:27 smithi193 ceph-mon[36716]: pgmap v229: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 204 op/s 2024-06-10T00:35:27.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:27 smithi121 ceph-mon[32378]: pgmap v229: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 204 op/s 2024-06-10T00:35:27.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:27 smithi135 ceph-mon[36424]: pgmap v229: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 204 op/s 2024-06-10T00:35:29.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:29 smithi193 ceph-mon[36716]: pgmap v230: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 150 op/s 2024-06-10T00:35:29.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:29 smithi121 ceph-mon[32378]: pgmap v230: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 150 op/s 2024-06-10T00:35:29.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:29 smithi135 ceph-mon[36424]: pgmap v230: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 150 op/s 2024-06-10T00:35:31.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:31 smithi193 ceph-mon[36716]: pgmap v231: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 234 op/s 2024-06-10T00:35:31.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:31 smithi121 ceph-mon[32378]: pgmap v231: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 234 op/s 2024-06-10T00:35:31.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:31 smithi135 ceph-mon[36424]: pgmap v231: 129 pgs: 129 active+clean; 1.4 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 234 op/s 2024-06-10T00:35:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:33 smithi193 ceph-mon[36716]: pgmap v232: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 209 op/s 2024-06-10T00:35:33.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:33 smithi121 ceph-mon[32378]: pgmap v232: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 209 op/s 2024-06-10T00:35:33.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:33 smithi135 ceph-mon[36424]: pgmap v232: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 209 op/s 2024-06-10T00:35:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:35 smithi193 ceph-mon[36716]: pgmap v233: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-06-10T00:35:35.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:35 smithi121 ceph-mon[32378]: pgmap v233: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-06-10T00:35:35.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:35 smithi135 ceph-mon[36424]: pgmap v233: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-06-10T00:35:37.336 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:35:37.592 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:35:37.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:37 smithi193 ceph-mon[36716]: pgmap v234: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 277 op/s 2024-06-10T00:35:37.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:37 smithi121 ceph-mon[32378]: pgmap v234: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 277 op/s 2024-06-10T00:35:37.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:37 smithi135 ceph-mon[36424]: pgmap v234: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.3 MiB/s wr, 277 op/s 2024-06-10T00:35:39.629 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:39 smithi121 ceph-mon[32378]: pgmap v235: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 248 op/s 2024-06-10T00:35:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:39 smithi193 ceph-mon[36716]: pgmap v235: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 248 op/s 2024-06-10T00:35:39.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:39 smithi135 ceph-mon[36424]: pgmap v235: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 248 op/s 2024-06-10T00:35:40.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.026+0000 7efdaca1c700 1 -- 172.21.15.121:0/1305700865 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda8101d50 msgr2=0x7efda810e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:40.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.026+0000 7efdaca1c700 1 --2- 172.21.15.121:0/1305700865 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda8101d50 0x7efda810e240 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7efda000b0a0 tx=0x7efda000b3b0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.026+0000 7efdaca1c700 1 -- 172.21.15.121:0/1305700865 shutdown_connections 2024-06-10T00:35:40.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.026+0000 7efdaca1c700 1 --2- 172.21.15.121:0/1305700865 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda8101d50 0x7efda810e240 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.026+0000 7efdaca1c700 1 --2- 172.21.15.121:0/1305700865 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efda8107400 0x7efda81077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.027 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.026+0000 7efdaca1c700 1 --2- 172.21.15.121:0/1305700865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efda8101390 0x7efda8101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.026+0000 7efdaca1c700 1 -- 172.21.15.121:0/1305700865 >> 172.21.15.121:0/1305700865 conn(0x7efda8078340 msgr2=0x7efda8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:40.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.026+0000 7efdaca1c700 1 -- 172.21.15.121:0/1305700865 shutdown_connections 2024-06-10T00:35:40.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.026+0000 7efdaca1c700 1 -- 172.21.15.121:0/1305700865 wait complete. 2024-06-10T00:35:40.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.027+0000 7efdaca1c700 1 Processor -- start 2024-06-10T00:35:40.028 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.027+0000 7efdaca1c700 1 -- start start 2024-06-10T00:35:40.029 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.027+0000 7efdaca1c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efda8101390 0x7efda81999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.027+0000 7efdaca1c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda8101d50 0x7efda8199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.027+0000 7efdaca1c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efda8107400 0x7efda81a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.027+0000 7efdaca1c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efda8111670 con 0x7efda8101d50 2024-06-10T00:35:40.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.027+0000 7efdaca1c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7efda81114f0 con 0x7efda8107400 2024-06-10T00:35:40.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.027+0000 7efdaca1c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efda81117f0 con 0x7efda8101390 2024-06-10T00:35:40.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efd9ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda8101d50 0x7efda8199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efda659c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efda8101390 0x7efda81999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efd9ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda8101d50 0x7efda8199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42962/0 (socket says 172.21.15.121:42962) 2024-06-10T00:35:40.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efd9ffff700 1 -- 172.21.15.121:0/2598850520 learned_addr learned my addr 172.21.15.121:0/2598850520 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:40.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efda6d9d700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efda8107400 0x7efda81a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.030 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efda659c700 1 -- 172.21.15.121:0/2598850520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efda8107400 msgr2=0x7efda81a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efda659c700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efda8107400 0x7efda81a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efda659c700 1 -- 172.21.15.121:0/2598850520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda8101d50 msgr2=0x7efda8199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efda659c700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda8101d50 0x7efda8199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efda659c700 1 -- 172.21.15.121:0/2598850520 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda81a4f90 con 0x7efda8101390 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efda6d9d700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efda8107400 0x7efda81a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efd9ffff700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda8101d50 0x7efda8199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efda659c700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efda8101390 0x7efda81999d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efd9800d950 tx=0x7efd9800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efd9f7fe700 1 -- 172.21.15.121:0/2598850520 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98017070 con 0x7efda8101390 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efdaca1c700 1 -- 172.21.15.121:0/2598850520 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efda81a51c0 con 0x7efda8101390 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efd9f7fe700 1 -- 172.21.15.121:0/2598850520 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd98009e10 con 0x7efda8101390 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efd9f7fe700 1 -- 172.21.15.121:0/2598850520 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9800f3c0 con 0x7efda8101390 2024-06-10T00:35:40.031 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.028+0000 7efdaca1c700 1 -- 172.21.15.121:0/2598850520 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efda81a5750 con 0x7efda8101390 2024-06-10T00:35:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.031+0000 7efd9f7fe700 1 -- 172.21.15.121:0/2598850520 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7efd9801a070 con 0x7efda8101390 2024-06-10T00:35:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.031+0000 7efd9f7fe700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7efd940a2d90 0x7efd940a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.031+0000 7efd9ffff700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7efd940a2d90 0x7efd940a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.032+0000 7efd9f7fe700 1 -- 172.21.15.121:0/2598850520 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7efd980cf1a0 con 0x7efda8101390 2024-06-10T00:35:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.032+0000 7efd9ffff700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7efd940a2d90 0x7efd940a5250 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efda80788d0 tx=0x7efd90009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.032+0000 7efdb729fb80 1 Processor -- start 2024-06-10T00:35:40.034 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.032+0000 7efdb729fb80 1 -- start start 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.032+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3ad640 0x562fbe3ada20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.032+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3adf60 0x562fbe3a79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.032+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3a8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.032+0000 7efdb729fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562fbe2f9810 con 0x562fbe3ad640 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.032+0000 7efdb729fb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x562fbe2a1720 con 0x562fbe3adf60 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.032+0000 7efdb729fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562fbe2f57d0 con 0x562fbe3a7f20 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efda659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3ad640 0x562fbe3ada20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efda659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3ad640 0x562fbe3ada20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42966/0 (socket says 172.21.15.121:42966) 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efda659c700 1 -- 172.21.15.121:0/1405960228 learned_addr learned my addr 172.21.15.121:0/1405960228 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efda6d9d700 1 --2- 172.21.15.121:0/1405960228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3a8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efd9ffff700 1 --2- 172.21.15.121:0/1405960228 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3adf60 0x562fbe3a79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efda659c700 1 -- 172.21.15.121:0/1405960228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 msgr2=0x562fbe3a8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:40.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efda659c700 1 --2- 172.21.15.121:0/1405960228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3a8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efda659c700 1 -- 172.21.15.121:0/1405960228 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3adf60 msgr2=0x562fbe3a79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efda659c700 1 --2- 172.21.15.121:0/1405960228 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3adf60 0x562fbe3a79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efda659c700 1 -- 172.21.15.121:0/1405960228 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fbe297f50 con 0x562fbe3ad640 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efda659c700 1 --2- 172.21.15.121:0/1405960228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3ad640 0x562fbe3ada20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7efd98097c40 tx=0x7efd9809aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f1a076b0fa4fc server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efd9cff9700 1 -- 172.21.15.121:0/1405960228 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98021070 con 0x562fbe3ad640 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efd9cff9700 1 -- 172.21.15.121:0/1405960228 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd9809f620 con 0x562fbe3ad640 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.033+0000 7efd9cff9700 1 -- 172.21.15.121:0/1405960228 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd980a3c50 con 0x562fbe3ad640 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.034+0000 7efdb729fb80 1 -- 172.21.15.121:0/1405960228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3ad640 msgr2=0x562fbe3ada20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.034+0000 7efdb729fb80 1 --2- 172.21.15.121:0/1405960228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3ad640 0x562fbe3ada20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7efd98097c40 tx=0x7efd9809aeb0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.034+0000 7efdb729fb80 1 -- 172.21.15.121:0/1405960228 shutdown_connections 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.034+0000 7efdb729fb80 1 --2- 172.21.15.121:0/1405960228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3a8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.034+0000 7efdb729fb80 1 --2- 172.21.15.121:0/1405960228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3ad640 0x562fbe3ada20 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.034+0000 7efdb729fb80 1 --2- 172.21.15.121:0/1405960228 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3adf60 0x562fbe3a79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.034+0000 7efdb729fb80 1 -- 172.21.15.121:0/1405960228 >> 172.21.15.121:0/1405960228 conn(0x562fbe2d2cc0 msgr2=0x562fbe3a2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:40.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.034+0000 7efdb729fb80 1 -- 172.21.15.121:0/1405960228 shutdown_connections 2024-06-10T00:35:40.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.034+0000 7efdb729fb80 1 -- 172.21.15.121:0/1405960228 wait complete. 2024-06-10T00:35:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.034+0000 7efdb729fb80 1 -- start start 2024-06-10T00:35:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efdb729fb80 1 Processor -- start 2024-06-10T00:35:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3a7f20 0x562fbe3b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3b1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3adf60 0x562fbe3b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efdb729fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562fbe3b4910 con 0x562fbe3a7f20 2024-06-10T00:35:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efdb729fb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x562fbe299bf0 con 0x562fbe3ad640 2024-06-10T00:35:40.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efdb729fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562fbe3b4a90 con 0x562fbe3adf60 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efdb729fb80 10 client.0 _ll_register_callbacks cb 0x562fbe293920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efda659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3a7f20 0x562fbe3b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efda659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3a7f20 0x562fbe3b3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42980/0 (socket says 172.21.15.121:42980) 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efda659c700 1 -- 172.21.15.121:0/2825523814 learned_addr learned my addr 172.21.15.121:0/2825523814 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efda6d9d700 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3adf60 0x562fbe3b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efd9ffff700 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efda659c700 1 -- 172.21.15.121:0/2825523814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3adf60 msgr2=0x562fbe3b15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efda659c700 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3adf60 0x562fbe3b15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efda659c700 1 -- 172.21.15.121:0/2825523814 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 msgr2=0x562fbe3b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efda659c700 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efda659c700 1 -- 172.21.15.121:0/2825523814 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fbe2488d0 con 0x562fbe3a7f20 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efda659c700 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3a7f20 0x562fbe3b3d80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7efd98005cf0 tx=0x7efd98010460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d134d0127fc145a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:40.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efd8dffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.035+0000 7efd8dffb700 1 -- 172.21.15.121:0/2825523814 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd980b3070 con 0x562fbe3a7f20 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efd8dffb700 1 -- 172.21.15.121:0/2825523814 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7efd9809b6e0 con 0x562fbe3a7f20 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efdb729fb80 1 -- 172.21.15.121:0/2825523814 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd9809f620 con 0x562fbe3a7f20 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efd8dffb700 1 -- 172.21.15.121:0/2825523814 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efd98021050 con 0x562fbe3a7f20 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efdb729fb80 10 client.16581 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efdb729fb80 1 -- 172.21.15.121:0/2825523814 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562fbe3bf2d0 con 0x562fbe3a7f20 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efd8dffb700 1 -- 172.21.15.121:0/2825523814 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd980a6080 con 0x562fbe3a7f20 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efd8dffb700 1 -- 172.21.15.121:0/2825523814 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7efd9809f170 con 0x562fbe3a7f20 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efdb729fb80 10 client.16581 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efdb729fb80 10 client.16581 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efdb729fb80 20 client.16581 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562fbe3c17a0 0x562fbe3c3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efdb729fb80 4 client.16581 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:35:40.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efdb729fb80 1 -- 172.21.15.121:0/2825523814 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fbe306d10 con 0x562fbe3c17a0 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.036+0000 7efd9ffff700 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562fbe3c17a0 0x562fbe3c3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.037+0000 7efd8dffb700 10 client.16581 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.037+0000 7efd9ffff700 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562fbe3c17a0 0x562fbe3c3b90 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=693451912ede7b12 server_cookie=38fed78de1f8b6cf in_seq=0 out_seq=0 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.037+0000 7efd8dffb700 1 -- 172.21.15.121:0/2825523814 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x562fbe306d10 con 0x562fbe3c17a0 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.037+0000 7efd8dffb700 10 client.16581 handle_command_reply: tid=0 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.037+0000 7efdb729fb80 1 client.16581 shutdown 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efdb729fb80 1 -- 172.21.15.121:0/2825523814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3a7f20 msgr2=0x562fbe3b3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3a7f20 0x562fbe3b3d80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7efd98005cf0 tx=0x7efd98010460 comp rx=0 tx=0).stop 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efdb729fb80 1 -- 172.21.15.121:0/2825523814 shutdown_connections 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562fbe3c17a0 0x562fbe3c3b90 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3adf60 0x562fbe3b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3a7f20 0x562fbe3b3d80 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2825523814 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efdb729fb80 1 -- 172.21.15.121:0/2825523814 >> 172.21.15.121:0/2825523814 conn(0x562fbe2d2cc0 msgr2=0x562fbe3a25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:40.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efd8dffb700 0 client.16581 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:40.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efdb729fb80 1 -- 172.21.15.121:0/2825523814 shutdown_connections 2024-06-10T00:35:40.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efdb729fb80 1 -- 172.21.15.121:0/2825523814 wait complete. 2024-06-10T00:35:40.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.038+0000 7efdb729fb80 20 client.16581 trim_cache size 0 max 16384 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.058+0000 7efdb729fb80 1 Processor -- start 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efdb729fb80 1 -- start start 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3a79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3aea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efdb729fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efda81b10f0 con 0x562fbe3adf60 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efdb729fb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7efda81ae8c0 con 0x562fbe3ad640 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efdb729fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efda81ac730 con 0x562fbe3a7f20 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efda6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efda659c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3a79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efda6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42992/0 (socket says 172.21.15.121:42992) 2024-06-10T00:35:40.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efda6d9d700 1 -- 172.21.15.121:0/673401924 learned_addr learned my addr 172.21.15.121:0/673401924 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efd9ffff700 1 --2- 172.21.15.121:0/673401924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efda6d9d700 1 -- 172.21.15.121:0/673401924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 msgr2=0x562fbe3a79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efda6d9d700 1 --2- 172.21.15.121:0/673401924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3a79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efda6d9d700 1 -- 172.21.15.121:0/673401924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 msgr2=0x562fbe3aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efda6d9d700 1 --2- 172.21.15.121:0/673401924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3aea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efda6d9d700 1 -- 172.21.15.121:0/673401924 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fbe3af5f0 con 0x562fbe3adf60 2024-06-10T00:35:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efda659c700 1 --2- 172.21.15.121:0/673401924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3a79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.059+0000 7efda6d9d700 1 --2- 172.21.15.121:0/673401924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c6bb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7efda0015000 tx=0x7efda0009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a85b2f1153976b5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efd8ffff700 1 -- 172.21.15.121:0/673401924 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda002b070 con 0x562fbe3adf60 2024-06-10T00:35:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efd8ffff700 1 -- 172.21.15.121:0/673401924 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efda000bb30 con 0x562fbe3adf60 2024-06-10T00:35:40.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efd8ffff700 1 -- 172.21.15.121:0/673401924 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda0007d40 con 0x562fbe3adf60 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efdb729fb80 1 -- 172.21.15.121:0/673401924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 msgr2=0x562fbe3c6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efdb729fb80 1 --2- 172.21.15.121:0/673401924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c6bb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7efda0015000 tx=0x7efda0009520 comp rx=0 tx=0).stop 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efdb729fb80 1 -- 172.21.15.121:0/673401924 shutdown_connections 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efdb729fb80 1 --2- 172.21.15.121:0/673401924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c6bb0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efdb729fb80 1 --2- 172.21.15.121:0/673401924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3a79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efdb729fb80 1 --2- 172.21.15.121:0/673401924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efdb729fb80 1 -- 172.21.15.121:0/673401924 >> 172.21.15.121:0/673401924 conn(0x562fbe2d2cc0 msgr2=0x562fbe3ae360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efdb729fb80 1 -- 172.21.15.121:0/673401924 shutdown_connections 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.060+0000 7efdb729fb80 1 -- 172.21.15.121:0/673401924 wait complete. 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efdb729fb80 1 -- start start 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efdb729fb80 1 Processor -- start 2024-06-10T00:35:40.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efdb729fb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efdb729fb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efda81acd80 con 0x562fbe3adf60 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efdb729fb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7efda81ae1f0 con 0x562fbe3ad640 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efdb729fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efda81ad450 con 0x562fbe3a7f20 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efdb729fb80 10 client.0 _ll_register_callbacks cb 0x562fbe293920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efda6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efda659c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efda6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43006/0 (socket says 172.21.15.121:43006) 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efda6d9d700 1 -- 172.21.15.121:0/2040812508 learned_addr learned my addr 172.21.15.121:0/2040812508 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efd9ffff700 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efda6d9d700 1 -- 172.21.15.121:0/2040812508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 msgr2=0x562fbe3c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:40.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efda6d9d700 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3c4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efda6d9d700 1 -- 172.21.15.121:0/2040812508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 msgr2=0x562fbe3c4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efda6d9d700 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3c4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efda6d9d700 1 -- 172.21.15.121:0/2040812508 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fbe42b190 con 0x562fbe3adf60 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.061+0000 7efda659c700 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efda6d9d700 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c54a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7efda00078c0 tx=0x7efda0007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248efa953b8f56de server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efd8dffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efd8dffb700 1 -- 172.21.15.121:0/2040812508 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda002b050 con 0x562fbe3adf60 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efdb729fb80 1 -- 172.21.15.121:0/2040812508 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efda000bb30 con 0x562fbe3adf60 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efd8dffb700 1 -- 172.21.15.121:0/2040812508 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7efda0005250 con 0x562fbe3adf60 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efd8dffb700 1 -- 172.21.15.121:0/2040812508 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7efda000b700 con 0x562fbe3adf60 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efdb729fb80 10 client.16587 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efdb729fb80 1 -- 172.21.15.121:0/2040812508 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562fbe42c480 con 0x562fbe3adf60 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efd8dffb700 1 -- 172.21.15.121:0/2040812508 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efda000bb30 con 0x562fbe3adf60 2024-06-10T00:35:40.067 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efd8dffb700 1 -- 172.21.15.121:0/2040812508 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7efda000bce0 con 0x562fbe3adf60 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efdb729fb80 10 client.16587 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efdb729fb80 10 client.16587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efdb729fb80 20 client.16587 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562fbe42cad0 0x562fbe42eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efdb729fb80 4 client.16587 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.062+0000 7efdb729fb80 1 -- 172.21.15.121:0/2040812508 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562fbe3abbd0 con 0x562fbe42cad0 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efda659c700 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562fbe42cad0 0x562fbe42eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efda659c700 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562fbe42cad0 0x562fbe42eef0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=597ebd16b966d00c server_cookie=6b22b72ebbc77514 in_seq=0 out_seq=0 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efd8dffb700 10 client.16587 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efd8dffb700 1 -- 172.21.15.121:0/2040812508 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562fbe3abbd0 con 0x562fbe42cad0 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efd8dffb700 10 client.16587 handle_command_reply: tid=0 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efdb729fb80 1 client.16587 shutdown 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efdb729fb80 1 -- 172.21.15.121:0/2040812508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 msgr2=0x562fbe3c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:40.068 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c54a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7efda00078c0 tx=0x7efda0007c70 comp rx=0 tx=0).stop 2024-06-10T00:35:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efdb729fb80 1 -- 172.21.15.121:0/2040812508 shutdown_connections 2024-06-10T00:35:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562fbe42cad0 0x562fbe42eef0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562fbe3adf60 0x562fbe3c54a0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fbe3a7f20 0x562fbe3c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efdb729fb80 1 --2- 172.21.15.121:0/2040812508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562fbe3ad640 0x562fbe3c4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efdb729fb80 1 -- 172.21.15.121:0/2040812508 >> 172.21.15.121:0/2040812508 conn(0x562fbe2d2cc0 msgr2=0x562fbe3aebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.063+0000 7efd8dffb700 0 client.16587 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdb729fb80 1 -- 172.21.15.121:0/2040812508 shutdown_connections 2024-06-10T00:35:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdb729fb80 1 -- 172.21.15.121:0/2040812508 wait complete. 2024-06-10T00:35:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdb729fb80 20 client.16587 trim_cache size 0 max 16384 2024-06-10T00:35:40.069 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:35:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "no active scrubs running", 2024-06-10T00:35:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": {} 2024-06-10T00:35:40.070 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:35:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 -- 172.21.15.121:0/2598850520 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7efd940a2d90 msgr2=0x7efd940a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7efd940a2d90 0x7efd940a5250 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efda80788d0 tx=0x7efd90009260 comp rx=0 tx=0).stop 2024-06-10T00:35:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 -- 172.21.15.121:0/2598850520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efda8101390 msgr2=0x7efda81999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efda8101390 0x7efda81999d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efd9800d950 tx=0x7efd9800dc60 comp rx=0 tx=0).stop 2024-06-10T00:35:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 -- 172.21.15.121:0/2598850520 shutdown_connections 2024-06-10T00:35:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7efd940a2d90 0x7efd940a5250 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efda8107400 0x7efda81a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efda8101390 0x7efda81999d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 --2- 172.21.15.121:0/2598850520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda8101d50 0x7efda8199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 -- 172.21.15.121:0/2598850520 >> 172.21.15.121:0/2598850520 conn(0x7efda8078340 msgr2=0x7efda80ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 -- 172.21.15.121:0/2598850520 shutdown_connections 2024-06-10T00:35:40.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:40.064+0000 7efdaca1c700 1 -- 172.21.15.121:0/2598850520 wait complete. 2024-06-10T00:35:40.853 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-10T00:35:40.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:35:41.118 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:35:41.513 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:41 smithi121 ceph-mon[32378]: pgmap v236: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-06-10T00:35:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:41 smithi193 ceph-mon[36716]: pgmap v236: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-06-10T00:35:41.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:41 smithi135 ceph-mon[36424]: pgmap v236: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-06-10T00:35:43.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.003+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/4071908024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a38107570 msgr2=0x7f5a38107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:43.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.003+0000 7f5a3d3ed700 1 --2- 172.21.15.121:0/4071908024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a38107570 0x7f5a38107950 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009990 tx=0x7f5a2c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:35:43.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.003+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/4071908024 shutdown_connections 2024-06-10T00:35:43.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.003+0000 7f5a3d3ed700 1 --2- 172.21.15.121:0/4071908024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a38101810 0x7f5a3810f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:43.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.003+0000 7f5a3d3ed700 1 --2- 172.21.15.121:0/4071908024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a38107570 0x7f5a38107950 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:43.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.003+0000 7f5a3d3ed700 1 --2- 172.21.15.121:0/4071908024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a38100e50 0x7f5a381012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:43.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.003+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/4071908024 >> 172.21.15.121:0/4071908024 conn(0x7f5a380766b0 msgr2=0x7f5a38076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:43.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.003+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/4071908024 shutdown_connections 2024-06-10T00:35:43.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.003+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/4071908024 wait complete. 2024-06-10T00:35:43.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.004+0000 7f5a3d3ed700 1 Processor -- start 2024-06-10T00:35:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.004+0000 7f5a3d3ed700 1 -- start start 2024-06-10T00:35:43.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a3d3ed700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a38100e50 0x7f5a3819e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a3d3ed700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a38101810 0x7f5a3819ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a3d3ed700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a38107570 0x7f5a38198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a3d3ed700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5a381127d0 con 0x7f5a38101810 2024-06-10T00:35:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a3d3ed700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f5a38112650 con 0x7f5a38100e50 2024-06-10T00:35:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a3d3ed700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5a38112950 con 0x7f5a38107570 2024-06-10T00:35:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a36ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a38100e50 0x7f5a3819e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:43.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a36ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a38100e50 0x7f5a3819e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:45772/0 (socket says 172.21.15.121:45772) 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a36ffd700 1 -- 172.21.15.121:0/3046525637 learned_addr learned my addr 172.21.15.121:0/3046525637 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a367fc700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a38101810 0x7f5a3819ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a377fe700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a38107570 0x7f5a38198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a36ffd700 1 -- 172.21.15.121:0/3046525637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a38107570 msgr2=0x7f5a38198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a36ffd700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a38107570 0x7f5a38198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a36ffd700 1 -- 172.21.15.121:0/3046525637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a38101810 msgr2=0x7f5a3819ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a36ffd700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a38101810 0x7f5a3819ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a36ffd700 1 -- 172.21.15.121:0/3046525637 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a381991b0 con 0x7f5a38100e50 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.005+0000 7f5a36ffd700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a38100e50 0x7f5a3819e840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c000c00 tx=0x7f5a2c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.006+0000 7f5a1ffff700 1 -- 172.21.15.121:0/3046525637 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c027070 con 0x7f5a38100e50 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.006+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/3046525637 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a38199440 con 0x7f5a38100e50 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.006+0000 7f5a1ffff700 1 -- 172.21.15.121:0/3046525637 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a2c003d70 con 0x7f5a38100e50 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.006+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/3046525637 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a381a5710 con 0x7f5a38100e50 2024-06-10T00:35:43.008 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.006+0000 7f5a1ffff700 1 -- 172.21.15.121:0/3046525637 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00f460 con 0x7f5a38100e50 2024-06-10T00:35:43.009 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.006+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/3046525637 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a38069180 con 0x7f5a38100e50 2024-06-10T00:35:43.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.008+0000 7f5a1ffff700 1 -- 172.21.15.121:0/3046525637 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f5a2c02a070 con 0x7f5a38100e50 2024-06-10T00:35:43.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.009+0000 7f5a1ffff700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5a200a2d90 0x7f5a200a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:43.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.009+0000 7f5a367fc700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5a200a2d90 0x7f5a200a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:43.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.009+0000 7f5a1ffff700 1 -- 172.21.15.121:0/3046525637 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7f5a2c0d0ab0 con 0x7f5a38100e50 2024-06-10T00:35:43.010 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.009+0000 7f5a367fc700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5a200a2d90 0x7f5a200a5250 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5a38076c20 tx=0x7f5a28009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:43.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.017+0000 7f5a1ffff700 1 -- 172.21.15.121:0/3046525637 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5a2c013070 con 0x7f5a38100e50 2024-06-10T00:35:43.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.276+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/3046525637 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a381a58e0 con 0x7f5a38100e50 2024-06-10T00:35:43.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.277+0000 7f5a1ffff700 1 -- 172.21.15.121:0/3046525637 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11894 (secure 0 0 0) 0x7f5a2c09d310 con 0x7f5a38100e50 2024-06-10T00:35:43.277 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:35:43.277 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:35:02.741011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":25,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24472,24479]},"id":1}]} 2024-06-10T00:35:43.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/3046525637 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5a200a2d90 msgr2=0x7f5a200a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:43.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5a200a2d90 0x7f5a200a5250 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5a38076c20 tx=0x7f5a28009260 comp rx=0 tx=0).stop 2024-06-10T00:35:43.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/3046525637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a38100e50 msgr2=0x7f5a3819e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:43.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a38100e50 0x7f5a3819e840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c000c00 tx=0x7f5a2c00bab0 comp rx=0 tx=0).stop 2024-06-10T00:35:43.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/3046525637 shutdown_connections 2024-06-10T00:35:43.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f5a200a2d90 0x7f5a200a5250 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:43.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a38107570 0x7f5a38198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:43.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5a38100e50 0x7f5a3819e840 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:43.280 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 --2- 172.21.15.121:0/3046525637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a38101810 0x7f5a3819ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:43.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/3046525637 >> 172.21.15.121:0/3046525637 conn(0x7f5a380766b0 msgr2=0x7f5a380ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:43.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/3046525637 shutdown_connections 2024-06-10T00:35:43.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:43.280+0000 7f5a3d3ed700 1 -- 172.21.15.121:0/3046525637 wait complete. 2024-06-10T00:35:43.281 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 32 2024-06-10T00:35:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:43 smithi193 ceph-mon[36716]: pgmap v237: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-06-10T00:35:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:43 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3046525637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:35:43.701 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:43 smithi121 ceph-mon[32378]: pgmap v237: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-06-10T00:35:43.702 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:43 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3046525637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:35:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:43 smithi135 ceph-mon[36424]: pgmap v237: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 189 op/s 2024-06-10T00:35:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:43 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3046525637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:35:44.216 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-10T00:35:44.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:44 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:35:44.535 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:35:44.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:44 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:35:44.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:44 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:35:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:45 smithi193 ceph-mon[36716]: pgmap v238: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-06-10T00:35:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:45 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:35:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:45 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:35:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:45 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:35:45.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:45 smithi121 ceph-mon[32378]: pgmap v238: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-06-10T00:35:45.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:45 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:35:45.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:45 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:35:45.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:45 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:35:45.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:45 smithi135 ceph-mon[36424]: pgmap v238: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-06-10T00:35:45.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:45 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:35:45.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:45 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:35:45.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:45 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:35:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.409+0000 7fea3355b700 1 -- 172.21.15.121:0/2905182563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea2c107860 msgr2=0x7fea2c10f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.409+0000 7fea3355b700 1 --2- 172.21.15.121:0/2905182563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea2c107860 0x7fea2c10f3e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fea2400b0d0 tx=0x7fea2400b3e0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.409+0000 7fea3355b700 1 -- 172.21.15.121:0/2905182563 shutdown_connections 2024-06-10T00:35:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.409+0000 7fea3355b700 1 --2- 172.21.15.121:0/2905182563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea2c107860 0x7fea2c10f3e0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.409+0000 7fea3355b700 1 --2- 172.21.15.121:0/2905182563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea2c102770 0x7fea2c102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.409+0000 7fea3355b700 1 --2- 172.21.15.121:0/2905182563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea2c103120 0x7fea2c107130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.409+0000 7fea3355b700 1 -- 172.21.15.121:0/2905182563 >> 172.21.15.121:0/2905182563 conn(0x7fea2c079600 msgr2=0x7fea2c079a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.409+0000 7fea3355b700 1 -- 172.21.15.121:0/2905182563 shutdown_connections 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.409+0000 7fea3355b700 1 -- 172.21.15.121:0/2905182563 wait complete. 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea3355b700 1 Processor -- start 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea3355b700 1 -- start start 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea3355b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea2c102770 0x7fea2c074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea3355b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea2c103120 0x7fea2c074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea3355b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea2c107860 0x7fea2c075210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea3355b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c1127a0 con 0x7fea2c107860 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea3355b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c112620 con 0x7fea2c103120 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea3355b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c112920 con 0x7fea2c102770 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea31af8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea2c107860 0x7fea2c075210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea30af6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea2c103120 0x7fea2c074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea30af6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea2c103120 0x7fea2c074cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:45784/0 (socket says 172.21.15.121:45784) 2024-06-10T00:35:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea31af8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea2c107860 0x7fea2c075210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43040/0 (socket says 172.21.15.121:43040) 2024-06-10T00:35:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea312f7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea2c102770 0x7fea2c074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea30af6700 1 -- 172.21.15.121:0/3138165663 learned_addr learned my addr 172.21.15.121:0/3138165663 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:46.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea30af6700 1 -- 172.21.15.121:0/3138165663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea2c102770 msgr2=0x7fea2c074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea30af6700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea2c102770 0x7fea2c074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea30af6700 1 -- 172.21.15.121:0/3138165663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea2c107860 msgr2=0x7fea2c075210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea30af6700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea2c107860 0x7fea2c075210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea30af6700 1 -- 172.21.15.121:0/3138165663 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2c1a70e0 con 0x7fea2c103120 2024-06-10T00:35:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.410+0000 7fea31af8700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea2c107860 0x7fea2c075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.411+0000 7fea30af6700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea2c103120 0x7fea2c074cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00e9c0 tx=0x7fea1c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.411+0000 7fea227fc700 1 -- 172.21.15.121:0/3138165663 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c00cbd0 con 0x7fea2c103120 2024-06-10T00:35:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.411+0000 7fea3355b700 1 -- 172.21.15.121:0/3138165663 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2c1a73d0 con 0x7fea2c103120 2024-06-10T00:35:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.411+0000 7fea227fc700 1 -- 172.21.15.121:0/3138165663 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea1c00ee60 con 0x7fea2c103120 2024-06-10T00:35:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.411+0000 7fea227fc700 1 -- 172.21.15.121:0/3138165663 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c005630 con 0x7fea2c103120 2024-06-10T00:35:46.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.411+0000 7fea3355b700 1 -- 172.21.15.121:0/3138165663 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea2c1a7910 con 0x7fea2c103120 2024-06-10T00:35:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.413+0000 7fea227fc700 1 -- 172.21.15.121:0/3138165663 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fea1c004750 con 0x7fea2c103120 2024-06-10T00:35:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.414+0000 7fea227fc700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea180a2d90 0x7fea180a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.414+0000 7fea312f7700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea180a2d90 0x7fea180a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.414+0000 7fea227fc700 1 -- 172.21.15.121:0/3138165663 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fea1c0cfa90 con 0x7fea2c103120 2024-06-10T00:35:46.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.414+0000 7fea312f7700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea180a2d90 0x7fea180a5250 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fea28009c60 tx=0x7fea28009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea3dddeb80 1 Processor -- start 2024-06-10T00:35:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea3dddeb80 1 -- start start 2024-06-10T00:35:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5b640 0x5577bcb5ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 0x5577bcb559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 0x5577bcb56340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5577bcaa7810 con 0x5577bcb5b640 2024-06-10T00:35:46.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5577bca4f720 con 0x5577bcb5bf60 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5577bcaa37d0 con 0x5577bcb55f20 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea312f7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5b640 0x5577bcb5ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea312f7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5b640 0x5577bcb5ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43046/0 (socket says 172.21.15.121:43046) 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea312f7700 1 -- 172.21.15.121:0/2106129691 learned_addr learned my addr 172.21.15.121:0/2106129691 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea30af6700 1 --2- 172.21.15.121:0/2106129691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 0x5577bcb559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea312f7700 1 -- 172.21.15.121:0/2106129691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 msgr2=0x5577bcb56340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea31af8700 1 --2- 172.21.15.121:0/2106129691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 0x5577bcb56340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea312f7700 1 --2- 172.21.15.121:0/2106129691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 0x5577bcb56340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea312f7700 1 -- 172.21.15.121:0/2106129691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 msgr2=0x5577bcb559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea312f7700 1 --2- 172.21.15.121:0/2106129691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 0x5577bcb559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea312f7700 1 -- 172.21.15.121:0/2106129691 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577bca45f50 con 0x5577bcb5b640 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea30af6700 1 --2- 172.21.15.121:0/2106129691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 0x5577bcb559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:35:46.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.415+0000 7fea312f7700 1 --2- 172.21.15.121:0/2106129691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5b640 0x5577bcb5ba20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fea2800e4e0 tx=0x7fea28014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a852a69af57ba8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.416+0000 7fea177fe700 1 -- 172.21.15.121:0/2106129691 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28026430 con 0x5577bcb5b640 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.416+0000 7fea177fe700 1 -- 172.21.15.121:0/2106129691 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea2801c8f0 con 0x5577bcb5b640 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.416+0000 7fea177fe700 1 -- 172.21.15.121:0/2106129691 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2801d970 con 0x5577bcb5b640 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.416+0000 7fea3dddeb80 1 -- 172.21.15.121:0/2106129691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5b640 msgr2=0x5577bcb5ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.416+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/2106129691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5b640 0x5577bcb5ba20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fea2800e4e0 tx=0x7fea28014000 comp rx=0 tx=0).stop 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.416+0000 7fea3dddeb80 1 -- 172.21.15.121:0/2106129691 shutdown_connections 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.416+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/2106129691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 0x5577bcb56340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.416+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/2106129691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5b640 0x5577bcb5ba20 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.416+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/2106129691 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 0x5577bcb559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.416+0000 7fea3dddeb80 1 -- 172.21.15.121:0/2106129691 >> 172.21.15.121:0/2106129691 conn(0x5577bca80cc0 msgr2=0x5577bcb50850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.417+0000 7fea3dddeb80 1 -- 172.21.15.121:0/2106129691 shutdown_connections 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.417+0000 7fea3dddeb80 1 -- 172.21.15.121:0/2106129691 wait complete. 2024-06-10T00:35:46.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.417+0000 7fea3dddeb80 1 -- start start 2024-06-10T00:35:46.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.417+0000 7fea3dddeb80 1 Processor -- start 2024-06-10T00:35:46.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.417+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb55f20 0x5577bcb61ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.417+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 0x5577bcb5efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.417+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 0x5577bcb5f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.417+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5577bcb629d0 con 0x5577bcb55f20 2024-06-10T00:35:46.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.417+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5577bca47bf0 con 0x5577bcb5bf60 2024-06-10T00:35:46.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.417+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5577bcb62b50 con 0x5577bcb5b640 2024-06-10T00:35:46.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.417+0000 7fea3dddeb80 10 client.0 _ll_register_callbacks cb 0x5577bca41920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:46.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea312f7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb55f20 0x5577bcb61ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea312f7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb55f20 0x5577bcb61ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43048/0 (socket says 172.21.15.121:43048) 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea31af8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 0x5577bcb5f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea312f7700 1 -- 172.21.15.121:0/470189066 learned_addr learned my addr 172.21.15.121:0/470189066 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea30af6700 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 0x5577bcb5efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea312f7700 1 -- 172.21.15.121:0/470189066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 msgr2=0x5577bcb5efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea312f7700 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 0x5577bcb5efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea312f7700 1 -- 172.21.15.121:0/470189066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 msgr2=0x5577bcb5f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea312f7700 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 0x5577bcb5f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea312f7700 1 -- 172.21.15.121:0/470189066 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577bc9f68d0 con 0x5577bcb55f20 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea31af8700 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 0x5577bcb5f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea312f7700 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb55f20 0x5577bcb61ca0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fea2800eda0 tx=0x7fea2800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26e20db4ea8441b server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea14ff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea14ff9700 1 -- 172.21.15.121:0/470189066 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28033060 con 0x5577bcb55f20 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea14ff9700 1 -- 172.21.15.121:0/470189066 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fea2801c620 con 0x5577bcb55f20 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.418+0000 7fea3dddeb80 1 -- 172.21.15.121:0/470189066 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea2801c8f0 con 0x5577bcb55f20 2024-06-10T00:35:46.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea14ff9700 1 -- 172.21.15.121:0/470189066 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fea2803a050 con 0x5577bcb55f20 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea3dddeb80 10 client.16605 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea3dddeb80 1 -- 172.21.15.121:0/470189066 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5577bcb6d2b0 con 0x5577bcb55f20 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea14ff9700 1 -- 172.21.15.121:0/470189066 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea2801c8f0 con 0x5577bcb55f20 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea14ff9700 1 -- 172.21.15.121:0/470189066 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fea280036f0 con 0x5577bcb55f20 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea3dddeb80 10 client.16605 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea3dddeb80 10 client.16605 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea3dddeb80 20 client.16605 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5577bcb6f830 0x5577bcb71c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea3dddeb80 4 client.16605 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea3dddeb80 1 -- 172.21.15.121:0/470189066 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577bcab4d10 con 0x5577bcb6f830 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.419+0000 7fea30af6700 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5577bcb6f830 0x5577bcb71c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.420+0000 7fea14ff9700 10 client.16605 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.420+0000 7fea30af6700 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5577bcb6f830 0x5577bcb71c20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bad386743ef54801 server_cookie=22c8fa4ee62f2d2b in_seq=0 out_seq=0 2024-06-10T00:35:46.426 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea14ff9700 1 -- 172.21.15.121:0/470189066 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5577bcab4d10 con 0x5577bcb6f830 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea14ff9700 10 client.16605 handle_command_reply: tid=0 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 1 client.16605 shutdown 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 1 -- 172.21.15.121:0/470189066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb55f20 msgr2=0x5577bcb61ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb55f20 0x5577bcb61ca0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fea2800eda0 tx=0x7fea2800edd0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 1 -- 172.21.15.121:0/470189066 shutdown_connections 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5577bcb6f830 0x5577bcb71c20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5bf60 0x5577bcb5f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb55f20 0x5577bcb61ca0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/470189066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 0x5577bcb5efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 1 -- 172.21.15.121:0/470189066 >> 172.21.15.121:0/470189066 conn(0x5577bca80cc0 msgr2=0x5577bcb549d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea14ff9700 0 client.16605 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 1 -- 172.21.15.121:0/470189066 shutdown_connections 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 1 -- 172.21.15.121:0/470189066 wait complete. 2024-06-10T00:35:46.427 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.421+0000 7fea3dddeb80 20 client.16605 trim_cache size 0 max 16384 2024-06-10T00:35:46.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.437+0000 7fea3dddeb80 1 Processor -- start 2024-06-10T00:35:46.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea3dddeb80 1 -- start start 2024-06-10T00:35:46.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 0x5577bcb559e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5b640 0x5577bcb5cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 0x5577bcb59730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c1b32e0 con 0x5577bcb5bf60 2024-06-10T00:35:46.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c1b0ab0 con 0x5577bcb5b640 2024-06-10T00:35:46.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c1ae920 con 0x5577bcb55f20 2024-06-10T00:35:46.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea312f7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 0x5577bcb559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea312f7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 0x5577bcb559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:56006/0 (socket says 172.21.15.121:56006) 2024-06-10T00:35:46.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea312f7700 1 -- 172.21.15.121:0/2424589005 learned_addr learned my addr 172.21.15.121:0/2424589005 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:46.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea31af8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 0x5577bcb59730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea30af6700 1 --2- 172.21.15.121:0/2424589005 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5b640 0x5577bcb5cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea312f7700 1 -- 172.21.15.121:0/2424589005 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5b640 msgr2=0x5577bcb5cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:46.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.438+0000 7fea312f7700 1 --2- 172.21.15.121:0/2424589005 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5b640 0x5577bcb5cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.439+0000 7fea312f7700 1 -- 172.21.15.121:0/2424589005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 msgr2=0x5577bcb59730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:46.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.439+0000 7fea312f7700 1 --2- 172.21.15.121:0/2424589005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 0x5577bcb59730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.439+0000 7fea312f7700 1 -- 172.21.15.121:0/2424589005 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577bcb5d510 con 0x5577bcb55f20 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.439+0000 7fea30af6700 1 --2- 172.21.15.121:0/2424589005 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5b640 0x5577bcb5cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.439+0000 7fea31af8700 1 --2- 172.21.15.121:0/2424589005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 0x5577bcb59730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.439+0000 7fea312f7700 1 --2- 172.21.15.121:0/2424589005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 0x5577bcb559e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fea28009b20 tx=0x7fea28005b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ad25e302703d1c0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.439+0000 7fea16ffd700 1 -- 172.21.15.121:0/2424589005 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28033040 con 0x5577bcb55f20 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.439+0000 7fea16ffd700 1 -- 172.21.15.121:0/2424589005 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea280055a0 con 0x5577bcb55f20 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.439+0000 7fea16ffd700 1 -- 172.21.15.121:0/2424589005 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2801ca80 con 0x5577bcb55f20 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.440+0000 7fea3dddeb80 1 -- 172.21.15.121:0/2424589005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 msgr2=0x5577bcb559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.440+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/2424589005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 0x5577bcb559e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fea28009b20 tx=0x7fea28005b40 comp rx=0 tx=0).stop 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.440+0000 7fea3dddeb80 1 -- 172.21.15.121:0/2424589005 shutdown_connections 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.440+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/2424589005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 0x5577bcb59730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.440+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/2424589005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb55f20 0x5577bcb559e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.440+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/2424589005 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb5b640 0x5577bcb5cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.440+0000 7fea3dddeb80 1 -- 172.21.15.121:0/2424589005 >> 172.21.15.121:0/2424589005 conn(0x5577bca80cc0 msgr2=0x5577bcb49e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.440+0000 7fea3dddeb80 1 -- 172.21.15.121:0/2424589005 shutdown_connections 2024-06-10T00:35:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.440+0000 7fea3dddeb80 1 -- 172.21.15.121:0/2424589005 wait complete. 2024-06-10T00:35:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.440+0000 7fea3dddeb80 1 -- start start 2024-06-10T00:35:46.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.440+0000 7fea3dddeb80 1 Processor -- start 2024-06-10T00:35:46.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb55f20 0x5577bcb750e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 0x5577bcb71470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea3dddeb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 0x5577bcb71a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c1aef70 con 0x5577bcb5bf60 2024-06-10T00:35:46.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c1b03e0 con 0x5577bcb55f20 2024-06-10T00:35:46.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea3dddeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea2c1af640 con 0x5577bcb5b640 2024-06-10T00:35:46.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea312f7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb55f20 0x5577bcb750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea30af6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 0x5577bcb71470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea3dddeb80 10 client.0 _ll_register_callbacks cb 0x5577bca41920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:46.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea30af6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 0x5577bcb71470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:56018/0 (socket says 172.21.15.121:56018) 2024-06-10T00:35:46.445 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea30af6700 1 -- 172.21.15.121:0/1418548887 learned_addr learned my addr 172.21.15.121:0/1418548887 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea312f7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb55f20 0x5577bcb750e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:45836/0 (socket says 172.21.15.121:45836) 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea31af8700 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 0x5577bcb71a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea30af6700 1 -- 172.21.15.121:0/1418548887 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb55f20 msgr2=0x5577bcb750e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea30af6700 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb55f20 0x5577bcb750e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea30af6700 1 -- 172.21.15.121:0/1418548887 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 msgr2=0x5577bcb71a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea30af6700 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 0x5577bcb71a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea30af6700 1 -- 172.21.15.121:0/1418548887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577bcbd8160 con 0x5577bcb5b640 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea312f7700 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb55f20 0x5577bcb750e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.441+0000 7fea31af8700 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 0x5577bcb71a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea30af6700 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 0x5577bcb71470 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00f010 tx=0x7fea1c097eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e52786fd77847af8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea14ff9700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea14ff9700 1 -- 172.21.15.121:0/1418548887 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1c00cd70 con 0x5577bcb5b640 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea3dddeb80 1 -- 172.21.15.121:0/1418548887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea280055a0 con 0x5577bcb5b640 2024-06-10T00:35:46.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea14ff9700 1 -- 172.21.15.121:0/1418548887 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fea1c0cd010 con 0x5577bcb5b640 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea14ff9700 1 -- 172.21.15.121:0/1418548887 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fea1c081050 con 0x5577bcb5b640 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea3dddeb80 10 client.26074 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea3dddeb80 1 -- 172.21.15.121:0/1418548887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5577bcbd9300 con 0x5577bcb5b640 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea14ff9700 1 -- 172.21.15.121:0/1418548887 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea280055a0 con 0x5577bcb5b640 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea14ff9700 1 -- 172.21.15.121:0/1418548887 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fea1c0a4440 con 0x5577bcb5b640 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea3dddeb80 10 client.26074 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea3dddeb80 10 client.26074 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea3dddeb80 20 client.26074 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5577bcbd9990 0x5577bcbdbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea3dddeb80 4 client.26074 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.442+0000 7fea3dddeb80 1 -- 172.21.15.121:0/1418548887 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5577bcaaf290 con 0x5577bcbd9990 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.443+0000 7fea312f7700 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5577bcbd9990 0x5577bcbdbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:46.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.443+0000 7fea312f7700 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5577bcbd9990 0x5577bcbdbdb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ca79e3a4e6d7485 server_cookie=1da6bd6d65222a38 in_seq=0 out_seq=0 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.443+0000 7fea14ff9700 10 client.26074 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.443+0000 7fea14ff9700 1 -- 172.21.15.121:0/1418548887 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5577bcaaf290 con 0x5577bcbd9990 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.443+0000 7fea14ff9700 10 client.26074 handle_command_reply: tid=0 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.443+0000 7fea3dddeb80 1 client.26074 shutdown 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.443+0000 7fea3dddeb80 1 -- 172.21.15.121:0/1418548887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 msgr2=0x5577bcb71470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.443+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 0x5577bcb71470 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00f010 tx=0x7fea1c097eb0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.443+0000 7fea3dddeb80 1 -- 172.21.15.121:0/1418548887 shutdown_connections 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.443+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5577bcbd9990 0x5577bcbdbdb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.443+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5577bcb5bf60 0x5577bcb71a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5577bcb55f20 0x5577bcb750e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3dddeb80 1 --2- 172.21.15.121:0/1418548887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5577bcb5b640 0x5577bcb71470 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3dddeb80 1 -- 172.21.15.121:0/1418548887 >> 172.21.15.121:0/1418548887 conn(0x5577bca80cc0 msgr2=0x5577bcb607d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea14ff9700 0 client.26074 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:35:46.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3dddeb80 1 -- 172.21.15.121:0/1418548887 shutdown_connections 2024-06-10T00:35:46.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3dddeb80 1 -- 172.21.15.121:0/1418548887 wait complete. 2024-06-10T00:35:46.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3dddeb80 20 client.26074 trim_cache size 0 max 16384 2024-06-10T00:35:46.449 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:35:46.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3355b700 1 -- 172.21.15.121:0/3138165663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea180a2d90 msgr2=0x7fea180a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3355b700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea180a2d90 0x7fea180a5250 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fea28009c60 tx=0x7fea28009260 comp rx=0 tx=0).stop 2024-06-10T00:35:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3355b700 1 -- 172.21.15.121:0/3138165663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea2c103120 msgr2=0x7fea2c074cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3355b700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea2c103120 0x7fea2c074cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea1c00e9c0 tx=0x7fea1c00ecd0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3355b700 1 -- 172.21.15.121:0/3138165663 shutdown_connections 2024-06-10T00:35:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3355b700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fea180a2d90 0x7fea180a5250 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3355b700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea2c107860 0x7fea2c075210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3355b700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea2c102770 0x7fea2c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3355b700 1 --2- 172.21.15.121:0/3138165663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea2c103120 0x7fea2c074cd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.444+0000 7fea3355b700 1 -- 172.21.15.121:0/3138165663 >> 172.21.15.121:0/3138165663 conn(0x7fea2c079600 msgr2=0x7fea2c1004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.445+0000 7fea3355b700 1 -- 172.21.15.121:0/3138165663 shutdown_connections 2024-06-10T00:35:46.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:46.445+0000 7fea3355b700 1 -- 172.21.15.121:0/3138165663 wait complete. 2024-06-10T00:35:47.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-10T00:35:47.541 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:35:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:47 smithi193 ceph-mon[36716]: pgmap v239: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 229 op/s 2024-06-10T00:35:47.663 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:47 smithi121 ceph-mon[32378]: pgmap v239: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 229 op/s 2024-06-10T00:35:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:47 smithi135 ceph-mon[36424]: pgmap v239: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 229 op/s 2024-06-10T00:35:49.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.435+0000 7fd1d455f700 1 -- 172.21.15.121:0/370748155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1cc10eef0 msgr2=0x7fd1cc1112e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:49.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.435+0000 7fd1d455f700 1 --2- 172.21.15.121:0/370748155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1cc10eef0 0x7fd1cc1112e0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400b0a0 tx=0x7fd1c400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.435+0000 7fd1d455f700 1 -- 172.21.15.121:0/370748155 shutdown_connections 2024-06-10T00:35:49.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.435+0000 7fd1d455f700 1 --2- 172.21.15.121:0/370748155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1cc10eef0 0x7fd1cc1112e0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.435+0000 7fd1d455f700 1 --2- 172.21.15.121:0/370748155 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1cc0746c0 0x7fd1cc074aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.435+0000 7fd1d455f700 1 --2- 172.21.15.121:0/370748155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1cc074fe0 0x7fd1cc10e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.435+0000 7fd1d455f700 1 -- 172.21.15.121:0/370748155 >> 172.21.15.121:0/370748155 conn(0x7fd1cc079570 msgr2=0x7fd1cc079980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:49.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.435+0000 7fd1d455f700 1 -- 172.21.15.121:0/370748155 shutdown_connections 2024-06-10T00:35:49.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.436+0000 7fd1d455f700 1 -- 172.21.15.121:0/370748155 wait complete. 2024-06-10T00:35:49.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.436+0000 7fd1d455f700 1 Processor -- start 2024-06-10T00:35:49.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.436+0000 7fd1d455f700 1 -- start start 2024-06-10T00:35:49.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d455f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1cc0746c0 0x7fd1cc19ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d455f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1cc074fe0 0x7fd1cc19b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d455f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1cc10eef0 0x7fd1cc19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d455f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1149c0 con 0x7fd1cc074fe0 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d455f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc114840 con 0x7fd1cc0746c0 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d455f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc114b40 con 0x7fd1cc10eef0 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d1afa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1cc074fe0 0x7fd1cc19b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d22fb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1cc0746c0 0x7fd1cc19ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d22fb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1cc0746c0 0x7fd1cc19ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:49234/0 (socket says 172.21.15.121:49234) 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d1afa700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1cc074fe0 0x7fd1cc19b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56930/0 (socket says 172.21.15.121:56930) 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d22fb700 1 -- 172.21.15.121:0/2912812025 learned_addr learned my addr 172.21.15.121:0/2912812025 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d2afc700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1cc10eef0 0x7fd1cc19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d22fb700 1 -- 172.21.15.121:0/2912812025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1cc10eef0 msgr2=0x7fd1cc19f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:49.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1cc10eef0 0x7fd1cc19f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d22fb700 1 -- 172.21.15.121:0/2912812025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1cc074fe0 msgr2=0x7fd1cc19b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1cc074fe0 0x7fd1cc19b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d22fb700 1 -- 172.21.15.121:0/2912812025 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1cc19fe80 con 0x7fd1cc0746c0 2024-06-10T00:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1cc074fe0 0x7fd1cc19b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1cc0746c0 0x7fd1cc19ae70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00eb30 tx=0x7fd1bc00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1c37fe700 1 -- 172.21.15.121:0/2912812025 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc00cbd0 con 0x7fd1cc0746c0 2024-06-10T00:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1c37fe700 1 -- 172.21.15.121:0/2912812025 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1bc004500 con 0x7fd1cc0746c0 2024-06-10T00:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.437+0000 7fd1d455f700 1 -- 172.21.15.121:0/2912812025 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1cc1a0170 con 0x7fd1cc0746c0 2024-06-10T00:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.438+0000 7fd1c37fe700 1 -- 172.21.15.121:0/2912812025 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc010430 con 0x7fd1cc0746c0 2024-06-10T00:35:49.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.438+0000 7fd1d455f700 1 -- 172.21.15.121:0/2912812025 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1cc1a79f0 con 0x7fd1cc0746c0 2024-06-10T00:35:49.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.440+0000 7fd1c37fe700 1 -- 172.21.15.121:0/2912812025 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd1bc003f00 con 0x7fd1cc0746c0 2024-06-10T00:35:49.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.440+0000 7fd1c37fe700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1b80a2d90 0x7fd1b80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.440+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1b80a2d90 0x7fd1b80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.440+0000 7fd1c37fe700 1 -- 172.21.15.121:0/2912812025 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fd1bc0cfba0 con 0x7fd1cc0746c0 2024-06-10T00:35:49.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.441+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1b80a2d90 0x7fd1b80a5250 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc110ad0 tx=0x7fd1c80073d0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:49.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.441+0000 7fd1dede2b80 1 Processor -- start 2024-06-10T00:35:49.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.441+0000 7fd1dede2b80 1 -- start start 2024-06-10T00:35:49.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.441+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5640 0x564af57e5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.441+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5f60 0x564af57df9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.441+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57e0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.441+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564af5731810 con 0x564af57e5640 2024-06-10T00:35:49.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.441+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x564af56d9720 con 0x564af57dff20 2024-06-10T00:35:49.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.441+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564af572d7d0 con 0x564af57e5f60 2024-06-10T00:35:49.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d22fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5640 0x564af57e5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d22fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5640 0x564af57e5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56946/0 (socket says 172.21.15.121:56946) 2024-06-10T00:35:49.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d2afc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57e0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d22fb700 1 -- 172.21.15.121:0/4232412741 learned_addr learned my addr 172.21.15.121:0/4232412741 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:49.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/4232412741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5f60 0x564af57df9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d2afc700 1 -- 172.21.15.121:0/4232412741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5f60 msgr2=0x564af57df9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:49.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d2afc700 1 --2- 172.21.15.121:0/4232412741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5f60 0x564af57df9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d2afc700 1 -- 172.21.15.121:0/4232412741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5640 msgr2=0x564af57e5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:49.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d2afc700 1 --2- 172.21.15.121:0/4232412741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5640 0x564af57e5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d2afc700 1 -- 172.21.15.121:0/4232412741 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564af56cff50 con 0x564af57dff20 2024-06-10T00:35:49.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/4232412741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5f60 0x564af57df9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:35:49.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1d2afc700 1 --2- 172.21.15.121:0/4232412741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57e0340 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400b070 tx=0x7fd1c4009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f75a46fcb2840e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:49.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1c0ff9700 1 -- 172.21.15.121:0/4232412741 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4020070 con 0x564af57dff20 2024-06-10T00:35:49.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1c0ff9700 1 -- 172.21.15.121:0/4232412741 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c400bcb0 con 0x564af57dff20 2024-06-10T00:35:49.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.442+0000 7fd1c0ff9700 1 -- 172.21.15.121:0/4232412741 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4007cb0 con 0x564af57dff20 2024-06-10T00:35:49.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.443+0000 7fd1dede2b80 1 -- 172.21.15.121:0/4232412741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 msgr2=0x564af57e0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:49.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.443+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/4232412741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57e0340 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd1c400b070 tx=0x7fd1c4009360 comp rx=0 tx=0).stop 2024-06-10T00:35:49.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.443+0000 7fd1dede2b80 1 -- 172.21.15.121:0/4232412741 shutdown_connections 2024-06-10T00:35:49.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.443+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/4232412741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57e0340 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.443+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/4232412741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5640 0x564af57e5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.443+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/4232412741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5f60 0x564af57df9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.443+0000 7fd1dede2b80 1 -- 172.21.15.121:0/4232412741 >> 172.21.15.121:0/4232412741 conn(0x564af570acc0 msgr2=0x564af57da850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:49.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.443+0000 7fd1dede2b80 1 -- 172.21.15.121:0/4232412741 shutdown_connections 2024-06-10T00:35:49.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.443+0000 7fd1dede2b80 1 -- 172.21.15.121:0/4232412741 wait complete. 2024-06-10T00:35:49.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.443+0000 7fd1dede2b80 1 -- start start 2024-06-10T00:35:49.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1dede2b80 1 Processor -- start 2024-06-10T00:35:49.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57ebcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57e8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564af57ec970 con 0x564af57e5f60 2024-06-10T00:35:49.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x564af56d1bf0 con 0x564af57dff20 2024-06-10T00:35:49.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564af57ecaf0 con 0x564af57e5640 2024-06-10T00:35:49.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1dede2b80 10 client.0 _ll_register_callbacks cb 0x564af56cb920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:49.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d2afc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d2afc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56952/0 (socket says 172.21.15.121:56952) 2024-06-10T00:35:49.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d22fb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57ebcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d2afc700 1 -- 172.21.15.121:0/3477051585 learned_addr learned my addr 172.21.15.121:0/3477051585 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:49.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57e8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d2afc700 1 -- 172.21.15.121:0/3477051585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 msgr2=0x564af57e8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:49.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d2afc700 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57e8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d2afc700 1 -- 172.21.15.121:0/3477051585 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 msgr2=0x564af57ebcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:49.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d2afc700 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57ebcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d2afc700 1 -- 172.21.15.121:0/3477051585 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564af56808d0 con 0x564af57e5f60 2024-06-10T00:35:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57e8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:35:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57ebcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1d2afc700 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e9530 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009610 tx=0x7fd1c40076d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298465b54051b334 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.444+0000 7fd1b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:35:49.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/3477051585 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4020040 con 0x564af57e5f60 2024-06-10T00:35:49.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1dede2b80 1 -- 172.21.15.121:0/3477051585 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c400bcb0 con 0x564af57e5f60 2024-06-10T00:35:49.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/3477051585 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fd1c400bab0 con 0x564af57e5f60 2024-06-10T00:35:49.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/3477051585 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd1c4037050 con 0x564af57e5f60 2024-06-10T00:35:49.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1dede2b80 10 client.16623 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:49.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1dede2b80 1 -- 172.21.15.121:0/3477051585 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564af57f72a0 con 0x564af57e5f60 2024-06-10T00:35:49.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/3477051585 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1c400bcb0 con 0x564af57e5f60 2024-06-10T00:35:49.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/3477051585 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd1c4011070 con 0x564af57e5f60 2024-06-10T00:35:49.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1dede2b80 10 client.16623 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:49.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1dede2b80 10 client.16623 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:35:49.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1dede2b80 20 client.16623 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:49.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x564af57f9770 0x564af57fbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1dede2b80 4 client.16623 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:35:49.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.445+0000 7fd1dede2b80 1 -- 172.21.15.121:0/3477051585 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564af573ed10 con 0x564af57f9770 2024-06-10T00:35:49.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.446+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x564af57f9770 0x564af57fbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.446+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x564af57f9770 0x564af57fbb60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1040dfe46552f7e0 server_cookie=9954a9dda5681cff in_seq=0 out_seq=0 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.446+0000 7fd1b5ffb700 10 client.16623 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/3477051585 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x564af573ed10 con 0x564af57f9770 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1b5ffb700 10 client.16623 handle_command_reply: tid=0 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 1 client.16623 shutdown 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 1 -- 172.21.15.121:0/3477051585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 msgr2=0x564af57e9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e9530 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009610 tx=0x7fd1c40076d0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 1 -- 172.21.15.121:0/3477051585 shutdown_connections 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x564af57f9770 0x564af57fbb60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e9530 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57ebcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/3477051585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57e8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 1 -- 172.21.15.121:0/3477051585 >> 172.21.15.121:0/3477051585 conn(0x564af570acc0 msgr2=0x564af57de9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:49.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1b5ffb700 0 client.16623 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:35:49.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 1 -- 172.21.15.121:0/3477051585 shutdown_connections 2024-06-10T00:35:49.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 1 -- 172.21.15.121:0/3477051585 wait complete. 2024-06-10T00:35:49.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.447+0000 7fd1dede2b80 20 client.16623 trim_cache size 0 max 16384 2024-06-10T00:35:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1dede2b80 1 Processor -- start 2024-06-10T00:35:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1dede2b80 1 -- start start 2024-06-10T00:35:49.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57df9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57e8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1b3190 con 0x564af57e5f60 2024-06-10T00:35:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1b0960 con 0x564af57dff20 2024-06-10T00:35:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1ae7d0 con 0x564af57e5640 2024-06-10T00:35:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1d2afc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1d22fb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57df9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1d22fb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57df9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:49262/0 (socket says 172.21.15.121:49262) 2024-06-10T00:35:49.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1d2afc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56968/0 (socket says 172.21.15.121:56968) 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.462+0000 7fd1d22fb700 1 -- 172.21.15.121:0/1464052516 learned_addr learned my addr 172.21.15.121:0/1464052516 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/1464052516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57e8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1d22fb700 1 -- 172.21.15.121:0/1464052516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 msgr2=0x564af57e8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/1464052516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57e8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1d22fb700 1 -- 172.21.15.121:0/1464052516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 msgr2=0x564af57e94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/1464052516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1d22fb700 1 -- 172.21.15.121:0/1464052516 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564af57e9a30 con 0x564af57dff20 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1d2afc700 1 --2- 172.21.15.121:0/1464052516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/1464052516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57e8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/1464052516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57df9e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009fd0 tx=0x7fd1bc098090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e91eddf22e82bf36 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1b7fff700 1 -- 172.21.15.121:0/1464052516 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc09c3d0 con 0x564af57dff20 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1b7fff700 1 -- 172.21.15.121:0/1464052516 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1bc09bcd0 con 0x564af57dff20 2024-06-10T00:35:49.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.463+0000 7fd1b7fff700 1 -- 172.21.15.121:0/1464052516 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc098340 con 0x564af57dff20 2024-06-10T00:35:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1464052516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 msgr2=0x564af57df9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/1464052516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57df9e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009fd0 tx=0x7fd1bc098090 comp rx=0 tx=0).stop 2024-06-10T00:35:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1464052516 shutdown_connections 2024-06-10T00:35:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/1464052516 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af57e94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/1464052516 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57df9e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/1464052516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57e8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1464052516 >> 172.21.15.121:0/1464052516 conn(0x564af570acc0 msgr2=0x564af57e78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:49.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1464052516 shutdown_connections 2024-06-10T00:35:49.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1464052516 wait complete. 2024-06-10T00:35:49.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 -- start start 2024-06-10T00:35:49.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 Processor -- start 2024-06-10T00:35:49.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af5800180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1aee20 con 0x564af57e5f60 2024-06-10T00:35:49.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1b0290 con 0x564af57dff20 2024-06-10T00:35:49.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc1af4f0 con 0x564af57e5640 2024-06-10T00:35:49.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1dede2b80 10 client.0 _ll_register_callbacks cb 0x564af56cb920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:49.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1d2afc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af5800180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1d2afc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af5800180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56982/0 (socket says 172.21.15.121:56982) 2024-06-10T00:35:49.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1d2afc700 1 -- 172.21.15.121:0/1612696864 learned_addr learned my addr 172.21.15.121:0/1612696864 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:49.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.464+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1d22fb700 1 -- 172.21.15.121:0/1612696864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 msgr2=0x564af57f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:49.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1d22fb700 1 -- 172.21.15.121:0/1612696864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 msgr2=0x564af5800180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:49.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af5800180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1d22fb700 1 -- 172.21.15.121:0/1612696864 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564af58006c0 con 0x564af57dff20 2024-06-10T00:35:49.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1d2afc700 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af5800180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:49.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:35:49.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1d22fb700 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57f1db0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009fd0 tx=0x7fd1bc0978e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1fa68caf605a3f7 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:49.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:35:49.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/1612696864 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc0a4940 con 0x564af57dff20 2024-06-10T00:35:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1612696864 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1bc09bcd0 con 0x564af57dff20 2024-06-10T00:35:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/1612696864 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fd1bc01fdd0 con 0x564af57dff20 2024-06-10T00:35:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/1612696864 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd1bc0a0080 con 0x564af57dff20 2024-06-10T00:35:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1dede2b80 10 client.26186 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1612696864 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564af5863870 con 0x564af57dff20 2024-06-10T00:35:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.465+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/1612696864 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1bc09bcd0 con 0x564af57dff20 2024-06-10T00:35:49.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/1612696864 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd1bc0a45b0 con 0x564af57dff20 2024-06-10T00:35:49.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 10 client.26186 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:49.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 10 client.26186 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:35:49.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 20 client.26186 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:49.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x564af5863ec0 0x564af58662b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:49.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 4 client.26186 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:35:49.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1612696864 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564af5739290 con 0x564af5863ec0 2024-06-10T00:35:49.476 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x564af5863ec0 0x564af58662b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1b5ffb700 10 client.26186 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:35:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1d1afa700 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x564af5863ec0 0x564af58662b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2ab0886bacd720a9 server_cookie=9e487f7a5a32f6d0 in_seq=0 out_seq=0 2024-06-10T00:35:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1b5ffb700 1 -- 172.21.15.121:0/1612696864 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564af5739290 con 0x564af5863ec0 2024-06-10T00:35:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1b5ffb700 10 client.26186 handle_command_reply: tid=0 2024-06-10T00:35:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 1 client.26186 shutdown 2024-06-10T00:35:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1612696864 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 msgr2=0x564af57f1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57f1db0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc009fd0 tx=0x7fd1bc0978e0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1612696864 shutdown_connections 2024-06-10T00:35:49.477 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x564af5863ec0 0x564af58662b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.466+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564af57e5f60 0x564af5800180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x564af57dff20 0x564af57f1db0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1b5ffb700 0 client.26186 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:35:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1dede2b80 1 --2- 172.21.15.121:0/1612696864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564af57e5640 0x564af57f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1612696864 >> 172.21.15.121:0/1612696864 conn(0x564af570acc0 msgr2=0x564af57e8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1612696864 shutdown_connections 2024-06-10T00:35:49.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1dede2b80 1 -- 172.21.15.121:0/1612696864 wait complete. 2024-06-10T00:35:49.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1dede2b80 20 client.26186 trim_cache size 0 max 16384 2024-06-10T00:35:49.479 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:35:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 -- 172.21.15.121:0/2912812025 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1b80a2d90 msgr2=0x7fd1b80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:49.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1b80a2d90 0x7fd1b80a5250 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc110ad0 tx=0x7fd1c80073d0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 -- 172.21.15.121:0/2912812025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1cc0746c0 msgr2=0x7fd1cc19ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1cc0746c0 0x7fd1cc19ae70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00eb30 tx=0x7fd1bc00ee40 comp rx=0 tx=0).stop 2024-06-10T00:35:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 -- 172.21.15.121:0/2912812025 shutdown_connections 2024-06-10T00:35:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd1b80a2d90 0x7fd1b80a5250 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1cc10eef0 0x7fd1cc19f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd1cc0746c0 0x7fd1cc19ae70 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 --2- 172.21.15.121:0/2912812025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1cc074fe0 0x7fd1cc19b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 -- 172.21.15.121:0/2912812025 >> 172.21.15.121:0/2912812025 conn(0x7fd1cc079570 msgr2=0x7fd1cc0ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 -- 172.21.15.121:0/2912812025 shutdown_connections 2024-06-10T00:35:49.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:49.467+0000 7fd1d455f700 1 -- 172.21.15.121:0/2912812025 wait complete. 2024-06-10T00:35:49.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:49 smithi193 ceph-mon[36716]: pgmap v240: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-06-10T00:35:49.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:49 smithi121 ceph-mon[32378]: pgmap v240: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-06-10T00:35:49.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:49 smithi135 ceph-mon[36424]: pgmap v240: 129 pgs: 129 active+clean; 1.4 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-06-10T00:35:50.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-10T00:35:50.519 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:35:51.599 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:51 smithi121 ceph-mon[32378]: pgmap v241: 129 pgs: 129 active+clean; 1.4 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 255 op/s 2024-06-10T00:35:51.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:51 smithi135 ceph-mon[36424]: pgmap v241: 129 pgs: 129 active+clean; 1.4 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 255 op/s 2024-06-10T00:35:51.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:51 smithi193 ceph-mon[36716]: pgmap v241: 129 pgs: 129 active+clean; 1.4 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 255 op/s 2024-06-10T00:35:52.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.404+0000 7f1118cb7700 1 -- 172.21.15.121:0/3082072370 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1114101390 msgr2=0x7f11141017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:52.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.404+0000 7f1118cb7700 1 --2- 172.21.15.121:0/3082072370 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1114101390 0x7f11141017f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1108009990 tx=0x7f1108009ca0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.405+0000 7f1118cb7700 1 -- 172.21.15.121:0/3082072370 shutdown_connections 2024-06-10T00:35:52.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.405+0000 7f1118cb7700 1 --2- 172.21.15.121:0/3082072370 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1114101d30 0x7f111410e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.405+0000 7f1118cb7700 1 --2- 172.21.15.121:0/3082072370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11141073d0 0x7f11141077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.405+0000 7f1118cb7700 1 --2- 172.21.15.121:0/3082072370 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1114101390 0x7f11141017f0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.405+0000 7f1118cb7700 1 -- 172.21.15.121:0/3082072370 >> 172.21.15.121:0/3082072370 conn(0x7f11140782c0 msgr2=0x7f11140786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:52.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.405+0000 7f1118cb7700 1 -- 172.21.15.121:0/3082072370 shutdown_connections 2024-06-10T00:35:52.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.405+0000 7f1118cb7700 1 -- 172.21.15.121:0/3082072370 wait complete. 2024-06-10T00:35:52.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.405+0000 7f1118cb7700 1 Processor -- start 2024-06-10T00:35:52.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1118cb7700 1 -- start start 2024-06-10T00:35:52.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1118cb7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1114101390 0x7f1114199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1118cb7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1114101d30 0x7f111419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1118cb7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11141073d0 0x7f11141a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1118cb7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1114111680 con 0x7f1114101390 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1118cb7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f1114111500 con 0x7f1114101d30 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1118cb7700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1114111800 con 0x7f11141073d0 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f111259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1114101390 0x7f1114199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1111d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1114101d30 0x7f111419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f111259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1114101390 0x7f1114199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57000/0 (socket says 172.21.15.121:57000) 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1111d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1114101d30 0x7f111419a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:49292/0 (socket says 172.21.15.121:49292) 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f111259c700 1 -- 172.21.15.121:0/1398117118 learned_addr learned my addr 172.21.15.121:0/1398117118 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1112d9d700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11141073d0 0x7f11141a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1111d9b700 1 -- 172.21.15.121:0/1398117118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11141073d0 msgr2=0x7f11141a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1111d9b700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11141073d0 0x7f11141a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1111d9b700 1 -- 172.21.15.121:0/1398117118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1114101390 msgr2=0x7f1114199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:52.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1111d9b700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1114101390 0x7f1114199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1111d9b700 1 -- 172.21.15.121:0/1398117118 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11141a50a0 con 0x7f1114101d30 2024-06-10T00:35:52.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.406+0000 7f1111d9b700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1114101d30 0x7f111419a020 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f110800b2e0 tx=0x7f110800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:52.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.407+0000 7f11037fe700 1 -- 172.21.15.121:0/1398117118 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110800fa90 con 0x7f1114101d30 2024-06-10T00:35:52.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.407+0000 7f1118cb7700 1 -- 172.21.15.121:0/1398117118 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11141a5270 con 0x7f1114101d30 2024-06-10T00:35:52.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.407+0000 7f11037fe700 1 -- 172.21.15.121:0/1398117118 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1108004d10 con 0x7f1114101d30 2024-06-10T00:35:52.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.407+0000 7f11037fe700 1 -- 172.21.15.121:0/1398117118 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11080055e0 con 0x7f1114101d30 2024-06-10T00:35:52.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.407+0000 7f1118cb7700 1 -- 172.21.15.121:0/1398117118 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11141a5660 con 0x7f1114101d30 2024-06-10T00:35:52.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.411+0000 7f11037fe700 1 -- 172.21.15.121:0/1398117118 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1108028060 con 0x7f1114101d30 2024-06-10T00:35:52.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.411+0000 7f11037fe700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f10fc0a2d90 0x7f10fc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.411+0000 7f112353ab80 1 Processor -- start 2024-06-10T00:35:52.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.411+0000 7f11037fe700 1 -- 172.21.15.121:0/1398117118 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7f11080d0290 con 0x7f1114101d30 2024-06-10T00:35:52.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.411+0000 7f111259c700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f10fc0a2d90 0x7f10fc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.411+0000 7f112353ab80 1 -- start start 2024-06-10T00:35:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f112353ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70a850 0x55807c70ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f112353ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70b170 0x55807c70efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f112353ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70f510 0x55807c70f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f112353ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55807c5f8f60 con 0x55807c70b170 2024-06-10T00:35:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f112353ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55807c657850 con 0x55807c70f510 2024-06-10T00:35:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f112353ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55807c5bf460 con 0x55807c70a850 2024-06-10T00:35:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f111259c700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f10fc0a2d90 0x7f10fc0a5250 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f110400bcb0 tx=0x7f110400b480 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f1111d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70b170 0x55807c70efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f111259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70a850 0x55807c70ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f1111d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70b170 0x55807c70efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57010/0 (socket says 172.21.15.121:57010) 2024-06-10T00:35:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f1111d9b700 1 -- 172.21.15.121:0/964377727 learned_addr learned my addr 172.21.15.121:0/964377727 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f1112d9d700 1 --2- 172.21.15.121:0/964377727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70f510 0x55807c70f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f111259c700 1 -- 172.21.15.121:0/964377727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70f510 msgr2=0x55807c70f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f111259c700 1 --2- 172.21.15.121:0/964377727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70f510 0x55807c70f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f111259c700 1 -- 172.21.15.121:0/964377727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70b170 msgr2=0x55807c70efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f111259c700 1 --2- 172.21.15.121:0/964377727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70b170 0x55807c70efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f111259c700 1 -- 172.21.15.121:0/964377727 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55807c5a68d0 con 0x55807c70a850 2024-06-10T00:35:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.412+0000 7f111259c700 1 --2- 172.21.15.121:0/964377727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70a850 0x55807c70ac30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1104010f70 tx=0x7f1104013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42678ac8adf4022b server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.413+0000 7f1100ff9700 1 -- 172.21.15.121:0/964377727 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104011db0 con 0x55807c70a850 2024-06-10T00:35:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.413+0000 7f1100ff9700 1 -- 172.21.15.121:0/964377727 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1104004d10 con 0x55807c70a850 2024-06-10T00:35:52.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.413+0000 7f1100ff9700 1 -- 172.21.15.121:0/964377727 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110401e680 con 0x55807c70a850 2024-06-10T00:35:52.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.413+0000 7f112353ab80 1 -- 172.21.15.121:0/964377727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70a850 msgr2=0x55807c70ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:52.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.413+0000 7f112353ab80 1 --2- 172.21.15.121:0/964377727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70a850 0x55807c70ac30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1104010f70 tx=0x7f1104013000 comp rx=0 tx=0).stop 2024-06-10T00:35:52.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.413+0000 7f112353ab80 1 -- 172.21.15.121:0/964377727 shutdown_connections 2024-06-10T00:35:52.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.413+0000 7f112353ab80 1 --2- 172.21.15.121:0/964377727 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70f510 0x55807c70f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.413+0000 7f112353ab80 1 --2- 172.21.15.121:0/964377727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70a850 0x55807c70ac30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.417 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.413+0000 7f112353ab80 1 --2- 172.21.15.121:0/964377727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70b170 0x55807c70efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.413+0000 7f112353ab80 1 -- 172.21.15.121:0/964377727 >> 172.21.15.121:0/964377727 conn(0x55807c633080 msgr2=0x55807c701220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f112353ab80 1 -- 172.21.15.121:0/964377727 shutdown_connections 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f112353ab80 1 -- 172.21.15.121:0/964377727 wait complete. 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f112353ab80 1 -- start start 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f112353ab80 1 Processor -- start 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f112353ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70a850 0x55807c7169c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f112353ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70b170 0x55807c713d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f112353ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 0x55807c714270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f112353ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55807c5f7fc0 con 0x55807c70f510 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f112353ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55807c653810 con 0x55807c70a850 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f112353ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55807c717660 con 0x55807c70b170 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f112353ab80 10 client.0 _ll_register_callbacks cb 0x55807c5ed5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f111259c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70a850 0x55807c7169c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.418 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f1112d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 0x55807c714270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f111259c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70a850 0x55807c7169c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:49300/0 (socket says 172.21.15.121:49300) 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f111259c700 1 -- 172.21.15.121:0/702443835 learned_addr learned my addr 172.21.15.121:0/702443835 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.414+0000 7f1111d9b700 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70b170 0x55807c713d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f1112d9d700 1 -- 172.21.15.121:0/702443835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70b170 msgr2=0x55807c713d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f1112d9d700 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70b170 0x55807c713d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f1112d9d700 1 -- 172.21.15.121:0/702443835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70a850 msgr2=0x55807c7169c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f1112d9d700 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70a850 0x55807c7169c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f1112d9d700 1 -- 172.21.15.121:0/702443835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55807c61d750 con 0x55807c70f510 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f111259c700 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70a850 0x55807c7169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f1111d9b700 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70b170 0x55807c713d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f1112d9d700 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 0x55807c714270 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f110c011e90 tx=0x7f110c014600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f6b65a8ca25613 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f10f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f10f9ffb700 1 -- 172.21.15.121:0/702443835 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c004db0 con 0x55807c70f510 2024-06-10T00:35:52.419 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f112353ab80 1 -- 172.21.15.121:0/702443835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1104004d10 con 0x55807c70f510 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f10f9ffb700 1 -- 172.21.15.121:0/702443835 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7f110c014a80 con 0x55807c70f510 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f10f9ffb700 1 -- 172.21.15.121:0/702443835 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f110c026050 con 0x55807c70f510 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f112353ab80 10 client.16641 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f112353ab80 1 -- 172.21.15.121:0/702443835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55807c721e30 con 0x55807c70f510 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.415+0000 7f10f9ffb700 1 -- 172.21.15.121:0/702443835 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1104004d10 con 0x55807c70f510 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.416+0000 7f10f9ffb700 1 -- 172.21.15.121:0/702443835 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f110c004350 con 0x55807c70f510 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.416+0000 7f112353ab80 10 client.16641 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.416+0000 7f112353ab80 10 client.16641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.416+0000 7f112353ab80 20 client.16641 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.416+0000 7f112353ab80 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55807c7244c0 0x55807c7268e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.416+0000 7f112353ab80 4 client.16641 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.416+0000 7f112353ab80 1 -- 172.21.15.121:0/702443835 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55807c664d50 con 0x55807c7244c0 2024-06-10T00:35:52.420 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.416+0000 7f111259c700 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55807c7244c0 0x55807c7268e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.416+0000 7f111259c700 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55807c7244c0 0x55807c7268e0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1d5b711a4789958 server_cookie=60a1c166b4be9da in_seq=0 out_seq=0 2024-06-10T00:35:52.421 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.416+0000 7f10f9ffb700 10 client.16641 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:52.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f10f9ffb700 1 -- 172.21.15.121:0/702443835 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55807c664d50 con 0x55807c7244c0 2024-06-10T00:35:52.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f10f9ffb700 10 client.16641 handle_command_reply: tid=0 2024-06-10T00:35:52.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f112353ab80 1 client.16641 shutdown 2024-06-10T00:35:52.931 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f112353ab80 1 -- 172.21.15.121:0/702443835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 msgr2=0x55807c714270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:52.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f112353ab80 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 0x55807c714270 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f110c011e90 tx=0x7f110c014600 comp rx=0 tx=0).stop 2024-06-10T00:35:52.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f112353ab80 1 -- 172.21.15.121:0/702443835 shutdown_connections 2024-06-10T00:35:52.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f112353ab80 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55807c7244c0 0x55807c7268e0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f112353ab80 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 0x55807c714270 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f112353ab80 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70a850 0x55807c7169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f112353ab80 1 --2- 172.21.15.121:0/702443835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70b170 0x55807c713d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f112353ab80 1 -- 172.21.15.121:0/702443835 >> 172.21.15.121:0/702443835 conn(0x55807c633080 msgr2=0x55807c700700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:52.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f10f9ffb700 0 client.16641 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:52.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f112353ab80 1 -- 172.21.15.121:0/702443835 shutdown_connections 2024-06-10T00:35:52.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.930+0000 7f112353ab80 1 -- 172.21.15.121:0/702443835 wait complete. 2024-06-10T00:35:52.932 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.931+0000 7f112353ab80 20 client.16641 trim_cache size 0 max 16384 2024-06-10T00:35:52.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.949+0000 7f112353ab80 1 Processor -- start 2024-06-10T00:35:52.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.949+0000 7f112353ab80 1 -- start start 2024-06-10T00:35:52.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.949+0000 7f112353ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70a850 0x55807c7142c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.949+0000 7f112353ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70b170 0x55807c715c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.949+0000 7f112353ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 0x55807c700e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.949+0000 7f112353ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f11141aed60 con 0x55807c70f510 2024-06-10T00:35:52.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.949+0000 7f112353ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f11141ac530 con 0x55807c70b170 2024-06-10T00:35:52.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.949+0000 7f112353ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f11141aa3a0 con 0x55807c70a850 2024-06-10T00:35:52.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.949+0000 7f1112d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 0x55807c700e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f1111d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70b170 0x55807c715c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f1112d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 0x55807c700e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57026/0 (socket says 172.21.15.121:57026) 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f1112d9d700 1 -- 172.21.15.121:0/3105735034 learned_addr learned my addr 172.21.15.121:0/3105735034 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f111259c700 1 --2- 172.21.15.121:0/3105735034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70a850 0x55807c7142c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f1112d9d700 1 -- 172.21.15.121:0/3105735034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70a850 msgr2=0x55807c7142c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f1112d9d700 1 --2- 172.21.15.121:0/3105735034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70a850 0x55807c7142c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f1112d9d700 1 -- 172.21.15.121:0/3105735034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70b170 msgr2=0x55807c715c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f1112d9d700 1 --2- 172.21.15.121:0/3105735034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70b170 0x55807c715c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f1112d9d700 1 -- 172.21.15.121:0/3105735034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55807c7122c0 con 0x55807c70f510 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f1112d9d700 1 --2- 172.21.15.121:0/3105735034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 0x55807c700e80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f110c009e50 tx=0x7f110c009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f60fb90f2c7c31 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f10fbfff700 1 -- 172.21.15.121:0/3105735034 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c0049b0 con 0x55807c70f510 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f10fbfff700 1 -- 172.21.15.121:0/3105735034 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f110c023070 con 0x55807c70f510 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.950+0000 7f10fbfff700 1 -- 172.21.15.121:0/3105735034 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c007bf0 con 0x55807c70f510 2024-06-10T00:35:52.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.951+0000 7f112353ab80 1 -- 172.21.15.121:0/3105735034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 msgr2=0x55807c700e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.951+0000 7f112353ab80 1 --2- 172.21.15.121:0/3105735034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 0x55807c700e80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f110c009e50 tx=0x7f110c009e80 comp rx=0 tx=0).stop 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.951+0000 7f112353ab80 1 -- 172.21.15.121:0/3105735034 shutdown_connections 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.951+0000 7f112353ab80 1 --2- 172.21.15.121:0/3105735034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70f510 0x55807c700e80 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.951+0000 7f112353ab80 1 --2- 172.21.15.121:0/3105735034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70a850 0x55807c7142c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.951+0000 7f112353ab80 1 --2- 172.21.15.121:0/3105735034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70b170 0x55807c715c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.951+0000 7f112353ab80 1 -- 172.21.15.121:0/3105735034 >> 172.21.15.121:0/3105735034 conn(0x55807c633080 msgr2=0x55807c715560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.951+0000 7f112353ab80 1 -- 172.21.15.121:0/3105735034 shutdown_connections 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.951+0000 7f112353ab80 1 -- 172.21.15.121:0/3105735034 wait complete. 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f112353ab80 1 -- start start 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f112353ab80 1 Processor -- start 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f112353ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70a850 0x55807c710820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f112353ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70b170 0x55807c710d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f112353ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70f510 0x55807c71b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f112353ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f11141aa9f0 con 0x55807c70a850 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f112353ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f11141abe60 con 0x55807c70f510 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f112353ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f11141ab0c0 con 0x55807c70b170 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f111259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70a850 0x55807c710820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f112353ab80 10 client.0 _ll_register_callbacks cb 0x55807c5ed5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f111259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70a850 0x55807c710820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57030/0 (socket says 172.21.15.121:57030) 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f111259c700 1 -- 172.21.15.121:0/3664281273 learned_addr learned my addr 172.21.15.121:0/3664281273 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f1112d9d700 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70f510 0x55807c71b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f1111d9b700 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70b170 0x55807c710d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f111259c700 1 -- 172.21.15.121:0/3664281273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70b170 msgr2=0x55807c710d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f111259c700 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70b170 0x55807c710d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f111259c700 1 -- 172.21.15.121:0/3664281273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70f510 msgr2=0x55807c71b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f111259c700 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70f510 0x55807c71b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:52.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.952+0000 7f111259c700 1 -- 172.21.15.121:0/3664281273 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55807c71be80 con 0x55807c70a850 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.953+0000 7f111259c700 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70a850 0x55807c710820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f110400ba40 tx=0x7f110400ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c9ca1e4ec5d692 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.953+0000 7f10f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.953+0000 7f10f9ffb700 1 -- 172.21.15.121:0/3664281273 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104037020 con 0x55807c70a850 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.953+0000 7f10f9ffb700 1 -- 172.21.15.121:0/3664281273 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7f1104005250 con 0x55807c70a850 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.953+0000 7f112353ab80 1 -- 172.21.15.121:0/3664281273 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f110c023070 con 0x55807c70a850 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.953+0000 7f10f9ffb700 1 -- 172.21.15.121:0/3664281273 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1104042050 con 0x55807c70a850 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.953+0000 7f112353ab80 10 client.16647 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.953+0000 7f112353ab80 1 -- 172.21.15.121:0/3664281273 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55807c71d100 con 0x55807c70a850 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.953+0000 7f10f9ffb700 1 -- 172.21.15.121:0/3664281273 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f110c023070 con 0x55807c70a850 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.954+0000 7f10f9ffb700 1 -- 172.21.15.121:0/3664281273 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f1104033540 con 0x55807c70a850 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.954+0000 7f112353ab80 10 client.16647 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.954+0000 7f112353ab80 10 client.16647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.954+0000 7f112353ab80 20 client.16647 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.954+0000 7f112353ab80 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55807c78f160 0x55807c791550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:52.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.954+0000 7f112353ab80 4 client.16647 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:35:52.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.954+0000 7f112353ab80 1 -- 172.21.15.121:0/3664281273 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55807c65f2d0 con 0x55807c78f160 2024-06-10T00:35:52.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.954+0000 7f1111d9b700 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55807c78f160 0x55807c791550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:52.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.954+0000 7f1111d9b700 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55807c78f160 0x55807c791550 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=355124c86f47f572 server_cookie=a27283270e399218 in_seq=0 out_seq=0 2024-06-10T00:35:52.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:52.954+0000 7f10f9ffb700 10 client.16647 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f10f9ffb700 1 -- 172.21.15.121:0/3664281273 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55807c65f2d0 con 0x55807c78f160 2024-06-10T00:35:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f10f9ffb700 10 client.16647 handle_command_reply: tid=0 2024-06-10T00:35:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 1 client.16647 shutdown 2024-06-10T00:35:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 1 -- 172.21.15.121:0/3664281273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70a850 msgr2=0x55807c710820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70a850 0x55807c710820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f110400ba40 tx=0x7f110400ba70 comp rx=0 tx=0).stop 2024-06-10T00:35:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 1 -- 172.21.15.121:0/3664281273 shutdown_connections 2024-06-10T00:35:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55807c78f160 0x55807c791550 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55807c70f510 0x55807c71b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:53.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55807c70a850 0x55807c710820 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:53.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 1 --2- 172.21.15.121:0/3664281273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55807c70b170 0x55807c710d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:53.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f10f9ffb700 0 client.16647 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:53.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 1 -- 172.21.15.121:0/3664281273 >> 172.21.15.121:0/3664281273 conn(0x55807c633080 msgr2=0x55807c715df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:53.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 1 -- 172.21.15.121:0/3664281273 shutdown_connections 2024-06-10T00:35:53.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 1 -- 172.21.15.121:0/3664281273 wait complete. 2024-06-10T00:35:53.479 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.477+0000 7f112353ab80 20 client.16647 trim_cache size 0 max 16384 2024-06-10T00:35:53.479 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:35:53.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 -- 172.21.15.121:0/1398117118 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f10fc0a2d90 msgr2=0x7f10fc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:53.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f10fc0a2d90 0x7f10fc0a5250 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f110400bcb0 tx=0x7f110400b480 comp rx=0 tx=0).stop 2024-06-10T00:35:53.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 -- 172.21.15.121:0/1398117118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1114101d30 msgr2=0x7f111419a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:53.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1114101d30 0x7f111419a020 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f110800b2e0 tx=0x7f110800b840 comp rx=0 tx=0).stop 2024-06-10T00:35:53.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 -- 172.21.15.121:0/1398117118 shutdown_connections 2024-06-10T00:35:53.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f10fc0a2d90 0x7f10fc0a5250 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:53.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11141073d0 0x7f11141a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:53.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1114101390 0x7f1114199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:53.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 --2- 172.21.15.121:0/1398117118 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1114101d30 0x7f111419a020 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:53.480 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 -- 172.21.15.121:0/1398117118 >> 172.21.15.121:0/1398117118 conn(0x7f11140782c0 msgr2=0x7f11141000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:53.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 -- 172.21.15.121:0/1398117118 shutdown_connections 2024-06-10T00:35:53.481 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:53.478+0000 7f1118cb7700 1 -- 172.21.15.121:0/1398117118 wait complete. 2024-06-10T00:35:53.682 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:53 smithi193 ceph-mon[36716]: pgmap v242: 129 pgs: 129 active+clean; 1.5 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 309 op/s 2024-06-10T00:35:53.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:53 smithi121 ceph-mon[32378]: pgmap v242: 129 pgs: 129 active+clean; 1.5 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 309 op/s 2024-06-10T00:35:53.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:53 smithi135 ceph-mon[36424]: pgmap v242: 129 pgs: 129 active+clean; 1.5 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 309 op/s 2024-06-10T00:35:54.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:54 smithi121 ceph-mon[32378]: pgmap v243: 129 pgs: 129 active+clean; 1.5 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 356 op/s 2024-06-10T00:35:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:54 smithi135 ceph-mon[36424]: pgmap v243: 129 pgs: 129 active+clean; 1.5 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 356 op/s 2024-06-10T00:35:54.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:54 smithi193 ceph-mon[36716]: pgmap v243: 129 pgs: 129 active+clean; 1.5 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 356 op/s 2024-06-10T00:35:55.257 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-06-10T00:35:55.257 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-06-10T00:35:55.504 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:35:56.934 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:56 smithi121 ceph-mon[32378]: pgmap v244: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 393 op/s 2024-06-10T00:35:57.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:56 smithi135 ceph-mon[36424]: pgmap v244: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 393 op/s 2024-06-10T00:35:57.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:56 smithi193 ceph-mon[36716]: pgmap v244: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 393 op/s 2024-06-10T00:35:57.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.325+0000 7fdc026ee700 1 -- 172.21.15.121:0/1435265532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbfc10ee90 msgr2=0x7fdbfc111280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:57.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.325+0000 7fdc026ee700 1 --2- 172.21.15.121:0/1435265532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbfc10ee90 0x7fdbfc111280 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400b0a0 tx=0x7fdbf400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.326+0000 7fdc026ee700 1 -- 172.21.15.121:0/1435265532 shutdown_connections 2024-06-10T00:35:57.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.326+0000 7fdc026ee700 1 --2- 172.21.15.121:0/1435265532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbfc10ee90 0x7fdbfc111280 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.326+0000 7fdc026ee700 1 --2- 172.21.15.121:0/1435265532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbfc074700 0x7fdbfc074ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.326+0000 7fdc026ee700 1 --2- 172.21.15.121:0/1435265532 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdbfc075020 0x7fdbfc10e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.326+0000 7fdc026ee700 1 -- 172.21.15.121:0/1435265532 >> 172.21.15.121:0/1435265532 conn(0x7fdbfc079680 msgr2=0x7fdbfc079a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:57.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.326+0000 7fdc026ee700 1 -- 172.21.15.121:0/1435265532 shutdown_connections 2024-06-10T00:35:57.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.326+0000 7fdc026ee700 1 -- 172.21.15.121:0/1435265532 wait complete. 2024-06-10T00:35:57.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.326+0000 7fdc026ee700 1 Processor -- start 2024-06-10T00:35:57.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.326+0000 7fdc026ee700 1 -- start start 2024-06-10T00:35:57.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdc026ee700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdbfc074700 0x7fdbfc19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdc026ee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbfc075020 0x7fdbfc19b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdc026ee700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbfc10ee90 0x7fdbfc19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdc026ee700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc114a00 con 0x7fdbfc075020 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdc026ee700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc114880 con 0x7fdbfc074700 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdc026ee700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc114b80 con 0x7fdbfc10ee90 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbfb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbfc075020 0x7fdbfc19b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbfb7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbfc075020 0x7fdbfc19b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57046/0 (socket says 172.21.15.121:57046) 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbfbfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdbfc074700 0x7fdbfc19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbfb7fe700 1 -- 172.21.15.121:0/399931366 learned_addr learned my addr 172.21.15.121:0/399931366 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdc00c8b700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbfc10ee90 0x7fdbfc19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbfb7fe700 1 -- 172.21.15.121:0/399931366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbfc10ee90 msgr2=0x7fdbfc19f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbfc10ee90 0x7fdbfc19f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbfb7fe700 1 -- 172.21.15.121:0/399931366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdbfc074700 msgr2=0x7fdbfc19ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:57.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdbfc074700 0x7fdbfc19ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbfb7fe700 1 -- 172.21.15.121:0/399931366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbfc19fdd0 con 0x7fdbfc075020 2024-06-10T00:35:57.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbfc075020 0x7fdbfc19b350 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000d980 tx=0x7fdbf000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:57.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbf97fa700 1 -- 172.21.15.121:0/399931366 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0016070 con 0x7fdbfc075020 2024-06-10T00:35:57.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbf97fa700 1 -- 172.21.15.121:0/399931366 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf0004d10 con 0x7fdbfc075020 2024-06-10T00:35:57.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.327+0000 7fdbf97fa700 1 -- 172.21.15.121:0/399931366 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00055f0 con 0x7fdbfc075020 2024-06-10T00:35:57.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.328+0000 7fdc026ee700 1 -- 172.21.15.121:0/399931366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbfc1a00c0 con 0x7fdbfc075020 2024-06-10T00:35:57.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.328+0000 7fdc026ee700 1 -- 172.21.15.121:0/399931366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbfc1a7940 con 0x7fdbfc075020 2024-06-10T00:35:57.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.330+0000 7fdbf97fa700 1 -- 172.21.15.121:0/399931366 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fdbf0004750 con 0x7fdbfc075020 2024-06-10T00:35:57.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.330+0000 7fdbf97fa700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdbe40a2d90 0x7fdbe40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.330+0000 7fdbf97fa700 1 -- 172.21.15.121:0/399931366 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fdbf00cf5f0 con 0x7fdbfc075020 2024-06-10T00:35:57.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.330+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdbe40a2d90 0x7fdbe40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.331+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdbe40a2d90 0x7fdbe40a5250 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00ac50 tx=0x7fdbec00a380 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:57.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdc0cf71b80 1 Processor -- start 2024-06-10T00:35:57.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdc0cf71b80 1 -- start start 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc215fe0 0x5642dc2163c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc216900 0x5642dc2102e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc210820 0x5642dc210c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5642dc102fe0 con 0x5642dc215fe0 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5642dc1618d0 con 0x5642dc210820 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5642dc0c94e0 con 0x5642dc216900 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdbfbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc215fe0 0x5642dc2163c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdbfbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc215fe0 0x5642dc2163c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57054/0 (socket says 172.21.15.121:57054) 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdbfbfff700 1 -- 172.21.15.121:0/1714410839 learned_addr learned my addr 172.21.15.121:0/1714410839 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/1714410839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc216900 0x5642dc2102e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdc00c8b700 1 --2- 172.21.15.121:0/1714410839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc210820 0x5642dc210c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdbfbfff700 1 -- 172.21.15.121:0/1714410839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc216900 msgr2=0x5642dc2102e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:57.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/1714410839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc216900 0x5642dc2102e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdbfbfff700 1 -- 172.21.15.121:0/1714410839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc210820 msgr2=0x5642dc210c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:57.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/1714410839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc210820 0x5642dc210c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdbfbfff700 1 -- 172.21.15.121:0/1714410839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642dc0b0950 con 0x5642dc215fe0 2024-06-10T00:35:57.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdc00c8b700 1 --2- 172.21.15.121:0/1714410839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc210820 0x5642dc210c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:35:57.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.332+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/1714410839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc215fe0 0x5642dc2163c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00ff90 tx=0x7fdbec012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28963f849defd6bb server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:57.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdbe27fc700 1 -- 172.21.15.121:0/1714410839 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec016070 con 0x5642dc215fe0 2024-06-10T00:35:57.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdbe27fc700 1 -- 172.21.15.121:0/1714410839 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec014070 con 0x5642dc215fe0 2024-06-10T00:35:57.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdbe27fc700 1 -- 172.21.15.121:0/1714410839 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec01d400 con 0x5642dc215fe0 2024-06-10T00:35:57.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/1714410839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc215fe0 msgr2=0x5642dc2163c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:57.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/1714410839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc215fe0 0x5642dc2163c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00ff90 tx=0x7fdbec012000 comp rx=0 tx=0).stop 2024-06-10T00:35:57.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/1714410839 shutdown_connections 2024-06-10T00:35:57.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/1714410839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc210820 0x5642dc210c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/1714410839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc215fe0 0x5642dc2163c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/1714410839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc216900 0x5642dc2102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/1714410839 >> 172.21.15.121:0/1714410839 conn(0x5642dc13d100 msgr2=0x5642dc20b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:57.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/1714410839 shutdown_connections 2024-06-10T00:35:57.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.333+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/1714410839 wait complete. 2024-06-10T00:35:57.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdc0cf71b80 1 -- start start 2024-06-10T00:35:57.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdc0cf71b80 1 Processor -- start 2024-06-10T00:35:57.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 0x5642dc219990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc216900 0x5642dc219ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5642dc102040 con 0x5642dc210820 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5642dc15d890 con 0x5642dc215fe0 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5642dc21d2e0 con 0x5642dc216900 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdc0cf71b80 10 client.0 _ll_register_callbacks cb 0x5642dc0f7630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdbfbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdbfbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57068/0 (socket says 172.21.15.121:57068) 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdbfbfff700 1 -- 172.21.15.121:0/3766983881 learned_addr learned my addr 172.21.15.121:0/3766983881 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.334+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 0x5642dc219990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdc00c8b700 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc216900 0x5642dc219ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdbfbfff700 1 -- 172.21.15.121:0/3766983881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc216900 msgr2=0x5642dc219ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:57.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc216900 0x5642dc219ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdbfbfff700 1 -- 172.21.15.121:0/3766983881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 msgr2=0x5642dc219990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 0x5642dc219990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdbfbfff700 1 -- 172.21.15.121:0/3766983881 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642dc1277d0 con 0x5642dc210820 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21c690 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00fa30 tx=0x7fdbec010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5853ebe27a2244e server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdbdb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3766983881 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec034060 con 0x5642dc210820 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3766983881 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fdbec010d70 con 0x5642dc210820 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3766983881 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbec014070 con 0x5642dc210820 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3766983881 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdbec03b050 con 0x5642dc210820 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdc0cf71b80 10 client.16659 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.335+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3766983881 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5642dc227a90 con 0x5642dc210820 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.336+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3766983881 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec014070 con 0x5642dc210820 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.336+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3766983881 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fdbec01d850 con 0x5642dc210820 2024-06-10T00:35:57.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.336+0000 7fdc0cf71b80 10 client.16659 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.336+0000 7fdc0cf71b80 10 client.16659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.336+0000 7fdc0cf71b80 20 client.16659 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.336+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5642dc22a010 0x5642dc22c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.336+0000 7fdc0cf71b80 4 client.16659 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.336+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3766983881 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642dc16edd0 con 0x5642dc22a010 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.336+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5642dc22a010 0x5642dc22c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.336+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5642dc22a010 0x5642dc22c400 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61f026e73355207a server_cookie=1ae10e88ef82015 in_seq=0 out_seq=0 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.336+0000 7fdbdb7fe700 10 client.16659 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3766983881 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5642dc16edd0 con 0x5642dc22a010 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdbdb7fe700 10 client.16659 handle_command_reply: tid=0 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdc0cf71b80 1 client.16659 shutdown 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3766983881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 msgr2=0x5642dc21c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:57.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21c690 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00fa30 tx=0x7fdbec010660 comp rx=0 tx=0).stop 2024-06-10T00:35:57.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3766983881 shutdown_connections 2024-06-10T00:35:57.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5642dc22a010 0x5642dc22c400 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc216900 0x5642dc219ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21c690 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdbdb7fe700 0 client.16659 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:57.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3766983881 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 0x5642dc219990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.337+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3766983881 >> 172.21.15.121:0/3766983881 conn(0x5642dc13d100 msgr2=0x5642dc20a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:57.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.338+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3766983881 shutdown_connections 2024-06-10T00:35:57.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.338+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3766983881 wait complete. 2024-06-10T00:35:57.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.338+0000 7fdc0cf71b80 20 client.16659 trim_cache size 0 max 16384 2024-06-10T00:35:57.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdc0cf71b80 1 Processor -- start 2024-06-10T00:35:57.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdc0cf71b80 1 -- start start 2024-06-10T00:35:57.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc215fe0 0x5642dc20af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc216900 0x5642dc20b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1b32e0 con 0x5642dc210820 2024-06-10T00:35:57.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1b0ab0 con 0x5642dc216900 2024-06-10T00:35:57.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1ae920 con 0x5642dc215fe0 2024-06-10T00:35:57.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdbfbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdbfbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57078/0 (socket says 172.21.15.121:57078) 2024-06-10T00:35:57.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdbfbfff700 1 -- 172.21.15.121:0/132411981 learned_addr learned my addr 172.21.15.121:0/132411981 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:57.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdc00c8b700 1 --2- 172.21.15.121:0/132411981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc216900 0x5642dc20b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/132411981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc215fe0 0x5642dc20af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdbfbfff700 1 -- 172.21.15.121:0/132411981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc215fe0 msgr2=0x5642dc20af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:57.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/132411981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc215fe0 0x5642dc20af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.358+0000 7fdbfbfff700 1 -- 172.21.15.121:0/132411981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc216900 msgr2=0x5642dc20b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:57.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.359+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/132411981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc216900 0x5642dc20b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.359+0000 7fdbfbfff700 1 -- 172.21.15.121:0/132411981 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642dc21a8c0 con 0x5642dc210820 2024-06-10T00:35:57.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.359+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/132411981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc215fe0 0x5642dc20af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:57.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.359+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/132411981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21ba50 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00ad30 tx=0x7fdbec00ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd519c93fccf9a5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:57.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.359+0000 7fdbe1ffb700 1 -- 172.21.15.121:0/132411981 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec016030 con 0x5642dc210820 2024-06-10T00:35:57.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.359+0000 7fdbe1ffb700 1 -- 172.21.15.121:0/132411981 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec014040 con 0x5642dc210820 2024-06-10T00:35:57.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.359+0000 7fdbe1ffb700 1 -- 172.21.15.121:0/132411981 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec034070 con 0x5642dc210820 2024-06-10T00:35:57.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/132411981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 msgr2=0x5642dc21ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:57.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/132411981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21ba50 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00ad30 tx=0x7fdbec00ad60 comp rx=0 tx=0).stop 2024-06-10T00:35:57.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/132411981 shutdown_connections 2024-06-10T00:35:57.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/132411981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc216900 0x5642dc20b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/132411981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc210820 0x5642dc21ba50 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/132411981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc215fe0 0x5642dc20af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/132411981 >> 172.21.15.121:0/132411981 conn(0x5642dc13d100 msgr2=0x5642dc217350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:57.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/132411981 shutdown_connections 2024-06-10T00:35:57.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/132411981 wait complete. 2024-06-10T00:35:57.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 -- start start 2024-06-10T00:35:57.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 Processor -- start 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc210820 0x5642dc222980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 0x5642dc222ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.360+0000 7fdc0cf71b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc216900 0x5642dc21f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1aef70 con 0x5642dc216900 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1b03e0 con 0x5642dc215fe0 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdc0cf71b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdbfc1af640 con 0x5642dc210820 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdc0cf71b80 10 client.0 _ll_register_callbacks cb 0x5642dc0f7630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdc00c8b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc216900 0x5642dc21f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbfb7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 0x5642dc222ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbfb7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 0x5642dc222ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:49370/0 (socket says 172.21.15.121:49370) 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdc00c8b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc216900 0x5642dc21f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57092/0 (socket says 172.21.15.121:57092) 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbfb7fe700 1 -- 172.21.15.121:0/3743318338 learned_addr learned my addr 172.21.15.121:0/3743318338 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc210820 0x5642dc222980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbfb7fe700 1 -- 172.21.15.121:0/3743318338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc210820 msgr2=0x5642dc222980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc210820 0x5642dc222980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbfb7fe700 1 -- 172.21.15.121:0/3743318338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc216900 msgr2=0x5642dc21f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc216900 0x5642dc21f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbfb7fe700 1 -- 172.21.15.121:0/3743318338 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642dc21ffb0 con 0x5642dc215fe0 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc210820 0x5642dc222980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdc00c8b700 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc216900 0x5642dc21f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbfb7fe700 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 0x5642dc222ec0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdbf00cc9b0 tx=0x7fdbf0004640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c2af9d022445381 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbdb7fe700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3743318338 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf009b720 con 0x5642dc215fe0 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3743318338 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbec014040 con 0x5642dc215fe0 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.361+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3743318338 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fdbf0099a20 con 0x5642dc215fe0 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3743318338 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdbf0004a40 con 0x5642dc215fe0 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdc0cf71b80 10 client.26222 fetch_fsmap learned FSMap version 32 2024-06-10T00:35:57.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3743318338 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5642dc295720 con 0x5642dc215fe0 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3743318338 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec014040 con 0x5642dc215fe0 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3743318338 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fdbf009b980 con 0x5642dc215fe0 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdc0cf71b80 10 client.26222 fetch_fsmap finished waiting for FSMap version 32 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdc0cf71b80 10 client.26222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdc0cf71b80 20 client.26222 populate_metadata read hostname 'smithi121' 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5642dc295ec0 0x5642dc2982b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdc0cf71b80 4 client.26222 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3743318338 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5642dc169350 con 0x5642dc295ec0 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.362+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5642dc295ec0 0x5642dc2982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.363+0000 7fdbdb7fe700 10 client.26222 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.363+0000 7fdbfbfff700 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5642dc295ec0 0x5642dc2982b0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8343164145fd0c1 server_cookie=e9f94551b08a8031 in_seq=0 out_seq=0 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdbdb7fe700 1 -- 172.21.15.121:0/3743318338 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5642dc169350 con 0x5642dc295ec0 2024-06-10T00:35:57.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdbdb7fe700 10 client.26222 handle_command_reply: tid=0 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 1 client.26222 shutdown 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3743318338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 msgr2=0x5642dc222ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 0x5642dc222ec0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdbf00cc9b0 tx=0x7fdbf0004640 comp rx=0 tx=0).stop 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3743318338 shutdown_connections 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5642dc295ec0 0x5642dc2982b0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5642dc216900 0x5642dc21f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5642dc210820 0x5642dc222980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 1 --2- 172.21.15.121:0/3743318338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5642dc215fe0 0x5642dc222ec0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdbdb7fe700 0 client.26222 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3743318338 >> 172.21.15.121:0/3743318338 conn(0x5642dc13d100 msgr2=0x5642dc21ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3743318338 shutdown_connections 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 1 -- 172.21.15.121:0/3743318338 wait complete. 2024-06-10T00:35:57.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.364+0000 7fdc0cf71b80 20 client.26222 trim_cache size 0 max 16384 2024-06-10T00:35:57.371 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:35:57.371 INFO:teuthology.orchestra.run.smithi121.stdout: "return_code": 0, 2024-06-10T00:35:57.371 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_tag": "e235dce8-6480-42f8-bb4d-df404081360e", 2024-06-10T00:35:57.371 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": "asynchronous" 2024-06-10T00:35:57.371 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:35:57.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 -- 172.21.15.121:0/399931366 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdbe40a2d90 msgr2=0x7fdbe40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:57.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdbe40a2d90 0x7fdbe40a5250 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fdbec00ac50 tx=0x7fdbec00a380 comp rx=0 tx=0).stop 2024-06-10T00:35:57.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 -- 172.21.15.121:0/399931366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbfc075020 msgr2=0x7fdbfc19b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:35:57.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbfc075020 0x7fdbfc19b350 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000d980 tx=0x7fdbf000dc90 comp rx=0 tx=0).stop 2024-06-10T00:35:57.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 -- 172.21.15.121:0/399931366 shutdown_connections 2024-06-10T00:35:57.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fdbe40a2d90 0x7fdbe40a5250 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdbfc10ee90 0x7fdbfc19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdbfc074700 0x7fdbfc19ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 --2- 172.21.15.121:0/399931366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbfc075020 0x7fdbfc19b350 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:35:57.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 -- 172.21.15.121:0/399931366 >> 172.21.15.121:0/399931366 conn(0x7fdbfc079680 msgr2=0x7fdbfc0ff600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:35:57.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 -- 172.21.15.121:0/399931366 shutdown_connections 2024-06-10T00:35:57.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:35:57.365+0000 7fdc026ee700 1 -- 172.21.15.121:0/399931366 wait complete. 2024-06-10T00:35:57.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:57 smithi135 ceph-mon[36424]: scrub queued for path: / 2024-06-10T00:35:57.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:57 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:57.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:57 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:35:57.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:57 smithi135 ceph-mon[36424]: scrub summary: active paths [/] 2024-06-10T00:35:57.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:57 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001d4a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-42) see mds.h log and `damage ls` output for details 2024-06-10T00:35:57.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:57 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000018c3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34) see mds.h log and `damage ls` output for details 2024-06-10T00:35:57.759 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:57 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:57.432+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001d4a [...7,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-42/ auth{1=1} fragtree_t(*^3) v149577 ap=1 f(v6 m2024-06-10T00:33:23.570379+0000 166=166+0) n(v11 rc2024-06-10T00:33:23.570379+0000 b4066047 167=166+1) old_inodes=3 (inest mix) caps={24871=pAsLsXsFs/pAsLsXsFs@1} mcw={1=AsLsXsFs} | lock=0 importingcaps=0 dirfrag=8 caps=1 dirwaiter=0 replicated=1 dirty=1 authpin=1 discoverbase=0 scrubqueue=0 0x558e1ee70000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000001d4a:[<0x10000000399/blog-42 v149577>,<0x10000000208/blogtest_in v2423>,<0x10000000201/src v2702>,<0x10000000006/blogbench-1.0 v2285>,<0x10000000005/tmp v519>,<0x100000001fb/client.0 v506>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v499>,<0x10000000001/sv_1 v514>,<0x10000000000/qa v497>,<0x1/volumes v486>]//[]","memoryvalue":"(2)0x10000001d4a:[<0x10000000399/blog-42 v149577>,<0x10000000208/blogtest_in v2444>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-10T00:33:23.570379+0000 166=166+0)","ondisk_value.rstat":"n(v0 rc2024-06-10T00:33:23.570379+0000 b4066047 167=166+1)","memory_value.dirstat":"f(v6 m2024-06-10T00:33:23.570379+0000 166=166+0)","memory_value.rstat":"n(v11 rc2024-06-10T00:33:23.570379+0000 b4066047 167=166+1)","error_str":""},"return_code":0} 2024-06-10T00:35:57.759 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:57 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:57.433+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000018c3 [...7,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/ auth fragtree_t(*^3) v146605 ap=1 f(v0 m2024-06-10T00:33:21.236409+0000 147=147+0) n(v0 rc2024-06-10T00:33:21.236409+0000 b4046279 148=147+1) old_inodes=3 (inest mix) (ifile excl) caps={24871=pAsLsXsFsx/pAsLsXsFsxcral@1},l=24871 | lock=0 importingcaps=0 dirfrag=8 caps=1 dirwaiter=0 replicated=0 dirty=1 authpin=1 discoverbase=0 scrubqueue=0 0x558e2b061b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x100000018c3:[<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[]","memoryvalue":"(2)0x100000018c3:[<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2444>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-06-10T00:33:21.236409+0000 147=147+0)","ondisk_value.rstat":"n(v0 rc2024-06-10T00:33:21.236409+0000 b4046279 148=147+1)","memory_value.dirstat":"f(v0 m2024-06-10T00:33:21.236409+0000 147=147+0)","memory_value.rstat":"n(v0 rc2024-06-10T00:33:21.236409+0000 b4046279 148=147+1)","error_str":""},"return_code":0} 2024-06-10T00:35:57.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:57 smithi121 ceph-mon[32378]: scrub queued for path: / 2024-06-10T00:35:57.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:57 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:57.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:57 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:35:57.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:57 smithi121 ceph-mon[32378]: scrub summary: active paths [/] 2024-06-10T00:35:57.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:57 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001d4a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-42) see mds.h log and `damage ls` output for details 2024-06-10T00:35:57.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:57 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000018c3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:57 smithi193 ceph-mon[36716]: scrub queued for path: / 2024-06-10T00:35:58.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:57 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:58.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:57 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:35:58.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:57 smithi193 ceph-mon[36716]: scrub summary: active paths [/] 2024-06-10T00:35:58.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:57 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001d4a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-42) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:57 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000018c3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:35:58.546 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.269+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001926 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-59.xml auth v970 ap=1 s=14943 n(v0 rc2024-06-10T00:30:42.348469+0000 b14943 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c956b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001926:[<0x100000018c3/article-59.xml v970>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001926:[<0x100000018c3/article-59.xml v970>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.546 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.269+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b3b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-13.jpg auth v972 ap=1 s=14733 n(v0 rc2024-06-10T00:30:46.654413+0000 b14733 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c957600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b3b:[<0x100000018c3/picture-13.jpg v972>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b3b:[<0x100000018c3/picture-13.jpg v972>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.546 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.269+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019cb [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-18.xml auth v978 ap=1 s=15968 n(v0 rc2024-06-10T00:30:45.199432+0000 b15968 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c960000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019cb:[<0x100000018c3/article-18.xml v978>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019cb:[<0x100000018c3/article-18.xml v978>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.547 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.269+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b27 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-18.jpg auth v974 ap=1 s=30326 n(v0 rc2024-06-10T00:30:46.588414+0000 b30326 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c958100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b27:[<0x100000018c3/picture-18.jpg v974>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b27:[<0x100000018c3/picture-18.jpg v974>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.549 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.269+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000194b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-51.xml auth v982 ap=1 s=15970 n(v0 rc2024-06-10T00:30:44.310443+0000 b15970 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c961600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000194b:[<0x100000018c3/article-51.xml v982>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000194b:[<0x100000018c3/article-51.xml v982>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.549 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.270+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b23 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-19.jpg auth v980 ap=1 s=23517 n(v0 rc2024-06-10T00:30:46.575414+0000 b23517 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c960b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b23:[<0x100000018c3/picture-19.jpg v980>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b23:[<0x100000018c3/picture-19.jpg v980>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.549 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.270+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019a7 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-27.xml auth v984 ap=1 s=31479 n(v0 rc2024-06-10T00:30:45.125433+0000 b31479 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c962100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019a7:[<0x100000018c3/article-27.xml v984>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x100000019a7:[<0x100000018c3/article-27.xml v984>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.549 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.270+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b5e [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-4.jpg auth v986 ap=1 s=46726 n(v0 rc2024-06-10T00:30:46.769411+0000 b46726 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c962c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b5e:[<0x100000018c3/picture-4.jpg v986>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b5e:[<0x100000018c3/picture-4.jpg v986>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.549 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.270+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ae8 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-35.jpg auth v988 ap=1 s=39127 n(v0 rc2024-06-10T00:30:46.306417+0000 b39127 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c963700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ae8:[<0x100000018c3/picture-35.jpg v988>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ae8:[<0x100000018c3/picture-35.jpg v988>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.549 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.270+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000018e8 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-70.xml auth v992 ap=1 s=7315 n(v0 rc2024-06-10T00:30:39.509505+0000 b7315 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c969080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000018e8:[<0x100000018c3/article-70.xml v992>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x100000018e8:[<0x100000018c3/article-70.xml v992>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.549 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.270+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b64 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-2.jpg auth v990 ap=1 s=61827 n(v0 rc2024-06-10T00:30:46.786411+0000 b61827 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c968580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b64:[<0x100000018c3/picture-2.jpg v990>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b64:[<0x100000018c3/picture-2.jpg v990>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.549 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.270+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b37 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-14.jpg auth v996 ap=1 s=16538 n(v0 rc2024-06-10T00:30:46.641413+0000 b16538 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c96ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b37:[<0x100000018c3/picture-14.jpg v996>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b37:[<0x100000018c3/picture-14.jpg v996>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.549 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.271+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000191b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-61.xml auth v998 ap=1 s=7863 n(v0 rc2024-06-10T00:30:39.795502+0000 b7863 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c96b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000191b:[<0x100000018c3/article-61.xml v998>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x1000000191b:[<0x100000018c3/article-61.xml v998>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.549 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.271+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001971 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-41.xml auth v1000 ap=1 s=22019 n(v0 rc2024-06-10T00:30:44.918435+0000 b22019 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c970580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001971:[<0x100000018c3/article-41.xml v1000>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001971:[<0x100000018c3/article-41.xml v1000>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.549 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.271+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001961 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-45.xml auth v1002 ap=1 s=25155 n(v0 rc2024-06-10T00:30:44.567440+0000 b25155 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c971080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001961:[<0x100000018c3/article-45.xml v1002>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001961:[<0x100000018c3/article-45.xml v1002>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.550 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.271+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a5c [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-61.jpg auth v1006 ap=1 s=58724 n(v0 rc2024-06-10T00:30:45.787424+0000 b58724 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c972c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a5c:[<0x100000018c3/picture-61.jpg v1006>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a5c:[<0x100000018c3/picture-61.jpg v1006>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.550 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.271+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b53 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-7.jpg auth v1008 ap=1 s=35060 n(v0 rc2024-06-10T00:30:46.738412+0000 b35060 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c973700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b53:[<0x100000018c3/picture-7.jpg v1008>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b53:[<0x100000018c3/picture-7.jpg v1008>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.550 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.271+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b4b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-9.jpg auth v1012 ap=1 s=32548 n(v0 rc2024-06-10T00:30:46.709412+0000 b32548 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c979080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b4b:[<0x100000018c3/picture-9.jpg v1012>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b4b:[<0x100000018c3/picture-9.jpg v1012>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.550 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.271+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001997 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-31.xml auth v1010 ap=1 s=4489 n(v0 rc2024-06-10T00:30:45.092433+0000 b4489 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c978580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001997:[<0x100000018c3/article-31.xml v1010>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001997:[<0x100000018c3/article-31.xml v1010>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.550 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.272+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a42 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-64.jpg auth v955 ap=1 s=29163 n(v0 rc2024-06-10T00:30:45.719425+0000 b29163 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c979b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a42:[<0x100000018c3/picture-64.jpg v955>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a42:[<0x100000018c3/picture-64.jpg v955>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.550 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.272+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b47 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-10.jpg auth v957 ap=1 s=22983 n(v0 rc2024-06-10T00:30:46.694412+0000 b22983 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c97a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b47:[<0x100000018c3/picture-10.jpg v957>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b47:[<0x100000018c3/picture-10.jpg v957>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.550 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.272+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b68 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-1.jpg auth v959 ap=1 s=64842 n(v0 rc2024-06-10T00:30:46.795411+0000 b64842 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c97b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b68:[<0x100000018c3/picture-1.jpg v959>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b68:[<0x100000018c3/picture-1.jpg v959>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.550 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.272+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019ef [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-9.xml auth v961 ap=1 s=6595 n(v0 rc2024-06-10T00:30:45.265431+0000 b6595 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c980000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019ef:[<0x100000018c3/article-9.xml v961>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019ef:[<0x100000018c3/article-9.xml v961>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.550 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.272+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001965 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-44.xml auth v963 ap=1 s=22476 n(v0 rc2024-06-10T00:30:44.692438+0000 b22476 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c980b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001965:[<0x100000018c3/article-44.xml v963>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001965:[<0x100000018c3/article-44.xml v963>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.550 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.273+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b57 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-6.jpg auth v965 ap=1 s=58350 n(v0 rc2024-06-10T00:30:46.752411+0000 b58350 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c981600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b57:[<0x100000018c3/picture-6.jpg v965>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b57:[<0x100000018c3/picture-6.jpg v965>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.550 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.273+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001904 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-65.xml auth v967 ap=1 s=20540 n(v0 rc2024-06-10T00:30:39.621504+0000 b20540 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c982100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001904:[<0x100000018c3/article-65.xml v967>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x10000001904:[<0x100000018c3/article-65.xml v967>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.273+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001915 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-62.xml auth v969 ap=1 s=14494 n(v0 rc2024-06-10T00:30:39.723503+0000 b14494 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c982c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001915:[<0x100000018c3/article-62.xml v969>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x10000001915:[<0x100000018c3/article-62.xml v969>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.273+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019b3 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-24.xml auth v971 ap=1 s=2300 n(v0 rc2024-06-10T00:30:45.151432+0000 b2300 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c983700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019b3:[<0x100000018c3/article-24.xml v971>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x100000019b3:[<0x100000018c3/article-24.xml v971>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.273+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001aff [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-28.jpg auth v973 ap=1 s=33141 n(v0 rc2024-06-10T00:30:46.445415+0000 b33141 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c98a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001aff:[<0x100000018c3/picture-28.jpg v973>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001aff:[<0x100000018c3/picture-28.jpg v973>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.273+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000018f9 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-67.xml auth v975 ap=1 s=17622 n(v0 rc2024-06-10T00:30:39.575505+0000 b17622 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c98b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000018f9:[<0x100000018c3/article-67.xml v975>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x100000018f9:[<0x100000018c3/article-67.xml v975>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.274+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ae2 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-37.jpg auth v977 ap=1 s=55605 n(v0 rc2024-06-10T00:30:46.286417+0000 b55605 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c98bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ae2:[<0x100000018c3/picture-37.jpg v977>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ae2:[<0x100000018c3/picture-37.jpg v977>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.274+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b2b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-17.jpg auth v979 ap=1 s=46107 n(v0 rc2024-06-10T00:30:46.601413+0000 b46107 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c98c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b2b:[<0x100000018c3/picture-17.jpg v979>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b2b:[<0x100000018c3/picture-17.jpg v979>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.274+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a07 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-3.xml auth v981 ap=1 s=8473 n(v0 rc2024-06-10T00:30:45.311430+0000 b8473 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c98d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a07:[<0x100000018c3/article-3.xml v981>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a07:[<0x100000018c3/article-3.xml v981>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.274+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b0f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-24.jpg auth v983 ap=1 s=33687 n(v0 rc2024-06-10T00:30:46.524414+0000 b33687 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c992000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b0f:[<0x100000018c3/picture-24.jpg v983>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b0f:[<0x100000018c3/picture-24.jpg v983>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.274+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001aee [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-33.jpg auth v985 ap=1 s=28995 n(v0 rc2024-06-10T00:30:46.338417+0000 b28995 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c992b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001aee:[<0x100000018c3/picture-33.jpg v985>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001aee:[<0x100000018c3/picture-33.jpg v985>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.274+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a7f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-57.jpg auth v987 ap=1 s=13476 n(v0 rc2024-06-10T00:30:45.957422+0000 b13476 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c993600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a7f:[<0x100000018c3/picture-57.jpg v987>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a7f:[<0x100000018c3/picture-57.jpg v987>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.274+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019f7 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-7.xml auth v989 ap=1 s=32691 n(v0 rc2024-06-10T00:30:45.280431+0000 b32691 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c994100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019f7:[<0x100000018c3/article-7.xml v989>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019f7:[<0x100000018c3/article-7.xml v989>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.551 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.275+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ad7 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-42.jpg auth v991 ap=1 s=30489 n(v0 rc2024-06-10T00:30:46.261418+0000 b30489 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c994c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ad7:[<0x100000018c3/picture-42.jpg v991>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ad7:[<0x100000018c3/picture-42.jpg v991>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.556 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.275+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a1e [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-68.jpg auth v993 ap=1 s=54437 n(v0 rc2024-06-10T00:30:45.509428+0000 b54437 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c995700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a1e:[<0x100000018c3/picture-68.jpg v993>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a1e:[<0x100000018c3/picture-68.jpg v993>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.556 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.275+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001add [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-39.jpg auth v995 ap=1 s=49378 n(v0 rc2024-06-10T00:30:46.274418+0000 b49378 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c99a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001add:[<0x100000018c3/picture-39.jpg v995>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001add:[<0x100000018c3/picture-39.jpg v995>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.557 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.275+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001937 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-55.xml auth v947 ap=1 s=1659 n(v0 rc2024-06-10T00:30:43.751450+0000 b1659 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c99b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001937:[<0x100000018c3/article-55.xml v947>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001937:[<0x100000018c3/article-55.xml v947>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.557 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.275+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b0b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-25.jpg auth v949 ap=1 s=49134 n(v0 rc2024-06-10T00:30:46.512415+0000 b49134 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c99bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b0b:[<0x100000018c3/picture-25.jpg v949>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b0b:[<0x100000018c3/picture-25.jpg v949>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.557 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.275+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019f3 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-8.xml auth v951 ap=1 s=26265 n(v0 rc2024-06-10T00:30:45.272431+0000 b26265 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c99c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019f3:[<0x100000018c3/article-8.xml v951>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019f3:[<0x100000018c3/article-8.xml v951>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.557 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.276+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001af4 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-31.jpg auth v953 ap=1 s=60375 n(v0 rc2024-06-10T00:30:46.377416+0000 b60375 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c99d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001af4:[<0x100000018c3/picture-31.jpg v953>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001af4:[<0x100000018c3/picture-31.jpg v953>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.557 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.276+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000018f3 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-68.xml auth v955 ap=1 s=24140 n(v0 rc2024-06-10T00:30:39.558505+0000 b24140 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9a2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000018f3:[<0x100000018c3/article-68.xml v955>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x100000018f3:[<0x100000018c3/article-68.xml v955>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.557 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.276+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ac6 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-49.jpg auth v957 ap=1 s=43281 n(v0 rc2024-06-10T00:30:46.223418+0000 b43281 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9a2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ac6:[<0x100000018c3/picture-49.jpg v957>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ac6:[<0x100000018c3/picture-49.jpg v957>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.557 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.276+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b4f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-8.jpg auth v961 ap=1 s=11162 n(v0 rc2024-06-10T00:30:46.724412+0000 b11162 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9a4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b4f:[<0x100000018c3/picture-8.jpg v961>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b4f:[<0x100000018c3/picture-8.jpg v961>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.557 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.276+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000018c5 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-76.xml auth v959 ap=1 s=26235 n(v0 rc2024-06-10T00:30:39.364507+0000 b26235 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9a3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000018c5:[<0x100000018c3/article-76.xml v959>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x100000018c5:[<0x100000018c3/article-76.xml v959>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.557 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.276+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b07 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-26.jpg auth v963 ap=1 s=56979 n(v0 rc2024-06-10T00:30:46.489415+0000 b56979 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9a4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b07:[<0x100000018c3/picture-26.jpg v963>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b07:[<0x100000018c3/picture-26.jpg v963>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.557 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.276+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001993 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-32.xml auth v965 ap=1 s=25617 n(v0 rc2024-06-10T00:30:45.084433+0000 b25617 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9a5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001993:[<0x100000018c3/article-32.xml v965>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001993:[<0x100000018c3/article-32.xml v965>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.557 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.277+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019bf [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-21.xml auth v967 ap=1 s=12444 n(v0 rc2024-06-10T00:30:45.175432+0000 b12444 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9aa580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019bf:[<0x100000018c3/article-21.xml v967>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x100000019bf:[<0x100000018c3/article-21.xml v967>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.558 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.277+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000199b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-30.xml auth v969 ap=1 s=19526 n(v0 rc2024-06-10T00:30:45.101433+0000 b19526 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9ab080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000199b:[<0x100000018c3/article-30.xml v969>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000199b:[<0x100000018c3/article-30.xml v969>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.558 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.277+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000194f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-50.xml auth v973 ap=1 s=4564 n(v0 rc2024-06-10T00:30:44.378442+0000 b4564 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9ac680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000194f:[<0x100000018c3/article-50.xml v973>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000194f:[<0x100000018c3/article-50.xml v973>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.558 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.277+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001e5d [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-13.xml auth v971 ap=1 s=17906 n(v0 rc2024-06-10T00:30:49.691373+0000 b17906 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9abb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001e5d:[<0x100000018c3/article-13.xml v971>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2423>,<0x10000000201/src v2702>,<0x10000000006/blogbench-1.0 v2285>,<0x10000000005/tmp v519>,<0x100000001fb/client.0 v506>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v499>,<0x10000000001/sv_1 v514>,<0x10000000000/qa v497>,<0x1/volumes v486>]//[3]","memoryvalue":"(4)0x10000001e5d:[<0x100000018c3/article-13.xml v971>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.558 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.277+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019fb [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-6.xml auth v975 ap=1 s=26351 n(v0 rc2024-06-10T00:30:45.288430+0000 b26351 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9ad180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019fb:[<0x100000018c3/article-6.xml v975>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019fb:[<0x100000018c3/article-6.xml v975>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.558 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.277+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a0b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-2.xml auth v977 ap=1 s=26841 n(v0 rc2024-06-10T00:30:45.323430+0000 b26841 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9b2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a0b:[<0x100000018c3/article-2.xml v977>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a0b:[<0x100000018c3/article-2.xml v977>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.558 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.278+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a9b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-54.jpg auth v979 ap=1 s=30263 n(v0 rc2024-06-10T00:30:46.108420+0000 b30263 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9b2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a9b:[<0x100000018c3/picture-54.jpg v979>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a9b:[<0x100000018c3/picture-54.jpg v979>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.558 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.278+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001afb [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-29.jpg auth v981 ap=1 s=17916 n(v0 rc2024-06-10T00:30:46.423416+0000 b17916 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9b3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001afb:[<0x100000018c3/picture-29.jpg v981>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001afb:[<0x100000018c3/picture-29.jpg v981>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.558 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.278+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001933 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-56.xml auth v983 ap=1 s=11563 n(v0 rc2024-06-10T00:30:43.633452+0000 b11563 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9b4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001933:[<0x100000018c3/article-56.xml v983>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001933:[<0x100000018c3/article-56.xml v983>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.558 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.278+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001969 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-43.xml auth v966 ap=1 s=10415 n(v0 rc2024-06-10T00:30:44.763437+0000 b10415 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2c9b5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001969:[<0x100000018c3/article-43.xml v966>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001969:[<0x100000018c3/article-43.xml v966>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.558 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.278+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000198f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-33.xml auth v968 ap=1 s=17709 n(v0 rc2024-06-10T00:30:45.076433+0000 b17709 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed38000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000198f:[<0x100000018c3/article-33.xml v968>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000198f:[<0x100000018c3/article-33.xml v968>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.558 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.278+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b5a [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-5.jpg auth v970 ap=1 s=48083 n(v0 rc2024-06-10T00:30:46.762411+0000 b48083 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed38b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b5a:[<0x100000018c3/picture-5.jpg v970>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b5a:[<0x100000018c3/picture-5.jpg v970>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.559 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.279+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ab5 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-50.jpg auth v972 ap=1 s=40447 n(v0 rc2024-06-10T00:30:46.177419+0000 b40447 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed39600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ab5:[<0x100000018c3/picture-50.jpg v972>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ab5:[<0x100000018c3/picture-50.jpg v972>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.559 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.279+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a77 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-58.jpg auth v974 ap=1 s=39334 n(v0 rc2024-06-10T00:30:45.901422+0000 b39334 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed3a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a77:[<0x100000018c3/picture-58.jpg v974>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a77:[<0x100000018c3/picture-58.jpg v974>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.559 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.279+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000192e [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-57.xml auth v976 ap=1 s=27121 n(v0 rc2024-06-10T00:30:43.452454+0000 b27121 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed3ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000192e:[<0x100000018c3/article-57.xml v976>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000192e:[<0x100000018c3/article-57.xml v976>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.559 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.279+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a38 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-65.jpg auth v978 ap=1 s=63710 n(v0 rc2024-06-10T00:30:45.697425+0000 b63710 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed3b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a38:[<0x100000018c3/picture-65.jpg v978>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a38:[<0x100000018c3/picture-65.jpg v978>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.559 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.279+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000192a [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-58.xml auth v980 ap=1 s=25869 n(v0 rc2024-06-10T00:30:43.349456+0000 b25869 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed3e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000192a:[<0x100000018c3/article-58.xml v980>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000192a:[<0x100000018c3/article-58.xml v980>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.559 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.279+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001940 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-53.xml auth v982 ap=1 s=23048 n(v0 rc2024-06-10T00:30:44.125446+0000 b23048 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed3f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001940:[<0x100000018c3/article-53.xml v982>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001940:[<0x100000018c3/article-53.xml v982>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.559 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.279+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019a3 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-28.xml auth v984 ap=1 s=22680 n(v0 rc2024-06-10T00:30:45.117433+0000 b22680 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed3fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019a3:[<0x100000018c3/article-28.xml v984>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x100000019a3:[<0x100000018c3/article-28.xml v984>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.559 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.280+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001947 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-52.xml auth v986 ap=1 s=2176 n(v0 rc2024-06-10T00:30:44.235444+0000 b2176 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed40680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001947:[<0x100000018c3/article-52.xml v986>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001947:[<0x100000018c3/article-52.xml v986>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.559 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.280+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a13 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-69.jpg auth v988 ap=1 s=42246 n(v0 rc2024-06-10T00:30:45.400429+0000 b42246 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed41180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a13:[<0x100000018c3/picture-69.jpg v988>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a13:[<0x100000018c3/picture-69.jpg v988>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.559 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.280+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ab1 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-51.jpg auth v990 ap=1 s=31093 n(v0 rc2024-06-10T00:30:46.166419+0000 b31093 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed48000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ab1:[<0x100000018c3/picture-51.jpg v990>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ab1:[<0x100000018c3/picture-51.jpg v990>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.561 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.280+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a53 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-62.jpg auth v992 ap=1 s=25507 n(v0 rc2024-06-10T00:30:45.757424+0000 b25507 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed48b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a53:[<0x100000018c3/picture-62.jpg v992>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a53:[<0x100000018c3/picture-62.jpg v992>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.561 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.280+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a8f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-55.jpg auth v972 ap=1 s=41928 n(v0 rc2024-06-10T00:30:46.072420+0000 b41928 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed4ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a8f:[<0x100000018c3/picture-55.jpg v972>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a8f:[<0x100000018c3/picture-55.jpg v972>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.561 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.281+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b61 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-3.jpg auth v970 ap=1 s=8320 n(v0 rc2024-06-10T00:30:46.777411+0000 b8320 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed4a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b61:[<0x100000018c3/picture-3.jpg v970>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b61:[<0x100000018c3/picture-3.jpg v970>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.281+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b43 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-11.jpg auth v974 ap=1 s=34705 n(v0 rc2024-06-10T00:30:46.680412+0000 b34705 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed4b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b43:[<0x100000018c3/picture-11.jpg v974>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b43:[<0x100000018c3/picture-11.jpg v974>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.281+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000198b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-34.xml auth v978 ap=1 s=12987 n(v0 rc2024-06-10T00:30:45.064433+0000 b12987 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed4f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000198b:[<0x100000018c3/article-34.xml v978>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000198b:[<0x100000018c3/article-34.xml v978>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.281+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000199f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-29.xml auth v980 ap=1 s=21843 n(v0 rc2024-06-10T00:30:45.109433+0000 b21843 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed50100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000199f:[<0x100000018c3/article-29.xml v980>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000199f:[<0x100000018c3/article-29.xml v980>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.281+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001920 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-60.xml auth v982 ap=1 s=16024 n(v0 rc2024-06-10T00:30:39.855501+0000 b16024 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed50c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001920:[<0x100000018c3/article-60.xml v982>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2393>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x10000001920:[<0x100000018c3/article-60.xml v982>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.281+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a4a [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-63.jpg auth v984 ap=1 s=42187 n(v0 rc2024-06-10T00:30:45.735425+0000 b42187 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed51700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a4a:[<0x100000018c3/picture-63.jpg v984>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a4a:[<0x100000018c3/picture-63.jpg v984>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.281+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001adb [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-40.jpg auth v986 ap=1 s=48791 n(v0 rc2024-06-10T00:30:46.271418+0000 b48791 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed56580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001adb:[<0x100000018c3/picture-40.jpg v986>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001adb:[<0x100000018c3/picture-40.jpg v986>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.282+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019bb [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-22.xml auth v988 ap=1 s=5565 n(v0 rc2024-06-10T00:30:45.167432+0000 b5565 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed57080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019bb:[<0x100000018c3/article-22.xml v988>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x100000019bb:[<0x100000018c3/article-22.xml v988>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.282+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001953 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-49.xml auth v990 ap=1 s=15586 n(v0 rc2024-06-10T00:30:44.436441+0000 b15586 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed57b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001953:[<0x100000018c3/article-49.xml v990>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001953:[<0x100000018c3/article-49.xml v990>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.282+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b13 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-23.jpg auth v963 ap=1 s=47917 n(v0 rc2024-06-10T00:30:46.533414+0000 b47917 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed58680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b13:[<0x100000018c3/picture-23.jpg v963>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b13:[<0x100000018c3/picture-23.jpg v963>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.282+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019db [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-14.xml auth v965 ap=1 s=15147 n(v0 rc2024-06-10T00:30:45.228431+0000 b15147 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed59180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019db:[<0x100000018c3/article-14.xml v965>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019db:[<0x100000018c3/article-14.xml v965>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.282+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ad0 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-45.jpg auth v967 ap=1 s=60142 n(v0 rc2024-06-10T00:30:46.246418+0000 b60142 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed60000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ad0:[<0x100000018c3/picture-45.jpg v967>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ad0:[<0x100000018c3/picture-45.jpg v967>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.562 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.282+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ad9 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-41.jpg auth v969 ap=1 s=39360 n(v0 rc2024-06-10T00:30:46.265418+0000 b39360 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed60b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ad9:[<0x100000018c3/picture-41.jpg v969>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ad9:[<0x100000018c3/picture-41.jpg v969>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.563 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.282+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a64 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-60.jpg auth v971 ap=1 s=63508 n(v0 rc2024-06-10T00:30:45.809424+0000 b63508 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed61600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a64:[<0x100000018c3/picture-60.jpg v971>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a64:[<0x100000018c3/picture-60.jpg v971>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.563 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.283+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a30 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-66.jpg auth v975 ap=1 s=38727 n(v0 rc2024-06-10T00:30:45.623426+0000 b38727 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed63180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a30:[<0x100000018c3/picture-66.jpg v975>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a30:[<0x100000018c3/picture-66.jpg v975>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.563 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.283+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000195a [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-47.xml auth v977 ap=1 s=15409 n(v0 rc2024-06-10T00:30:44.475441+0000 b15409 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed6a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000195a:[<0x100000018c3/article-47.xml v977>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000195a:[<0x100000018c3/article-47.xml v977>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.563 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.283+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000018e2 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-71.xml auth v979 ap=1 s=26180 n(v0 rc2024-06-10T00:30:39.489506+0000 b26180 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed6ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000018e2:[<0x100000018c3/article-71.xml v979>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x100000018e2:[<0x100000018c3/article-71.xml v979>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.563 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.283+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b17 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-22.jpg auth v981 ap=1 s=18030 n(v0 rc2024-06-10T00:30:46.547414+0000 b18030 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed6b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b17:[<0x100000018c3/picture-22.jpg v981>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b17:[<0x100000018c3/picture-22.jpg v981>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.563 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.283+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019e7 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-11.xml auth v983 ap=1 s=11889 n(v0 rc2024-06-10T00:30:45.251431+0000 b11889 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed6c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019e7:[<0x100000018c3/article-11.xml v983>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019e7:[<0x100000018c3/article-11.xml v983>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.563 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.283+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019d3 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-16.xml auth v985 ap=1 s=3182 n(v0 rc2024-06-10T00:30:45.214431+0000 b3182 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed6cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019d3:[<0x100000018c3/article-16.xml v985>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019d3:[<0x100000018c3/article-16.xml v985>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.563 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.283+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000193b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-54.xml auth v987 ap=1 s=25863 n(v0 rc2024-06-10T00:30:43.860449+0000 b25863 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed6d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000193b:[<0x100000018c3/article-54.xml v987>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000193b:[<0x100000018c3/article-54.xml v987>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.563 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.284+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019ab [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-26.xml auth v989 ap=1 s=19322 n(v0 rc2024-06-10T00:30:45.133432+0000 b19322 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed70580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019ab:[<0x100000018c3/article-26.xml v989>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x100000019ab:[<0x100000018c3/article-26.xml v989>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.563 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.284+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b33 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-15.jpg auth v991 ap=1 s=64428 n(v0 rc2024-06-10T00:30:46.628413+0000 b64428 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed71080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b33:[<0x100000018c3/picture-15.jpg v991>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b33:[<0x100000018c3/picture-15.jpg v991>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.563 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.284+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a86 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-56.jpg auth v993 ap=1 s=63671 n(v0 rc2024-06-10T00:30:46.008421+0000 b63671 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed71b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a86:[<0x100000018c3/picture-56.jpg v993>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a86:[<0x100000018c3/picture-56.jpg v993>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.564 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.284+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001acb [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-47.jpg auth v995 ap=1 s=62146 n(v0 rc2024-06-10T00:30:46.234418+0000 b62146 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed72680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001acb:[<0x100000018c3/picture-47.jpg v995>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001acb:[<0x100000018c3/picture-47.jpg v995>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.564 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.284+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019eb [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-10.xml auth v997 ap=1 s=11348 n(v0 rc2024-06-10T00:30:45.258431+0000 b11348 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed73180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019eb:[<0x100000018c3/article-10.xml v997>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019eb:[<0x100000018c3/article-10.xml v997>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.564 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.284+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b3f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-12.jpg auth v999 ap=1 s=14343 n(v0 rc2024-06-10T00:30:46.667413+0000 b14343 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed78000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b3f:[<0x100000018c3/picture-12.jpg v999>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b3f:[<0x100000018c3/picture-12.jpg v999>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.564 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.284+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ad2 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-44.jpg auth v1001 ap=1 s=23623 n(v0 rc2024-06-10T00:30:46.251418+0000 b23623 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed78b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ad2:[<0x100000018c3/picture-44.jpg v1001>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ad2:[<0x100000018c3/picture-44.jpg v1001>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.564 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.285+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001956 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-48.xml auth v1003 ap=1 s=22690 n(v0 rc2024-06-10T00:30:44.450441+0000 b22690 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed79600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001956:[<0x100000018c3/article-48.xml v1003>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001956:[<0x100000018c3/article-48.xml v1003>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.564 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.285+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a03 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-4.xml auth v1005 ap=1 s=27210 n(v0 rc2024-06-10T00:30:45.303430+0000 b27210 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed7a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a03:[<0x100000018c3/article-4.xml v1005>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a03:[<0x100000018c3/article-4.xml v1005>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.564 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.285+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019af [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-25.xml auth v1007 ap=1 s=27765 n(v0 rc2024-06-10T00:30:45.142432+0000 b27765 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed7ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019af:[<0x100000018c3/article-25.xml v1007>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x100000019af:[<0x100000018c3/article-25.xml v1007>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.564 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.285+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a6d [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-59.jpg auth v1009 ap=1 s=20729 n(v0 rc2024-06-10T00:30:45.855423+0000 b20729 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed7b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a6d:[<0x100000018c3/picture-59.jpg v1009>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a6d:[<0x100000018c3/picture-59.jpg v1009>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.564 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.285+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000197b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-38.xml auth v1011 ap=1 s=8481 n(v0 rc2024-06-10T00:30:45.021434+0000 b8481 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed80580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000197b:[<0x100000018c3/article-38.xml v1011>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000197b:[<0x100000018c3/article-38.xml v1011>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.564 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.285+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000018dd [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-72.xml auth v963 ap=1 s=19070 n(v0 rc2024-06-10T00:30:39.462506+0000 b19070 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed81080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000018dd:[<0x100000018c3/article-72.xml v963>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x100000018dd:[<0x100000018c3/article-72.xml v963>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.564 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.286+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001987 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-35.xml auth v967 ap=1 s=26065 n(v0 rc2024-06-10T00:30:45.053433+0000 b26065 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed82c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001987:[<0x100000018c3/article-35.xml v967>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001987:[<0x100000018c3/article-35.xml v967>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.565 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.286+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001975 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-40.xml auth v969 ap=1 s=14845 n(v0 rc2024-06-10T00:30:44.989434+0000 b14845 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed83700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001975:[<0x100000018c3/article-40.xml v969>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001975:[<0x100000018c3/article-40.xml v969>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.565 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.286+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019d7 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-15.xml auth v973 ap=1 s=17674 n(v0 rc2024-06-10T00:30:45.221431+0000 b17674 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed89600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019d7:[<0x100000018c3/article-15.xml v973>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019d7:[<0x100000018c3/article-15.xml v973>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.565 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.286+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ae5 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-36.jpg auth v975 ap=1 s=12155 n(v0 rc2024-06-10T00:30:46.293417+0000 b12155 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed8a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ae5:[<0x100000018c3/picture-36.jpg v975>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ae5:[<0x100000018c3/picture-36.jpg v975>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.565 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.286+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ad5 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-43.jpg auth v977 ap=1 s=32381 n(v0 rc2024-06-10T00:30:46.257418+0000 b32381 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed8ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ad5:[<0x100000018c3/picture-43.jpg v977>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ad5:[<0x100000018c3/picture-43.jpg v977>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.565 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.287+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b1f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-20.jpg auth v979 ap=1 s=39387 n(v0 rc2024-06-10T00:30:46.565414+0000 b39387 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed90580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b1f:[<0x100000018c3/picture-20.jpg v979>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b1f:[<0x100000018c3/picture-20.jpg v979>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.565 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.287+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000196d [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-42.xml auth v981 ap=1 s=7333 n(v0 rc2024-06-10T00:30:44.838436+0000 b7333 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed91080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000196d:[<0x100000018c3/article-42.xml v981>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000196d:[<0x100000018c3/article-42.xml v981>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.565 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.287+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001adf [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-38.jpg auth v983 ap=1 s=62697 n(v0 rc2024-06-10T00:30:46.281418+0000 b62697 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed91b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001adf:[<0x100000018c3/picture-38.jpg v983>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001adf:[<0x100000018c3/picture-38.jpg v983>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.565 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.287+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000190a [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-64.xml auth v972 ap=1 s=25856 n(v0 rc2024-06-10T00:30:39.644504+0000 b25856 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed92680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000190a:[<0x100000018c3/article-64.xml v972>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x1000000190a:[<0x100000018c3/article-64.xml v972>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.565 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.287+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000018cc [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-75.xml auth v974 ap=1 s=8238 n(v0 rc2024-06-10T00:30:39.377507+0000 b8238 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed93180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000018cc:[<0x100000018c3/article-75.xml v974>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x100000018cc:[<0x100000018c3/article-75.xml v974>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.565 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.287+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000197f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-37.xml auth v976 ap=1 s=9402 n(v0 rc2024-06-10T00:30:45.032434+0000 b9402 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed9a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000197f:[<0x100000018c3/article-37.xml v976>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x1000000197f:[<0x100000018c3/article-37.xml v976>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.566 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.287+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b03 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-27.jpg auth v978 ap=1 s=61756 n(v0 rc2024-06-10T00:30:46.469415+0000 b61756 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed9ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b03:[<0x100000018c3/picture-27.jpg v978>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b03:[<0x100000018c3/picture-27.jpg v978>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.566 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.288+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000018d7 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-73.xml auth v980 ap=1 s=18016 n(v0 rc2024-06-10T00:30:39.439506+0000 b18016 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed9b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000018d7:[<0x100000018c3/article-73.xml v980>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x100000018d7:[<0x100000018c3/article-73.xml v980>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.566 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.288+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001977 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-39.xml auth v982 ap=1 s=25174 n(v0 rc2024-06-10T00:30:45.013434+0000 b25174 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed9c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001977:[<0x100000018c3/article-39.xml v982>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001977:[<0x100000018c3/article-39.xml v982>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.566 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.288+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001983 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-36.xml auth v984 ap=1 s=11526 n(v0 rc2024-06-10T00:30:45.043434+0000 b11526 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed9cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001983:[<0x100000018c3/article-36.xml v984>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x10000001983:[<0x100000018c3/article-36.xml v984>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.566 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.288+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019e3 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-12.xml auth v986 ap=1 s=10777 n(v0 rc2024-06-10T00:30:45.243431+0000 b10777 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ed9d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019e3:[<0x100000018c3/article-12.xml v986>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019e3:[<0x100000018c3/article-12.xml v986>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.566 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.288+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001aeb [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-34.jpg auth v990 ap=1 s=26601 n(v0 rc2024-06-10T00:30:46.323417+0000 b26601 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2eda3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001aeb:[<0x100000018c3/picture-34.jpg v990>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001aeb:[<0x100000018c3/picture-34.jpg v990>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.566 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.289+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a0f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-1.xml auth v988 ap=1 s=9864 n(v0 rc2024-06-10T00:30:45.344430+0000 b9864 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2eda3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a0f:[<0x100000018c3/article-1.xml v988>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a0f:[<0x100000018c3/article-1.xml v988>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.568 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.289+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001aa2 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-53.jpg auth v992 ap=1 s=49731 n(v0 rc2024-06-10T00:30:46.128420+0000 b49731 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2eda4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001aa2:[<0x100000018c3/picture-53.jpg v992>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001aa2:[<0x100000018c3/picture-53.jpg v992>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.568 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.289+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001a29 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-67.jpg auth v994 ap=1 s=32243 n(v0 rc2024-06-10T00:30:45.574427+0000 b32243 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2eda5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001a29:[<0x100000018c3/picture-67.jpg v994>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001a29:[<0x100000018c3/picture-67.jpg v994>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.568 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.289+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000018ff [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-66.xml auth v996 ap=1 s=29539 n(v0 rc2024-06-10T00:30:39.597504+0000 b29539 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2edaa000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000018ff:[<0x100000018c3/article-66.xml v996>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x100000018ff:[<0x100000018c3/article-66.xml v996>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.568 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.289+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001b2f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-16.jpg auth v998 ap=1 s=40156 n(v0 rc2024-06-10T00:30:46.614413+0000 b40156 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2edaab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001b2f:[<0x100000018c3/picture-16.jpg v998>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001b2f:[<0x100000018c3/picture-16.jpg v998>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.568 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.289+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019b7 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-23.xml auth v1002 ap=1 s=19775 n(v0 rc2024-06-10T00:30:45.159432+0000 b19775 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2edac100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019b7:[<0x100000018c3/article-23.xml v1002>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2403>,<0x10000000201/src v2691>,<0x10000000006/blogbench-1.0 v2275>,<0x10000000005/tmp v511>,<0x100000001fb/client.0 v500>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v493>,<0x10000000001/sv_1 v508>,<0x10000000000/qa v491>,<0x1/volumes v480>]//[3]","memoryvalue":"(4)0x100000019b7:[<0x100000018c3/article-23.xml v1002>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.568 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.290+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000019cf [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-17.xml auth v1004 ap=1 s=28689 n(v0 rc2024-06-10T00:30:45.206431+0000 b28689 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2edacc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000019cf:[<0x100000018c3/article-17.xml v1004>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2413>,<0x10000000201/src v2696>,<0x10000000006/blogbench-1.0 v2279>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x100000019cf:[<0x100000018c3/article-17.xml v1004>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.568 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.290+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001af1 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-32.jpg auth v1006 ap=1 s=11116 n(v0 rc2024-06-10T00:30:46.354417+0000 b11116 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2edad700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001af1:[<0x100000018c3/picture-32.jpg v1006>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001af1:[<0x100000018c3/picture-32.jpg v1006>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.568 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.290+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001910 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-63.xml auth v1008 ap=1 s=15958 n(v0 rc2024-06-10T00:30:39.676503+0000 b15958 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2edb2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001910:[<0x100000018c3/article-63.xml v1008>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2391>,<0x10000000201/src v2687>,<0x10000000006/blogbench-1.0 v2271>,<0x10000000005/tmp v507>,<0x100000001fb/client.0 v496>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v489>,<0x10000000001/sv_1 v504>,<0x10000000000/qa v487>,<0x1/volumes v476>]//[3]","memoryvalue":"(4)0x10000001910:[<0x100000018c3/article-63.xml v1008>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.568 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.290+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001ac9 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-48.jpg auth v1010 ap=1 s=20355 n(v0 rc2024-06-10T00:30:46.230418+0000 b20355 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2edb3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001ac9:[<0x100000018c3/picture-48.jpg v1010>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001ac9:[<0x100000018c3/picture-48.jpg v1010>,<0x10000000399/blog-34 v146605>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.477+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001581 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg auth v1420 ap=1 s=43665 n(v0 rc2024-06-10T00:30:35.898552+0000 b43665 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef0a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001581:[<0x100000007cd/picture-13.jpg v1420>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v471>,<0x100000001fb/client.0 v458>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v451>,<0x10000000001/sv_1 v465>,<0x10000000000/qa v444>,<0x1/volumes v434>]//[3]","memoryvalue":"(4)0x10000001581:[<0x100000007cd/picture-13.jpg v1420>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.477+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001422 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-56.xml auth v1414 ap=1 s=4977 n(v0 rc2024-06-10T00:30:34.224574+0000 b4977 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef0a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001422:[<0x100000007cd/comment-56.xml v1414>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v469>,<0x100000001fb/client.0 v455>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v449>,<0x10000000001/sv_1 v463>,<0x10000000000/qa v441>,<0x1/volumes v432>]//[3]","memoryvalue":"(4)0x10000001422:[<0x100000007cd/comment-56.xml v1414>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.480+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001440 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-50.xml auth v1385 ap=1 s=2199 n(v0 rc2024-06-10T00:30:34.368572+0000 b2199 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ef77080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001440:[<0x100000007cd/comment-50.xml v1385>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v469>,<0x100000001fb/client.0 v455>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v449>,<0x10000000001/sv_1 v463>,<0x10000000000/qa v441>,<0x1/volumes v432>]//[3]","memoryvalue":"(4)0x10000001440:[<0x100000007cd/comment-50.xml v1385>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.481+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001449 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-48.xml auth v1387 ap=1 s=9321 n(v0 rc2024-06-10T00:30:34.410572+0000 b9321 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x558e2ef77b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001449:[<0x100000007cd/comment-48.xml v1387>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v471>,<0x100000001fb/client.0 v458>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v451>,<0x10000000001/sv_1 v465>,<0x10000000000/qa v444>,<0x1/volumes v434>]//[3]","memoryvalue":"(4)0x10000001449:[<0x100000007cd/comment-48.xml v1387>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.482+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000140f [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-58.xml auth v1408 ap=1 s=8657 n(v0 rc2024-06-10T00:30:34.080576+0000 b8657 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef1cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000140f:[<0x100000007cd/comment-58.xml v1408>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2245>,<0x10000000201/src v2582>,<0x10000000006/blogbench-1.0 v2227>,<0x10000000005/tmp v467>,<0x100000001fb/client.0 v453>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v447>,<0x10000000001/sv_1 v461>,<0x10000000000/qa v439>,<0x1/volumes v430>]//[3]","memoryvalue":"(4)0x1000000140f:[<0x100000007cd/comment-58.xml v1408>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.482+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001429 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-55.xml auth v1410 ap=1 s=3520 n(v0 rc2024-06-10T00:30:34.258574+0000 b3520 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef1d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001429:[<0x100000007cd/comment-55.xml v1410>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v469>,<0x100000001fb/client.0 v455>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v449>,<0x10000000001/sv_1 v463>,<0x10000000000/qa v441>,<0x1/volumes v432>]//[3]","memoryvalue":"(4)0x10000001429:[<0x100000007cd/comment-55.xml v1410>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.484+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001433 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-53.xml auth v1430 ap=1 s=2505 n(v0 rc2024-06-10T00:30:34.303573+0000 b2505 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef4f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001433:[<0x100000007cd/comment-53.xml v1430>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v469>,<0x100000001fb/client.0 v455>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v449>,<0x10000000001/sv_1 v463>,<0x10000000000/qa v441>,<0x1/volumes v432>]//[3]","memoryvalue":"(4)0x10000001433:[<0x100000007cd/comment-53.xml v1430>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.485+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001418 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-57.xml auth v1428 ap=1 s=6515 n(v0 rc2024-06-10T00:30:34.137575+0000 b6515 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef58580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001418:[<0x100000007cd/comment-57.xml v1428>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2245>,<0x10000000201/src v2582>,<0x10000000006/blogbench-1.0 v2227>,<0x10000000005/tmp v467>,<0x100000001fb/client.0 v453>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v447>,<0x10000000001/sv_1 v461>,<0x10000000000/qa v439>,<0x1/volumes v430>]//[3]","memoryvalue":"(4)0x10000001418:[<0x100000007cd/comment-57.xml v1428>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.486+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000145d [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-44.xml auth v1441 ap=1 s=7270 n(v0 rc2024-06-10T00:30:34.489571+0000 b7270 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef42100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000145d:[<0x100000007cd/comment-44.xml v1441>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v471>,<0x100000001fb/client.0 v458>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v451>,<0x10000000001/sv_1 v465>,<0x10000000000/qa v444>,<0x1/volumes v434>]//[3]","memoryvalue":"(4)0x1000000145d:[<0x100000007cd/comment-44.xml v1441>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.487+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001435 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-52.xml auth v1437 ap=1 s=9232 n(v0 rc2024-06-10T00:30:34.325573+0000 b9232 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef46580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001435:[<0x100000007cd/comment-52.xml v1437>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v469>,<0x100000001fb/client.0 v455>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v449>,<0x10000000001/sv_1 v463>,<0x10000000000/qa v441>,<0x1/volumes v432>]//[3]","memoryvalue":"(4)0x10000001435:[<0x100000007cd/comment-52.xml v1437>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.487+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001460 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-43.xml auth v1443 ap=1 s=7527 n(v0 rc2024-06-10T00:30:34.510570+0000 b7527 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef47b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001460:[<0x100000007cd/comment-43.xml v1443>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v471>,<0x100000001fb/client.0 v458>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v451>,<0x10000000001/sv_1 v465>,<0x10000000000/qa v444>,<0x1/volumes v434>]//[3]","memoryvalue":"(4)0x10000001460:[<0x100000007cd/comment-43.xml v1443>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.569 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.488+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001457 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-45.xml auth v1439 ap=1 s=9988 n(v0 rc2024-06-10T00:30:34.470571+0000 b9988 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef49700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001457:[<0x100000007cd/comment-45.xml v1439>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v471>,<0x100000001fb/client.0 v458>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v451>,<0x10000000001/sv_1 v465>,<0x10000000000/qa v444>,<0x1/volumes v434>]//[3]","memoryvalue":"(4)0x10000001457:[<0x100000007cd/comment-45.xml v1439>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.570 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.515+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001452 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-46.xml auth v1422 ap=1 s=3284 n(v0 rc2024-06-10T00:30:34.451571+0000 b3284 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef2f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001452:[<0x100000007cd/comment-46.xml v1422>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v471>,<0x100000001fb/client.0 v458>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v451>,<0x10000000001/sv_1 v465>,<0x10000000000/qa v444>,<0x1/volumes v434>]//[3]","memoryvalue":"(4)0x10000001452:[<0x100000007cd/comment-46.xml v1422>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.570 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.515+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000144e [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-47.xml auth v1420 ap=1 s=6996 n(v0 rc2024-06-10T00:30:34.430571+0000 b6996 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef2eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000144e:[<0x100000007cd/comment-47.xml v1420>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v471>,<0x100000001fb/client.0 v458>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v451>,<0x10000000001/sv_1 v465>,<0x10000000000/qa v444>,<0x1/volumes v434>]//[3]","memoryvalue":"(4)0x1000000144e:[<0x100000007cd/comment-47.xml v1420>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.570 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.545+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000142d [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-54.xml auth v1429 ap=1 s=8240 n(v0 rc2024-06-10T00:30:34.282573+0000 b8240 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef8eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000142d:[<0x100000007cd/comment-54.xml v1429>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v469>,<0x100000001fb/client.0 v455>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v449>,<0x10000000001/sv_1 v463>,<0x10000000000/qa v441>,<0x1/volumes v432>]//[3]","memoryvalue":"(4)0x1000000142d:[<0x100000007cd/comment-54.xml v1429>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.570 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.545+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001beb [4,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-42.xml auth v1265 ap=1 s=5575 n(v0 rc2024-06-10T00:30:47.136406+0000 b5575 1=1+0) | authpin=1 scrubqueue=0 0x558e2c64ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001beb:[<0x100000007cd/article-42.xml v1265>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2417>,<0x10000000201/src v2698>,<0x10000000006/blogbench-1.0 v2281>,<0x10000000005/tmp v515>,<0x100000001fb/client.0 v502>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v495>,<0x10000000001/sv_1 v510>,<0x10000000000/qa v493>,<0x1/volumes v482>]//[3]","memoryvalue":"(4)0x10000001beb:[<0x100000007cd/article-42.xml v1265>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.570 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.558+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000001443 [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-49.xml auth v1425 ap=1 s=5617 n(v0 rc2024-06-10T00:30:34.389572+0000 b5617 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2ef9d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000001443:[<0x100000007cd/comment-49.xml v1425>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v471>,<0x100000001fb/client.0 v458>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v451>,<0x10000000001/sv_1 v465>,<0x10000000000/qa v444>,<0x1/volumes v434>]//[3]","memoryvalue":"(4)0x10000001443:[<0x100000007cd/comment-49.xml v1425>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:58.852 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:35:58.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: pgmap v245: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 307 op/s 2024-06-10T00:35:58.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:58.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001926 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-59.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b3b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019cb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-18.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b27 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000194b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-51.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b23 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019a7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-27.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b5e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ae8 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000018e8 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-70.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b64 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b37 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000191b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-61.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001971 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-41.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001961 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-45.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a5c (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-61.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b53 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.951 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b4b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001997 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-31.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a42 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-64.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b47 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b68 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019ef (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-9.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001965 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-44.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b57 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001904 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-65.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001915 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-62.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019b3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-24.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001aff (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000018f9 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-67.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.952 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ae2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b2b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a07 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-3.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b0f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001aee (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a7f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019f7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-7.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ad7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-42.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a1e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-68.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001add (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-39.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001937 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-55.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b0b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-25.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019f3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-8.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.953 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001af4 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-31.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000018f3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-68.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ac6 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-49.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b4f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-8.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000018c5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-76.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b07 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-26.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001993 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-32.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019bf (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-21.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000199b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-30.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000194f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-50.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001e5d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-13.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019fb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-6.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.954 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a0b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-2.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a9b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-54.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001afb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-29.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001933 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-56.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001969 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-43.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000198f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-33.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b5a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-5.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ab5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-50.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.955 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a77 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-58.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000192e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-57.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a38 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-65.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000192a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-58.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001940 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-53.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019a3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-28.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.956 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001947 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-52.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a13 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-69.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ab1 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-51.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a53 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-62.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a8f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-55.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b61 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-3.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b43 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-11.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000198b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-34.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000199f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-29.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001920 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-60.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a4a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-63.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001adb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-40.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019bb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-22.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001953 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-49.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.957 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b13 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-23.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019db (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-14.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ad0 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-45.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ad9 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-41.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a64 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-60.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a30 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-66.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000195a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-47.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000018e2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-71.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b17 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-22.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019e7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-11.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019d3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-16.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000193b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-54.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.958 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019ab (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-26.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b33 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-15.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a86 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-56.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001acb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-47.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019eb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-10.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b3f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-12.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ad2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-44.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001956 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-48.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a03 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-4.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019af (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-25.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a6d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-59.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000197b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-38.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000018dd (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-72.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.959 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001987 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-35.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001975 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-40.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019d7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-15.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ae5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-36.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ad5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-43.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b1f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-20.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000196d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-42.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001adf (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-38.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000190a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-64.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000018cc (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-75.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000197f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-37.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b03 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-27.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000018d7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-73.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001977 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-39.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.960 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001983 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-36.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019e3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-12.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001aeb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-34.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a0f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-1.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001aa2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-53.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001a29 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-67.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000018ff (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-66.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001b2f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-16.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019b7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-23.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x100000019cf (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-17.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001af1 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-32.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001910 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-63.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001ac9 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-48.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001581 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.961 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001422 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-56.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001440 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-50.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001449 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-48.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000140f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-58.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001429 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-55.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001433 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-53.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001418 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-57.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000145d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-44.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001435 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-52.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001460 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-43.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001457 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-45.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001452 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-46.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000144e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-47.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.962 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000142d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-54.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.963 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001beb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-42.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.963 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x10000001443 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-49.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:58.963 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:58 smithi121 ceph-mon[32378]: Scrub error on inode 0x1000000143b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-51.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: pgmap v245: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 307 op/s 2024-06-10T00:35:59.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:59.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001926 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-59.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b3b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019cb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-18.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b27 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000194b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-51.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b23 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019a7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-27.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b5e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ae8 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000018e8 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-70.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b64 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b37 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000191b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-61.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001971 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-41.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.009 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001961 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-45.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a5c (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-61.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b53 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b4b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001997 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-31.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a42 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-64.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b47 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b68 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019ef (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-9.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001965 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-44.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.010 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b57 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001904 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-65.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001915 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-62.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019b3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-24.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001aff (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000018f9 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-67.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.011 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ae2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.011 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:35:58.559+0000 7f8fbedb5700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000143b [6,head] /volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-51.xml auth v1423 ap=1 s=7330 n(v0 rc2024-06-10T00:30:34.347572+0000 b7330 1=1+0) caps={24871=pAsLsXsFscr/pFscr@1},l=24871 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x558e2efa3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000143b:[<0x100000007cd/comment-51.xml v1423>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2247>,<0x10000000201/src v2585>,<0x10000000006/blogbench-1.0 v2229>,<0x10000000005/tmp v469>,<0x100000001fb/client.0 v455>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v449>,<0x10000000001/sv_1 v463>,<0x10000000000/qa v441>,<0x1/volumes v432>]//[3]","memoryvalue":"(4)0x1000000143b:[<0x100000007cd/comment-51.xml v1423>,<0x10000000399/blog-12 v147797>,<0x10000000208/blogtest_in v2452>,<0x10000000201/src v2713>,<0x10000000006/blogbench-1.0 v2293>,<0x10000000005/tmp v528>,<0x100000001fb/client.0 v510>,<0x100000001fa/3fc9b7f7-a431-4fce-8235-be0cc96de6ca v453>,<0x10000000001/sv_1 v458>,<0x10000000000/qa v446>,<0x1/volumes v452>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-06-10T00:35:59.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b2b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a07 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-3.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b0f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.012 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001aee (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a7f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019f7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-7.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ad7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-42.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a1e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-68.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001add (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-39.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001937 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-55.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b0b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-25.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019f3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-8.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001af4 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-31.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000018f3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-68.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ac6 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-49.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b4f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-8.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000018c5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-76.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.013 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b07 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-26.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001993 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-32.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019bf (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-21.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000199b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-30.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000194f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-50.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001e5d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-13.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019fb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-6.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a0b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-2.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a9b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-54.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001afb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-29.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001933 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-56.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001969 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-43.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000198f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-33.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.014 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b5a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-5.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ab5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-50.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a77 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-58.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000192e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-57.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a38 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-65.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000192a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-58.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001940 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-53.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019a3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-28.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001947 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-52.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a13 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-69.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ab1 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-51.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a53 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-62.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a8f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-55.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.015 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b61 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-3.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b43 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-11.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000198b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-34.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000199f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-29.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001920 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-60.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a4a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-63.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001adb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-40.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019bb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-22.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001953 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-49.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b13 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-23.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019db (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-14.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ad0 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-45.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ad9 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-41.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a64 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-60.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a30 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-66.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.016 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000195a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-47.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000018e2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-71.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b17 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-22.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019e7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-11.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019d3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-16.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000193b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-54.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019ab (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-26.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b33 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-15.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a86 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-56.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001acb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-47.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019eb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-10.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b3f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-12.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ad2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-44.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001956 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-48.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.017 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a03 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-4.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019af (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-25.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a6d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-59.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000197b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-38.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000018dd (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-72.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001987 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-35.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001975 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-40.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019d7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-15.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ae5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-36.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ad5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-43.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b1f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-20.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000196d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-42.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001adf (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-38.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000190a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-64.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.018 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000018cc (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-75.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000197f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-37.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b03 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-27.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000018d7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-73.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001977 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-39.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001983 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-36.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019e3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-12.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001aeb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-34.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a0f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-1.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001aa2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-53.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001a29 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-67.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000018ff (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-66.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001b2f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-16.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019b7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-23.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.019 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x100000019cf (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-17.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001af1 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-32.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001910 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-63.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001ac9 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-48.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001581 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001422 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-56.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001440 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-50.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001449 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-48.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000140f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-58.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001429 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-55.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001433 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-53.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001418 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-57.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000145d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-44.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001435 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-52.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.020 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001460 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-43.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.021 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001457 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-45.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.021 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001452 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-46.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.021 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000144e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-47.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.021 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000142d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-54.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.021 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001beb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-42.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.021 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x10000001443 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-49.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.021 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:58 smithi135 ceph-mon[36424]: Scrub error on inode 0x1000000143b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-51.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: pgmap v245: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 307 op/s 2024-06-10T00:35:59.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:35:59.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001926 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-59.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b3b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019cb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-18.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b27 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-18.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000194b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-51.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b23 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-19.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019a7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-27.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b5e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-4.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ae8 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-35.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000018e8 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-70.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b64 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-2.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b37 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-14.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000191b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-61.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001971 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-41.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001961 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-45.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a5c (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-61.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b53 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-7.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b4b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-9.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001997 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-31.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a42 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-64.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b47 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-10.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b68 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-1.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019ef (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-9.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001965 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-44.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b57 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-6.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001904 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-65.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001915 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-62.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019b3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-24.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001aff (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-28.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000018f9 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-67.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ae2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-37.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b2b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-17.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a07 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-3.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b0f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-24.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001aee (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-33.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a7f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-57.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.134 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019f7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-7.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ad7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-42.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a1e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-68.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001add (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-39.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001937 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-55.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b0b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-25.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019f3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-8.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001af4 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-31.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000018f3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-68.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ac6 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-49.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b4f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-8.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000018c5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-76.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b07 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-26.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001993 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-32.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019bf (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-21.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000199b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-30.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000194f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-50.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001e5d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-13.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019fb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-6.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a0b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-2.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a9b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-54.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001afb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-29.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001933 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-56.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001969 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-43.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000198f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-33.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b5a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-5.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ab5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-50.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a77 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-58.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000192e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-57.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a38 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-65.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000192a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-58.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001940 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-53.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019a3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-28.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001947 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-52.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a13 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-69.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ab1 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-51.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a53 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-62.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a8f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-55.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b61 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-3.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b43 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-11.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000198b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-34.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000199f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-29.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.138 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001920 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-60.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a4a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-63.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001adb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-40.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019bb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-22.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001953 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-49.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b13 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-23.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019db (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-14.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ad0 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-45.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ad9 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-41.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a64 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-60.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a30 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-66.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000195a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-47.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000018e2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-71.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b17 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-22.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.139 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019e7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-11.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019d3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-16.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000193b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-54.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019ab (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-26.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b33 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-15.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a86 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-56.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001acb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-47.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019eb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-10.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b3f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-12.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ad2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-44.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001956 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-48.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a03 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-4.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019af (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-25.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.140 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a6d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-59.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000197b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-38.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000018dd (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-72.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001987 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-35.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001975 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-40.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019d7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-15.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ae5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-36.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ad5 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-43.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b1f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-20.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000196d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-42.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001adf (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-38.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000190a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-64.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000018cc (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-75.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000197f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-37.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.141 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b03 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-27.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000018d7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-73.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001977 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-39.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001983 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-36.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019e3 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-12.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001aeb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-34.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a0f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-1.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001aa2 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-53.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001a29 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-67.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000018ff (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-66.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001b2f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-16.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019b7 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-23.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.142 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x100000019cf (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-17.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001af1 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-32.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001910 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-63.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001ac9 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-48.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001581 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001422 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-56.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001440 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-50.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001449 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-48.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000140f (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-58.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001429 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-55.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001433 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-53.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001418 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-57.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000145d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-44.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.143 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001435 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-52.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.144 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001460 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-43.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.144 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001457 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-45.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.144 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001452 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-46.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.144 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000144e (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-47.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.144 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000142d (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-54.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.144 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001beb (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-42.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.144 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x10000001443 (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-49.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.144 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:58 smithi193 ceph-mon[36716]: Scrub error on inode 0x1000000143b (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-51.xml) see mds.h log and `damage ls` output for details 2024-06-10T00:35:59.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a[32374]: 2024-06-10T00:35:59.684+0000 7f460c847700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-06-10T00:35:59.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:59 smithi121 ceph-mon[32378]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-06-10T00:35:59.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:59 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] up:active 2024-06-10T00:35:59.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:35:59 smithi121 ceph-mon[32378]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:36:00.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:59 smithi135 ceph-mon[36424]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-06-10T00:36:00.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:59 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] up:active 2024-06-10T00:36:00.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:35:59 smithi135 ceph-mon[36424]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:36:00.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:59 smithi193 ceph-mon[36716]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-06-10T00:36:00.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:59 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] up:active 2024-06-10T00:36:00.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:35:59 smithi193 ceph-mon[36716]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-06-10T00:36:00.935 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:00 smithi121 ceph-mon[32378]: pgmap v246: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 380 op/s 2024-06-10T00:36:00.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.944+0000 7fd61fb36700 1 -- 172.21.15.121:0/2835754046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd618108520 msgr2=0x7fd618108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:00.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.944+0000 7fd61fb36700 1 --2- 172.21.15.121:0/2835754046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd618108520 0x7fd618108900 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fd60800b0a0 tx=0x7fd60800b3b0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.944+0000 7fd61fb36700 1 -- 172.21.15.121:0/2835754046 shutdown_connections 2024-06-10T00:36:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.944+0000 7fd61fb36700 1 --2- 172.21.15.121:0/2835754046 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd618102ed0 0x7fd61810f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.944+0000 7fd61fb36700 1 --2- 172.21.15.121:0/2835754046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd618108520 0x7fd618108900 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.944+0000 7fd61fb36700 1 --2- 172.21.15.121:0/2835754046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd618102510 0x7fd618102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.944+0000 7fd61fb36700 1 -- 172.21.15.121:0/2835754046 >> 172.21.15.121:0/2835754046 conn(0x7fd618079560 msgr2=0x7fd618079970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.944+0000 7fd61fb36700 1 -- 172.21.15.121:0/2835754046 shutdown_connections 2024-06-10T00:36:00.945 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.944+0000 7fd61fb36700 1 -- 172.21.15.121:0/2835754046 wait complete. 2024-06-10T00:36:00.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61fb36700 1 Processor -- start 2024-06-10T00:36:00.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61fb36700 1 -- start start 2024-06-10T00:36:00.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61fb36700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd618102510 0x7fd61819e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61fb36700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd618102ed0 0x7fd61819ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61fb36700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd618108520 0x7fd6181989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61fb36700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd6181127b0 con 0x7fd618102ed0 2024-06-10T00:36:00.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61fb36700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd618112630 con 0x7fd618108520 2024-06-10T00:36:00.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61fb36700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd618112930 con 0x7fd618102510 2024-06-10T00:36:00.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61d0d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd618102ed0 0x7fd61819ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61d0d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd618102ed0 0x7fd61819ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43488/0 (socket says 172.21.15.121:43488) 2024-06-10T00:36:00.947 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61d0d1700 1 -- 172.21.15.121:0/2849459793 learned_addr learned my addr 172.21.15.121:0/2849459793 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61d8d2700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd618102510 0x7fd61819e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61e0d3700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd618108520 0x7fd6181989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61d0d1700 1 -- 172.21.15.121:0/2849459793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd618102510 msgr2=0x7fd61819e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd618102510 0x7fd61819e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61d0d1700 1 -- 172.21.15.121:0/2849459793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd618108520 msgr2=0x7fd6181989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd618108520 0x7fd6181989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.945+0000 7fd61d0d1700 1 -- 172.21.15.121:0/2849459793 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd618199110 con 0x7fd618102ed0 2024-06-10T00:36:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.946+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd618102ed0 0x7fd61819ed30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd60c00c970 tx=0x7fd60c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.946+0000 7fd616ffd700 1 -- 172.21.15.121:0/2849459793 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c0078a0 con 0x7fd618102ed0 2024-06-10T00:36:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.946+0000 7fd616ffd700 1 -- 172.21.15.121:0/2849459793 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd60c004830 con 0x7fd618102ed0 2024-06-10T00:36:00.948 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.946+0000 7fd61fb36700 1 -- 172.21.15.121:0/2849459793 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd618199400 con 0x7fd618102ed0 2024-06-10T00:36:00.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.946+0000 7fd616ffd700 1 -- 172.21.15.121:0/2849459793 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c005540 con 0x7fd618102ed0 2024-06-10T00:36:00.949 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.946+0000 7fd61fb36700 1 -- 172.21.15.121:0/2849459793 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6181a56c0 con 0x7fd618102ed0 2024-06-10T00:36:00.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.948+0000 7fd616ffd700 1 -- 172.21.15.121:0/2849459793 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd60c0049e0 con 0x7fd618102ed0 2024-06-10T00:36:00.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.949+0000 7fd616ffd700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6000a2e10 0x7fd6000a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.949+0000 7fd61d8d2700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6000a2e10 0x7fd6000a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.949+0000 7fd616ffd700 1 -- 172.21.15.121:0/2849459793 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fd60c0cf5e0 con 0x7fd618102ed0 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.949+0000 7fd61d8d2700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6000a2e10 0x7fd6000a52d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd608009fd0 tx=0x7fd60800b7b0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd62a3b9b80 1 Processor -- start 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd62a3b9b80 1 -- start start 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40ddaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40dd49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40dd5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562b40d26810 con 0x562b40dd4f20 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x562b40cce720 con 0x562b40dda640 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562b40d227d0 con 0x562b40ddaf60 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd61e0d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40dd5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd61d8d2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40ddaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd61e0d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40dd5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43504/0 (socket says 172.21.15.121:43504) 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd61d8d2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40ddaa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:42908/0 (socket says 172.21.15.121:42908) 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd61e0d3700 1 -- 172.21.15.121:0/576452915 learned_addr learned my addr 172.21.15.121:0/576452915 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/576452915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40dd49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd61e0d3700 1 -- 172.21.15.121:0/576452915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 msgr2=0x562b40dd49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd61e0d3700 1 --2- 172.21.15.121:0/576452915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40dd49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd61e0d3700 1 -- 172.21.15.121:0/576452915 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 msgr2=0x562b40ddaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd61e0d3700 1 --2- 172.21.15.121:0/576452915 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40ddaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.950+0000 7fd61e0d3700 1 -- 172.21.15.121:0/576452915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b40cc4f50 con 0x562b40dd4f20 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.951+0000 7fd61e0d3700 1 --2- 172.21.15.121:0/576452915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40dd5340 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd610007c90 tx=0x7fd6100103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3586ec8e840e06e5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.951+0000 7fd5fffff700 1 -- 172.21.15.121:0/576452915 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610010bf0 con 0x562b40dd4f20 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.951+0000 7fd5fffff700 1 -- 172.21.15.121:0/576452915 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd610015070 con 0x562b40dd4f20 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.951+0000 7fd5fffff700 1 -- 172.21.15.121:0/576452915 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610003df0 con 0x562b40dd4f20 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.952+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/576452915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 msgr2=0x562b40dd5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:00.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.952+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/576452915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40dd5340 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd610007c90 tx=0x7fd6100103d0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.952+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/576452915 shutdown_connections 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.952+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/576452915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40dd5340 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.952+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/576452915 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40ddaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.952+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/576452915 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40dd49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.952+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/576452915 >> 172.21.15.121:0/576452915 conn(0x562b40cffcc0 msgr2=0x562b40dcf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.952+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/576452915 shutdown_connections 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.952+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/576452915 wait complete. 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.952+0000 7fd62a3b9b80 1 -- start start 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.952+0000 7fd62a3b9b80 1 Processor -- start 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40dddff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40dde530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562b40de19d0 con 0x562b40dd4f20 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x562b40cc6bf0 con 0x562b40dda640 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562b40de1b50 con 0x562b40ddaf60 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd62a3b9b80 10 client.0 _ll_register_callbacks cb 0x562b40cc0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd61d8d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd61d8d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43516/0 (socket says 172.21.15.121:43516) 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd61d8d2700 1 -- 172.21.15.121:0/1650111839 learned_addr learned my addr 172.21.15.121:0/1650111839 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40dddff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd61e0d3700 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40dde530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd61d8d2700 1 -- 172.21.15.121:0/1650111839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 msgr2=0x562b40dde530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd61d8d2700 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40dde530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd61d8d2700 1 -- 172.21.15.121:0/1650111839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 msgr2=0x562b40dddff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd61d8d2700 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40dddff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd61d8d2700 1 -- 172.21.15.121:0/1650111839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b40c758d0 con 0x562b40dd4f20 2024-06-10T00:36:00.960 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd61d8d2700 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de0cf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd60800b0a0 tx=0x7fd6080099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de65d5a865eb92b server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd5fd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/1650111839 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60802d070 con 0x562b40dd4f20 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/1650111839 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fd608007dd0 con 0x562b40dd4f20 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.953+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/1650111839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd610015070 con 0x562b40dd4f20 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/1650111839 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd60803b050 con 0x562b40dd4f20 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd62a3b9b80 10 client.16677 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/1650111839 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562b40dec330 con 0x562b40dd4f20 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/1650111839 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd610015070 con 0x562b40dd4f20 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/1650111839 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7fd608011070 con 0x562b40dd4f20 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd62a3b9b80 10 client.16677 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd62a3b9b80 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd62a3b9b80 20 client.16677 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562b40dee970 0x562b40df0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd62a3b9b80 4 client.16677 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:36:00.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/1650111839 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b40d33d10 con 0x562b40dee970 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.954+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562b40dee970 0x562b40df0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.955+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562b40dee970 0x562b40df0d80 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e03b97b04fdca104 server_cookie=30dded153078fc7e in_seq=0 out_seq=0 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.955+0000 7fd5fd7fa700 10 client.16677 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/1650111839 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x562b40d33d10 con 0x562b40dee970 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd5fd7fa700 10 client.16677 handle_command_reply: tid=0 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 1 client.16677 shutdown 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/1650111839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 msgr2=0x562b40de0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de0cf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd60800b0a0 tx=0x7fd6080099e0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/1650111839 shutdown_connections 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562b40dee970 0x562b40df0d80 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40dde530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de0cf0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/1650111839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40dddff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.962 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd5fd7fa700 0 client.16677 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:00.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/1650111839 >> 172.21.15.121:0/1650111839 conn(0x562b40cffcc0 msgr2=0x562b40dd39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:00.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/1650111839 shutdown_connections 2024-06-10T00:36:00.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/1650111839 wait complete. 2024-06-10T00:36:00.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.956+0000 7fd62a3b9b80 20 client.16677 trim_cache size 0 max 16384 2024-06-10T00:36:00.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.978+0000 7fd62a3b9b80 1 Processor -- start 2024-06-10T00:36:00.979 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.978+0000 7fd62a3b9b80 1 -- start start 2024-06-10T00:36:00.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.978+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dd4f20 0x562b40dd49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.978+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40dda640 0x562b40ddcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.978+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40ddaf60 0x562b40dc9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.978+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd6181b0f90 con 0x562b40ddaf60 2024-06-10T00:36:00.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.978+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd6181ae760 con 0x562b40dd4f20 2024-06-10T00:36:00.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.978+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6181ac5d0 con 0x562b40dda640 2024-06-10T00:36:00.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.978+0000 7fd61e0d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40ddaf60 0x562b40dc9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd61e0d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40ddaf60 0x562b40dc9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43530/0 (socket says 172.21.15.121:43530) 2024-06-10T00:36:00.980 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd61d0d1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40dda640 0x562b40ddcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd61e0d3700 1 -- 172.21.15.121:0/545940061 learned_addr learned my addr 172.21.15.121:0/545940061 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd61d8d2700 1 --2- 172.21.15.121:0/545940061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dd4f20 0x562b40dd49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd61d0d1700 1 -- 172.21.15.121:0/545940061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dd4f20 msgr2=0x562b40dd49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/545940061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dd4f20 0x562b40dd49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd61d0d1700 1 -- 172.21.15.121:0/545940061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40ddaf60 msgr2=0x562b40dc9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/545940061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40ddaf60 0x562b40dc9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd61d0d1700 1 -- 172.21.15.121:0/545940061 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b40ddc560 con 0x562b40dda640 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd61e0d3700 1 --2- 172.21.15.121:0/545940061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40ddaf60 0x562b40dc9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/545940061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40dda640 0x562b40ddcf50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0cc940 tx=0x7fd60c09e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c719200468a8eb84 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd5ff7fe700 1 -- 172.21.15.121:0/545940061 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c09fb50 con 0x562b40dda640 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd5ff7fe700 1 -- 172.21.15.121:0/545940061 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd60c0a4380 con 0x562b40dda640 2024-06-10T00:36:00.981 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.979+0000 7fd5ff7fe700 1 -- 172.21.15.121:0/545940061 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c09bc60 con 0x562b40dda640 2024-06-10T00:36:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.980+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/545940061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40dda640 msgr2=0x562b40ddcf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.980+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/545940061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40dda640 0x562b40ddcf50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0cc940 tx=0x7fd60c09e040 comp rx=0 tx=0).stop 2024-06-10T00:36:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.980+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/545940061 shutdown_connections 2024-06-10T00:36:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.980+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/545940061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40ddaf60 0x562b40dc9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.980+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/545940061 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dd4f20 0x562b40dd49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.980+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/545940061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40dda640 0x562b40ddcf50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.980+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/545940061 >> 172.21.15.121:0/545940061 conn(0x562b40cffcc0 msgr2=0x562b40ddb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.980+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/545940061 shutdown_connections 2024-06-10T00:36:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.980+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/545940061 wait complete. 2024-06-10T00:36:00.985 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.980+0000 7fd62a3b9b80 1 -- start start 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.980+0000 7fd62a3b9b80 1 Processor -- start 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40de7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd62a3b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40de3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd6181acc20 con 0x562b40dd4f20 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd6181ae090 con 0x562b40dda640 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd62a3b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6181ad2f0 con 0x562b40ddaf60 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd62a3b9b80 10 client.0 _ll_register_callbacks cb 0x562b40cc0920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd61d8d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd61d8d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:43532/0 (socket says 172.21.15.121:43532) 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd61d0d1700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40de7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd61d8d2700 1 -- 172.21.15.121:0/2533752510 learned_addr learned my addr 172.21.15.121:0/2533752510 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:00.986 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd61e0d3700 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40de3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd61d8d2700 1 -- 172.21.15.121:0/2533752510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 msgr2=0x562b40de3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd61d8d2700 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40de3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd61d8d2700 1 -- 172.21.15.121:0/2533752510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 msgr2=0x562b40de7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd61d8d2700 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40de7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd61d8d2700 1 -- 172.21.15.121:0/2533752510 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b40de4350 con 0x562b40dd4f20 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.981+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40de7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd61d8d2700 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de6e40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd608009e60 tx=0x7fd608029a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3deb75f6e36e4136 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd5fd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/2533752510 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd608044060 con 0x562b40dd4f20 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/2533752510 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9525+0+0 (secure 0 0 0) 0x7fd608007dd0 con 0x562b40dd4f20 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/2533752510 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd60c0a4380 con 0x562b40dd4f20 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/2533752510 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fd60804b050 con 0x562b40dd4f20 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd62a3b9b80 10 client.16683 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:00.987 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/2533752510 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562b40e5e1f0 con 0x562b40dd4f20 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/2533752510 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd60c0a4380 con 0x562b40dd4f20 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/2533752510 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7fd60803e030 con 0x562b40dd4f20 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd62a3b9b80 10 client.16683 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd62a3b9b80 10 client.16683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd62a3b9b80 20 client.16683 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562b40e5e8d0 0x562b40e60cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd62a3b9b80 4 client.16683 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.982+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/2533752510 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562b40d2e290 con 0x562b40e5e8d0 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562b40e5e8d0 0x562b40e60cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd5fd7fa700 10 client.16683 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd61d0d1700 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562b40e5e8d0 0x562b40e60cf0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6ba46e4842223c3 server_cookie=a70f50b7a0bdf418 in_seq=0 out_seq=0 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd5fd7fa700 1 -- 172.21.15.121:0/2533752510 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562b40d2e290 con 0x562b40e5e8d0 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd5fd7fa700 10 client.16683 handle_command_reply: tid=0 2024-06-10T00:36:00.988 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd62a3b9b80 1 client.16683 shutdown 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/2533752510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 msgr2=0x562b40de6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de6e40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd608009e60 tx=0x7fd608029a70 comp rx=0 tx=0).stop 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/2533752510 shutdown_connections 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x562b40e5e8d0 0x562b40e60cf0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b40ddaf60 0x562b40de3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b40dd4f20 0x562b40de6e40 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd62a3b9b80 1 --2- 172.21.15.121:0/2533752510 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x562b40dda640 0x562b40de7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.983+0000 7fd5fd7fa700 0 client.16683 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/2533752510 >> 172.21.15.121:0/2533752510 conn(0x562b40cffcc0 msgr2=0x562b40dc9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/2533752510 shutdown_connections 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd62a3b9b80 1 -- 172.21.15.121:0/2533752510 wait complete. 2024-06-10T00:36:00.989 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd62a3b9b80 20 client.16683 trim_cache size 0 max 16384 2024-06-10T00:36:00.990 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:36:00.990 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "IDLE (waiting for more scrubs)", 2024-06-10T00:36:00.990 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": { 2024-06-10T00:36:00.990 INFO:teuthology.orchestra.run.smithi121.stdout: "e235dce8-6480-42f8-bb4d-df404081360e": { 2024-06-10T00:36:00.990 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/", 2024-06-10T00:36:00.990 INFO:teuthology.orchestra.run.smithi121.stdout: "tag": "e235dce8-6480-42f8-bb4d-df404081360e", 2024-06-10T00:36:00.990 INFO:teuthology.orchestra.run.smithi121.stdout: "options": "recursive,force" 2024-06-10T00:36:00.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:00.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:00.990 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:36:00.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 -- 172.21.15.121:0/2849459793 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6000a2e10 msgr2=0x7fd6000a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:00.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6000a2e10 0x7fd6000a52d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd608009fd0 tx=0x7fd60800b7b0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 -- 172.21.15.121:0/2849459793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd618102ed0 msgr2=0x7fd61819ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:00.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd618102ed0 0x7fd61819ed30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd60c00c970 tx=0x7fd60c00cc80 comp rx=0 tx=0).stop 2024-06-10T00:36:00.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 -- 172.21.15.121:0/2849459793 shutdown_connections 2024-06-10T00:36:00.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6000a2e10 0x7fd6000a52d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd618108520 0x7fd6181989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd618102510 0x7fd61819e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.991 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 --2- 172.21.15.121:0/2849459793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd618102ed0 0x7fd61819ed30 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:00.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 -- 172.21.15.121:0/2849459793 >> 172.21.15.121:0/2849459793 conn(0x7fd618079560 msgr2=0x7fd618101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:00.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 -- 172.21.15.121:0/2849459793 shutdown_connections 2024-06-10T00:36:00.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:00.984+0000 7fd61fb36700 1 -- 172.21.15.121:0/2849459793 wait complete. 2024-06-10T00:36:01.007 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:00 smithi135 ceph-mon[36424]: pgmap v246: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 380 op/s 2024-06-10T00:36:01.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:00 smithi193 ceph-mon[36716]: pgmap v246: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 380 op/s 2024-06-10T00:36:01.768 INFO:tasks.cephfs.filesystem:scrub status for tag:e235dce8-6480-42f8-bb4d-df404081360e - {'path': '/', 'tag': 'e235dce8-6480-42f8-bb4d-df404081360e', 'options': 'recursive,force'} 2024-06-10T00:36:03.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:02 smithi135 ceph-mon[36424]: pgmap v247: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 335 op/s 2024-06-10T00:36:03.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:02 smithi193 ceph-mon[36716]: pgmap v247: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 335 op/s 2024-06-10T00:36:03.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:02 smithi121 ceph-mon[32378]: pgmap v247: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 335 op/s 2024-06-10T00:36:05.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:04 smithi135 ceph-mon[36424]: pgmap v248: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 300 op/s 2024-06-10T00:36:05.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:04 smithi193 ceph-mon[36716]: pgmap v248: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 300 op/s 2024-06-10T00:36:05.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:04 smithi121 ceph-mon[32378]: pgmap v248: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 300 op/s 2024-06-10T00:36:07.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:06 smithi135 ceph-mon[36424]: pgmap v249: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 281 op/s 2024-06-10T00:36:07.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:06 smithi193 ceph-mon[36716]: pgmap v249: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 281 op/s 2024-06-10T00:36:07.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:06 smithi121 ceph-mon[32378]: pgmap v249: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 281 op/s 2024-06-10T00:36:08.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:07 smithi193 ceph-mon[36716]: osdmap e104: 12 total, 12 up, 12 in 2024-06-10T00:36:08.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:07 smithi121 ceph-mon[32378]: osdmap e104: 12 total, 12 up, 12 in 2024-06-10T00:36:08.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:07 smithi135 ceph-mon[36424]: osdmap e104: 12 total, 12 up, 12 in 2024-06-10T00:36:09.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:08 smithi193 ceph-mon[36716]: pgmap v251: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 243 op/s 2024-06-10T00:36:09.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:08 smithi193 ceph-mon[36716]: scrub summary: idle+waiting paths [/] 2024-06-10T00:36:09.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:08 smithi121 ceph-mon[32378]: pgmap v251: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 243 op/s 2024-06-10T00:36:09.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:08 smithi121 ceph-mon[32378]: scrub summary: idle+waiting paths [/] 2024-06-10T00:36:09.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:08 smithi135 ceph-mon[36424]: pgmap v251: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 243 op/s 2024-06-10T00:36:09.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:08 smithi135 ceph-mon[36424]: scrub summary: idle+waiting paths [/] 2024-06-10T00:36:11.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:10 smithi193 ceph-mon[36716]: pgmap v252: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-06-10T00:36:11.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:10 smithi121 ceph-mon[32378]: pgmap v252: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-06-10T00:36:11.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:10 smithi135 ceph-mon[36424]: pgmap v252: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 186 op/s 2024-06-10T00:36:13.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:12 smithi193 ceph-mon[36716]: pgmap v253: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 169 op/s 2024-06-10T00:36:13.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:12 smithi121 ceph-mon[32378]: pgmap v253: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 169 op/s 2024-06-10T00:36:13.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:12 smithi135 ceph-mon[36424]: pgmap v253: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 169 op/s 2024-06-10T00:36:15.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:14 smithi193 ceph-mon[36716]: pgmap v254: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 222 op/s 2024-06-10T00:36:15.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:14 smithi121 ceph-mon[32378]: pgmap v254: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 222 op/s 2024-06-10T00:36:15.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:14 smithi135 ceph-mon[36424]: pgmap v254: 129 pgs: 129 active+clean; 1.5 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.9 MiB/s wr, 222 op/s 2024-06-10T00:36:16.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:16 smithi135 ceph-mon[36424]: osdmap e105: 12 total, 12 up, 12 in 2024-06-10T00:36:16.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:16 smithi193 ceph-mon[36716]: osdmap e105: 12 total, 12 up, 12 in 2024-06-10T00:36:16.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:16 smithi121 ceph-mon[32378]: osdmap e105: 12 total, 12 up, 12 in 2024-06-10T00:36:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:17 smithi135 ceph-mon[36424]: pgmap v256: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 334 op/s 2024-06-10T00:36:17.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:17 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:36:17.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:17 smithi193 ceph-mon[36716]: pgmap v256: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 334 op/s 2024-06-10T00:36:17.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:17 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:36:17.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:17 smithi121 ceph-mon[32378]: pgmap v256: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 334 op/s 2024-06-10T00:36:17.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:17 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:36:18.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:18 smithi135 ceph-mon[36424]: scrub summary: idle 2024-06-10T00:36:18.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:18 smithi193 ceph-mon[36716]: scrub summary: idle 2024-06-10T00:36:18.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:18 smithi121 ceph-mon[32378]: scrub summary: idle 2024-06-10T00:36:19.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:19 smithi135 ceph-mon[36424]: pgmap v257: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 296 op/s 2024-06-10T00:36:19.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:19 smithi193 ceph-mon[36716]: pgmap v257: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 296 op/s 2024-06-10T00:36:19.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:19 smithi121 ceph-mon[32378]: pgmap v257: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 296 op/s 2024-06-10T00:36:21.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:21 smithi135 ceph-mon[36424]: pgmap v258: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 334 op/s 2024-06-10T00:36:21.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:21 smithi193 ceph-mon[36716]: pgmap v258: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 334 op/s 2024-06-10T00:36:21.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:21 smithi121 ceph-mon[32378]: pgmap v258: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 334 op/s 2024-06-10T00:36:23.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:23 smithi135 ceph-mon[36424]: pgmap v259: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 331 op/s 2024-06-10T00:36:23.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:23 smithi193 ceph-mon[36716]: pgmap v259: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 331 op/s 2024-06-10T00:36:23.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:23 smithi121 ceph-mon[32378]: pgmap v259: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 331 op/s 2024-06-10T00:36:25.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:25 smithi135 ceph-mon[36424]: pgmap v260: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 294 op/s 2024-06-10T00:36:25.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:25 smithi193 ceph-mon[36716]: pgmap v260: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 294 op/s 2024-06-10T00:36:25.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:25 smithi121 ceph-mon[32378]: pgmap v260: 129 pgs: 129 active+clean; 1.5 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 294 op/s 2024-06-10T00:36:27.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:27 smithi135 ceph-mon[36424]: pgmap v261: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 258 op/s 2024-06-10T00:36:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:27 smithi193 ceph-mon[36716]: pgmap v261: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 258 op/s 2024-06-10T00:36:27.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:27 smithi121 ceph-mon[32378]: pgmap v261: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.4 MiB/s wr, 258 op/s 2024-06-10T00:36:29.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:29 smithi135 ceph-mon[36424]: pgmap v262: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-06-10T00:36:29.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:29 smithi193 ceph-mon[36716]: pgmap v262: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-06-10T00:36:29.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:29 smithi121 ceph-mon[32378]: pgmap v262: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-06-10T00:36:31.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:31 smithi135 ceph-mon[36424]: pgmap v263: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 300 op/s 2024-06-10T00:36:31.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:31 smithi193 ceph-mon[36716]: pgmap v263: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 300 op/s 2024-06-10T00:36:31.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:31 smithi121 ceph-mon[32378]: pgmap v263: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.1 MiB/s wr, 300 op/s 2024-06-10T00:36:31.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-06-10T00:36:32.050 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:36:33.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:33 smithi135 ceph-mon[36424]: pgmap v264: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 285 op/s 2024-06-10T00:36:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:33 smithi193 ceph-mon[36716]: pgmap v264: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 285 op/s 2024-06-10T00:36:33.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:33 smithi121 ceph-mon[32378]: pgmap v264: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 285 op/s 2024-06-10T00:36:35.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:35 smithi135 ceph-mon[36424]: pgmap v265: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 294 op/s 2024-06-10T00:36:35.528 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:35 smithi121 ceph-mon[32378]: pgmap v265: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 294 op/s 2024-06-10T00:36:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.526+0000 7f4efb391700 1 -- 172.21.15.121:0/3937926872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef4101d30 msgr2=0x7f4ef410e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.526+0000 7f4efb391700 1 --2- 172.21.15.121:0/3937926872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef4101d30 0x7f4ef410e210 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00b0a0 tx=0x7f4eec00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.526+0000 7f4efb391700 1 -- 172.21.15.121:0/3937926872 shutdown_connections 2024-06-10T00:36:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.526+0000 7f4efb391700 1 --2- 172.21.15.121:0/3937926872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef4101d30 0x7f4ef410e210 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.526+0000 7f4efb391700 1 --2- 172.21.15.121:0/3937926872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ef41073d0 0x7f4ef41077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.526+0000 7f4efb391700 1 --2- 172.21.15.121:0/3937926872 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ef4101390 0x7f4ef41017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.526+0000 7f4efb391700 1 -- 172.21.15.121:0/3937926872 >> 172.21.15.121:0/3937926872 conn(0x7f4ef40782c0 msgr2=0x7f4ef40786d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.526+0000 7f4efb391700 1 -- 172.21.15.121:0/3937926872 shutdown_connections 2024-06-10T00:36:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.526+0000 7f4efb391700 1 -- 172.21.15.121:0/3937926872 wait complete. 2024-06-10T00:36:35.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.527+0000 7f4efb391700 1 Processor -- start 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.527+0000 7f4efb391700 1 -- start start 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4efb391700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ef4101390 0x7f4ef4199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4efb391700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef4101d30 0x7f4ef4199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4efb391700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ef41073d0 0x7f4ef41a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4efb391700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef4111650 con 0x7f4ef4101d30 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4efb391700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41114d0 con 0x7f4ef41073d0 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4efb391700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41117d0 con 0x7f4ef4101390 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef892c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef4101d30 0x7f4ef4199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef892c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef4101d30 0x7f4ef4199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57448/0 (socket says 172.21.15.121:57448) 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef992e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ef41073d0 0x7f4ef41a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef892c700 1 -- 172.21.15.121:0/1307392587 learned_addr learned my addr 172.21.15.121:0/1307392587 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:35.604 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef912d700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ef4101390 0x7f4ef4199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef892c700 1 -- 172.21.15.121:0/1307392587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ef4101390 msgr2=0x7f4ef4199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef892c700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ef4101390 0x7f4ef4199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef892c700 1 -- 172.21.15.121:0/1307392587 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ef41073d0 msgr2=0x7f4ef41a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef892c700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ef41073d0 0x7f4ef41a4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef892c700 1 -- 172.21.15.121:0/1307392587 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef41a5050 con 0x7f4ef4101d30 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef992e700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ef41073d0 0x7f4ef41a4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef912d700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ef4101390 0x7f4ef4199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.528+0000 7f4ef892c700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef4101d30 0x7f4ef4199fd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400e9a0 tx=0x7f4ee400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.529+0000 7f4eea7fc700 1 -- 172.21.15.121:0/1307392587 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee400ca10 con 0x7f4ef4101d30 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.529+0000 7f4eea7fc700 1 -- 172.21.15.121:0/1307392587 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee400cbb0 con 0x7f4ef4101d30 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.529+0000 7f4efb391700 1 -- 172.21.15.121:0/1307392587 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef41a5280 con 0x7f4ef4101d30 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.529+0000 7f4eea7fc700 1 -- 172.21.15.121:0/1307392587 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4018680 con 0x7f4ef4101d30 2024-06-10T00:36:35.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.529+0000 7f4efb391700 1 -- 172.21.15.121:0/1307392587 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef41a57e0 con 0x7f4ef4101d30 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.531+0000 7f4eea7fc700 1 -- 172.21.15.121:0/1307392587 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4ee4010430 con 0x7f4ef4101d30 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.532+0000 7f4eea7fc700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ee00a2e10 0x7f4ee00a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.532+0000 7f4ef912d700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ee00a2e10 0x7f4ee00a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.532+0000 7f4eea7fc700 1 -- 172.21.15.121:0/1307392587 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f4ee40cf9e0 con 0x7f4ef4101d30 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.532+0000 7f4ef912d700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ee00a2e10 0x7f4ee00a52d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009fd0 tx=0x7f4ef0009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4f05c14b80 1 Processor -- start 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4f05c14b80 1 -- start start 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 0x557274739a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 0x5572747339e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x557274734340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557274685810 con 0x557274739f60 2024-06-10T00:36:35.606 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55727462d720 con 0x557274739640 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5572746817d0 con 0x557274733f20 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef912d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 0x557274739a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef892c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 0x5572747339e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef912d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 0x557274739a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:46448/0 (socket says 172.21.15.121:46448) 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef912d700 1 -- 172.21.15.121:0/2964839892 learned_addr learned my addr 172.21.15.121:0/2964839892 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef992e700 1 --2- 172.21.15.121:0/2964839892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x557274734340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef892c700 1 -- 172.21.15.121:0/2964839892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 msgr2=0x557274734340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef892c700 1 --2- 172.21.15.121:0/2964839892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x557274734340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef892c700 1 -- 172.21.15.121:0/2964839892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 msgr2=0x557274739a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef892c700 1 --2- 172.21.15.121:0/2964839892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 0x557274739a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef892c700 1 -- 172.21.15.121:0/2964839892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557274623f50 con 0x557274739f60 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef992e700 1 --2- 172.21.15.121:0/2964839892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x557274734340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:36:35.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef912d700 1 --2- 172.21.15.121:0/2964839892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 0x557274739a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4ef892c700 1 --2- 172.21.15.121:0/2964839892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 0x5572747339e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40ccd40 tx=0x7f4ee409c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab4ec71554a849a0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.533+0000 7f4edf7fe700 1 -- 172.21.15.121:0/2964839892 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40a4780 con 0x557274739f60 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.534+0000 7f4edf7fe700 1 -- 172.21.15.121:0/2964839892 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee4099e10 con 0x557274739f60 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.534+0000 7f4edf7fe700 1 -- 172.21.15.121:0/2964839892 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4010980 con 0x557274739f60 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.534+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2964839892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 msgr2=0x5572747339e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.534+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2964839892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 0x5572747339e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40ccd40 tx=0x7f4ee409c2e0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.534+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2964839892 shutdown_connections 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.534+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2964839892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x557274734340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.534+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2964839892 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 0x557274739a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.534+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2964839892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 0x5572747339e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.534+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2964839892 >> 172.21.15.121:0/2964839892 conn(0x55727465ecc0 msgr2=0x55727472e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.534+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2964839892 shutdown_connections 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.534+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2964839892 wait complete. 2024-06-10T00:36:35.608 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.535+0000 7f4f05c14b80 1 -- start start 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.535+0000 7f4f05c14b80 1 Processor -- start 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.535+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x55727473fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.535+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 0x55727473d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.535+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 0x55727473d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.535+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557274740970 con 0x557274739f60 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.535+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x557274625bf0 con 0x557274739640 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.535+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557274740af0 con 0x557274733f20 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.535+0000 7f4f05c14b80 10 client.0 _ll_register_callbacks cb 0x55727461f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.535+0000 7f4ef992e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 0x55727473d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4ef992e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 0x55727473d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57464/0 (socket says 172.21.15.121:57464) 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4ef992e700 1 -- 172.21.15.121:0/3354606252 learned_addr learned my addr 172.21.15.121:0/3354606252 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4ef892c700 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 0x55727473d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4ef912d700 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x55727473fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.609 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4ef992e700 1 -- 172.21.15.121:0/3354606252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 msgr2=0x55727473fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4ef992e700 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x55727473fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4ef992e700 1 -- 172.21.15.121:0/3354606252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 msgr2=0x55727473d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4ef992e700 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 0x55727473d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4ef992e700 1 -- 172.21.15.121:0/3354606252 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572745d48d0 con 0x557274739f60 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4ef912d700 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x55727473fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4ef992e700 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 0x55727473d5a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f4eec004d10 tx=0x7f4eec004d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50f613b95374acb server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4edcff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4edcff9700 1 -- 172.21.15.121:0/3354606252 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec0076f0 con 0x557274739f60 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4edcff9700 1 -- 172.21.15.121:0/3354606252 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f4eec007890 con 0x557274739f60 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4f05c14b80 1 -- 172.21.15.121:0/3354606252 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ee4099e10 con 0x557274739f60 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4edcff9700 1 -- 172.21.15.121:0/3354606252 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4eec007ad0 con 0x557274739f60 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4f05c14b80 10 client.16695 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4f05c14b80 1 -- 172.21.15.121:0/3354606252 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55727474b320 con 0x557274739f60 2024-06-10T00:36:35.610 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.536+0000 7f4edcff9700 1 -- 172.21.15.121:0/3354606252 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee4099e10 con 0x557274739f60 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.537+0000 7f4edcff9700 1 -- 172.21.15.121:0/3354606252 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f4eec011070 con 0x557274739f60 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.537+0000 7f4f05c14b80 10 client.16695 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.537+0000 7f4f05c14b80 10 client.16695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.537+0000 7f4f05c14b80 20 client.16695 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.537+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55727474d9b0 0x55727474fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.537+0000 7f4f05c14b80 4 client.16695 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.537+0000 7f4f05c14b80 1 -- 172.21.15.121:0/3354606252 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557274692d10 con 0x55727474d9b0 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.537+0000 7f4ef912d700 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55727474d9b0 0x55727474fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.537+0000 7f4ef912d700 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55727474d9b0 0x55727474fe00 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c647b01cffdb1b42 server_cookie=2a7d1ef94f050ddb in_seq=0 out_seq=0 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.537+0000 7f4edcff9700 10 client.16695 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4edcff9700 1 -- 172.21.15.121:0/3354606252 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x557274692d10 con 0x55727474d9b0 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4edcff9700 10 client.16695 handle_command_reply: tid=0 2024-06-10T00:36:35.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 1 client.16695 shutdown 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 1 -- 172.21.15.121:0/3354606252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 msgr2=0x55727473d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 0x55727473d5a0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f4eec004d10 tx=0x7f4eec004d40 comp rx=0 tx=0).stop 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 1 -- 172.21.15.121:0/3354606252 shutdown_connections 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55727474d9b0 0x55727474fe00 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739f60 0x55727473d5a0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x55727473fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4edcff9700 0 client.16695 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/3354606252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739640 0x55727473d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 1 -- 172.21.15.121:0/3354606252 >> 172.21.15.121:0/3354606252 conn(0x55727465ecc0 msgr2=0x5572747329d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 1 -- 172.21.15.121:0/3354606252 shutdown_connections 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 1 -- 172.21.15.121:0/3354606252 wait complete. 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.539+0000 7f4f05c14b80 20 client.16695 trim_cache size 0 max 16384 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4f05c14b80 1 Processor -- start 2024-06-10T00:36:35.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4f05c14b80 1 -- start start 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x5572747339e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x557274749c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739f60 0x557274755710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41b1090 con 0x557274739640 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41ae860 con 0x557274739f60 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41ac6d0 con 0x557274733f20 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef892c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x557274749c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef892c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x557274749c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57480/0 (socket says 172.21.15.121:57480) 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef892c700 1 -- 172.21.15.121:0/2386971757 learned_addr learned my addr 172.21.15.121:0/2386971757 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef992e700 1 --2- 172.21.15.121:0/2386971757 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739f60 0x557274755710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef912d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x5572747339e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef892c700 1 -- 172.21.15.121:0/2386971757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 msgr2=0x5572747339e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef892c700 1 --2- 172.21.15.121:0/2386971757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x5572747339e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef892c700 1 -- 172.21.15.121:0/2386971757 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739f60 msgr2=0x557274755710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef892c700 1 --2- 172.21.15.121:0/2386971757 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739f60 0x557274755710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef892c700 1 -- 172.21.15.121:0/2386971757 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557274739a20 con 0x557274739640 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef912d700 1 --2- 172.21.15.121:0/2386971757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x5572747339e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.560+0000 7f4ef892c700 1 --2- 172.21.15.121:0/2386971757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x557274749c10 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f4ee409b810 tx=0x7f4ee400e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c60c11df6caede7 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4edeffd700 1 -- 172.21.15.121:0/2386971757 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40b9070 con 0x557274739640 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4edeffd700 1 -- 172.21.15.121:0/2386971757 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee4099e10 con 0x557274739640 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4edeffd700 1 -- 172.21.15.121:0/2386971757 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4010980 con 0x557274739640 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2386971757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 msgr2=0x557274749c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2386971757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x557274749c10 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f4ee409b810 tx=0x7f4ee400e8f0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2386971757 shutdown_connections 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2386971757 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274739f60 0x557274755710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2386971757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274733f20 0x5572747339e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2386971757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x557274749c10 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2386971757 >> 172.21.15.121:0/2386971757 conn(0x55727465ecc0 msgr2=0x55727473b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2386971757 shutdown_connections 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.561+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2386971757 wait complete. 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4f05c14b80 1 -- start start 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4f05c14b80 1 Processor -- start 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274733f20 0x5572747552b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x5572747557f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4f05c14b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274739f60 0x557274755d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41acd20 con 0x557274739640 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41ae190 con 0x557274733f20 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4f05c14b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef41ad3f0 con 0x557274739f60 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4f05c14b80 10 client.0 _ll_register_callbacks cb 0x55727461f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4ef892c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x5572747557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4ef912d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274733f20 0x5572747552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4ef892c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x5572747557f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:57484/0 (socket says 172.21.15.121:57484) 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4ef892c700 1 -- 172.21.15.121:0/2591521026 learned_addr learned my addr 172.21.15.121:0/2591521026 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.562+0000 7f4ef992e700 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274739f60 0x557274755d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4ef892c700 1 -- 172.21.15.121:0/2591521026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274739f60 msgr2=0x557274755d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4ef892c700 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274739f60 0x557274755d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4ef892c700 1 -- 172.21.15.121:0/2591521026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274733f20 msgr2=0x5572747552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4ef892c700 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274733f20 0x5572747552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4ef892c700 1 -- 172.21.15.121:0/2591521026 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572747b76e0 con 0x557274739640 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4ef892c700 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x5572747557f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4097f70 tx=0x7f4ee40a4680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fdac2e9e71e7fa server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4edcff9700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4edcff9700 1 -- 172.21.15.121:0/2591521026 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40b9070 con 0x557274739640 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4edcff9700 1 -- 172.21.15.121:0/2591521026 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f4ee40ba4b0 con 0x557274739640 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2591521026 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ee4099e10 con 0x557274739640 2024-06-10T00:36:35.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4edcff9700 1 -- 172.21.15.121:0/2591521026 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4ee4010720 con 0x557274739640 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4f05c14b80 10 client.16701 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2591521026 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5572747b8970 con 0x557274739640 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4edcff9700 1 -- 172.21.15.121:0/2591521026 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ee4099e10 con 0x557274739640 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4edcff9700 1 -- 172.21.15.121:0/2591521026 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f4ee409a9d0 con 0x557274739640 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4f05c14b80 10 client.16701 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4f05c14b80 10 client.16701 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4f05c14b80 20 client.16701 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55727474d9b0 0x5572747bac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4f05c14b80 4 client.16701 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2591521026 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55727468d290 con 0x55727474d9b0 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.563+0000 7f4ef912d700 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55727474d9b0 0x5572747bac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.564+0000 7f4ef912d700 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55727474d9b0 0x5572747bac00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3110368afa999ace server_cookie=99b0096381b30a73 in_seq=0 out_seq=0 2024-06-10T00:36:35.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.564+0000 7f4edcff9700 10 client.16701 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:35.618 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:36:35.618 INFO:teuthology.orchestra.run.smithi121.stdout: "status": "no active scrubs running", 2024-06-10T00:36:35.618 INFO:teuthology.orchestra.run.smithi121.stdout: "scrubs": {} 2024-06-10T00:36:35.618 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:36:35.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4edcff9700 1 -- 172.21.15.121:0/2591521026 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55727468d290 con 0x55727474d9b0 2024-06-10T00:36:35.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4edcff9700 10 client.16701 handle_command_reply: tid=0 2024-06-10T00:36:35.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 1 client.16701 shutdown 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2591521026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 msgr2=0x5572747557f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x5572747557f0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4097f70 tx=0x7f4ee40a4680 comp rx=0 tx=0).stop 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2591521026 shutdown_connections 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55727474d9b0 0x5572747bac00 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557274739f60 0x557274755d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4edcff9700 0 client.16701 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x557274733f20 0x5572747552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 1 --2- 172.21.15.121:0/2591521026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557274739640 0x5572747557f0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2591521026 >> 172.21.15.121:0/2591521026 conn(0x55727465ecc0 msgr2=0x557274727e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2591521026 shutdown_connections 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 1 -- 172.21.15.121:0/2591521026 wait complete. 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.566+0000 7f4f05c14b80 20 client.16701 trim_cache size 0 max 16384 2024-06-10T00:36:35.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 -- 172.21.15.121:0/1307392587 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ee00a2e10 msgr2=0x7f4ee00a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ee00a2e10 0x7f4ee00a52d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009fd0 tx=0x7f4ef0009260 comp rx=0 tx=0).stop 2024-06-10T00:36:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 -- 172.21.15.121:0/1307392587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef4101d30 msgr2=0x7f4ef4199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef4101d30 0x7f4ef4199fd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400e9a0 tx=0x7f4ee400ecb0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 -- 172.21.15.121:0/1307392587 shutdown_connections 2024-06-10T00:36:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f4ee00a2e10 0x7f4ee00a52d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4ef41073d0 0x7f4ef41a4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ef4101390 0x7f4ef4199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 --2- 172.21.15.121:0/1307392587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef4101d30 0x7f4ef4199fd0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 -- 172.21.15.121:0/1307392587 >> 172.21.15.121:0/1307392587 conn(0x7f4ef40782c0 msgr2=0x7f4ef40fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 -- 172.21.15.121:0/1307392587 shutdown_connections 2024-06-10T00:36:35.620 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:35.567+0000 7f4efb391700 1 -- 172.21.15.121:0/1307392587 wait complete. 2024-06-10T00:36:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:35 smithi193 ceph-mon[36716]: pgmap v265: 129 pgs: 129 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.3 MiB/s wr, 294 op/s 2024-06-10T00:36:37.385 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-06-10T00:36:37.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:36:37.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:37 smithi135 ceph-mon[36424]: pgmap v266: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 326 op/s 2024-06-10T00:36:37.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:37 smithi193 ceph-mon[36716]: pgmap v266: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 326 op/s 2024-06-10T00:36:37.637 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:36:37.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:37 smithi121 ceph-mon[32378]: pgmap v266: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 326 op/s 2024-06-10T00:36:38.359 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 16.0 2024-06-10T00:36:38.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:36:38.887 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:36:39.374 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:39 smithi121 ceph-mon[32378]: pgmap v267: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-06-10T00:36:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.431+0000 7f0ef2ffd700 1 -- 172.21.15.121:0/2930049945 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee800b760 con 0x7f0ef40772e0 2024-06-10T00:36:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.432+0000 7f0efa762700 1 -- 172.21.15.121:0/2930049945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ef40772e0 msgr2=0x7f0ef4112dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.432+0000 7f0efa762700 1 --2- 172.21.15.121:0/2930049945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ef40772e0 0x7f0ef4112dc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80098f0 tx=0x7f0ee8009c00 comp rx=0 tx=0).stop 2024-06-10T00:36:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.432+0000 7f0efa762700 1 -- 172.21.15.121:0/2930049945 shutdown_connections 2024-06-10T00:36:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.432+0000 7f0efa762700 1 --2- 172.21.15.121:0/2930049945 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ef4113300 0x7f0ef41156f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.432+0000 7f0efa762700 1 --2- 172.21.15.121:0/2930049945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef40769c0 0x7f0ef4076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.432+0000 7f0efa762700 1 --2- 172.21.15.121:0/2930049945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ef40772e0 0x7f0ef4112dc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:39.506 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.432+0000 7f0efa762700 1 -- 172.21.15.121:0/2930049945 >> 172.21.15.121:0/2930049945 conn(0x7f0ef4100170 msgr2=0x7f0ef4102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.433+0000 7f0efa762700 1 -- 172.21.15.121:0/2930049945 shutdown_connections 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.433+0000 7f0efa762700 1 -- 172.21.15.121:0/2930049945 wait complete. 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.434+0000 7f0efa762700 1 Processor -- start 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.434+0000 7f0efa762700 1 -- start start 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.435+0000 7f0efa762700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ef40769c0 0x7f0ef4104740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.435+0000 7f0efa762700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ef40772e0 0x7f0ef4104c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.435+0000 7f0efa762700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4113300 0x7f0ef41088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.435+0000 7f0efa762700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef406c4c0 con 0x7f0ef4113300 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.435+0000 7f0efa762700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef406c340 con 0x7f0ef40769c0 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.435+0000 7f0efa762700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef406c640 con 0x7f0ef40772e0 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.435+0000 7f0ef8cff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4113300 0x7f0ef41088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:39.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.435+0000 7f0ef8cff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4113300 0x7f0ef41088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56372/0 (socket says 172.21.15.121:56372) 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.435+0000 7f0ef8cff700 1 -- 172.21.15.121:0/345293639 learned_addr learned my addr 172.21.15.121:0/345293639 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.436+0000 7f0ef8cff700 1 -- 172.21.15.121:0/345293639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ef40772e0 msgr2=0x7f0ef4104c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.436+0000 7f0ef8cff700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ef40772e0 0x7f0ef4104c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.436+0000 7f0ef8cff700 1 -- 172.21.15.121:0/345293639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ef40769c0 msgr2=0x7f0ef4104740 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.436+0000 7f0ef8cff700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ef40769c0 0x7f0ef4104740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.436+0000 7f0ef8cff700 1 -- 172.21.15.121:0/345293639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef4109020 con 0x7f0ef4113300 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.436+0000 7f0ef8cff700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4113300 0x7f0ef41088e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009fd0 tx=0x7f0eec00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.437+0000 7f0ef17fa700 1 -- 172.21.15.121:0/345293639 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec0092e0 con 0x7f0ef4113300 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.437+0000 7f0ef17fa700 1 -- 172.21.15.121:0/345293639 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0eec004bb0 con 0x7f0ef4113300 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.437+0000 7f0ef17fa700 1 -- 172.21.15.121:0/345293639 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec004280 con 0x7f0ef4113300 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.437+0000 7f0efa762700 1 -- 172.21.15.121:0/345293639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef41092b0 con 0x7f0ef4113300 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.437+0000 7f0efa762700 1 -- 172.21.15.121:0/345293639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef41096e0 con 0x7f0ef4113300 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.439+0000 7f0edaffd700 1 -- 172.21.15.121:0/345293639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef4069180 con 0x7f0ef4113300 2024-06-10T00:36:39.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.443+0000 7f0ef17fa700 1 -- 172.21.15.121:0/345293639 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0eec01b030 con 0x7f0ef4113300 2024-06-10T00:36:39.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.444+0000 7f0ef17fa700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0edc0a2f60 0x7f0edc0a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:39.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.444+0000 7f0ef17fa700 1 -- 172.21.15.121:0/345293639 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f0eec0cfdd0 con 0x7f0ef4113300 2024-06-10T00:36:39.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.445+0000 7f0ef3fff700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0edc0a2f60 0x7f0edc0a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:39.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.446+0000 7f0ef3fff700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0edc0a2f60 0x7f0edc0a5420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009730 tx=0x7f0ee4006cb0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:39.509 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.449+0000 7f0ef17fa700 1 -- 172.21.15.121:0/345293639 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0eec022020 con 0x7f0ef4113300 2024-06-10T00:36:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:39 smithi193 ceph-mon[36716]: pgmap v267: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-06-10T00:36:39.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:39 smithi135 ceph-mon[36424]: pgmap v267: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-06-10T00:36:39.806 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.806+0000 7f0edaffd700 1 -- 172.21.15.121:0/345293639 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ef4063140 con 0x7f0ef4113300 2024-06-10T00:36:39.807 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.807+0000 7f0ef17fa700 1 -- 172.21.15.121:0/345293639 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11895 (secure 0 0 0) 0x7f0eec010040 con 0x7f0ef4113300 2024-06-10T00:36:39.807 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:36:39.807 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:35:59.291154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":189,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:36:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.809+0000 7f0edaffd700 1 -- 172.21.15.121:0/345293639 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0edc0a2f60 msgr2=0x7f0edc0a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.809+0000 7f0edaffd700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0edc0a2f60 0x7f0edc0a5420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009730 tx=0x7f0ee4006cb0 comp rx=0 tx=0).stop 2024-06-10T00:36:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.809+0000 7f0edaffd700 1 -- 172.21.15.121:0/345293639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4113300 msgr2=0x7f0ef41088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.809+0000 7f0edaffd700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4113300 0x7f0ef41088e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009fd0 tx=0x7f0eec00ef90 comp rx=0 tx=0).stop 2024-06-10T00:36:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.809+0000 7f0edaffd700 1 -- 172.21.15.121:0/345293639 shutdown_connections 2024-06-10T00:36:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.809+0000 7f0edaffd700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0edc0a2f60 0x7f0edc0a5420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.809+0000 7f0edaffd700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ef4113300 0x7f0ef41088e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.809+0000 7f0edaffd700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ef40769c0 0x7f0ef4104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.809+0000 7f0edaffd700 1 --2- 172.21.15.121:0/345293639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ef40772e0 0x7f0ef4104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.809+0000 7f0edaffd700 1 -- 172.21.15.121:0/345293639 >> 172.21.15.121:0/345293639 conn(0x7f0ef4100170 msgr2=0x7f0ef4102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:39.810 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.810+0000 7f0edaffd700 1 -- 172.21.15.121:0/345293639 shutdown_connections 2024-06-10T00:36:39.811 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:39.810+0000 7f0edaffd700 1 -- 172.21.15.121:0/345293639 wait complete. 2024-06-10T00:36:39.811 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 33 2024-06-10T00:36:40.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:40 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/345293639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:36:40.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:40 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/345293639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:36:40.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:40 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/345293639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:36:40.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-06-10T00:36:41.527 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:41 smithi121 ceph-mon[32378]: pgmap v268: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 314 op/s 2024-06-10T00:36:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:41 smithi193 ceph-mon[36716]: pgmap v268: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 314 op/s 2024-06-10T00:36:41.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:41 smithi135 ceph-mon[36424]: pgmap v268: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 314 op/s 2024-06-10T00:36:41.944 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.938+0000 7fb4d510b700 1 -- 172.21.15.121:0/550161958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4d0102550 msgr2=0x7fb4d01029d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:42.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.938+0000 7fb4d510b700 1 --2- 172.21.15.121:0/550161958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4d0102550 0x7fb4d01029d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009990 tx=0x7fb4c4009ca0 comp rx=0 tx=0).stop 2024-06-10T00:36:42.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.938+0000 7fb4d510b700 1 -- 172.21.15.121:0/550161958 shutdown_connections 2024-06-10T00:36:42.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.938+0000 7fb4d510b700 1 --2- 172.21.15.121:0/550161958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0102f10 0x7fb4d010f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.938+0000 7fb4d510b700 1 --2- 172.21.15.121:0/550161958 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4d01084f0 0x7fb4d01088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.938+0000 7fb4d510b700 1 --2- 172.21.15.121:0/550161958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4d0102550 0x7fb4d01029d0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.938+0000 7fb4d510b700 1 -- 172.21.15.121:0/550161958 >> 172.21.15.121:0/550161958 conn(0x7fb4d0079650 msgr2=0x7fb4d0079a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.938+0000 7fb4d510b700 1 -- 172.21.15.121:0/550161958 shutdown_connections 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.938+0000 7fb4d510b700 1 -- 172.21.15.121:0/550161958 wait complete. 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.939+0000 7fb4d510b700 1 Processor -- start 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.939+0000 7fb4d510b700 1 -- start start 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.939+0000 7fb4d510b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4d0102550 0x7fb4d019e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.939+0000 7fb4d510b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0102f10 0x7fb4d019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.939+0000 7fb4d510b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4d01084f0 0x7fb4d0198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.939+0000 7fb4d510b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d0112760 con 0x7fb4d0102550 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.939+0000 7fb4d510b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d01125e0 con 0x7fb4d01084f0 2024-06-10T00:36:42.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.939+0000 7fb4d510b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d01128e0 con 0x7fb4d0102f10 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4ced9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4d0102550 0x7fb4d019e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4ced9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4d0102550 0x7fb4d019e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56392/0 (socket says 172.21.15.121:56392) 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4ced9d700 1 -- 172.21.15.121:0/2148360705 learned_addr learned my addr 172.21.15.121:0/2148360705 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4cf59e700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4d01084f0 0x7fb4d0198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4ce59c700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0102f10 0x7fb4d019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4ced9d700 1 -- 172.21.15.121:0/2148360705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0102f10 msgr2=0x7fb4d019edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4ced9d700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0102f10 0x7fb4d019edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4ced9d700 1 -- 172.21.15.121:0/2148360705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4d01084f0 msgr2=0x7fb4d0198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4ced9d700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4d01084f0 0x7fb4d0198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4ced9d700 1 -- 172.21.15.121:0/2148360705 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d01991a0 con 0x7fb4d0102550 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4ce59c700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0102f10 0x7fb4d019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:36:42.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4cf59e700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4d01084f0 0x7fb4d0198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4ced9d700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4d0102550 0x7fb4d019e880 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000e9a0 tx=0x7fb4c000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4b7fff700 1 -- 172.21.15.121:0/2148360705 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c000cbd0 con 0x7fb4d0102550 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4b7fff700 1 -- 172.21.15.121:0/2148360705 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c000cd70 con 0x7fb4d0102550 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4d510b700 1 -- 172.21.15.121:0/2148360705 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d0199490 con 0x7fb4d0102550 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4b7fff700 1 -- 172.21.15.121:0/2148360705 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0018750 con 0x7fb4d0102550 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.940+0000 7fb4d510b700 1 -- 172.21.15.121:0/2148360705 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4d01a5750 con 0x7fb4d0102550 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.941+0000 7fb4d510b700 1 -- 172.21.15.121:0/2148360705 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4d0069180 con 0x7fb4d0102550 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.942+0000 7fb4b7fff700 1 -- 172.21.15.121:0/2148360705 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb4c0010430 con 0x7fb4d0102550 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.943+0000 7fb4b7fff700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb4b80a2d90 0x7fb4b80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.943+0000 7fb4ce59c700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb4b80a2d90 0x7fb4b80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.943+0000 7fb4b7fff700 1 -- 172.21.15.121:0/2148360705 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7fb4c00cfa80 con 0x7fb4d0102550 2024-06-10T00:36:42.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.943+0000 7fb4ce59c700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb4b80a2d90 0x7fb4b80a5250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400b2e0 tx=0x7fb4c4005e80 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:42.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:41.948+0000 7fb4b7fff700 1 -- 172.21.15.121:0/2148360705 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb4c009c2b0 con 0x7fb4d0102550 2024-06-10T00:36:42.020 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:36:42.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.321+0000 7fb4d510b700 1 -- 172.21.15.121:0/2148360705 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4d002dca0 con 0x7fb4d0102550 2024-06-10T00:36:42.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.322+0000 7fb4b7fff700 1 -- 172.21.15.121:0/2148360705 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11895 (secure 0 0 0) 0x7fb4c009bbb0 con 0x7fb4d0102550 2024-06-10T00:36:42.323 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:36:42.323 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:35:59.291154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":189,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:36:42.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.324+0000 7fb4d510b700 1 -- 172.21.15.121:0/2148360705 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb4b80a2d90 msgr2=0x7fb4b80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:42.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.324+0000 7fb4d510b700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb4b80a2d90 0x7fb4b80a5250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb4c400b2e0 tx=0x7fb4c4005e80 comp rx=0 tx=0).stop 2024-06-10T00:36:42.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.324+0000 7fb4d510b700 1 -- 172.21.15.121:0/2148360705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4d0102550 msgr2=0x7fb4d019e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:42.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.324+0000 7fb4d510b700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4d0102550 0x7fb4d019e880 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000e9a0 tx=0x7fb4c000ecb0 comp rx=0 tx=0).stop 2024-06-10T00:36:42.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.324+0000 7fb4d510b700 1 -- 172.21.15.121:0/2148360705 shutdown_connections 2024-06-10T00:36:42.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.324+0000 7fb4d510b700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb4b80a2d90 0x7fb4b80a5250 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:42.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.324+0000 7fb4d510b700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb4d01084f0 0x7fb4d0198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:42.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.324+0000 7fb4d510b700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4d0102550 0x7fb4d019e880 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:42.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.324+0000 7fb4d510b700 1 --2- 172.21.15.121:0/2148360705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d0102f10 0x7fb4d019edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:42.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.324+0000 7fb4d510b700 1 -- 172.21.15.121:0/2148360705 >> 172.21.15.121:0/2148360705 conn(0x7fb4d0079650 msgr2=0x7fb4d01005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:42.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.325+0000 7fb4d510b700 1 -- 172.21.15.121:0/2148360705 shutdown_connections 2024-06-10T00:36:42.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:42.325+0000 7fb4d510b700 1 -- 172.21.15.121:0/2148360705 wait complete. 2024-06-10T00:36:42.326 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 33 2024-06-10T00:36:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:43 smithi121 ceph-mon[32378]: pgmap v269: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-06-10T00:36:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:43 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2148360705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:36:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:43 smithi193 ceph-mon[36716]: pgmap v269: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-06-10T00:36:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:43 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2148360705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:36:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:43 smithi135 ceph-mon[36424]: pgmap v269: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-06-10T00:36:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:43 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2148360705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:36:44.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.202+0000 7f22a7b86700 1 -- 172.21.15.121:0/4100455313 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22a0110c70 msgr2=0x7f22a0109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:44.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.202+0000 7f22a7b86700 1 --2- 172.21.15.121:0/4100455313 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22a0110c70 0x7f22a0109490 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2298009f90 tx=0x7f229800b330 comp rx=0 tx=0).stop 2024-06-10T00:36:44.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.202+0000 7f22a7b86700 1 -- 172.21.15.121:0/4100455313 shutdown_connections 2024-06-10T00:36:44.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.202+0000 7f22a7b86700 1 --2- 172.21.15.121:0/4100455313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a011bc70 0x7f22a011e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.202+0000 7f22a7b86700 1 --2- 172.21.15.121:0/4100455313 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22a0110c70 0x7f22a0109490 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.202+0000 7f22a7b86700 1 --2- 172.21.15.121:0/4100455313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a01099d0 0x7f22a0109e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.202+0000 7f22a7b86700 1 -- 172.21.15.121:0/4100455313 >> 172.21.15.121:0/4100455313 conn(0x7f22a006d600 msgr2=0x7f22a006da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:44.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.202+0000 7f22a7b86700 1 -- 172.21.15.121:0/4100455313 shutdown_connections 2024-06-10T00:36:44.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.202+0000 7f22a7b86700 1 -- 172.21.15.121:0/4100455313 wait complete. 2024-06-10T00:36:44.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.203+0000 7f22a7b86700 1 Processor -- start 2024-06-10T00:36:44.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.203+0000 7f22a7b86700 1 -- start start 2024-06-10T00:36:44.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a7b86700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22a01099d0 0x7f22a011b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a7b86700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a0110c70 0x7f22a0116780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a7b86700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a011bc70 0x7f22a0116ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a7b86700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f22a0113230 con 0x7f22a0110c70 2024-06-10T00:36:44.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a7b86700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f22a01130b0 con 0x7f22a01099d0 2024-06-10T00:36:44.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a7b86700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f22a01133b0 con 0x7f22a011bc70 2024-06-10T00:36:44.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a5121700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a0110c70 0x7f22a0116780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a5121700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a0110c70 0x7f22a0116780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56420/0 (socket says 172.21.15.121:56420) 2024-06-10T00:36:44.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a5121700 1 -- 172.21.15.121:0/3514278744 learned_addr learned my addr 172.21.15.121:0/3514278744 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:44.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a6123700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a011bc70 0x7f22a0116ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a5121700 1 -- 172.21.15.121:0/3514278744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a011bc70 msgr2=0x7f22a0116ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:44.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a5121700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a011bc70 0x7f22a0116ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a5121700 1 -- 172.21.15.121:0/3514278744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22a01099d0 msgr2=0x7f22a011b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:44.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a5121700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22a01099d0 0x7f22a011b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.204+0000 7f22a5121700 1 -- 172.21.15.121:0/3514278744 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a01bc650 con 0x7f22a0110c70 2024-06-10T00:36:44.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.205+0000 7f22a5121700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a0110c70 0x7f22a0116780 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f229c0309b0 tx=0x7f229c030d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:44.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.205+0000 7f2296ffd700 1 -- 172.21.15.121:0/3514278744 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c02ebe0 con 0x7f22a0110c70 2024-06-10T00:36:44.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.205+0000 7f22a7b86700 1 -- 172.21.15.121:0/3514278744 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a011f950 con 0x7f22a0110c70 2024-06-10T00:36:44.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.205+0000 7f22a7b86700 1 -- 172.21.15.121:0/3514278744 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a011fee0 con 0x7f22a0110c70 2024-06-10T00:36:44.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.206+0000 7f2296ffd700 1 -- 172.21.15.121:0/3514278744 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f229c02ed80 con 0x7f22a0110c70 2024-06-10T00:36:44.207 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.206+0000 7f2296ffd700 1 -- 172.21.15.121:0/3514278744 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c048060 con 0x7f22a0110c70 2024-06-10T00:36:44.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.210+0000 7f22b2409b80 1 Processor -- start 2024-06-10T00:36:44.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.211+0000 7f2296ffd700 1 -- 172.21.15.121:0/3514278744 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f229c03a590 con 0x7f22a0110c70 2024-06-10T00:36:44.211 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.211+0000 7f22b2409b80 1 -- start start 2024-06-10T00:36:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.211+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.211+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4c880 0x5629b1c46260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.211+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c467a0 0x5629b1c46bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.211+0000 7f22b2409b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5629b1b38f60 con 0x5629b1c4bf60 2024-06-10T00:36:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.211+0000 7f22b2409b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5629b1b97850 con 0x5629b1c467a0 2024-06-10T00:36:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.211+0000 7f22b2409b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5629b1aff460 con 0x5629b1c4c880 2024-06-10T00:36:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.212+0000 7f22a6123700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.214 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.212+0000 7f22a6123700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56422/0 (socket says 172.21.15.121:56422) 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.212+0000 7f22a6123700 1 -- 172.21.15.121:0/3881021492 learned_addr learned my addr 172.21.15.121:0/3881021492 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.212+0000 7f22a5922700 1 --2- 172.21.15.121:0/3881021492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4c880 0x5629b1c46260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.212+0000 7f22a6123700 1 -- 172.21.15.121:0/3881021492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4c880 msgr2=0x5629b1c46260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.212+0000 7f22a6123700 1 --2- 172.21.15.121:0/3881021492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4c880 0x5629b1c46260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.212+0000 7f22a6123700 1 -- 172.21.15.121:0/3881021492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c467a0 msgr2=0x5629b1c46bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.213+0000 7f22a5121700 1 --2- 172.21.15.121:0/3881021492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c467a0 0x5629b1c46bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.213+0000 7f22a6123700 1 --2- 172.21.15.121:0/3881021492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c467a0 0x5629b1c46bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.213+0000 7f22a6123700 1 -- 172.21.15.121:0/3881021492 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629b1ae68d0 con 0x5629b1c4bf60 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.213+0000 7f2296ffd700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f228c0a3190 0x7f228c0a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.213+0000 7f22a5121700 1 --2- 172.21.15.121:0/3881021492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c467a0 0x5629b1c46bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.213+0000 7f2296ffd700 1 -- 172.21.15.121:0/3514278744 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f229c02e9e0 con 0x7f22a0110c70 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.213+0000 7f22a6123700 1 --2- 172.21.15.121:0/3881021492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4c340 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f2290009ae0 tx=0x7f2290009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeea86319279e603 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:44.215 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.213+0000 7f22a5922700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f228c0a3190 0x7f228c0a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.214+0000 7f228bfff700 1 -- 172.21.15.121:0/3881021492 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2290010b20 con 0x5629b1c4bf60 2024-06-10T00:36:44.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.214+0000 7f228bfff700 1 -- 172.21.15.121:0/3881021492 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2290010cc0 con 0x5629b1c4bf60 2024-06-10T00:36:44.216 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.214+0000 7f228bfff700 1 -- 172.21.15.121:0/3881021492 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2290012810 con 0x5629b1c4bf60 2024-06-10T00:36:44.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.216+0000 7f22b2409b80 1 -- 172.21.15.121:0/3881021492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 msgr2=0x5629b1c4c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:44.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.216+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3881021492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4c340 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f2290009ae0 tx=0x7f2290009df0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.217+0000 7f22b2409b80 1 -- 172.21.15.121:0/3881021492 shutdown_connections 2024-06-10T00:36:44.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.217+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3881021492 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c467a0 0x5629b1c46bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.217+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3881021492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4c340 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.217+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3881021492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4c880 0x5629b1c46260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.217+0000 7f22b2409b80 1 -- 172.21.15.121:0/3881021492 >> 172.21.15.121:0/3881021492 conn(0x5629b1b73080 msgr2=0x5629b1c41220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.218+0000 7f22a5922700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f228c0a3190 0x7f228c0a5650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2298009f90 tx=0x7f2298000f40 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:44.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.220+0000 7f22b2409b80 1 -- 172.21.15.121:0/3881021492 shutdown_connections 2024-06-10T00:36:44.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.221+0000 7f22b2409b80 1 -- 172.21.15.121:0/3881021492 wait complete. 2024-06-10T00:36:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.222+0000 7f22b2409b80 1 -- start start 2024-06-10T00:36:44.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.222+0000 7f22b2409b80 1 Processor -- start 2024-06-10T00:36:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.222+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c467a0 0x5629b1c525e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.222+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.222+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4c880 0x5629b1c4fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.222+0000 7f22b2409b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5629b1b37fc0 con 0x5629b1c4bf60 2024-06-10T00:36:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.222+0000 7f22b2409b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5629b1b93810 con 0x5629b1c4c880 2024-06-10T00:36:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.222+0000 7f22b2409b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5629b1c53260 con 0x5629b1c467a0 2024-06-10T00:36:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.222+0000 7f22b2409b80 10 client.0 _ll_register_callbacks cb 0x5629b1b2d5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:44.223 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.223+0000 7f22a6123700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4c880 0x5629b1c4fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.223+0000 7f22a5121700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.223+0000 7f22a5121700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56432/0 (socket says 172.21.15.121:56432) 2024-06-10T00:36:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.223+0000 7f22a5121700 1 -- 172.21.15.121:0/1758409905 learned_addr learned my addr 172.21.15.121:0/1758409905 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.223+0000 7f22a5121700 1 -- 172.21.15.121:0/1758409905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c467a0 msgr2=0x5629b1c525e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:36:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.223+0000 7f22a5121700 1 --2- 172.21.15.121:0/1758409905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c467a0 0x5629b1c525e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.223+0000 7f22a5121700 1 -- 172.21.15.121:0/1758409905 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4c880 msgr2=0x5629b1c4fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.223+0000 7f22a5121700 1 --2- 172.21.15.121:0/1758409905 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4c880 0x5629b1c4fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.223+0000 7f22a5121700 1 -- 172.21.15.121:0/1758409905 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629b1b5d750 con 0x5629b1c4bf60 2024-06-10T00:36:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.224+0000 7f22897fa700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.224+0000 7f22a5121700 1 --2- 172.21.15.121:0/1758409905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4f930 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f229c032ee0 tx=0x7f229c0bcfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa20e86704729e79 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.224+0000 7f22897fa700 1 -- 172.21.15.121:0/1758409905 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c0967e0 con 0x5629b1c4bf60 2024-06-10T00:36:44.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.224+0000 7f22b2409b80 1 -- 172.21.15.121:0/1758409905 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2290010cc0 con 0x5629b1c4bf60 2024-06-10T00:36:44.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.224+0000 7f22897fa700 1 -- 172.21.15.121:0/1758409905 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f229c0962b0 con 0x5629b1c4bf60 2024-06-10T00:36:44.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.227+0000 7f22897fa700 1 -- 172.21.15.121:0/1758409905 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f229c096020 con 0x5629b1c4bf60 2024-06-10T00:36:44.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.228+0000 7f22b2409b80 10 client.16722 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:44.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.228+0000 7f22b2409b80 1 -- 172.21.15.121:0/1758409905 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5629b1c5d7f0 con 0x5629b1c4bf60 2024-06-10T00:36:44.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.228+0000 7f22897fa700 1 -- 172.21.15.121:0/1758409905 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2290010cc0 con 0x5629b1c4bf60 2024-06-10T00:36:44.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.228+0000 7f22897fa700 1 -- 172.21.15.121:0/1758409905 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f229c0f11b0 con 0x5629b1c4bf60 2024-06-10T00:36:44.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.228+0000 7f22b2409b80 10 client.16722 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:44.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.228+0000 7f22b2409b80 10 client.16722 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:36:44.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.228+0000 7f22b2409b80 20 client.16722 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:44.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.228+0000 7f22b2409b80 1 --2- 172.21.15.121:0/1758409905 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5629b1c602b0 0x5629b1c626f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.229+0000 7f22b2409b80 4 client.16722 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:36:44.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.229+0000 7f22b2409b80 1 -- 172.21.15.121:0/1758409905 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629b1ba4d50 con 0x5629b1c602b0 2024-06-10T00:36:44.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.230+0000 7f22a5922700 1 --2- 172.21.15.121:0/1758409905 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5629b1c602b0 0x5629b1c626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.231+0000 7f22a5922700 1 --2- 172.21.15.121:0/1758409905 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5629b1c602b0 0x5629b1c626f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d29def33b311a65 server_cookie=e9a614b6869f220e in_seq=0 out_seq=0 2024-06-10T00:36:44.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.231+0000 7f22897fa700 10 client.16722 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:36:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.232+0000 7f22897fa700 1 -- 172.21.15.121:0/1758409905 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5629b1ba4d50 con 0x5629b1c602b0 2024-06-10T00:36:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.232+0000 7f22897fa700 10 client.16722 handle_command_reply: tid=0 2024-06-10T00:36:44.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.232+0000 7f22b2409b80 1 client.16722 shutdown 2024-06-10T00:36:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.232+0000 7f22b2409b80 1 -- 172.21.15.121:0/1758409905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 msgr2=0x5629b1c4f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.232+0000 7f22b2409b80 1 --2- 172.21.15.121:0/1758409905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4f930 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f229c032ee0 tx=0x7f229c0bcfb0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.233+0000 7f22b2409b80 1 -- 172.21.15.121:0/1758409905 shutdown_connections 2024-06-10T00:36:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.233+0000 7f22b2409b80 1 --2- 172.21.15.121:0/1758409905 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5629b1c602b0 0x5629b1c626f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.233+0000 7f22b2409b80 1 --2- 172.21.15.121:0/1758409905 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4c880 0x5629b1c4fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.233+0000 7f22b2409b80 1 --2- 172.21.15.121:0/1758409905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c467a0 0x5629b1c525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.233+0000 7f22b2409b80 1 --2- 172.21.15.121:0/1758409905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c4bf60 0x5629b1c4f930 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.233+0000 7f22b2409b80 1 -- 172.21.15.121:0/1758409905 >> 172.21.15.121:0/1758409905 conn(0x5629b1b73080 msgr2=0x5629b1c40700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.233+0000 7f22897fa700 0 client.16722 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:36:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.233+0000 7f22b2409b80 1 -- 172.21.15.121:0/1758409905 shutdown_connections 2024-06-10T00:36:44.234 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.233+0000 7f22b2409b80 1 -- 172.21.15.121:0/1758409905 wait complete. 2024-06-10T00:36:44.235 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.233+0000 7f22b2409b80 20 client.16722 trim_cache size 0 max 16384 2024-06-10T00:36:44.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.265+0000 7f22b2409b80 1 Processor -- start 2024-06-10T00:36:44.267 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.267+0000 7f22b2409b80 1 -- start start 2024-06-10T00:36:44.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.268+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c467a0 0x5629b1c46260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.268+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4bf60 0x5629b1c4fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.268 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.268+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4c880 0x5629b1c4d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.268+0000 7f22b2409b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2284011a10 con 0x5629b1c467a0 2024-06-10T00:36:44.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.268+0000 7f22b2409b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f228400f1e0 con 0x5629b1c4c880 2024-06-10T00:36:44.269 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.268+0000 7f22b2409b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f228400cfd0 con 0x5629b1c4bf60 2024-06-10T00:36:44.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.268+0000 7f22a5121700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4bf60 0x5629b1c4fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.269+0000 7f22a6123700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4c880 0x5629b1c4d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.269+0000 7f22a5121700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4bf60 0x5629b1c4fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:40646/0 (socket says 172.21.15.121:40646) 2024-06-10T00:36:44.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.269+0000 7f22a5121700 1 -- 172.21.15.121:0/3235837439 learned_addr learned my addr 172.21.15.121:0/3235837439 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:44.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.269+0000 7f22a5121700 1 -- 172.21.15.121:0/3235837439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4c880 msgr2=0x5629b1c4d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:44.270 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.269+0000 7f22a5121700 1 --2- 172.21.15.121:0/3235837439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4c880 0x5629b1c4d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.269+0000 7f22a5121700 1 -- 172.21.15.121:0/3235837439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c467a0 msgr2=0x5629b1c46260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:44.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.269+0000 7f22a5121700 1 --2- 172.21.15.121:0/3235837439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c467a0 0x5629b1c46260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.269+0000 7f22a5121700 1 -- 172.21.15.121:0/3235837439 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629b1c4dea0 con 0x5629b1c4bf60 2024-06-10T00:36:44.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.270+0000 7f22a5121700 1 --2- 172.21.15.121:0/3235837439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4bf60 0x5629b1c4fea0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f229c0b9ab0 tx=0x7f229c030900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb6300bcd402c1a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:44.271 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.271+0000 7f228b7fe700 1 -- 172.21.15.121:0/3235837439 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c096020 con 0x5629b1c4bf60 2024-06-10T00:36:44.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.271+0000 7f228b7fe700 1 -- 172.21.15.121:0/3235837439 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f229c0967e0 con 0x5629b1c4bf60 2024-06-10T00:36:44.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.271+0000 7f228b7fe700 1 -- 172.21.15.121:0/3235837439 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c0be620 con 0x5629b1c4bf60 2024-06-10T00:36:44.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.272+0000 7f22b2409b80 1 -- 172.21.15.121:0/3235837439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4bf60 msgr2=0x5629b1c4fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:44.272 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.272+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3235837439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4bf60 0x5629b1c4fea0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f229c0b9ab0 tx=0x7f229c030900 comp rx=0 tx=0).stop 2024-06-10T00:36:44.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.273+0000 7f22b2409b80 1 -- 172.21.15.121:0/3235837439 shutdown_connections 2024-06-10T00:36:44.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.273+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3235837439 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4c880 0x5629b1c4d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.273+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3235837439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c467a0 0x5629b1c46260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.273+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3235837439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4bf60 0x5629b1c4fea0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.273 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.273+0000 7f22b2409b80 1 -- 172.21.15.121:0/3235837439 >> 172.21.15.121:0/3235837439 conn(0x5629b1b73080 msgr2=0x5629b1c4cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:44.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.273+0000 7f22b2409b80 1 -- 172.21.15.121:0/3235837439 shutdown_connections 2024-06-10T00:36:44.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.273+0000 7f22b2409b80 1 -- 172.21.15.121:0/3235837439 wait complete. 2024-06-10T00:36:44.274 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.273+0000 7f22b2409b80 1 -- start start 2024-06-10T00:36:44.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.274+0000 7f22b2409b80 1 Processor -- start 2024-06-10T00:36:44.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.274+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c467a0 0x5629b1c58580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.275+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4bf60 0x5629b1c58ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.275+0000 7f22b2409b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4c880 0x5629b1c66ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.275+0000 7f22b2409b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f228400d6a0 con 0x5629b1c467a0 2024-06-10T00:36:44.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.275+0000 7f22b2409b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f228400eb10 con 0x5629b1c4bf60 2024-06-10T00:36:44.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.275+0000 7f22b2409b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f228400dd70 con 0x5629b1c4c880 2024-06-10T00:36:44.275 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.275+0000 7f22b2409b80 10 client.0 _ll_register_callbacks cb 0x5629b1b2d5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.276+0000 7f22a5922700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c467a0 0x5629b1c58580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.276+0000 7f22a5922700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c467a0 0x5629b1c58580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56454/0 (socket says 172.21.15.121:56454) 2024-06-10T00:36:44.276 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.276+0000 7f22a5922700 1 -- 172.21.15.121:0/3482152465 learned_addr learned my addr 172.21.15.121:0/3482152465 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.276+0000 7f22a5922700 1 -- 172.21.15.121:0/3482152465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4c880 msgr2=0x5629b1c66ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:36:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.276+0000 7f22a5922700 1 --2- 172.21.15.121:0/3482152465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4c880 0x5629b1c66ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.276+0000 7f22a5922700 1 -- 172.21.15.121:0/3482152465 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4bf60 msgr2=0x5629b1c58ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:36:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.276+0000 7f22a5922700 1 --2- 172.21.15.121:0/3482152465 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4bf60 0x5629b1c58ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.276+0000 7f22a5922700 1 -- 172.21.15.121:0/3482152465 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629b1c68b50 con 0x5629b1c467a0 2024-06-10T00:36:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.277+0000 7f22a5922700 1 --2- 172.21.15.121:0/3482152465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c467a0 0x5629b1c58580 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f2298009510 tx=0x7f229800baa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e22f4cff9639177 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:44.277 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.277+0000 7f22897fa700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:44.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.281+0000 7f22897fa700 1 -- 172.21.15.121:0/3482152465 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2298029870 con 0x5629b1c467a0 2024-06-10T00:36:44.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.281+0000 7f22897fa700 1 -- 172.21.15.121:0/3482152465 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f2298029a10 con 0x5629b1c467a0 2024-06-10T00:36:44.281 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.282+0000 7f22b2409b80 1 -- 172.21.15.121:0/3482152465 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f229c0967e0 con 0x5629b1c467a0 2024-06-10T00:36:44.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.285+0000 7f22897fa700 1 -- 172.21.15.121:0/3482152465 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2298029c10 con 0x5629b1c467a0 2024-06-10T00:36:44.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.286+0000 7f22b2409b80 10 client.16725 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:44.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.286+0000 7f22b2409b80 1 -- 172.21.15.121:0/3482152465 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5629b1cca510 con 0x5629b1c467a0 2024-06-10T00:36:44.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.288+0000 7f22897fa700 1 -- 172.21.15.121:0/3482152465 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2298037070 con 0x5629b1c467a0 2024-06-10T00:36:44.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.288+0000 7f22897fa700 1 -- 172.21.15.121:0/3482152465 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f2298011070 con 0x5629b1c467a0 2024-06-10T00:36:44.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.290+0000 7f22b2409b80 10 client.16725 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:44.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.290+0000 7f22b2409b80 10 client.16725 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-06-10T00:36:44.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.290+0000 7f22b2409b80 20 client.16725 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:44.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.290+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3482152465 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5629b1ccacb0 0x5629b1ccd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:44.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.290+0000 7f22b2409b80 4 client.16725 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:36:44.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.290+0000 7f22b2409b80 1 -- 172.21.15.121:0/3482152465 --> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f229c0967e0 con 0x5629b1ccacb0 2024-06-10T00:36:44.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.291+0000 7f22a5121700 1 --2- 172.21.15.121:0/3482152465 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5629b1ccacb0 0x5629b1ccd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:44.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.292+0000 7f22a5121700 1 --2- 172.21.15.121:0/3482152465 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5629b1ccacb0 0x5629b1ccd0d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=146ace738e851e7b server_cookie=26a5b9ddfa04ff1d in_seq=0 out_seq=0 2024-06-10T00:36:44.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.292+0000 7f22897fa700 10 client.16725 ms_handle_connect on v2:172.21.15.135:6836/1549198529 2024-06-10T00:36:44.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.293+0000 7f22897fa700 1 -- 172.21.15.121:0/3482152465 <== mds.2 v2:172.21.15.135:6836/1549198529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+37576 (crc 0 0 0) 0x7f229c0967e0 con 0x5629b1ccacb0 2024-06-10T00:36:44.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.293+0000 7f22897fa700 10 client.16725 handle_command_reply: tid=0 2024-06-10T00:36:44.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.294+0000 7f22b2409b80 1 client.16725 shutdown 2024-06-10T00:36:44.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.294+0000 7f22b2409b80 1 -- 172.21.15.121:0/3482152465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c467a0 msgr2=0x5629b1c58580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:44.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.294+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3482152465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c467a0 0x5629b1c58580 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f2298009510 tx=0x7f229800baa0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.295+0000 7f22b2409b80 1 -- 172.21.15.121:0/3482152465 shutdown_connections 2024-06-10T00:36:44.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.295+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3482152465 >> [v2:172.21.15.135:6836/1549198529,v1:172.21.15.135:6837/1549198529] conn(0x5629b1ccacb0 0x5629b1ccd0d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.295+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3482152465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5629b1c4c880 0x5629b1c66ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.295+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3482152465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5629b1c467a0 0x5629b1c58580 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.295+0000 7f22b2409b80 1 --2- 172.21.15.121:0/3482152465 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5629b1c4bf60 0x5629b1c58ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.295+0000 7f22b2409b80 1 -- 172.21.15.121:0/3482152465 >> 172.21.15.121:0/3482152465 conn(0x5629b1b73080 msgr2=0x5629b1c51110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:44.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.295+0000 7f22897fa700 0 client.16725 ms_handle_reset on v2:172.21.15.135:6836/1549198529 2024-06-10T00:36:44.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.295+0000 7f22b2409b80 1 -- 172.21.15.121:0/3482152465 shutdown_connections 2024-06-10T00:36:44.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.295+0000 7f22b2409b80 1 -- 172.21.15.121:0/3482152465 wait complete. 2024-06-10T00:36:44.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.295+0000 7f22b2409b80 20 client.16725 trim_cache size 0 max 16384 2024-06-10T00:36:44.296 INFO:teuthology.orchestra.run.smithi121.stdout:[ 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 34338598, 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634775, 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-6.jpg" 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 76697924, 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634649, 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-41.jpg" 2024-06-10T00:36:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 102277572, 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634403, 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-12.xml" 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 124889576, 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634635, 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-47.jpg" 2024-06-10T00:36:44.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.299 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.299 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 167324012, 2024-06-10T00:36:44.299 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634651, 2024-06-10T00:36:44.299 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-40.jpg" 2024-06-10T00:36:44.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.299 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.299 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.299 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 180342043, 2024-06-10T00:36:44.299 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634575, 2024-06-10T00:36:44.299 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-55.jpg" 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 281322038, 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632941, 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-54.xml" 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 304650276, 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634541, 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-59.jpg" 2024-06-10T00:36:44.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 351525597, 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634594, 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-53.jpg" 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 358327564, 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511635274, 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-42" 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.301 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 376517141, 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634661, 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-36.jpg" 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 386364377, 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634323, 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-32.xml" 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.302 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 391306149, 2024-06-10T00:36:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634727, 2024-06-10T00:36:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-18.jpg" 2024-06-10T00:36:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 412212068, 2024-06-10T00:36:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634175, 2024-06-10T00:36:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-66.xml" 2024-06-10T00:36:44.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 478585274, 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634383, 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-17.xml" 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 505461447, 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634691, 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-27.jpg" 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.304 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 508687041, 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634218, 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-58.xml" 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 534096139, 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634124, 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-75.xml" 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 553361710, 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634676, 2024-06-10T00:36:44.305 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-31.jpg" 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 566470830, 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634630, 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-49.jpg" 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 570466905, 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511633281, 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg" 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 583996357, 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634755, 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-11.jpg" 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 588954681, 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634551, 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-58.jpg" 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.307 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 591936727, 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634231, 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-55.xml" 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 600439603, 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632983, 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-45.xml" 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 604410212, 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634771, 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-7.jpg" 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.308 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 618023760, 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634355, 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-24.xml" 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 688466367, 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634419, 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-8.xml" 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 708474436, 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634642, 2024-06-10T00:36:44.309 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-44.jpg" 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 741998825, 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632920, 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-57.xml" 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 765537078, 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634462, 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-68.jpg" 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.310 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 775663768, 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634255, 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-50.xml" 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 776894961, 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634240, 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-53.xml" 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 786511926, 2024-06-10T00:36:44.311 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634208, 2024-06-10T00:36:44.312 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-60.xml" 2024-06-10T00:36:44.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.312 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.312 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.312 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 789400562, 2024-06-10T00:36:44.312 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634277, 2024-06-10T00:36:44.312 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-44.xml" 2024-06-10T00:36:44.312 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 803799586, 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634295, 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-39.xml" 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 809313725, 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632989, 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-44.xml" 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.313 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 832224897, 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634699, 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-25.jpg" 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 879760599, 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634532, 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-60.jpg" 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 890246669, 2024-06-10T00:36:44.314 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634609, 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-51.jpg" 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 900879738, 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634480, 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-66.jpg" 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 909928836, 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632978, 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-46.xml" 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.315 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 913271421, 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634747, 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-13.jpg" 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1104829424, 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634687, 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-28.jpg" 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1130597280, 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634347, 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-26.xml" 2024-06-10T00:36:44.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1165988883, 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634664, 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-35.jpg" 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1189738846, 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634227, 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-56.xml" 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.317 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1275139091, 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634299, 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-38.xml" 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1280400881, 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634488, 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-65.jpg" 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1333101433, 2024-06-10T00:36:44.318 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634289, 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-41.xml" 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1350418077, 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634391, 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-15.xml" 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1367139248, 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634473, 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-67.jpg" 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.319 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1373345681, 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634335, 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-29.xml" 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1383811342, 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634633, 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-48.jpg" 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1385567070, 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634559, 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-57.jpg" 2024-06-10T00:36:44.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1413699889, 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634293, 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-40.xml" 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1459959856, 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634415, 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-9.xml" 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.321 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1492204660, 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632930, 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-56.xml" 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1530106411, 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634566, 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-56.jpg" 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1588345072, 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634259, 2024-06-10T00:36:44.322 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-49.xml" 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1697398191, 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634731, 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-17.jpg" 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1722664619, 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634735, 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-16.jpg" 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.323 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1733916310, 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634135, 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-73.xml" 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1743677523, 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634247, 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-52.xml" 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.324 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1776587408, 2024-06-10T00:36:44.325 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634359, 2024-06-10T00:36:44.325 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-23.xml" 2024-06-10T00:36:44.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.325 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.325 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.325 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1776707318, 2024-06-10T00:36:44.325 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634655, 2024-06-10T00:36:44.325 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-38.jpg" 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1788937052, 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634331, 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-30.xml" 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1939042924, 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634423, 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-7.xml" 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.326 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1951329526, 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634387, 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-16.xml" 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1957222731, 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634303, 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-37.xml" 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.327 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1973612380, 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634186, 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-64.xml" 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1977938069, 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634339, 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-28.xml" 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1991142873, 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634587, 2024-06-10T00:36:44.328 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-54.jpg" 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 1995259393, 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632949, 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-52.xml" 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2019110139, 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634367, 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-21.xml" 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.329 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2059562121, 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634214, 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-59.xml" 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2064626047, 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634180, 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-65.xml" 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2076431525, 2024-06-10T00:36:44.330 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634751, 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-12.jpg" 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2104727018, 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634427, 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-6.xml" 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2116731428, 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634163, 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-68.xml" 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2135546774, 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634262, 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-48.xml" 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2200905500, 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634670, 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-33.jpg" 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2232223223, 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634311, 2024-06-10T00:36:44.332 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-35.xml" 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2256786380, 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634719, 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-20.jpg" 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2313440470, 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634792, 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-1.jpg" 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.333 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2390665378, 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634653, 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-39.jpg" 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2406395763, 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634498, 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-64.jpg" 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2447084590, 2024-06-10T00:36:44.334 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634146, 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-71.xml" 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2485414524, 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634767, 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-8.jpg" 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2548723451, 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634222, 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-57.xml" 2024-06-10T00:36:44.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2596689273, 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634343, 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-27.xml" 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2598931016, 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634192, 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-63.xml" 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.336 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2609159704, 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634315, 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-34.xml" 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2632384235, 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634506, 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-63.jpg" 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2686615640, 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634266, 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-47.xml" 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.337 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2705339567, 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634435, 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-4.xml" 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2730169823, 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634640, 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-45.jpg" 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.338 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2730333660, 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634707, 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-23.jpg" 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2809419974, 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634743, 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-14.jpg" 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2825511312, 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634778, 2024-06-10T00:36:44.339 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-5.jpg" 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2834586146, 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634723, 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-19.jpg" 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2845807658, 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634759, 2024-06-10T00:36:44.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.297+0000 7f22a7b86700 1 -- 172.21.15.121:0/3514278744 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f228c0a3190 msgr2=0x7f228c0a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:44.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.297+0000 7f22a7b86700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f228c0a3190 0x7f228c0a5650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2298009f90 tx=0x7f2298000f40 comp rx=0 tx=0).stop 2024-06-10T00:36:44.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.297+0000 7f22a7b86700 1 -- 172.21.15.121:0/3514278744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a0110c70 msgr2=0x7f22a0116780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:44.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.297+0000 7f22a7b86700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a0110c70 0x7f22a0116780 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f229c0309b0 tx=0x7f229c030d70 comp rx=0 tx=0).stop 2024-06-10T00:36:44.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.297+0000 7f22a7b86700 1 -- 172.21.15.121:0/3514278744 shutdown_connections 2024-06-10T00:36:44.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.297+0000 7f22a7b86700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f228c0a3190 0x7f228c0a5650 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.297+0000 7f22a7b86700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f22a011bc70 0x7f22a0116ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.297+0000 7f22a7b86700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22a01099d0 0x7f22a011b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.297+0000 7f22a7b86700 1 --2- 172.21.15.121:0/3514278744 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a0110c70 0x7f22a0116780 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:44.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.297+0000 7f22a7b86700 1 -- 172.21.15.121:0/3514278744 >> 172.21.15.121:0/3514278744 conn(0x7f22a006d600 msgr2=0x7f22a011d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:44.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.297+0000 7f22a7b86700 1 -- 172.21.15.121:0/3514278744 shutdown_connections 2024-06-10T00:36:44.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:44.298+0000 7f22a7b86700 1 -- 172.21.15.121:0/3514278744 wait complete. 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-10.jpg" 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2857928425, 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634115, 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34" 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2897514245, 2024-06-10T00:36:44.342 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634447, 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-1.xml" 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 2908880805, 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632963, 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-49.xml" 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3005429153, 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632937, 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-55.xml" 2024-06-10T00:36:44.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3011589996, 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634281, 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-43.xml" 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3107124885, 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634327, 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-31.xml" 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.344 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3130640380, 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634251, 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-51.xml" 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3132393222, 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632992, 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-43.xml" 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.345 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3152238432, 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634788, 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-2.jpg" 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3154987171, 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634647, 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-42.jpg" 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3156124799, 2024-06-10T00:36:44.346 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632947, 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-53.xml" 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3199530314, 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634235, 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-54.xml" 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3207045957, 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634117, 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-76.xml" 2024-06-10T00:36:44.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3245683064, 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634673, 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-32.jpg" 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3255360236, 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634319, 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-33.xml" 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3257844014, 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511635549, 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-13.xml" 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3270590008, 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634683, 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-29.jpg" 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3283843128, 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634169, 2024-06-10T00:36:44.349 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-67.xml" 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3307930268, 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634307, 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-36.xml" 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3326795728, 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634763, 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-9.jpg" 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.350 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3360518134, 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632960, 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-50.xml" 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3374640461, 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634524, 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-61.jpg" 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3378546672, 2024-06-10T00:36:44.351 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634363, 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-22.xml" 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3386533530, 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634785, 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-3.jpg" 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3407334236, 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632974, 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-47.xml" 2024-06-10T00:36:44.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3453727486, 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634203, 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-61.xml" 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3472591155, 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634613, 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-50.jpg" 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.353 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3486263159, 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634739, 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-15.jpg" 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3488449575, 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632969, 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-48.xml" 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3643606676, 2024-06-10T00:36:44.354 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634197, 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-62.xml" 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3649328185, 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632911, 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-58.xml" 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3652667385, 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634407, 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-11.xml" 2024-06-10T00:36:44.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3695208712, 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634411, 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-10.xml" 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3703163384, 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634443, 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-2.xml" 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.356 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3712101459, 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634782, 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-4.jpg" 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3713152167, 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634667, 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-34.jpg" 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3740728973, 2024-06-10T00:36:44.357 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634451, 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-69.jpg" 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3770120785, 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634923, 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-42.xml" 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3793631517, 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634141, 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-72.xml" 2024-06-10T00:36:44.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3799814763, 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634351, 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-25.xml" 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3809398512, 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634439, 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-3.xml" 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.359 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3848959133, 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634395, 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-14.xml" 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3867357855, 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634152, 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-70.xml" 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3917534529, 2024-06-10T00:36:44.360 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634379, 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-18.xml" 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 3972286589, 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634645, 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-43.jpg" 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4023566656, 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634515, 2024-06-10T00:36:44.361 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-62.jpg" 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4037569096, 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511632955, 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-51.xml" 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4123897479, 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634695, 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-26.jpg" 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.362 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4143208748, 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634658, 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-37.jpg" 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4184672691, 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634711, 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-22.jpg" 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.363 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4284426747, 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634703, 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-24.jpg" 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4292482900, 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634273, 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-45.xml" 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: "damage_type": "backtrace", 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4294749627, 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 1099511634285, 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: "path": "/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-42.xml" 2024-06-10T00:36:44.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:44.365 INFO:teuthology.orchestra.run.smithi121.stdout:] 2024-06-10T00:36:44.635 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db / --timeout 90 --expiration 136.0 --await-for 45.0' 2024-06-10T00:36:44.636 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db / --timeout 90 --expiration 136.0 --await-for 45.0 2024-06-10T00:36:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:45 smithi121 ceph-mon[32378]: pgmap v270: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 293 op/s 2024-06-10T00:36:45.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:45 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:36:45.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-06-10T00:36:45.599 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:36:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:45 smithi193 ceph-mon[36716]: pgmap v270: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 293 op/s 2024-06-10T00:36:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:45 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:36:45.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:45 smithi135 ceph-mon[36424]: pgmap v270: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 293 op/s 2024-06-10T00:36:45.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:45 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:36:46.794 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:36:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:47 smithi121 ceph-mon[32378]: pgmap v271: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 299 op/s 2024-06-10T00:36:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:47 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:36:47.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:47 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:36:47.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:47 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:36:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:47 smithi193 ceph-mon[36716]: pgmap v271: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 299 op/s 2024-06-10T00:36:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:47 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:36:47.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:47 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:36:47.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:47 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:36:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:47 smithi135 ceph-mon[36424]: pgmap v271: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 299 op/s 2024-06-10T00:36:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:47 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:36:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:47 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:36:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:47 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:36:48.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.001+0000 7ffaed037700 1 -- 172.21.15.121:0/60454286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffae8101d80 msgr2=0x7ffae810e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:48.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.001+0000 7ffaed037700 1 --2- 172.21.15.121:0/60454286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffae8101d80 0x7ffae810e220 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ffae000b0a0 tx=0x7ffae000b3b0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.001+0000 7ffaed037700 1 -- 172.21.15.121:0/60454286 shutdown_connections 2024-06-10T00:36:48.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.001+0000 7ffaed037700 1 --2- 172.21.15.121:0/60454286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffae8101d80 0x7ffae810e220 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.001 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.001+0000 7ffaed037700 1 --2- 172.21.15.121:0/60454286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae8107470 0x7ffae8107850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.001+0000 7ffaed037700 1 --2- 172.21.15.121:0/60454286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae81013c0 0x7ffae8101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.001+0000 7ffaed037700 1 -- 172.21.15.121:0/60454286 >> 172.21.15.121:0/60454286 conn(0x7ffae80783d0 msgr2=0x7ffae80787e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:48.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.001+0000 7ffaed037700 1 -- 172.21.15.121:0/60454286 shutdown_connections 2024-06-10T00:36:48.002 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.001+0000 7ffaed037700 1 -- 172.21.15.121:0/60454286 wait complete. 2024-06-10T00:36:48.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffaed037700 1 Processor -- start 2024-06-10T00:36:48.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffaed037700 1 -- start start 2024-06-10T00:36:48.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffaed037700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffae81013c0 0x7ffae8103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.004 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffaed037700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae8101d80 0x7ffae81035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffaed037700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae8107470 0x7ffae8103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffaed037700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffae8111680 con 0x7ffae81013c0 2024-06-10T00:36:48.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffaed037700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ffae8111500 con 0x7ffae8101d80 2024-06-10T00:36:48.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffaed037700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffae8111800 con 0x7ffae8107470 2024-06-10T00:36:48.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffae6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffae81013c0 0x7ffae8103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffae6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffae81013c0 0x7ffae8103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56480/0 (socket says 172.21.15.121:56480) 2024-06-10T00:36:48.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffae6d9d700 1 -- 172.21.15.121:0/1588901780 learned_addr learned my addr 172.21.15.121:0/1588901780 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:48.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffae659c700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae8101d80 0x7ffae81035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffae759e700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae8107470 0x7ffae8103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.005 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffae6d9d700 1 -- 172.21.15.121:0/1588901780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae8107470 msgr2=0x7ffae8103af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:48.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.002+0000 7ffae6d9d700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae8107470 0x7ffae8103af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.003+0000 7ffae6d9d700 1 -- 172.21.15.121:0/1588901780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae8101d80 msgr2=0x7ffae81035b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:48.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.003+0000 7ffae6d9d700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae8101d80 0x7ffae81035b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.003+0000 7ffae6d9d700 1 -- 172.21.15.121:0/1588901780 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae81a6f60 con 0x7ffae81013c0 2024-06-10T00:36:48.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.003+0000 7ffae659c700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae8101d80 0x7ffae81035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:48.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.003+0000 7ffae6d9d700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffae81013c0 0x7ffae8103070 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ffad800e9a0 tx=0x7ffad800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:48.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.003+0000 7ffacffff700 1 -- 172.21.15.121:0/1588901780 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad800ca10 con 0x7ffae81013c0 2024-06-10T00:36:48.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.003+0000 7ffaed037700 1 -- 172.21.15.121:0/1588901780 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae81a7250 con 0x7ffae81013c0 2024-06-10T00:36:48.006 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.003+0000 7ffacffff700 1 -- 172.21.15.121:0/1588901780 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffad800cbb0 con 0x7ffae81013c0 2024-06-10T00:36:48.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.003+0000 7ffacffff700 1 -- 172.21.15.121:0/1588901780 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad8018680 con 0x7ffae81013c0 2024-06-10T00:36:48.007 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.003+0000 7ffaed037700 1 -- 172.21.15.121:0/1588901780 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae81a77e0 con 0x7ffae81013c0 2024-06-10T00:36:48.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.005+0000 7ffacffff700 1 -- 172.21.15.121:0/1588901780 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ffad8010430 con 0x7ffae81013c0 2024-06-10T00:36:48.012 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.006+0000 7ffacffff700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffad00a2eb0 0x7ffad00a5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.006+0000 7ffae659c700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffad00a2eb0 0x7ffad00a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.006+0000 7ffacffff700 1 -- 172.21.15.121:0/1588901780 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7ffad80cf9d0 con 0x7ffae81013c0 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.007+0000 7ffae659c700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffad00a2eb0 0x7ffad00a5370 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ffae8078960 tx=0x7ffadc00b410 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.007+0000 7ffaf78bab80 1 Processor -- start 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.007+0000 7ffaf78bab80 1 -- start start 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.007+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4b0590 0x5560fa4b0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.007+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0eb0 0x5560fa4aa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.007+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4aace0 0x5560fa4ab100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.007+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5560fa39d0f0 con 0x5560fa4b0eb0 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.007+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5560fa3fb9e0 con 0x5560fa4b0590 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.007+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5560fa3635f0 con 0x5560fa4aace0 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.007+0000 7ffae659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0eb0 0x5560fa4aa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.013 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.007+0000 7ffae659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0eb0 0x5560fa4aa7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56494/0 (socket says 172.21.15.121:56494) 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.008+0000 7ffae659c700 1 -- 172.21.15.121:0/1730736692 learned_addr learned my addr 172.21.15.121:0/1730736692 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.008+0000 7ffae759e700 1 --2- 172.21.15.121:0/1730736692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4aace0 0x5560fa4ab100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.008+0000 7ffae659c700 1 -- 172.21.15.121:0/1730736692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4aace0 msgr2=0x5560fa4ab100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.008+0000 7ffae659c700 1 --2- 172.21.15.121:0/1730736692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4aace0 0x5560fa4ab100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.008+0000 7ffae659c700 1 -- 172.21.15.121:0/1730736692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4b0590 msgr2=0x5560fa4b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.008+0000 7ffae659c700 1 --2- 172.21.15.121:0/1730736692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4b0590 0x5560fa4b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.008+0000 7ffae659c700 1 -- 172.21.15.121:0/1730736692 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560fa34aa60 con 0x5560fa4b0eb0 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.008+0000 7ffae659c700 1 --2- 172.21.15.121:0/1730736692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0eb0 0x5560fa4aa7a0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ffadc011760 tx=0x7ffadc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0acb0fdb0804850 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.008+0000 7ffacd7fa700 1 -- 172.21.15.121:0/1730736692 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc01e620 con 0x5560fa4b0eb0 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.008+0000 7ffacd7fa700 1 -- 172.21.15.121:0/1730736692 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffadc010e70 con 0x5560fa4b0eb0 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.008+0000 7ffacd7fa700 1 -- 172.21.15.121:0/1730736692 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc026c10 con 0x5560fa4b0eb0 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.009+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1730736692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0eb0 msgr2=0x5560fa4aa7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:48.014 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.009+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1730736692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0eb0 0x5560fa4aa7a0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ffadc011760 tx=0x7ffadc013000 comp rx=0 tx=0).stop 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.009+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1730736692 shutdown_connections 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.009+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1730736692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4aace0 0x5560fa4ab100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.009+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1730736692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4b0590 0x5560fa4b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.009+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1730736692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0eb0 0x5560fa4aa7a0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.009+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1730736692 >> 172.21.15.121:0/1730736692 conn(0x5560fa3d7210 msgr2=0x5560fa4a5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.009+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1730736692 shutdown_connections 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.009+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1730736692 wait complete. 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffaf78bab80 1 -- start start 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffaf78bab80 1 Processor -- start 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4aace0 0x5560fa4b6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5560fa39c150 con 0x5560fa4b0590 2024-06-10T00:36:48.015 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5560fa3f79a0 con 0x5560fa4aace0 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5560fa4b7890 con 0x5560fa4b0eb0 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffaf78bab80 10 client.0 _ll_register_callbacks cb 0x5560fa391740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffae659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffae6d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4aace0 0x5560fa4b6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffae659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56502/0 (socket says 172.21.15.121:56502) 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffae659c700 1 -- 172.21.15.121:0/1130886420 learned_addr learned my addr 172.21.15.121:0/1130886420 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffae759e700 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.010+0000 7ffae6d9d700 1 -- 172.21.15.121:0/1130886420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 msgr2=0x5560fa4b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffae6d9d700 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffae6d9d700 1 -- 172.21.15.121:0/1130886420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 msgr2=0x5560fa4b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffae6d9d700 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffae6d9d700 1 -- 172.21.15.121:0/1130886420 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560fa3c18e0 con 0x5560fa4aace0 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffae659c700 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:48.016 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffae759e700 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffae6d9d700 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4aace0 0x5560fa4b6c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffad80cccd0 tx=0x7ffad809c1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93ea26cb938bbc97 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffacaff5700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffacaff5700 1 -- 172.21.15.121:0/1130886420 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad80a4480 con 0x5560fa4aace0 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffacaff5700 1 -- 172.21.15.121:0/1130886420 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7ffad8010720 con 0x5560fa4aace0 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1130886420 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffadc010e70 con 0x5560fa4aace0 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffacaff5700 1 -- 172.21.15.121:0/1130886420 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffad809fe50 con 0x5560fa4aace0 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffaf78bab80 10 client.26279 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1130886420 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5560fa4c2070 con 0x5560fa4aace0 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffacaff5700 1 -- 172.21.15.121:0/1130886420 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffadc010e70 con 0x5560fa4aace0 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.011+0000 7ffacaff5700 1 -- 172.21.15.121:0/1130886420 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7ffad809a960 con 0x5560fa4aace0 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.012+0000 7ffaf78bab80 10 client.26279 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:48.017 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.012+0000 7ffaf78bab80 10 client.26279 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:36:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.012+0000 7ffaf78bab80 20 client.26279 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.012+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560fa4c4970 0x5560fa4c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.012+0000 7ffaf78bab80 4 client.26279 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:36:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.012+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1130886420 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560fa408ee0 con 0x5560fa4c4970 2024-06-10T00:36:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.012+0000 7ffae659c700 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560fa4c4970 0x5560fa4c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.012+0000 7ffacaff5700 10 client.26279 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:48.018 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.012+0000 7ffae659c700 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560fa4c4970 0x5560fa4c6d60 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af665e6c15eb5053 server_cookie=d94abc8d0e1a1722 in_seq=0 out_seq=0 2024-06-10T00:36:48.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffacaff5700 1 -- 172.21.15.121:0/1130886420 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5560fa408ee0 con 0x5560fa4c4970 2024-06-10T00:36:48.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffacaff5700 10 client.26279 handle_command_reply: tid=0 2024-06-10T00:36:48.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 1 client.26279 shutdown 2024-06-10T00:36:48.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1130886420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4aace0 msgr2=0x5560fa4b6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:48.019 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4aace0 0x5560fa4b6c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffad80cccd0 tx=0x7ffad809c1a0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1130886420 shutdown_connections 2024-06-10T00:36:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560fa4c4970 0x5560fa4c6d60 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4aace0 0x5560fa4b6c80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffacaff5700 0 client.26279 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1130886420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1130886420 >> 172.21.15.121:0/1130886420 conn(0x5560fa3d7210 msgr2=0x5560fa4a43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1130886420 shutdown_connections 2024-06-10T00:36:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1130886420 wait complete. 2024-06-10T00:36:48.020 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.013+0000 7ffaf78bab80 20 client.26279 trim_cache size 0 max 16384 2024-06-10T00:36:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.033+0000 7ffaf78bab80 1 Processor -- start 2024-06-10T00:36:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.033+0000 7ffaf78bab80 1 -- start start 2024-06-10T00:36:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4aace0 0x5560fa4b67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4b0590 0x5560fa4c0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4c07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffae81b3220 con 0x5560fa4aace0 2024-06-10T00:36:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ffae81b09f0 con 0x5560fa4b0590 2024-06-10T00:36:48.035 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffae81ae860 con 0x5560fa4b0eb0 2024-06-10T00:36:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4aace0 0x5560fa4b67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae659c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4b0590 0x5560fa4c0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4aace0 0x5560fa4b67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56510/0 (socket says 172.21.15.121:56510) 2024-06-10T00:36:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae6d9d700 1 -- 172.21.15.121:0/2902532015 learned_addr learned my addr 172.21.15.121:0/2902532015 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae759e700 1 --2- 172.21.15.121:0/2902532015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4c07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae6d9d700 1 -- 172.21.15.121:0/2902532015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 msgr2=0x5560fa4c07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae6d9d700 1 --2- 172.21.15.121:0/2902532015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4c07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae6d9d700 1 -- 172.21.15.121:0/2902532015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4b0590 msgr2=0x5560fa4c0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae6d9d700 1 --2- 172.21.15.121:0/2902532015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4b0590 0x5560fa4c0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae6d9d700 1 -- 172.21.15.121:0/2902532015 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560fa4b2510 con 0x5560fa4aace0 2024-06-10T00:36:48.036 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae759e700 1 --2- 172.21.15.121:0/2902532015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4c07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:36:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae659c700 1 --2- 172.21.15.121:0/2902532015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4b0590 0x5560fa4c0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffae6d9d700 1 --2- 172.21.15.121:0/2902532015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4aace0 0x5560fa4b67c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ffad809ba30 tx=0x7ffad8099fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6344c6e863abe521 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.034+0000 7ffaccff9700 1 -- 172.21.15.121:0/2902532015 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad80860b0 con 0x5560fa4aace0 2024-06-10T00:36:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.035+0000 7ffaccff9700 1 -- 172.21.15.121:0/2902532015 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffad809a960 con 0x5560fa4aace0 2024-06-10T00:36:48.037 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.035+0000 7ffaccff9700 1 -- 172.21.15.121:0/2902532015 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad8010960 con 0x5560fa4aace0 2024-06-10T00:36:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.035+0000 7ffaf78bab80 1 -- 172.21.15.121:0/2902532015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4aace0 msgr2=0x5560fa4b67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.035+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/2902532015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4aace0 0x5560fa4b67c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7ffad809ba30 tx=0x7ffad8099fb0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.035+0000 7ffaf78bab80 1 -- 172.21.15.121:0/2902532015 shutdown_connections 2024-06-10T00:36:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.035+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/2902532015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4c07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.035+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/2902532015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4aace0 0x5560fa4b67c0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.035+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/2902532015 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4b0590 0x5560fa4c0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.035+0000 7ffaf78bab80 1 -- 172.21.15.121:0/2902532015 >> 172.21.15.121:0/2902532015 conn(0x5560fa3d7210 msgr2=0x5560fa4bfb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.035+0000 7ffaf78bab80 1 -- 172.21.15.121:0/2902532015 shutdown_connections 2024-06-10T00:36:48.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.035+0000 7ffaf78bab80 1 -- 172.21.15.121:0/2902532015 wait complete. 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffaf78bab80 1 -- start start 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffaf78bab80 1 Processor -- start 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4aace0 0x5560fa4c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4ca320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffaf78bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4c8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffae81aeeb0 con 0x5560fa4b0590 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ffae81b0320 con 0x5560fa4aace0 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffaf78bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffae81af580 con 0x5560fa4b0eb0 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffaf78bab80 10 client.0 _ll_register_callbacks cb 0x5560fa391740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffae659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffae6d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4aace0 0x5560fa4c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffae659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4ca320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56520/0 (socket says 172.21.15.121:56520) 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffae659c700 1 -- 172.21.15.121:0/1759692769 learned_addr learned my addr 172.21.15.121:0/1759692769 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:48.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffae759e700 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffae659c700 1 -- 172.21.15.121:0/1759692769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 msgr2=0x5560fa4c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffae659c700 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffae659c700 1 -- 172.21.15.121:0/1759692769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4aace0 msgr2=0x5560fa4c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffae659c700 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4aace0 0x5560fa4c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.036+0000 7ffae659c700 1 -- 172.21.15.121:0/1759692769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560fa4c8b40 con 0x5560fa4b0590 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffae659c700 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4ca320 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ffadc026840 tx=0x7ffadc010eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee974bd7638ef9aa server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffacaff5700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffacaff5700 1 -- 172.21.15.121:0/1759692769 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc026b90 con 0x5560fa4b0590 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffacaff5700 1 -- 172.21.15.121:0/1759692769 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7ffadc026d30 con 0x5560fa4b0590 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1759692769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffad809a960 con 0x5560fa4b0590 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffacaff5700 1 -- 172.21.15.121:0/1759692769 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ffadc043050 con 0x5560fa4b0590 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffaf78bab80 10 client.16743 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1759692769 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5560fa52fd60 con 0x5560fa4b0590 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffacaff5700 1 -- 172.21.15.121:0/1759692769 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffad809a960 con 0x5560fa4b0590 2024-06-10T00:36:48.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffacaff5700 1 -- 172.21.15.121:0/1759692769 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7ffadc0055e0 con 0x5560fa4b0590 2024-06-10T00:36:48.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffaf78bab80 10 client.16743 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:48.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffaf78bab80 10 client.16743 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:36:48.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffaf78bab80 20 client.16743 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:48.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560fa530230 0x5560fa532650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:48.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffaf78bab80 4 client.16743 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 136.0, "await_for": 45.0}] 2024-06-10T00:36:48.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.037+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1759692769 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 136.0, "await_for": 45.0}) -- 0x5560fa403460 con 0x5560fa530230 2024-06-10T00:36:48.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.038+0000 7ffae6d9d700 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560fa530230 0x5560fa532650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:48.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.038+0000 7ffae6d9d700 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560fa530230 0x5560fa532650 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=845a74988bd210e3 server_cookie=9799cb728592cda5 in_seq=0 out_seq=0 2024-06-10T00:36:48.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:48.038+0000 7ffacaff5700 10 client.16743 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.315+0000 7f8d1144d700 1 -- 172.21.15.121:0/1928505839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d0c108510 msgr2=0x7f8d0c1088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.315+0000 7f8d1144d700 1 --2- 172.21.15.121:0/1928505839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d0c108510 0x7f8d0c1088f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009990 tx=0x7f8cfc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.315+0000 7f8d1144d700 1 -- 172.21.15.121:0/1928505839 shutdown_connections 2024-06-10T00:36:49.317 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.315+0000 7f8d1144d700 1 --2- 172.21.15.121:0/1928505839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d0c1013c0 0x7f8d0c10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.315+0000 7f8d1144d700 1 --2- 172.21.15.121:0/1928505839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d0c108510 0x7f8d0c1088f0 secure :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009990 tx=0x7f8cfc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.315+0000 7f8d1144d700 1 --2- 172.21.15.121:0/1928505839 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d0c100a00 0x7f8d0c100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.315+0000 7f8d1144d700 1 -- 172.21.15.121:0/1928505839 >> 172.21.15.121:0/1928505839 conn(0x7f8d0c077990 msgr2=0x7f8d0c0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.315+0000 7f8d1144d700 1 -- 172.21.15.121:0/1928505839 shutdown_connections 2024-06-10T00:36:49.318 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.315+0000 7f8d1144d700 1 -- 172.21.15.121:0/1928505839 wait complete. 2024-06-10T00:36:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.316+0000 7f8d1144d700 1 Processor -- start 2024-06-10T00:36:49.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.316+0000 7f8d1144d700 1 -- start start 2024-06-10T00:36:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.317+0000 7f8d1144d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d0c100a00 0x7f8d0c19b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.317+0000 7f8d1144d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d0c1013c0 0x7f8d0c19b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.317+0000 7f8d1144d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d0c19f8c0 0x7f8d0c19fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.317+0000 7f8d1144d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c112790 con 0x7f8d0c1013c0 2024-06-10T00:36:49.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.317+0000 7f8d1144d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c112610 con 0x7f8d0c100a00 2024-06-10T00:36:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.317+0000 7f8d1144d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c112910 con 0x7f8d0c19f8c0 2024-06-10T00:36:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.317+0000 7f8d0a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d0c1013c0 0x7f8d0c19b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.317+0000 7f8d0a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d0c1013c0 0x7f8d0c19b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37004/0 (socket says 172.21.15.121:37004) 2024-06-10T00:36:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.317+0000 7f8d0a7fc700 1 -- 172.21.15.121:0/2837657477 learned_addr learned my addr 172.21.15.121:0/2837657477 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.318+0000 7f8d0a7fc700 1 -- 172.21.15.121:0/2837657477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d0c19f8c0 msgr2=0x7f8d0c19fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:49.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.318+0000 7f8d0a7fc700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d0c19f8c0 0x7f8d0c19fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.318+0000 7f8d0a7fc700 1 -- 172.21.15.121:0/2837657477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d0c100a00 msgr2=0x7f8d0c19b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.318+0000 7f8d0a7fc700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d0c100a00 0x7f8d0c19b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.318+0000 7f8d0a7fc700 1 -- 172.21.15.121:0/2837657477 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0c1a03e0 con 0x7f8d0c1013c0 2024-06-10T00:36:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.319+0000 7f8d0a7fc700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d0c1013c0 0x7f8d0c19b580 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000e9a0 tx=0x7f8d0000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.319+0000 7f8cf3fff700 1 -- 172.21.15.121:0/2837657477 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0000cbd0 con 0x7f8d0c1013c0 2024-06-10T00:36:49.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.319+0000 7f8cf3fff700 1 -- 172.21.15.121:0/2837657477 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d0000cd70 con 0x7f8d0c1013c0 2024-06-10T00:36:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.319+0000 7f8cf3fff700 1 -- 172.21.15.121:0/2837657477 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00018710 con 0x7f8d0c1013c0 2024-06-10T00:36:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.319+0000 7f8d1144d700 1 -- 172.21.15.121:0/2837657477 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0c1a06d0 con 0x7f8d0c1013c0 2024-06-10T00:36:49.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.320+0000 7f8d1144d700 1 -- 172.21.15.121:0/2837657477 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d0c1094f0 con 0x7f8d0c1013c0 2024-06-10T00:36:49.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.324+0000 7f8cf3fff700 1 -- 172.21.15.121:0/2837657477 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8d00010430 con 0x7f8d0c1013c0 2024-06-10T00:36:49.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.326+0000 7f8d1bcd0b80 1 Processor -- start 2024-06-10T00:36:49.333 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.326+0000 7f8cf3fff700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cf40a2f60 0x7f8cf40a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.326+0000 7f8d1bcd0b80 1 -- start start 2024-06-10T00:36:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.326+0000 7f8d0b7fe700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cf40a2f60 0x7f8cf40a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.326+0000 7f8cf3fff700 1 -- 172.21.15.121:0/2837657477 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f8d000cfad0 con 0x7f8d0c1013c0 2024-06-10T00:36:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.326+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x561535239a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.334 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.326+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 0x5615352339f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.326+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535234350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.326+0000 7f8d1bcd0b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561535174810 con 0x561535233f30 2024-06-10T00:36:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.326+0000 7f8d1bcd0b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x56153511c720 con 0x561535239670 2024-06-10T00:36:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.326+0000 7f8d1bcd0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5615351707d0 con 0x561535239f90 2024-06-10T00:36:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.327+0000 7f8d0a7fc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 0x5615352339f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.335 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.328+0000 7f8d0a7fc700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 0x5615352339f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:50550/0 (socket says 172.21.15.121:50550) 2024-06-10T00:36:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.328+0000 7f8d0a7fc700 1 -- 172.21.15.121:0/2370579411 learned_addr learned my addr 172.21.15.121:0/2370579411 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.328+0000 7f8d0a7fc700 1 -- 172.21.15.121:0/2370579411 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 msgr2=0x561535239a50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-06-10T00:36:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.328+0000 7f8d0a7fc700 1 --2- 172.21.15.121:0/2370579411 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x561535239a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.328+0000 7f8d0a7fc700 1 -- 172.21.15.121:0/2370579411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 msgr2=0x561535234350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:49.336 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.328+0000 7f8d0a7fc700 1 --2- 172.21.15.121:0/2370579411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535234350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.328+0000 7f8d0a7fc700 1 -- 172.21.15.121:0/2370579411 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561535112f50 con 0x561535239f90 2024-06-10T00:36:49.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.329+0000 7f8d0a7fc700 1 --2- 172.21.15.121:0/2370579411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 0x5615352339f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c5b0 tx=0x7f8d0009b580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cbc7fccc3e33763 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:49.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.329+0000 7f8d0b7fe700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cf40a2f60 0x7f8cf40a5420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400e3a0 tx=0x7f8d0400e760 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:49.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.329+0000 7f8cf17fa700 1 -- 172.21.15.121:0/2370579411 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0009ff50 con 0x561535239f90 2024-06-10T00:36:49.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.330+0000 7f8cf17fa700 1 -- 172.21.15.121:0/2370579411 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d000a4810 con 0x561535239f90 2024-06-10T00:36:49.337 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.330+0000 7f8cf17fa700 1 -- 172.21.15.121:0/2370579411 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d000a4380 con 0x561535239f90 2024-06-10T00:36:49.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.331+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2370579411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 msgr2=0x5615352339f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:49.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.331+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/2370579411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 0x5615352339f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c5b0 tx=0x7f8d0009b580 comp rx=0 tx=0).stop 2024-06-10T00:36:49.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.332+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2370579411 shutdown_connections 2024-06-10T00:36:49.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.332+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/2370579411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535234350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.332+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/2370579411 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x561535239a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.338 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.332+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/2370579411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 0x5615352339f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.332+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2370579411 >> 172.21.15.121:0/2370579411 conn(0x56153514dcc0 msgr2=0x56153521d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:49.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.332+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2370579411 shutdown_connections 2024-06-10T00:36:49.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.332+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2370579411 wait complete. 2024-06-10T00:36:49.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.333+0000 7f8d1bcd0b80 1 -- start start 2024-06-10T00:36:49.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.333+0000 7f8d1bcd0b80 1 Processor -- start 2024-06-10T00:36:49.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.333+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535233f30 0x56153523d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.333+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535239670 0x56153523dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.334+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 0x56153523e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.334+0000 7f8d0affd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535233f30 0x56153523d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.334+0000 7f8d0affd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535233f30 0x56153523d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:45484/0 (socket says 172.21.15.121:45484) 2024-06-10T00:36:49.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.334+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2274384011 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561535222d80 con 0x561535239670 2024-06-10T00:36:49.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.334+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2274384011 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x561535114bf0 con 0x561535233f30 2024-06-10T00:36:49.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.334+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2274384011 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561535222f00 con 0x561535239f90 2024-06-10T00:36:49.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.334+0000 7f8d1bcd0b80 10 client.0 _ll_register_callbacks cb 0x56153510e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:49.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.334+0000 7f8d0a7fc700 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535239670 0x56153523dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.334+0000 7f8d0affd700 1 -- 172.21.15.121:0/2274384011 learned_addr learned my addr 172.21.15.121:0/2274384011 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:49.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.334+0000 7f8d0b7fe700 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 0x56153523e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.335+0000 7f8d0a7fc700 1 -- 172.21.15.121:0/2274384011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 msgr2=0x56153523e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:49.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.335+0000 7f8d0a7fc700 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 0x56153523e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.335+0000 7f8d0a7fc700 1 -- 172.21.15.121:0/2274384011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535233f30 msgr2=0x56153523d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:49.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.335+0000 7f8d0a7fc700 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535233f30 0x56153523d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.335+0000 7f8d0a7fc700 1 -- 172.21.15.121:0/2274384011 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615350c38d0 con 0x561535239670 2024-06-10T00:36:49.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.335+0000 7f8d0a7fc700 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535239670 0x56153523dbb0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8d0009c200 tx=0x7f8d0001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92da98b94f59744d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:49.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.335+0000 7f8ceeff5700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:49.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.335+0000 7f8ceeff5700 1 -- 172.21.15.121:0/2274384011 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d000a4380 con 0x561535239670 2024-06-10T00:36:49.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.335+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2274384011 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d000a4810 con 0x561535239670 2024-06-10T00:36:49.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.335+0000 7f8ceeff5700 1 -- 172.21.15.121:0/2274384011 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f8d000a4520 con 0x561535239670 2024-06-10T00:36:49.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8ceeff5700 1 -- 172.21.15.121:0/2274384011 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8d000ba2f0 con 0x561535239670 2024-06-10T00:36:49.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8d1bcd0b80 10 client.16752 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:49.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2274384011 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56153522dcf0 con 0x561535239670 2024-06-10T00:36:49.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8ceeff5700 1 -- 172.21.15.121:0/2274384011 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d000a4810 con 0x561535239670 2024-06-10T00:36:49.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8ceeff5700 1 -- 172.21.15.121:0/2274384011 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f8d0009aa60 con 0x561535239670 2024-06-10T00:36:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8d1bcd0b80 10 client.16752 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8d1bcd0b80 10 client.16752 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:36:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8d1bcd0b80 20 client.16752 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x561535230280 0x561535232690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8d1bcd0b80 4 client.16752 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:36:49.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2274384011 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561535181d10 con 0x561535230280 2024-06-10T00:36:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.336+0000 7f8d0affd700 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x561535230280 0x561535232690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.337+0000 7f8ceeff5700 10 client.16752 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:36:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.337+0000 7f8d0affd700 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x561535230280 0x561535232690 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=676c47324dbcdcc8 server_cookie=6765ce617c0b3efd in_seq=0 out_seq=0 2024-06-10T00:36:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8ceeff5700 1 -- 172.21.15.121:0/2274384011 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x561535181d10 con 0x561535230280 2024-06-10T00:36:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8ceeff5700 10 client.16752 handle_command_reply: tid=0 2024-06-10T00:36:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 1 client.16752 shutdown 2024-06-10T00:36:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2274384011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535239670 msgr2=0x56153523dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535239670 0x56153523dbb0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f8d0009c200 tx=0x7f8d0001f000 comp rx=0 tx=0).stop 2024-06-10T00:36:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2274384011 shutdown_connections 2024-06-10T00:36:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x561535230280 0x561535232690 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535239f90 0x56153523e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8ceeff5700 0 client.16752 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:36:49.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535233f30 0x56153523d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/2274384011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535239670 0x56153523dbb0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2274384011 >> 172.21.15.121:0/2274384011 conn(0x56153514dcc0 msgr2=0x5615352219d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:49.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2274384011 shutdown_connections 2024-06-10T00:36:49.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/2274384011 wait complete. 2024-06-10T00:36:49.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.339+0000 7f8d1bcd0b80 20 client.16752 trim_cache size 0 max 16384 2024-06-10T00:36:49.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d1bcd0b80 1 Processor -- start 2024-06-10T00:36:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d1bcd0b80 1 -- start start 2024-06-10T00:36:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 0x56153523cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535233680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x561535238690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d1bcd0b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1af4e0 con 0x561535233f30 2024-06-10T00:36:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d1bcd0b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1accb0 con 0x561535239670 2024-06-10T00:36:49.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d1bcd0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1aab20 con 0x561535230280 2024-06-10T00:36:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d0a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535233680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d0affd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 0x56153523cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d0affd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 0x56153523cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:50568/0 (socket says 172.21.15.121:50568) 2024-06-10T00:36:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d0a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535233680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37034/0 (socket says 172.21.15.121:37034) 2024-06-10T00:36:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d0affd700 1 -- 172.21.15.121:0/4065734293 learned_addr learned my addr 172.21.15.121:0/4065734293 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:49.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.355+0000 7f8d0b7fe700 1 --2- 172.21.15.121:0/4065734293 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x561535238690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.356+0000 7f8d0affd700 1 -- 172.21.15.121:0/4065734293 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 msgr2=0x561535238690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.356+0000 7f8d0affd700 1 --2- 172.21.15.121:0/4065734293 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x561535238690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.356+0000 7f8d0affd700 1 -- 172.21.15.121:0/4065734293 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 msgr2=0x561535233680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.356+0000 7f8d0affd700 1 --2- 172.21.15.121:0/4065734293 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535233680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.356+0000 7f8d0affd700 1 -- 172.21.15.121:0/4065734293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56153522c7a0 con 0x561535230280 2024-06-10T00:36:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.356+0000 7f8d0a7fc700 1 --2- 172.21.15.121:0/4065734293 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535233680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.356+0000 7f8d0affd700 1 --2- 172.21.15.121:0/4065734293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 0x56153523cf20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009990 tx=0x7f8cfc00b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f27b6d2b408e9bf0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:49.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.356+0000 7f8cf0ff9700 1 -- 172.21.15.121:0/4065734293 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc028020 con 0x561535230280 2024-06-10T00:36:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.356+0000 7f8cf0ff9700 1 -- 172.21.15.121:0/4065734293 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cfc00be60 con 0x561535230280 2024-06-10T00:36:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.356+0000 7f8cf0ff9700 1 -- 172.21.15.121:0/4065734293 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc003b90 con 0x561535230280 2024-06-10T00:36:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.357+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/4065734293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 msgr2=0x56153523cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.357+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/4065734293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 0x56153523cf20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009990 tx=0x7f8cfc00b260 comp rx=0 tx=0).stop 2024-06-10T00:36:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.357+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/4065734293 shutdown_connections 2024-06-10T00:36:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.357+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/4065734293 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x561535238690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.357+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/4065734293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 0x56153523cf20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.357+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/4065734293 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535233680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.357+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/4065734293 >> 172.21.15.121:0/4065734293 conn(0x56153514dcc0 msgr2=0x56153523bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.357+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/4065734293 shutdown_connections 2024-06-10T00:36:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.357+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/4065734293 wait complete. 2024-06-10T00:36:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.357+0000 7f8d1bcd0b80 1 -- start start 2024-06-10T00:36:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.357+0000 7f8d1bcd0b80 1 Processor -- start 2024-06-10T00:36:49.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 0x561535267cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535228190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d1bcd0b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x5615352286d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d0a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535228190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d0affd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 0x561535267cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d0affd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 0x561535267cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:50570/0 (socket says 172.21.15.121:50570) 2024-06-10T00:36:49.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d0a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535228190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37040/0 (socket says 172.21.15.121:37040) 2024-06-10T00:36:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d0affd700 1 -- 172.21.15.121:0/284377034 learned_addr learned my addr 172.21.15.121:0/284377034 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/284377034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1ab170 con 0x561535233f30 2024-06-10T00:36:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/284377034 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1ac5e0 con 0x561535239670 2024-06-10T00:36:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/284377034 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c1ab840 con 0x561535230280 2024-06-10T00:36:49.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d1bcd0b80 10 client.0 _ll_register_callbacks cb 0x56153522ca70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d0b7fe700 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x5615352286d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d0b7fe700 1 -- 172.21.15.121:0/284377034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 msgr2=0x561535267cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d0b7fe700 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 0x561535267cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d0b7fe700 1 -- 172.21.15.121:0/284377034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 msgr2=0x561535228190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.358+0000 7f8d0b7fe700 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535228190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.359+0000 7f8d0b7fe700 1 -- 172.21.15.121:0/284377034 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561535232a70 con 0x561535239670 2024-06-10T00:36:49.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.359+0000 7f8d0b7fe700 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x5615352286d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x561535219f60 tx=0x7f8d04009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fff504565e22da8 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.359+0000 7f8ceeff5700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.359+0000 7f8ceeff5700 1 -- 172.21.15.121:0/284377034 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04026070 con 0x561535239670 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.359+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/284377034 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cfc00be60 con 0x561535239670 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.359+0000 7f8ceeff5700 1 -- 172.21.15.121:0/284377034 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f8d04004bb0 con 0x561535239670 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.359+0000 7f8ceeff5700 1 -- 172.21.15.121:0/284377034 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8d04031050 con 0x561535239670 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.359+0000 7f8d1bcd0b80 10 client.26297 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.359+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/284377034 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5615352c8770 con 0x561535239670 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.359+0000 7f8ceeff5700 1 -- 172.21.15.121:0/284377034 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cfc00be60 con 0x561535239670 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.360+0000 7f8ceeff5700 1 -- 172.21.15.121:0/284377034 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f8d04016050 con 0x561535239670 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.360+0000 7f8d1bcd0b80 10 client.26297 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.360+0000 7f8d1bcd0b80 10 client.26297 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.360+0000 7f8d1bcd0b80 20 client.26297 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.360+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x561535239f90 0x5615352caf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:49.373 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.360+0000 7f8d1bcd0b80 4 client.26297 mds_command: new command op to 24472 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.360+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/284377034 --> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56153517c290 con 0x561535239f90 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.360+0000 7f8d0affd700 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x561535239f90 0x5615352caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.360+0000 7f8d0affd700 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x561535239f90 0x5615352caf00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b4bee0b5ec2cdd33 server_cookie=107689a26ab24303 in_seq=0 out_seq=0 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.360+0000 7f8ceeff5700 10 client.26297 ms_handle_connect on v2:172.21.15.193:6838/2677954983 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8ceeff5700 1 -- 172.21.15.121:0/284377034 <== mds.1 v2:172.21.15.193:6838/2677954983 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56153517c290 con 0x561535239f90 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8ceeff5700 10 client.26297 handle_command_reply: tid=0 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 1 client.26297 shutdown 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/284377034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 msgr2=0x5615352286d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x5615352286d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x561535219f60 tx=0x7f8d04009920 comp rx=0 tx=0).stop 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/284377034 shutdown_connections 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.193:6838/2677954983,v1:172.21.15.193:6839/2677954983] conn(0x561535239f90 0x5615352caf00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8ceeff5700 0 client.26297 ms_handle_reset on v2:172.21.15.193:6838/2677954983 2024-06-10T00:36:49.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x561535239670 0x5615352286d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561535230280 0x561535267cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 1 --2- 172.21.15.121:0/284377034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561535233f30 0x561535228190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/284377034 >> 172.21.15.121:0/284377034 conn(0x56153514dcc0 msgr2=0x561535216e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:49.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/284377034 shutdown_connections 2024-06-10T00:36:49.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 1 -- 172.21.15.121:0/284377034 wait complete. 2024-06-10T00:36:49.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.362+0000 7f8d1bcd0b80 20 client.26297 trim_cache size 0 max 16384 2024-06-10T00:36:49.375 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:36:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 -- 172.21.15.121:0/2837657477 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cf40a2f60 msgr2=0x7f8cf40a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cf40a2f60 0x7f8cf40a5420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400e3a0 tx=0x7f8d0400e760 comp rx=0 tx=0).stop 2024-06-10T00:36:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 -- 172.21.15.121:0/2837657477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d0c1013c0 msgr2=0x7f8d0c19b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d0c1013c0 0x7f8d0c19b580 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000e9a0 tx=0x7f8d0000ed60 comp rx=0 tx=0).stop 2024-06-10T00:36:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 -- 172.21.15.121:0/2837657477 shutdown_connections 2024-06-10T00:36:49.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8cf40a2f60 0x7f8cf40a5420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d0c19f8c0 0x7f8d0c19fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d0c100a00 0x7f8d0c19b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 --2- 172.21.15.121:0/2837657477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d0c1013c0 0x7f8d0c19b580 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 -- 172.21.15.121:0/2837657477 >> 172.21.15.121:0/2837657477 conn(0x7f8d0c077990 msgr2=0x7f8d0c0feea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 -- 172.21.15.121:0/2837657477 shutdown_connections 2024-06-10T00:36:49.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:49.363+0000 7f8d1144d700 1 -- 172.21.15.121:0/2837657477 wait complete. 2024-06-10T00:36:49.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:49 smithi193 ceph-mon[36716]: pgmap v272: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 226 op/s 2024-06-10T00:36:49.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:49 smithi121 ceph-mon[32378]: pgmap v272: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 226 op/s 2024-06-10T00:36:49.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:49 smithi135 ceph-mon[36424]: pgmap v272: 129 pgs: 129 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 226 op/s 2024-06-10T00:36:50.240 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-06-10T00:36:50.487 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:36:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:51 smithi193 ceph-mon[36716]: pgmap v273: 129 pgs: 129 active+clean; 1.6 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 288 op/s 2024-06-10T00:36:51.647 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:51 smithi121 ceph-mon[32378]: pgmap v273: 129 pgs: 129 active+clean; 1.6 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 288 op/s 2024-06-10T00:36:51.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:51 smithi135 ceph-mon[36424]: pgmap v273: 129 pgs: 129 active+clean; 1.6 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 288 op/s 2024-06-10T00:36:52.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.189+0000 7f38723b5700 1 -- 172.21.15.121:0/1460340199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f386c101810 msgr2=0x7f386c10f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:52.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.189+0000 7f38723b5700 1 --2- 172.21.15.121:0/1460340199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f386c101810 0x7f386c10f280 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f386400b0a0 tx=0x7f386400b3b0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.189+0000 7f38723b5700 1 -- 172.21.15.121:0/1460340199 shutdown_connections 2024-06-10T00:36:52.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.189+0000 7f38723b5700 1 --2- 172.21.15.121:0/1460340199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f386c101810 0x7f386c10f280 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.189+0000 7f38723b5700 1 --2- 172.21.15.121:0/1460340199 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f386c107570 0x7f386c107950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.189+0000 7f38723b5700 1 --2- 172.21.15.121:0/1460340199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c100e50 0x7f386c1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.191 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.189+0000 7f38723b5700 1 -- 172.21.15.121:0/1460340199 >> 172.21.15.121:0/1460340199 conn(0x7f386c0766b0 msgr2=0x7f386c076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.189+0000 7f38723b5700 1 -- 172.21.15.121:0/1460340199 shutdown_connections 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.189+0000 7f38723b5700 1 -- 172.21.15.121:0/1460340199 wait complete. 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.189+0000 7f38723b5700 1 Processor -- start 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f38723b5700 1 -- start start 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f38723b5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f386c100e50 0x7f386c19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f38723b5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c101810 0x7f386c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f38723b5700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f386c107570 0x7f386c198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f38723b5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1127d0 con 0x7f386c100e50 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f38723b5700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f386c112650 con 0x7f386c107570 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f38723b5700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f386c112950 con 0x7f386c101810 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f386bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f386c100e50 0x7f386c19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f386bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f386c100e50 0x7f386c19e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37050/0 (socket says 172.21.15.121:37050) 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f386bfff700 1 -- 172.21.15.121:0/2294679268 learned_addr learned my addr 172.21.15.121:0/2294679268 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f3870952700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f386c107570 0x7f386c198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.192 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f386b7fe700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c101810 0x7f386c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f386bfff700 1 -- 172.21.15.121:0/2294679268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c101810 msgr2=0x7f386c19edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f386bfff700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c101810 0x7f386c19edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f386bfff700 1 -- 172.21.15.121:0/2294679268 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f386c107570 msgr2=0x7f386c198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f386bfff700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f386c107570 0x7f386c198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.190+0000 7f386bfff700 1 -- 172.21.15.121:0/2294679268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386c199190 con 0x7f386c100e50 2024-06-10T00:36:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.191+0000 7f386bfff700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f386c100e50 0x7f386c19e870 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f385c00e970 tx=0x7f385c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.191+0000 7f38697fa700 1 -- 172.21.15.121:0/2294679268 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c00cbd0 con 0x7f386c100e50 2024-06-10T00:36:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.191+0000 7f38697fa700 1 -- 172.21.15.121:0/2294679268 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f385c004500 con 0x7f386c100e50 2024-06-10T00:36:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.191+0000 7f38697fa700 1 -- 172.21.15.121:0/2294679268 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c003e30 con 0x7f386c100e50 2024-06-10T00:36:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.191+0000 7f38723b5700 1 -- 172.21.15.121:0/2294679268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f386c199480 con 0x7f386c100e50 2024-06-10T00:36:52.193 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.191+0000 7f38723b5700 1 -- 172.21.15.121:0/2294679268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f386c1a5740 con 0x7f386c100e50 2024-06-10T00:36:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.193+0000 7f38697fa700 1 -- 172.21.15.121:0/2294679268 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f385c004000 con 0x7f386c100e50 2024-06-10T00:36:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.194+0000 7f38697fa700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f38540a2d90 0x7f38540a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.194+0000 7f386b7fe700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f38540a2d90 0x7f38540a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.194+0000 7f38697fa700 1 -- 172.21.15.121:0/2294679268 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f385c0cfb50 con 0x7f386c100e50 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.194+0000 7f386b7fe700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f38540a2d90 0x7f38540a5250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f386c106490 tx=0x7f3860005cf0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.194+0000 7f387cc38b80 1 Processor -- start 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.194+0000 7f387cc38b80 1 -- start start 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586a5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 0x55ce5869faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a0030 0x55ce586ac210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f387cc38b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ce5858df60 con 0x55ce586a5470 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f387cc38b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55ce585ec850 con 0x55ce5869f670 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f387cc38b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ce58554460 con 0x55ce586a0030 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f386bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f386b7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 0x55ce5869faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.196 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f386bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586a5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37056/0 (socket says 172.21.15.121:37056) 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f386bfff700 1 -- 172.21.15.121:0/1082864877 learned_addr learned my addr 172.21.15.121:0/1082864877 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f3870952700 1 --2- 172.21.15.121:0/1082864877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a0030 0x55ce586ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f386bfff700 1 -- 172.21.15.121:0/1082864877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a0030 msgr2=0x55ce586ac210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f386bfff700 1 --2- 172.21.15.121:0/1082864877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a0030 0x55ce586ac210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f386bfff700 1 -- 172.21.15.121:0/1082864877 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 msgr2=0x55ce5869faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f386bfff700 1 --2- 172.21.15.121:0/1082864877 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 0x55ce5869faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f386bfff700 1 -- 172.21.15.121:0/1082864877 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce5853b8d0 con 0x55ce586a5470 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f386b7fe700 1 --2- 172.21.15.121:0/1082864877 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 0x55ce5869faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f386bfff700 1 --2- 172.21.15.121:0/1082864877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586a5850 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f385c0985f0 tx=0x7f385c09ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0ba0d3e2acd001 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f38527fc700 1 -- 172.21.15.121:0/1082864877 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c00ce00 con 0x55ce586a5470 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f38527fc700 1 -- 172.21.15.121:0/1082864877 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f385c09c320 con 0x55ce586a5470 2024-06-10T00:36:52.197 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.195+0000 7f38527fc700 1 -- 172.21.15.121:0/1082864877 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c098290 con 0x55ce586a5470 2024-06-10T00:36:52.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.196+0000 7f387cc38b80 1 -- 172.21.15.121:0/1082864877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 msgr2=0x55ce586a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:52.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.196+0000 7f387cc38b80 1 --2- 172.21.15.121:0/1082864877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586a5850 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f385c0985f0 tx=0x7f385c09ba70 comp rx=0 tx=0).stop 2024-06-10T00:36:52.200 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.196+0000 7f387cc38b80 1 -- 172.21.15.121:0/1082864877 shutdown_connections 2024-06-10T00:36:52.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.196+0000 7f387cc38b80 1 --2- 172.21.15.121:0/1082864877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a0030 0x55ce586ac210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.196+0000 7f387cc38b80 1 --2- 172.21.15.121:0/1082864877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586a5850 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.196+0000 7f387cc38b80 1 --2- 172.21.15.121:0/1082864877 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 0x55ce5869faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.196+0000 7f387cc38b80 1 -- 172.21.15.121:0/1082864877 >> 172.21.15.121:0/1082864877 conn(0x55ce585c8080 msgr2=0x55ce58696220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:52.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.196+0000 7f387cc38b80 1 -- 172.21.15.121:0/1082864877 shutdown_connections 2024-06-10T00:36:52.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.196+0000 7f387cc38b80 1 -- 172.21.15.121:0/1082864877 wait complete. 2024-06-10T00:36:52.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f387cc38b80 1 -- start start 2024-06-10T00:36:52.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f387cc38b80 1 Processor -- start 2024-06-10T00:36:52.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce5869f670 0x55ce586abef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce586a0030 0x55ce586a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a5470 0x55ce586a9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f387cc38b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ce5858cfc0 con 0x55ce5869f670 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f387cc38b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55ce585e8810 con 0x55ce586a0030 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f387cc38b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ce586ac750 con 0x55ce586a5470 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f387cc38b80 10 client.0 _ll_register_callbacks cb 0x55ce585825b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f386bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce5869f670 0x55ce586abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f386bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce5869f670 0x55ce586abef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37072/0 (socket says 172.21.15.121:37072) 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f386bfff700 1 -- 172.21.15.121:0/26744897 learned_addr learned my addr 172.21.15.121:0/26744897 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f386b7fe700 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce586a0030 0x55ce586a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.197+0000 7f3870952700 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a5470 0x55ce586a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f386bfff700 1 -- 172.21.15.121:0/26744897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a5470 msgr2=0x55ce586a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:52.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f386bfff700 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a5470 0x55ce586a9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f386bfff700 1 -- 172.21.15.121:0/26744897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce586a0030 msgr2=0x55ce586a91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f386bfff700 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce586a0030 0x55ce586a91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f386bfff700 1 -- 172.21.15.121:0/26744897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce585b2750 con 0x55ce5869f670 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f3870952700 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a5470 0x55ce586a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f386b7fe700 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce586a0030 0x55ce586a91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f386bfff700 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce5869f670 0x55ce586abef0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f385c003fd0 tx=0x7f385c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58372200365776bb server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f384b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f384b7fe700 1 -- 172.21.15.121:0/26744897 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c09bc20 con 0x55ce5869f670 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f387cc38b80 1 -- 172.21.15.121:0/26744897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f385c09c320 con 0x55ce5869f670 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f384b7fe700 1 -- 172.21.15.121:0/26744897 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f385c099f20 con 0x55ce5869f670 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f384b7fe700 1 -- 172.21.15.121:0/26744897 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f385c0a4890 con 0x55ce5869f670 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f387cc38b80 10 client.16770 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f387cc38b80 1 -- 172.21.15.121:0/26744897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ce586b6dd0 con 0x55ce5869f670 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f384b7fe700 1 -- 172.21.15.121:0/26744897 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f385c09c320 con 0x55ce5869f670 2024-06-10T00:36:52.203 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f384b7fe700 1 -- 172.21.15.121:0/26744897 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f385c0a4400 con 0x55ce5869f670 2024-06-10T00:36:52.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f387cc38b80 10 client.16770 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:52.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.198+0000 7f387cc38b80 10 client.16770 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:36:52.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.199+0000 7f387cc38b80 20 client.16770 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:52.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.199+0000 7f387cc38b80 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ce586b92a0 0x55ce586bb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.199+0000 7f387cc38b80 4 client.16770 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:36:52.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.199+0000 7f387cc38b80 1 -- 172.21.15.121:0/26744897 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce585f9d50 con 0x55ce586b92a0 2024-06-10T00:36:52.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.199+0000 7f386b7fe700 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ce586b92a0 0x55ce586bb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.199+0000 7f384b7fe700 10 client.16770 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:52.204 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.199+0000 7f386b7fe700 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ce586b92a0 0x55ce586bb690 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=662c223a9e478575 server_cookie=d45ec1a1e07dcc8 in_seq=0 out_seq=0 2024-06-10T00:36:52.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f384b7fe700 1 -- 172.21.15.121:0/26744897 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55ce585f9d50 con 0x55ce586b92a0 2024-06-10T00:36:52.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f384b7fe700 10 client.16770 handle_command_reply: tid=0 2024-06-10T00:36:52.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f387cc38b80 1 client.16770 shutdown 2024-06-10T00:36:52.205 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f387cc38b80 1 -- 172.21.15.121:0/26744897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce5869f670 msgr2=0x55ce586abef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:52.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f387cc38b80 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce5869f670 0x55ce586abef0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f385c003fd0 tx=0x7f385c003680 comp rx=0 tx=0).stop 2024-06-10T00:36:52.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f387cc38b80 1 -- 172.21.15.121:0/26744897 shutdown_connections 2024-06-10T00:36:52.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f387cc38b80 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ce586b92a0 0x55ce586bb690 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f384b7fe700 0 client.16770 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:52.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f387cc38b80 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a5470 0x55ce586a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f387cc38b80 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce5869f670 0x55ce586abef0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f387cc38b80 1 --2- 172.21.15.121:0/26744897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce586a0030 0x55ce586a91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f387cc38b80 1 -- 172.21.15.121:0/26744897 >> 172.21.15.121:0/26744897 conn(0x55ce585c8080 msgr2=0x55ce58695700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:52.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f387cc38b80 1 -- 172.21.15.121:0/26744897 shutdown_connections 2024-06-10T00:36:52.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.203+0000 7f387cc38b80 1 -- 172.21.15.121:0/26744897 wait complete. 2024-06-10T00:36:52.206 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.204+0000 7f387cc38b80 20 client.16770 trim_cache size 0 max 16384 2024-06-10T00:36:52.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.218+0000 7f387cc38b80 1 Processor -- start 2024-06-10T00:36:52.219 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.218+0000 7f387cc38b80 1 -- start start 2024-06-10T00:36:52.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 0x55ce586a4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a0030 0x55ce58695e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.220 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a5470 0x55ce586963c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f387cc38b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1aef60 con 0x55ce586a0030 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f387cc38b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1ac730 con 0x55ce5869f670 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f387cc38b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1aa5a0 con 0x55ce586a5470 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f386b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a0030 0x55ce58695e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f386bfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 0x55ce586a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f386b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a0030 0x55ce58695e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37084/0 (socket says 172.21.15.121:37084) 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f386b7fe700 1 -- 172.21.15.121:0/3681452874 learned_addr learned my addr 172.21.15.121:0/3681452874 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f3870952700 1 --2- 172.21.15.121:0/3681452874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a5470 0x55ce586963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f386b7fe700 1 -- 172.21.15.121:0/3681452874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a5470 msgr2=0x55ce586963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f386b7fe700 1 --2- 172.21.15.121:0/3681452874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a5470 0x55ce586963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f386b7fe700 1 -- 172.21.15.121:0/3681452874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 msgr2=0x55ce586a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:52.221 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f386b7fe700 1 --2- 172.21.15.121:0/3681452874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 0x55ce586a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f386b7fe700 1 -- 172.21.15.121:0/3681452874 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce586a7760 con 0x55ce586a0030 2024-06-10T00:36:52.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.219+0000 7f386bfff700 1 --2- 172.21.15.121:0/3681452874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 0x55ce586a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-10T00:36:52.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.220+0000 7f386b7fe700 1 --2- 172.21.15.121:0/3681452874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a0030 0x55ce58695e80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3860005950 tx=0x7f3860010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76fd1d23eb52e4b server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:52.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.220+0000 7f3851ffb700 1 -- 172.21.15.121:0/3681452874 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386000f9a0 con 0x55ce586a0030 2024-06-10T00:36:52.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.220+0000 7f3851ffb700 1 -- 172.21.15.121:0/3681452874 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3860004d10 con 0x55ce586a0030 2024-06-10T00:36:52.222 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.220+0000 7f3851ffb700 1 -- 172.21.15.121:0/3681452874 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3860005620 con 0x55ce586a0030 2024-06-10T00:36:52.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.221+0000 7f387cc38b80 1 -- 172.21.15.121:0/3681452874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a0030 msgr2=0x55ce58695e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:52.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.221+0000 7f387cc38b80 1 --2- 172.21.15.121:0/3681452874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a0030 0x55ce58695e80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3860005950 tx=0x7f3860010000 comp rx=0 tx=0).stop 2024-06-10T00:36:52.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.221+0000 7f387cc38b80 1 -- 172.21.15.121:0/3681452874 shutdown_connections 2024-06-10T00:36:52.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.221+0000 7f387cc38b80 1 --2- 172.21.15.121:0/3681452874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce586a5470 0x55ce586963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.221+0000 7f387cc38b80 1 --2- 172.21.15.121:0/3681452874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce5869f670 0x55ce586a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.221+0000 7f387cc38b80 1 --2- 172.21.15.121:0/3681452874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a0030 0x55ce58695e80 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.221+0000 7f387cc38b80 1 -- 172.21.15.121:0/3681452874 >> 172.21.15.121:0/3681452874 conn(0x55ce585c8080 msgr2=0x55ce5869aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.221+0000 7f387cc38b80 1 -- 172.21.15.121:0/3681452874 shutdown_connections 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.221+0000 7f387cc38b80 1 -- 172.21.15.121:0/3681452874 wait complete. 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.221+0000 7f387cc38b80 1 -- start start 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f387cc38b80 1 Processor -- start 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce5869f670 0x55ce586be640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce586a0030 0x55ce586beb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f387cc38b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586bbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f387cc38b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1aabf0 con 0x55ce586a5470 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f387cc38b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1ac060 con 0x55ce586a0030 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f387cc38b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f386c1ab2c0 con 0x55ce5869f670 2024-06-10T00:36:52.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f387cc38b80 10 client.0 _ll_register_callbacks cb 0x55ce585825b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f3870952700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586bbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f3870952700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586bbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37088/0 (socket says 172.21.15.121:37088) 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f3870952700 1 -- 172.21.15.121:0/1964766959 learned_addr learned my addr 172.21.15.121:0/1964766959 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f386b7fe700 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce586a0030 0x55ce586beb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f386bfff700 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce5869f670 0x55ce586be640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f3870952700 1 -- 172.21.15.121:0/1964766959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce5869f670 msgr2=0x55ce586be640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f3870952700 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce5869f670 0x55ce586be640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f3870952700 1 -- 172.21.15.121:0/1964766959 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce586a0030 msgr2=0x55ce586beb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f3870952700 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce586a0030 0x55ce586beb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.222+0000 7f3870952700 1 -- 172.21.15.121:0/1964766959 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce586bc3a0 con 0x55ce586a5470 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.223+0000 7f3870952700 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586bbc50 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f386400b0a0 tx=0x7f3864004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecb65cbbbe6a876 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.223+0000 7f384b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:52.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.223+0000 7f384b7fe700 1 -- 172.21.15.121:0/1964766959 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3864004e50 con 0x55ce586a5470 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.223+0000 7f384b7fe700 1 -- 172.21.15.121:0/1964766959 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f38640074e0 con 0x55ce586a5470 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.223+0000 7f387cc38b80 1 -- 172.21.15.121:0/1964766959 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3860004d10 con 0x55ce586a5470 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.223+0000 7f384b7fe700 1 -- 172.21.15.121:0/1964766959 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f38640076e0 con 0x55ce586a5470 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.223+0000 7f387cc38b80 10 client.16776 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.223+0000 7f387cc38b80 1 -- 172.21.15.121:0/1964766959 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55ce58723670 con 0x55ce586a5470 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.223+0000 7f384b7fe700 1 -- 172.21.15.121:0/1964766959 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3860004d10 con 0x55ce586a5470 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.223+0000 7f384b7fe700 1 -- 172.21.15.121:0/1964766959 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f3864011070 con 0x55ce586a5470 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.224+0000 7f387cc38b80 10 client.16776 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.224+0000 7f387cc38b80 10 client.16776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.224+0000 7f387cc38b80 20 client.16776 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.224+0000 7f387cc38b80 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ce586b92a0 0x55ce58725ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.224+0000 7f387cc38b80 4 client.16776 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-06-10T00:36:52.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.224+0000 7f387cc38b80 1 -- 172.21.15.121:0/1964766959 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ce585f42d0 con 0x55ce586b92a0 2024-06-10T00:36:52.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.224+0000 7f386bfff700 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ce586b92a0 0x55ce58725ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:52.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.224+0000 7f384b7fe700 10 client.16776 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:52.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.224+0000 7f386bfff700 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ce586b92a0 0x55ce58725ed0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5795b1deeb86e31 server_cookie=bbdfed68e53a2941 in_seq=0 out_seq=0 2024-06-10T00:36:52.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f384b7fe700 1 -- 172.21.15.121:0/1964766959 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ce585f42d0 con 0x55ce586b92a0 2024-06-10T00:36:52.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f384b7fe700 10 client.16776 handle_command_reply: tid=0 2024-06-10T00:36:52.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 1 client.16776 shutdown 2024-06-10T00:36:52.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 1 -- 172.21.15.121:0/1964766959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 msgr2=0x55ce586bbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:52.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586bbc50 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f386400b0a0 tx=0x7f3864004a10 comp rx=0 tx=0).stop 2024-06-10T00:36:52.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 1 -- 172.21.15.121:0/1964766959 shutdown_connections 2024-06-10T00:36:52.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55ce586b92a0 0x55ce58725ed0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ce586a5470 0x55ce586bbc50 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.231 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ce5869f670 0x55ce586be640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 1 --2- 172.21.15.121:0/1964766959 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55ce586a0030 0x55ce586beb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 1 -- 172.21.15.121:0/1964766959 >> 172.21.15.121:0/1964766959 conn(0x55ce585c8080 msgr2=0x55ce586a2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:52.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f384b7fe700 0 client.16776 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:52.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 1 -- 172.21.15.121:0/1964766959 shutdown_connections 2024-06-10T00:36:52.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 1 -- 172.21.15.121:0/1964766959 wait complete. 2024-06-10T00:36:52.232 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.229+0000 7f387cc38b80 20 client.16776 trim_cache size 0 max 16384 2024-06-10T00:36:52.232 INFO:teuthology.orchestra.run.smithi121.stdout:[] 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 -- 172.21.15.121:0/2294679268 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f38540a2d90 msgr2=0x7f38540a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f38540a2d90 0x7f38540a5250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f386c106490 tx=0x7f3860005cf0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 -- 172.21.15.121:0/2294679268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f386c100e50 msgr2=0x7f386c19e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f386c100e50 0x7f386c19e870 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f385c00e970 tx=0x7f385c00ec80 comp rx=0 tx=0).stop 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 -- 172.21.15.121:0/2294679268 shutdown_connections 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f38540a2d90 0x7f38540a5250 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f386c107570 0x7f386c198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f386c100e50 0x7f386c19e870 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 --2- 172.21.15.121:0/2294679268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f386c101810 0x7f386c19edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 -- 172.21.15.121:0/2294679268 >> 172.21.15.121:0/2294679268 conn(0x7f386c0766b0 msgr2=0x7f386c0ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 -- 172.21.15.121:0/2294679268 shutdown_connections 2024-06-10T00:36:52.233 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:52.230+0000 7f38723b5700 1 -- 172.21.15.121:0/2294679268 wait complete. 2024-06-10T00:36:53.036 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 34338598, 'ino': 1099511634775, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-6.jpg'}, {'damage_type': 'backtrace', 'id': 76697924, 'ino': 1099511634649, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-41.jpg'}, {'damage_type': 'backtrace', 'id': 102277572, 'ino': 1099511634403, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-12.xml'}, {'damage_type': 'backtrace', 'id': 124889576, 'ino': 1099511634635, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-47.jpg'}, {'damage_type': 'backtrace', 'id': 167324012, 'ino': 1099511634651, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-40.jpg'}, {'damage_type': 'backtrace', 'id': 180342043, 'ino': 1099511634575, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-55.jpg'}, {'damage_type': 'backtrace', 'id': 281322038, 'ino': 1099511632941, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-54.xml'}, {'damage_type': 'backtrace', 'id': 304650276, 'ino': 1099511634541, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-59.jpg'}, {'damage_type': 'backtrace', 'id': 351525597, 'ino': 1099511634594, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-53.jpg'}, {'damage_type': 'backtrace', 'id': 358327564, 'ino': 1099511635274, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-42'}, {'damage_type': 'backtrace', 'id': 376517141, 'ino': 1099511634661, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-36.jpg'}, {'damage_type': 'backtrace', 'id': 386364377, 'ino': 1099511634323, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-32.xml'}, {'damage_type': 'backtrace', 'id': 391306149, 'ino': 1099511634727, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-18.jpg'}, {'damage_type': 'backtrace', 'id': 412212068, 'ino': 1099511634175, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-66.xml'}, {'damage_type': 'backtrace', 'id': 478585274, 'ino': 1099511634383, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-17.xml'}, {'damage_type': 'backtrace', 'id': 505461447, 'ino': 1099511634691, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-27.jpg'}, {'damage_type': 'backtrace', 'id': 508687041, 'ino': 1099511634218, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-58.xml'}, {'damage_type': 'backtrace', 'id': 534096139, 'ino': 1099511634124, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-75.xml'}, {'damage_type': 'backtrace', 'id': 553361710, 'ino': 1099511634676, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-31.jpg'}, {'damage_type': 'backtrace', 'id': 566470830, 'ino': 1099511634630, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-49.jpg'}, {'damage_type': 'backtrace', 'id': 570466905, 'ino': 1099511633281, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/picture-13.jpg'}, {'damage_type': 'backtrace', 'id': 583996357, 'ino': 1099511634755, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-11.jpg'}, {'damage_type': 'backtrace', 'id': 588954681, 'ino': 1099511634551, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-58.jpg'}, {'damage_type': 'backtrace', 'id': 591936727, 'ino': 1099511634231, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-55.xml'}, {'damage_type': 'backtrace', 'id': 600439603, 'ino': 1099511632983, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-45.xml'}, {'damage_type': 'backtrace', 'id': 604410212, 'ino': 1099511634771, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-7.jpg'}, {'damage_type': 'backtrace', 'id': 618023760, 'ino': 1099511634355, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-24.xml'}, {'damage_type': 'backtrace', 'id': 688466367, 'ino': 1099511634419, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-8.xml'}, {'damage_type': 'backtrace', 'id': 708474436, 'ino': 1099511634642, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-44.jpg'}, {'damage_type': 'backtrace', 'id': 741998825, 'ino': 1099511632920, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-57.xml'}, {'damage_type': 'backtrace', 'id': 765537078, 'ino': 1099511634462, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-68.jpg'}, {'damage_type': 'backtrace', 'id': 775663768, 'ino': 1099511634255, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-50.xml'}, {'damage_type': 'backtrace', 'id': 776894961, 'ino': 1099511634240, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-53.xml'}, {'damage_type': 'backtrace', 'id': 786511926, 'ino': 1099511634208, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-60.xml'}, {'damage_type': 'backtrace', 'id': 789400562, 'ino': 1099511634277, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-44.xml'}, {'damage_type': 'backtrace', 'id': 803799586, 'ino': 1099511634295, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-39.xml'}, {'damage_type': 'backtrace', 'id': 809313725, 'ino': 1099511632989, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-44.xml'}, {'damage_type': 'backtrace', 'id': 832224897, 'ino': 1099511634699, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-25.jpg'}, {'damage_type': 'backtrace', 'id': 879760599, 'ino': 1099511634532, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-60.jpg'}, {'damage_type': 'backtrace', 'id': 890246669, 'ino': 1099511634609, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-51.jpg'}, {'damage_type': 'backtrace', 'id': 900879738, 'ino': 1099511634480, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-66.jpg'}, {'damage_type': 'backtrace', 'id': 909928836, 'ino': 1099511632978, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-46.xml'}, {'damage_type': 'backtrace', 'id': 913271421, 'ino': 1099511634747, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-13.jpg'}, {'damage_type': 'backtrace', 'id': 1104829424, 'ino': 1099511634687, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-28.jpg'}, {'damage_type': 'backtrace', 'id': 1130597280, 'ino': 1099511634347, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-26.xml'}, {'damage_type': 'backtrace', 'id': 1165988883, 'ino': 1099511634664, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-35.jpg'}, {'damage_type': 'backtrace', 'id': 1189738846, 'ino': 1099511634227, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-56.xml'}, {'damage_type': 'backtrace', 'id': 1275139091, 'ino': 1099511634299, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-38.xml'}, {'damage_type': 'backtrace', 'id': 1280400881, 'ino': 1099511634488, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-65.jpg'}, {'damage_type': 'backtrace', 'id': 1333101433, 'ino': 1099511634289, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-41.xml'}, {'damage_type': 'backtrace', 'id': 1350418077, 'ino': 1099511634391, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-15.xml'}, {'damage_type': 'backtrace', 'id': 1367139248, 'ino': 1099511634473, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-67.jpg'}, {'damage_type': 'backtrace', 'id': 1373345681, 'ino': 1099511634335, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-29.xml'}, {'damage_type': 'backtrace', 'id': 1383811342, 'ino': 1099511634633, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-48.jpg'}, {'damage_type': 'backtrace', 'id': 1385567070, 'ino': 1099511634559, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-57.jpg'}, {'damage_type': 'backtrace', 'id': 1413699889, 'ino': 1099511634293, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-40.xml'}, {'damage_type': 'backtrace', 'id': 1459959856, 'ino': 1099511634415, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-9.xml'}, {'damage_type': 'backtrace', 'id': 1492204660, 'ino': 1099511632930, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-56.xml'}, {'damage_type': 'backtrace', 'id': 1530106411, 'ino': 1099511634566, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-56.jpg'}, {'damage_type': 'backtrace', 'id': 1588345072, 'ino': 1099511634259, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-49.xml'}, {'damage_type': 'backtrace', 'id': 1697398191, 'ino': 1099511634731, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-17.jpg'}, {'damage_type': 'backtrace', 'id': 1722664619, 'ino': 1099511634735, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-16.jpg'}, {'damage_type': 'backtrace', 'id': 1733916310, 'ino': 1099511634135, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-73.xml'}, {'damage_type': 'backtrace', 'id': 1743677523, 'ino': 1099511634247, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-52.xml'}, {'damage_type': 'backtrace', 'id': 1776587408, 'ino': 1099511634359, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-23.xml'}, {'damage_type': 'backtrace', 'id': 1776707318, 'ino': 1099511634655, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-38.jpg'}, {'damage_type': 'backtrace', 'id': 1788937052, 'ino': 1099511634331, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-30.xml'}, {'damage_type': 'backtrace', 'id': 1939042924, 'ino': 1099511634423, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-7.xml'}, {'damage_type': 'backtrace', 'id': 1951329526, 'ino': 1099511634387, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-16.xml'}, {'damage_type': 'backtrace', 'id': 1957222731, 'ino': 1099511634303, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-37.xml'}, {'damage_type': 'backtrace', 'id': 1973612380, 'ino': 1099511634186, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-64.xml'}, {'damage_type': 'backtrace', 'id': 1977938069, 'ino': 1099511634339, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-28.xml'}, {'damage_type': 'backtrace', 'id': 1991142873, 'ino': 1099511634587, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-54.jpg'}, {'damage_type': 'backtrace', 'id': 1995259393, 'ino': 1099511632949, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-52.xml'}, {'damage_type': 'backtrace', 'id': 2019110139, 'ino': 1099511634367, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-21.xml'}, {'damage_type': 'backtrace', 'id': 2059562121, 'ino': 1099511634214, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-59.xml'}, {'damage_type': 'backtrace', 'id': 2064626047, 'ino': 1099511634180, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-65.xml'}, {'damage_type': 'backtrace', 'id': 2076431525, 'ino': 1099511634751, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-12.jpg'}, {'damage_type': 'backtrace', 'id': 2104727018, 'ino': 1099511634427, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-6.xml'}, {'damage_type': 'backtrace', 'id': 2116731428, 'ino': 1099511634163, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-68.xml'}, {'damage_type': 'backtrace', 'id': 2135546774, 'ino': 1099511634262, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-48.xml'}, {'damage_type': 'backtrace', 'id': 2200905500, 'ino': 1099511634670, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-33.jpg'}, {'damage_type': 'backtrace', 'id': 2232223223, 'ino': 1099511634311, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-35.xml'}, {'damage_type': 'backtrace', 'id': 2256786380, 'ino': 1099511634719, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-20.jpg'}, {'damage_type': 'backtrace', 'id': 2313440470, 'ino': 1099511634792, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-1.jpg'}, {'damage_type': 'backtrace', 'id': 2390665378, 'ino': 1099511634653, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-39.jpg'}, {'damage_type': 'backtrace', 'id': 2406395763, 'ino': 1099511634498, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-64.jpg'}, {'damage_type': 'backtrace', 'id': 2447084590, 'ino': 1099511634146, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-71.xml'}, {'damage_type': 'backtrace', 'id': 2485414524, 'ino': 1099511634767, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-8.jpg'}, {'damage_type': 'backtrace', 'id': 2548723451, 'ino': 1099511634222, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-57.xml'}, {'damage_type': 'backtrace', 'id': 2596689273, 'ino': 1099511634343, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-27.xml'}, {'damage_type': 'backtrace', 'id': 2598931016, 'ino': 1099511634192, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-63.xml'}, {'damage_type': 'backtrace', 'id': 2609159704, 'ino': 1099511634315, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-34.xml'}, {'damage_type': 'backtrace', 'id': 2632384235, 'ino': 1099511634506, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-63.jpg'}, {'damage_type': 'backtrace', 'id': 2686615640, 'ino': 1099511634266, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-47.xml'}, {'damage_type': 'backtrace', 'id': 2705339567, 'ino': 1099511634435, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-4.xml'}, {'damage_type': 'backtrace', 'id': 2730169823, 'ino': 1099511634640, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-45.jpg'}, {'damage_type': 'backtrace', 'id': 2730333660, 'ino': 1099511634707, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-23.jpg'}, {'damage_type': 'backtrace', 'id': 2809419974, 'ino': 1099511634743, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-14.jpg'}, {'damage_type': 'backtrace', 'id': 2825511312, 'ino': 1099511634778, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-5.jpg'}, {'damage_type': 'backtrace', 'id': 2834586146, 'ino': 1099511634723, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-19.jpg'}, {'damage_type': 'backtrace', 'id': 2845807658, 'ino': 1099511634759, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-10.jpg'}, {'damage_type': 'backtrace', 'id': 2857928425, 'ino': 1099511634115, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34'}, {'damage_type': 'backtrace', 'id': 2897514245, 'ino': 1099511634447, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-1.xml'}, {'damage_type': 'backtrace', 'id': 2908880805, 'ino': 1099511632963, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-49.xml'}, {'damage_type': 'backtrace', 'id': 3005429153, 'ino': 1099511632937, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-55.xml'}, {'damage_type': 'backtrace', 'id': 3011589996, 'ino': 1099511634281, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-43.xml'}, {'damage_type': 'backtrace', 'id': 3107124885, 'ino': 1099511634327, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-31.xml'}, {'damage_type': 'backtrace', 'id': 3130640380, 'ino': 1099511634251, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-51.xml'}, {'damage_type': 'backtrace', 'id': 3132393222, 'ino': 1099511632992, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-43.xml'}, {'damage_type': 'backtrace', 'id': 3152238432, 'ino': 1099511634788, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-2.jpg'}, {'damage_type': 'backtrace', 'id': 3154987171, 'ino': 1099511634647, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-42.jpg'}, {'damage_type': 'backtrace', 'id': 3156124799, 'ino': 1099511632947, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-53.xml'}, {'damage_type': 'backtrace', 'id': 3199530314, 'ino': 1099511634235, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-54.xml'}, {'damage_type': 'backtrace', 'id': 3207045957, 'ino': 1099511634117, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-76.xml'}, {'damage_type': 'backtrace', 'id': 3245683064, 'ino': 1099511634673, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-32.jpg'}, {'damage_type': 'backtrace', 'id': 3255360236, 'ino': 1099511634319, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-33.xml'}, {'damage_type': 'backtrace', 'id': 3257844014, 'ino': 1099511635549, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-13.xml'}, {'damage_type': 'backtrace', 'id': 3270590008, 'ino': 1099511634683, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-29.jpg'}, {'damage_type': 'backtrace', 'id': 3283843128, 'ino': 1099511634169, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-67.xml'}, {'damage_type': 'backtrace', 'id': 3307930268, 'ino': 1099511634307, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-36.xml'}, {'damage_type': 'backtrace', 'id': 3326795728, 'ino': 1099511634763, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-9.jpg'}, {'damage_type': 'backtrace', 'id': 3360518134, 'ino': 1099511632960, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-50.xml'}, {'damage_type': 'backtrace', 'id': 3374640461, 'ino': 1099511634524, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-61.jpg'}, {'damage_type': 'backtrace', 'id': 3378546672, 'ino': 1099511634363, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-22.xml'}, {'damage_type': 'backtrace', 'id': 3386533530, 'ino': 1099511634785, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-3.jpg'}, {'damage_type': 'backtrace', 'id': 3407334236, 'ino': 1099511632974, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-47.xml'}, {'damage_type': 'backtrace', 'id': 3453727486, 'ino': 1099511634203, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-61.xml'}, {'damage_type': 'backtrace', 'id': 3472591155, 'ino': 1099511634613, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-50.jpg'}, {'damage_type': 'backtrace', 'id': 3486263159, 'ino': 1099511634739, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-15.jpg'}, {'damage_type': 'backtrace', 'id': 3488449575, 'ino': 1099511632969, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-48.xml'}, {'damage_type': 'backtrace', 'id': 3643606676, 'ino': 1099511634197, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-62.xml'}, {'damage_type': 'backtrace', 'id': 3649328185, 'ino': 1099511632911, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-58.xml'}, {'damage_type': 'backtrace', 'id': 3652667385, 'ino': 1099511634407, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-11.xml'}, {'damage_type': 'backtrace', 'id': 3695208712, 'ino': 1099511634411, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-10.xml'}, {'damage_type': 'backtrace', 'id': 3703163384, 'ino': 1099511634443, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-2.xml'}, {'damage_type': 'backtrace', 'id': 3712101459, 'ino': 1099511634782, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-4.jpg'}, {'damage_type': 'backtrace', 'id': 3713152167, 'ino': 1099511634667, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-34.jpg'}, {'damage_type': 'backtrace', 'id': 3740728973, 'ino': 1099511634451, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-69.jpg'}, {'damage_type': 'backtrace', 'id': 3770120785, 'ino': 1099511634923, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/article-42.xml'}, {'damage_type': 'backtrace', 'id': 3793631517, 'ino': 1099511634141, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-72.xml'}, {'damage_type': 'backtrace', 'id': 3799814763, 'ino': 1099511634351, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-25.xml'}, {'damage_type': 'backtrace', 'id': 3809398512, 'ino': 1099511634439, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-3.xml'}, {'damage_type': 'backtrace', 'id': 3848959133, 'ino': 1099511634395, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-14.xml'}, {'damage_type': 'backtrace', 'id': 3867357855, 'ino': 1099511634152, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-70.xml'}, {'damage_type': 'backtrace', 'id': 3917534529, 'ino': 1099511634379, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-18.xml'}, {'damage_type': 'backtrace', 'id': 3972286589, 'ino': 1099511634645, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-43.jpg'}, {'damage_type': 'backtrace', 'id': 4023566656, 'ino': 1099511634515, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-62.jpg'}, {'damage_type': 'backtrace', 'id': 4037569096, 'ino': 1099511632955, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-12/comment-51.xml'}, {'damage_type': 'backtrace', 'id': 4123897479, 'ino': 1099511634695, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-26.jpg'}, {'damage_type': 'backtrace', 'id': 4143208748, 'ino': 1099511634658, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-37.jpg'}, {'damage_type': 'backtrace', 'id': 4184672691, 'ino': 1099511634711, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-22.jpg'}, {'damage_type': 'backtrace', 'id': 4284426747, 'ino': 1099511634703, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/picture-24.jpg'}, {'damage_type': 'backtrace', 'id': 4292482900, 'ino': 1099511634273, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-45.xml'}, {'damage_type': 'backtrace', 'id': 4294749627, 'ino': 1099511634285, 'path': '/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-34/article-42.xml'}] 2024-06-10T00:36:53.037 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/fwd_scrub.py", line 48, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/fwd_scrub.py", line 70, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/fwd_scrub.py", line 81, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-06-10T00:36:53.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffacaff5700 1 -- 172.21.15.121:0/1759692769 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5560fa403460 con 0x5560fa530230 2024-06-10T00:36:53.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffacaff5700 10 client.16743 handle_command_reply: tid=0 2024-06-10T00:36:53.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffaf78bab80 1 client.16743 shutdown 2024-06-10T00:36:53.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1759692769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 msgr2=0x5560fa4ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:53.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4ca320 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ffadc026840 tx=0x7ffadc010eb0 comp rx=0 tx=0).stop 2024-06-10T00:36:53.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1759692769 shutdown_connections 2024-06-10T00:36:53.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560fa530230 0x5560fa532650 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:53.070 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560fa4b0eb0 0x5560fa4c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:53.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560fa4aace0 0x5560fa4c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:53.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffacaff5700 0 client.16743 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:53.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffaf78bab80 1 --2- 172.21.15.121:0/1759692769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560fa4b0590 0x5560fa4ca320 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:53.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1759692769 >> 172.21.15.121:0/1759692769 conn(0x5560fa3d7210 msgr2=0x5560fa4af8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:53.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.069+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1759692769 shutdown_connections 2024-06-10T00:36:53.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaf78bab80 1 -- 172.21.15.121:0/1759692769 wait complete. 2024-06-10T00:36:53.071 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaf78bab80 20 client.16743 trim_cache size 0 max 16384 2024-06-10T00:36:53.071 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:36:53.071 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 33, 2024-06-10T00:36:53.071 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 22, 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "7b6c6e18": { 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 22, 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 136.0, 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:36:53.072 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:36:53.073 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:36:53.073 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:36:53.073 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:36:53.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:53.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:53.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:53.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:53.073 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:53.073 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:36:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaed037700 1 -- 172.21.15.121:0/1588901780 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffad00a2eb0 msgr2=0x7ffad00a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaed037700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffad00a2eb0 0x7ffad00a5370 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ffae8078960 tx=0x7ffadc00b410 comp rx=0 tx=0).stop 2024-06-10T00:36:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaed037700 1 -- 172.21.15.121:0/1588901780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffae81013c0 msgr2=0x7ffae8103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaed037700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffae81013c0 0x7ffae8103070 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ffad800e9a0 tx=0x7ffad800ecb0 comp rx=0 tx=0).stop 2024-06-10T00:36:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaed037700 1 -- 172.21.15.121:0/1588901780 shutdown_connections 2024-06-10T00:36:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaed037700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffad00a2eb0 0x7ffad00a5370 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaed037700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae8107470 0x7ffae8103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaed037700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffae81013c0 0x7ffae8103070 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaed037700 1 --2- 172.21.15.121:0/1588901780 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffae8101d80 0x7ffae81035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:53.074 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.070+0000 7ffaed037700 1 -- 172.21.15.121:0/1588901780 >> 172.21.15.121:0/1588901780 conn(0x7ffae80783d0 msgr2=0x7ffae8100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:53.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.071+0000 7ffaed037700 1 -- 172.21.15.121:0/1588901780 shutdown_connections 2024-06-10T00:36:53.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:53.071+0000 7ffaed037700 1 -- 172.21.15.121:0/1588901780 wait complete. 2024-06-10T00:36:53.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:53 smithi193 ceph-mon[36716]: pgmap v274: 129 pgs: 129 active+clean; 1.6 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-06-10T00:36:53.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:53 smithi121 ceph-mon[32378]: pgmap v274: 129 pgs: 129 active+clean; 1.6 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-06-10T00:36:53.729 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '7b6c6e18', quiesce took 15.4 seconds. Will release after: 0.5999999999999996 2024-06-10T00:36:53.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:53 smithi135 ceph-mon[36424]: pgmap v274: 129 pgs: 129 active+clean; 1.6 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 254 op/s 2024-06-10T00:36:54.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:36:54.580 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:36:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:55 smithi193 ceph-mon[36716]: pgmap v275: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2024-06-10T00:36:55.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:55 smithi121 ceph-mon[32378]: pgmap v275: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2024-06-10T00:36:55.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:55 smithi135 ceph-mon[36424]: pgmap v275: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 234 op/s 2024-06-10T00:36:56.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.395+0000 7fab6dedd700 1 -- 172.21.15.121:0/3693637578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab68107840 msgr2=0x7fab6810f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:56.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.395+0000 7fab6dedd700 1 --2- 172.21.15.121:0/3693637578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab68107840 0x7fab6810f3c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fab580099c0 tx=0x7fab58009cd0 comp rx=0 tx=0).stop 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.395+0000 7fab6dedd700 1 -- 172.21.15.121:0/3693637578 shutdown_connections 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.395+0000 7fab6dedd700 1 --2- 172.21.15.121:0/3693637578 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab68107840 0x7fab6810f3c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.395+0000 7fab6dedd700 1 --2- 172.21.15.121:0/3693637578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab68102740 0x7fab68102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.395+0000 7fab6dedd700 1 --2- 172.21.15.121:0/3693637578 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab681030f0 0x7fab68107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.395+0000 7fab6dedd700 1 -- 172.21.15.121:0/3693637578 >> 172.21.15.121:0/3693637578 conn(0x7fab680795f0 msgr2=0x7fab68079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.395+0000 7fab6dedd700 1 -- 172.21.15.121:0/3693637578 shutdown_connections 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.395+0000 7fab6dedd700 1 -- 172.21.15.121:0/3693637578 wait complete. 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.396+0000 7fab6dedd700 1 Processor -- start 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.396+0000 7fab6dedd700 1 -- start start 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.396+0000 7fab6dedd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab68102740 0x7fab6819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.396+0000 7fab6dedd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab681030f0 0x7fab6819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.396+0000 7fab6dedd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab68107840 0x7fab681988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.396+0000 7fab6dedd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fab68112780 con 0x7fab68102740 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.396+0000 7fab6dedd700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fab68112600 con 0x7fab681030f0 2024-06-10T00:36:56.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.396+0000 7fab6dedd700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fab68112900 con 0x7fab68107840 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.396+0000 7fab677fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab68102740 0x7fab6819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab677fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab68102740 0x7fab6819e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:37096/0 (socket says 172.21.15.121:37096) 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab677fe700 1 -- 172.21.15.121:0/1326925135 learned_addr learned my addr 172.21.15.121:0/1326925135 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab67fff700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab68107840 0x7fab681988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab66ffd700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab681030f0 0x7fab6819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab677fe700 1 -- 172.21.15.121:0/1326925135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab68107840 msgr2=0x7fab681988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab677fe700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab68107840 0x7fab681988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab677fe700 1 -- 172.21.15.121:0/1326925135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab681030f0 msgr2=0x7fab6819ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab677fe700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab681030f0 0x7fab6819ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab677fe700 1 -- 172.21.15.121:0/1326925135 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab68199020 con 0x7fab68102740 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab67fff700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab68107840 0x7fab681988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab677fe700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab68102740 0x7fab6819e700 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fab6000ada0 tx=0x7fab6000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab64ff9700 1 -- 172.21.15.121:0/1326925135 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60017020 con 0x7fab68102740 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab64ff9700 1 -- 172.21.15.121:0/1326925135 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab60004bb0 con 0x7fab68102740 2024-06-10T00:36:56.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab6dedd700 1 -- 172.21.15.121:0/1326925135 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab68199310 con 0x7fab68102740 2024-06-10T00:36:56.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab64ff9700 1 -- 172.21.15.121:0/1326925135 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab60005660 con 0x7fab68102740 2024-06-10T00:36:56.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.397+0000 7fab6dedd700 1 -- 172.21.15.121:0/1326925135 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab681a55d0 con 0x7fab68102740 2024-06-10T00:36:56.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.398+0000 7fab6dedd700 1 -- 172.21.15.121:0/1326925135 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab68069180 con 0x7fab68102740 2024-06-10T00:36:56.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.399+0000 7fab64ff9700 1 -- 172.21.15.121:0/1326925135 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fab60005800 con 0x7fab68102740 2024-06-10T00:36:56.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.400+0000 7fab64ff9700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fab540a2d90 0x7fab540a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:56.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.400+0000 7fab66ffd700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fab540a2d90 0x7fab540a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:56.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.400+0000 7fab64ff9700 1 -- 172.21.15.121:0/1326925135 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7fab600cf050 con 0x7fab68102740 2024-06-10T00:36:56.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.400+0000 7fab66ffd700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fab540a2d90 0x7fab540a5250 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fab50005ea0 tx=0x7fab50005e30 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:56.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.405+0000 7fab64ff9700 1 -- 172.21.15.121:0/1326925135 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fab60021020 con 0x7fab68102740 2024-06-10T00:36:56.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.646+0000 7fab6dedd700 1 -- 172.21.15.121:0/1326925135 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab681a57a0 con 0x7fab68102740 2024-06-10T00:36:56.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.647+0000 7fab64ff9700 1 -- 172.21.15.121:0/1326925135 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11895 (secure 0 0 0) 0x7fab6009b910 con 0x7fab68102740 2024-06-10T00:36:56.647 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:36:56.647 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:35:59.291154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":189,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:36:56.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 -- 172.21.15.121:0/1326925135 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fab540a2d90 msgr2=0x7fab540a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:56.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fab540a2d90 0x7fab540a5250 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fab50005ea0 tx=0x7fab50005e30 comp rx=0 tx=0).stop 2024-06-10T00:36:56.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 -- 172.21.15.121:0/1326925135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab68102740 msgr2=0x7fab6819e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:56.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab68102740 0x7fab6819e700 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fab6000ada0 tx=0x7fab6000c330 comp rx=0 tx=0).stop 2024-06-10T00:36:56.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 -- 172.21.15.121:0/1326925135 shutdown_connections 2024-06-10T00:36:56.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fab540a2d90 0x7fab540a5250 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:56.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab68107840 0x7fab681988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:56.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab68102740 0x7fab6819e700 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:56.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 --2- 172.21.15.121:0/1326925135 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fab681030f0 0x7fab6819ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:56.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 -- 172.21.15.121:0/1326925135 >> 172.21.15.121:0/1326925135 conn(0x7fab680795f0 msgr2=0x7fab68100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:56.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 -- 172.21.15.121:0/1326925135 shutdown_connections 2024-06-10T00:36:56.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:56.649+0000 7fab6dedd700 1 -- 172.21.15.121:0/1326925135 wait complete. 2024-06-10T00:36:56.650 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 33 2024-06-10T00:36:57.382 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db --set-id 7b6c6e18 --release --await' 2024-06-10T00:36:57.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db --set-id 7b6c6e18 --release --await 2024-06-10T00:36:57.560 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:57 smithi135 ceph-mon[36424]: pgmap v276: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-06-10T00:36:57.560 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:57 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1326925135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:36:57.624 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:36:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:57 smithi193 ceph-mon[36716]: pgmap v276: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-06-10T00:36:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:57 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1326925135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:36:57.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:57 smithi121 ceph-mon[32378]: pgmap v276: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-06-10T00:36:57.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:57 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1326925135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.288+0000 7f41e19a6700 1 -- 172.21.15.121:0/2408231545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41dc1046a0 msgr2=0x7f41dc104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.288+0000 7f41e19a6700 1 --2- 172.21.15.121:0/2408231545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41dc1046a0 0x7f41dc104b20 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009990 tx=0x7f41d0009ca0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.288+0000 7f41e19a6700 1 -- 172.21.15.121:0/2408231545 shutdown_connections 2024-06-10T00:36:59.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.288+0000 7f41e19a6700 1 --2- 172.21.15.121:0/2408231545 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41dc105060 0x7f41dc111570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.288+0000 7f41e19a6700 1 --2- 172.21.15.121:0/2408231545 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41dc10a6a0 0x7f41dc10aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.288+0000 7f41e19a6700 1 --2- 172.21.15.121:0/2408231545 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41dc1046a0 0x7f41dc104b20 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.288+0000 7f41e19a6700 1 -- 172.21.15.121:0/2408231545 >> 172.21.15.121:0/2408231545 conn(0x7f41dc100190 msgr2=0x7f41dc1025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.288+0000 7f41e19a6700 1 -- 172.21.15.121:0/2408231545 shutdown_connections 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.288+0000 7f41e19a6700 1 -- 172.21.15.121:0/2408231545 wait complete. 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.288+0000 7f41e19a6700 1 Processor -- start 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41e19a6700 1 -- start start 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41e19a6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41dc1046a0 0x7f41dc19aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41e19a6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41dc105060 0x7f41dc19b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41e19a6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41dc10a6a0 0x7f41dc19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41e19a6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc114a30 con 0x7f41dc1046a0 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41e19a6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1148b0 con 0x7f41dc105060 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41e19a6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc114bb0 con 0x7f41dc10a6a0 2024-06-10T00:36:59.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41dc1046a0 0x7f41dc19aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41dc1046a0 0x7f41dc19aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42042/0 (socket says 172.21.15.121:42042) 2024-06-10T00:36:59.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41daffd700 1 -- 172.21.15.121:0/457739781 learned_addr learned my addr 172.21.15.121:0/457739781 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:59.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41db7fe700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41dc10a6a0 0x7f41dc19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41da7fc700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41dc105060 0x7f41dc19b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41daffd700 1 -- 172.21.15.121:0/457739781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41dc10a6a0 msgr2=0x7f41dc19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:59.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41daffd700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41dc10a6a0 0x7f41dc19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41daffd700 1 -- 172.21.15.121:0/457739781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41dc105060 msgr2=0x7f41dc19b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:59.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41daffd700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41dc105060 0x7f41dc19b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41daffd700 1 -- 172.21.15.121:0/457739781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41dc19feb0 con 0x7f41dc1046a0 2024-06-10T00:36:59.291 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.289+0000 7f41da7fc700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41dc105060 0x7f41dc19b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:36:59.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.290+0000 7f41daffd700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41dc1046a0 0x7f41dc19aea0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00b530 tx=0x7f41cc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:59.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.290+0000 7f41e09a4700 1 -- 172.21.15.121:0/457739781 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc015020 con 0x7f41dc1046a0 2024-06-10T00:36:59.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.290+0000 7f41e19a6700 1 -- 172.21.15.121:0/457739781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41dc1a01a0 con 0x7f41dc1046a0 2024-06-10T00:36:59.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.290+0000 7f41e09a4700 1 -- 172.21.15.121:0/457739781 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41cc011d10 con 0x7f41dc1046a0 2024-06-10T00:36:59.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.290+0000 7f41e09a4700 1 -- 172.21.15.121:0/457739781 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc00f2e0 con 0x7f41dc1046a0 2024-06-10T00:36:59.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.290+0000 7f41e19a6700 1 -- 172.21.15.121:0/457739781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41dc1a7a20 con 0x7f41dc1046a0 2024-06-10T00:36:59.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.292+0000 7f41e09a4700 1 -- 172.21.15.121:0/457739781 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f41cc018020 con 0x7f41dc1046a0 2024-06-10T00:36:59.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.293+0000 7f41e09a4700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41c40a2d90 0x7f41c40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.293+0000 7f41da7fc700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41c40a2d90 0x7f41c40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.293+0000 7f41e09a4700 1 -- 172.21.15.121:0/457739781 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f41cc0cec60 con 0x7f41dc1046a0 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.293+0000 7f41da7fc700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41c40a2d90 0x7f41c40a5250 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f41d0006010 tx=0x7f41d000b540 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.293+0000 7f41ec229b80 1 Processor -- start 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.293+0000 7f41ec229b80 1 -- start start 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff09558e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff0955e20 0x563ff094f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff094fc50 0x563ff0950070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41ec229b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x563ff0842060 con 0x563ff0955500 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41ec229b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x563ff08a0950 con 0x563ff0955e20 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41ec229b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563ff0808560 con 0x563ff094fc50 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff09558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff09558e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42048/0 (socket says 172.21.15.121:42048) 2024-06-10T00:36:59.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41daffd700 1 -- 172.21.15.121:0/2672981276 learned_addr learned my addr 172.21.15.121:0/2672981276 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41da7fc700 1 --2- 172.21.15.121:0/2672981276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff0955e20 0x563ff094f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41db7fe700 1 --2- 172.21.15.121:0/2672981276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff094fc50 0x563ff0950070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41daffd700 1 -- 172.21.15.121:0/2672981276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff094fc50 msgr2=0x563ff0950070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41daffd700 1 --2- 172.21.15.121:0/2672981276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff094fc50 0x563ff0950070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41daffd700 1 -- 172.21.15.121:0/2672981276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff0955e20 msgr2=0x563ff094f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41daffd700 1 --2- 172.21.15.121:0/2672981276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff0955e20 0x563ff094f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41daffd700 1 -- 172.21.15.121:0/2672981276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ff07ef9d0 con 0x563ff0955500 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41db7fe700 1 --2- 172.21.15.121:0/2672981276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff094fc50 0x563ff0950070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41daffd700 1 --2- 172.21.15.121:0/2672981276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff09558e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f41cc097300 tx=0x7f41cc09a8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69bfdb14be928624 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41c1ffb700 1 -- 172.21.15.121:0/2672981276 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc09f250 con 0x563ff0955500 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.294+0000 7f41c1ffb700 1 -- 172.21.15.121:0/2672981276 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41cc0a3b10 con 0x563ff0955500 2024-06-10T00:36:59.296 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.295+0000 7f41c1ffb700 1 -- 172.21.15.121:0/2672981276 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0202b0 con 0x563ff0955500 2024-06-10T00:36:59.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.295+0000 7f41ec229b80 1 -- 172.21.15.121:0/2672981276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 msgr2=0x563ff09558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:59.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.295+0000 7f41ec229b80 1 --2- 172.21.15.121:0/2672981276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff09558e0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f41cc097300 tx=0x7f41cc09a8d0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.295+0000 7f41ec229b80 1 -- 172.21.15.121:0/2672981276 shutdown_connections 2024-06-10T00:36:59.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.295+0000 7f41ec229b80 1 --2- 172.21.15.121:0/2672981276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff094fc50 0x563ff0950070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.295+0000 7f41ec229b80 1 --2- 172.21.15.121:0/2672981276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff09558e0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.295+0000 7f41ec229b80 1 --2- 172.21.15.121:0/2672981276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff0955e20 0x563ff094f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.298 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.295+0000 7f41ec229b80 1 -- 172.21.15.121:0/2672981276 >> 172.21.15.121:0/2672981276 conn(0x563ff087c180 msgr2=0x563ff094aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.295+0000 7f41ec229b80 1 -- 172.21.15.121:0/2672981276 shutdown_connections 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.295+0000 7f41ec229b80 1 -- 172.21.15.121:0/2672981276 wait complete. 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41ec229b80 1 -- start start 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41ec229b80 1 Processor -- start 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff094fc50 0x563ff095bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff0955500 0x563ff0958eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff09593f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41ec229b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x563ff08410c0 con 0x563ff094fc50 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41ec229b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x563ff089c910 con 0x563ff0955500 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41ec229b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563ff095c800 con 0x563ff0955e20 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41ec229b80 10 client.0 _ll_register_callbacks cb 0x563ff08366b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff094fc50 0x563ff095bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41daffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff094fc50 0x563ff095bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42064/0 (socket says 172.21.15.121:42064) 2024-06-10T00:36:59.299 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41daffd700 1 -- 172.21.15.121:0/3158246884 learned_addr learned my addr 172.21.15.121:0/3158246884 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:59.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41da7fc700 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff0955500 0x563ff0958eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.296+0000 7f41db7fe700 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff09593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.297+0000 7f41daffd700 1 -- 172.21.15.121:0/3158246884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 msgr2=0x563ff09593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:59.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.297+0000 7f41daffd700 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff09593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.297+0000 7f41daffd700 1 -- 172.21.15.121:0/3158246884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff0955500 msgr2=0x563ff0958eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:59.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.297+0000 7f41daffd700 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff0955500 0x563ff0958eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.297+0000 7f41daffd700 1 -- 172.21.15.121:0/3158246884 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ff0866850 con 0x563ff094fc50 2024-06-10T00:36:59.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.297+0000 7f41db7fe700 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff09593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:36:59.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.297+0000 7f41daffd700 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff094fc50 0x563ff095bbb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f41cc097870 tx=0x7f41cc097270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc26619e070edfdb server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:59.300 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.297+0000 7f41baffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:59.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.300+0000 7f41baffd700 1 -- 172.21.15.121:0/3158246884 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41cc097510 con 0x563ff094fc50 2024-06-10T00:36:59.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.300+0000 7f41baffd700 1 -- 172.21.15.121:0/3158246884 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f41cc0202b0 con 0x563ff094fc50 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.300+0000 7f41ec229b80 1 -- 172.21.15.121:0/3158246884 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41cc0a3b10 con 0x563ff094fc50 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.300+0000 7f41baffd700 1 -- 172.21.15.121:0/3158246884 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f41cc0976b0 con 0x563ff094fc50 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.300+0000 7f41ec229b80 10 client.16794 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.300+0000 7f41ec229b80 1 -- 172.21.15.121:0/3158246884 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563ff0967010 con 0x563ff094fc50 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.301+0000 7f41baffd700 1 -- 172.21.15.121:0/3158246884 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41cc0a3b10 con 0x563ff094fc50 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.301+0000 7f41baffd700 1 -- 172.21.15.121:0/3158246884 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f41cc09a2f0 con 0x563ff094fc50 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.301+0000 7f41ec229b80 10 client.16794 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.301+0000 7f41ec229b80 10 client.16794 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.301+0000 7f41ec229b80 20 client.16794 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.301+0000 7f41ec229b80 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x563ff09694e0 0x563ff096b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.301+0000 7f41ec229b80 4 client.16794 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.301+0000 7f41ec229b80 1 -- 172.21.15.121:0/3158246884 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ff08ade50 con 0x563ff09694e0 2024-06-10T00:36:59.302 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.301+0000 7f41da7fc700 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x563ff09694e0 0x563ff096b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.301+0000 7f41da7fc700 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x563ff09694e0 0x563ff096b8d0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc0a4f38c188319d server_cookie=79f7a23479978607 in_seq=0 out_seq=0 2024-06-10T00:36:59.303 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.301+0000 7f41baffd700 10 client.16794 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41baffd700 1 -- 172.21.15.121:0/3158246884 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x563ff08ade50 con 0x563ff09694e0 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41baffd700 10 client.16794 handle_command_reply: tid=0 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41ec229b80 1 client.16794 shutdown 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41ec229b80 1 -- 172.21.15.121:0/3158246884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff094fc50 msgr2=0x563ff095bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41ec229b80 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff094fc50 0x563ff095bbb0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f41cc097870 tx=0x7f41cc097270 comp rx=0 tx=0).stop 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41ec229b80 1 -- 172.21.15.121:0/3158246884 shutdown_connections 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41ec229b80 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x563ff09694e0 0x563ff096b8d0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41baffd700 0 client.16794 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41ec229b80 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff09593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41ec229b80 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff094fc50 0x563ff095bbb0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41ec229b80 1 --2- 172.21.15.121:0/3158246884 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff0955500 0x563ff0958eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.302+0000 7f41ec229b80 1 -- 172.21.15.121:0/3158246884 >> 172.21.15.121:0/3158246884 conn(0x563ff087c180 msgr2=0x563ff0949bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:59.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.303+0000 7f41ec229b80 1 -- 172.21.15.121:0/3158246884 shutdown_connections 2024-06-10T00:36:59.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.303+0000 7f41ec229b80 1 -- 172.21.15.121:0/3158246884 wait complete. 2024-06-10T00:36:59.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.303+0000 7f41ec229b80 20 client.16794 trim_cache size 0 max 16384 2024-06-10T00:36:59.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.322+0000 7f41ec229b80 1 Processor -- start 2024-06-10T00:36:59.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.322+0000 7f41ec229b80 1 -- start start 2024-06-10T00:36:59.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.322+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff094fc50 0x563ff095b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.322+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff094a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.322+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff094a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.324 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41ec229b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1b3330 con 0x563ff0955500 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41ec229b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1b0b00 con 0x563ff094fc50 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41ec229b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1ae970 con 0x563ff0955e20 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41da7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff094a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41da7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff094a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42070/0 (socket says 172.21.15.121:42070) 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41da7fc700 1 -- 172.21.15.121:0/3985665126 learned_addr learned my addr 172.21.15.121:0/3985665126 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41daffd700 1 --2- 172.21.15.121:0/3985665126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff094fc50 0x563ff095b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41db7fe700 1 --2- 172.21.15.121:0/3985665126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff094a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41da7fc700 1 -- 172.21.15.121:0/3985665126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 msgr2=0x563ff094a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41da7fc700 1 --2- 172.21.15.121:0/3985665126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff094a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41da7fc700 1 -- 172.21.15.121:0/3985665126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff094fc50 msgr2=0x563ff095b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41da7fc700 1 --2- 172.21.15.121:0/3985665126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff094fc50 0x563ff095b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41da7fc700 1 -- 172.21.15.121:0/3985665126 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ff0954fa0 con 0x563ff0955500 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41da7fc700 1 --2- 172.21.15.121:0/3985665126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff094a330 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f41d0005f50 tx=0x7f41d001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a67596884dcb03 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:59.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41c17fa700 1 -- 172.21.15.121:0/3985665126 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d000f7e0 con 0x563ff0955500 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41c17fa700 1 -- 172.21.15.121:0/3985665126 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41d003c410 con 0x563ff0955500 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.323+0000 7f41c17fa700 1 -- 172.21.15.121:0/3985665126 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d000fe60 con 0x563ff0955500 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.324+0000 7f41ec229b80 1 -- 172.21.15.121:0/3985665126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 msgr2=0x563ff094a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.324+0000 7f41ec229b80 1 --2- 172.21.15.121:0/3985665126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff094a330 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f41d0005f50 tx=0x7f41d001f000 comp rx=0 tx=0).stop 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.324+0000 7f41ec229b80 1 -- 172.21.15.121:0/3985665126 shutdown_connections 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.324+0000 7f41ec229b80 1 --2- 172.21.15.121:0/3985665126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff094a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.324+0000 7f41ec229b80 1 --2- 172.21.15.121:0/3985665126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff094fc50 0x563ff095b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.324+0000 7f41ec229b80 1 --2- 172.21.15.121:0/3985665126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff094a330 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.324+0000 7f41ec229b80 1 -- 172.21.15.121:0/3985665126 >> 172.21.15.121:0/3985665126 conn(0x563ff087c180 msgr2=0x563ff0954890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.324+0000 7f41ec229b80 1 -- 172.21.15.121:0/3985665126 shutdown_connections 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.324+0000 7f41ec229b80 1 -- 172.21.15.121:0/3985665126 wait complete. 2024-06-10T00:36:59.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.324+0000 7f41ec229b80 1 -- start start 2024-06-10T00:36:59.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41ec229b80 1 Processor -- start 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff094fc50 0x563ff096cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff096d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41ec229b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff09d2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41ec229b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1aefc0 con 0x563ff0955500 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41ec229b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1b0430 con 0x563ff094fc50 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41ec229b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f41dc1af690 con 0x563ff0955e20 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41ec229b80 10 client.0 _ll_register_callbacks cb 0x563ff08366b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41da7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff096d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41da7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff096d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42084/0 (socket says 172.21.15.121:42084) 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41da7fc700 1 -- 172.21.15.121:0/1439493542 learned_addr learned my addr 172.21.15.121:0/1439493542 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41daffd700 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff094fc50 0x563ff096cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41da7fc700 1 -- 172.21.15.121:0/1439493542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 msgr2=0x563ff09d2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:59.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41db7fe700 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff09d2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41da7fc700 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff09d2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41da7fc700 1 -- 172.21.15.121:0/1439493542 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff094fc50 msgr2=0x563ff096cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41da7fc700 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff094fc50 0x563ff096cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41da7fc700 1 -- 172.21.15.121:0/1439493542 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ff09d3360 con 0x563ff0955500 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41da7fc700 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff096d280 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f41d00098e0 tx=0x7f41d000b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c4146aaf23ccea server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.325+0000 7f41baffd700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41baffd700 1 -- 172.21.15.121:0/1439493542 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d000f5e0 con 0x563ff0955500 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41ec229b80 1 -- 172.21.15.121:0/1439493542 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41d003c410 con 0x563ff0955500 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41baffd700 1 -- 172.21.15.121:0/1439493542 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9525+0+0 (secure 0 0 0) 0x7f41d000f780 con 0x563ff0955500 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41baffd700 1 -- 172.21.15.121:0/1439493542 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f41d004a050 con 0x563ff0955500 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41ec229b80 10 client.16800 fetch_fsmap learned FSMap version 33 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41ec229b80 1 -- 172.21.15.121:0/1439493542 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563ff09d4610 con 0x563ff0955500 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41baffd700 1 -- 172.21.15.121:0/1439493542 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41d003c410 con 0x563ff0955500 2024-06-10T00:36:59.330 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41baffd700 1 -- 172.21.15.121:0/1439493542 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f41d0024a40 con 0x563ff0955500 2024-06-10T00:36:59.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41ec229b80 10 client.16800 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:36:59.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41ec229b80 10 client.16800 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:36:59.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41ec229b80 20 client.16800 populate_metadata read hostname 'smithi121' 2024-06-10T00:36:59.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41ec229b80 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x563ff09694e0 0x563ff09d6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:36:59.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41ec229b80 4 client.16800 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7b6c6e18", "release": true, "await": true}] 2024-06-10T00:36:59.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41ec229b80 1 -- 172.21.15.121:0/1439493542 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7b6c6e18", "release": true, "await": true}) -- 0x563ff08a83d0 con 0x563ff09694e0 2024-06-10T00:36:59.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.326+0000 7f41daffd700 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x563ff09694e0 0x563ff09d6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:36:59.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.327+0000 7f41baffd700 10 client.16800 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:59.331 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.327+0000 7f41daffd700 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x563ff09694e0 0x563ff09d6ce0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=228d8b54eb372aaa server_cookie=355f330524a1087d in_seq=0 out_seq=0 2024-06-10T00:36:59.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.610+0000 7f41baffd700 1 -- 172.21.15.121:0/1439493542 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x563ff08a83d0 con 0x563ff09694e0 2024-06-10T00:36:59.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.610+0000 7f41baffd700 10 client.16800 handle_command_reply: tid=0 2024-06-10T00:36:59.611 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.610+0000 7f41ec229b80 1 client.16800 shutdown 2024-06-10T00:36:59.611 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:36:59 smithi121 ceph-mon[32378]: pgmap v277: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 153 op/s 2024-06-10T00:36:59.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41ec229b80 1 -- 172.21.15.121:0/1439493542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 msgr2=0x563ff096d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:59.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41ec229b80 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff096d280 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f41d00098e0 tx=0x7f41d000b930 comp rx=0 tx=0).stop 2024-06-10T00:36:59.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41ec229b80 1 -- 172.21.15.121:0/1439493542 shutdown_connections 2024-06-10T00:36:59.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41ec229b80 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x563ff09694e0 0x563ff09d6ce0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41ec229b80 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563ff0955e20 0x563ff09d2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41ec229b80 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x563ff094fc50 0x563ff096cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.612 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41ec229b80 1 --2- 172.21.15.121:0/1439493542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563ff0955500 0x563ff096d280 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41ec229b80 1 -- 172.21.15.121:0/1439493542 >> 172.21.15.121:0/1439493542 conn(0x563ff087c180 msgr2=0x563ff0958760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:59.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41baffd700 0 client.16800 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:36:59.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41ec229b80 1 -- 172.21.15.121:0/1439493542 shutdown_connections 2024-06-10T00:36:59.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41ec229b80 1 -- 172.21.15.121:0/1439493542 wait complete. 2024-06-10T00:36:59.613 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.611+0000 7f41ec229b80 20 client.16800 trim_cache size 0 max 16384 2024-06-10T00:36:59.613 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:36:59.613 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 33, 2024-06-10T00:36:59.613 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:36:59.613 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 24, 2024-06-10T00:36:59.613 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:36:59.613 INFO:teuthology.orchestra.run.smithi121.stdout: "7b6c6e18": { 2024-06-10T00:36:59.614 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 24, 2024-06-10T00:36:59.614 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:36:59.614 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:36:59.614 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:36:59.614 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:36:59.614 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:36:59.614 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:36:59.614 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 136.0, 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:36:59.615 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:36:59.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 -- 172.21.15.121:0/457739781 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41c40a2d90 msgr2=0x7f41c40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:59.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41c40a2d90 0x7f41c40a5250 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f41d0006010 tx=0x7f41d000b540 comp rx=0 tx=0).stop 2024-06-10T00:36:59.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 -- 172.21.15.121:0/457739781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41dc1046a0 msgr2=0x7f41dc19aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:36:59.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41dc1046a0 0x7f41dc19aea0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00b530 tx=0x7f41cc00b840 comp rx=0 tx=0).stop 2024-06-10T00:36:59.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 -- 172.21.15.121:0/457739781 shutdown_connections 2024-06-10T00:36:59.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f41c40a2d90 0x7f41c40a5250 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41dc10a6a0 0x7f41dc19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41dc1046a0 0x7f41dc19aea0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 --2- 172.21.15.121:0/457739781 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f41dc105060 0x7f41dc19b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:36:59.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 -- 172.21.15.121:0/457739781 >> 172.21.15.121:0/457739781 conn(0x7f41dc100190 msgr2=0x7f41dc1006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:36:59.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 -- 172.21.15.121:0/457739781 shutdown_connections 2024-06-10T00:36:59.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:36:59.612+0000 7f41e19a6700 1 -- 172.21.15.121:0/457739781 wait complete. 2024-06-10T00:36:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:36:59 smithi193 ceph-mon[36716]: pgmap v277: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 153 op/s 2024-06-10T00:36:59.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:36:59 smithi135 ceph-mon[36424]: pgmap v277: 129 pgs: 129 active+clean; 1.6 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 153 op/s 2024-06-10T00:37:00.354 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '7b6c6e18', total seconds elapsed: 22.0 2024-06-10T00:37:01.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:01 smithi193 ceph-mon[36716]: pgmap v278: 129 pgs: 129 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-06-10T00:37:01.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:01 smithi121 ceph-mon[32378]: pgmap v278: 129 pgs: 129 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-06-10T00:37:01.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:01 smithi135 ceph-mon[36424]: pgmap v278: 129 pgs: 129 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-06-10T00:37:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:03 smithi193 ceph-mon[36716]: pgmap v279: 129 pgs: 129 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 623 KiB/s wr, 99 op/s 2024-06-10T00:37:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:03 smithi193 ceph-mon[36716]: osdmap e106: 12 total, 12 up, 12 in 2024-06-10T00:37:03.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:03 smithi121 ceph-mon[32378]: pgmap v279: 129 pgs: 129 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 623 KiB/s wr, 99 op/s 2024-06-10T00:37:03.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:03 smithi121 ceph-mon[32378]: osdmap e106: 12 total, 12 up, 12 in 2024-06-10T00:37:03.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:03 smithi135 ceph-mon[36424]: pgmap v279: 129 pgs: 129 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 623 KiB/s wr, 99 op/s 2024-06-10T00:37:03.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:03 smithi135 ceph-mon[36424]: osdmap e106: 12 total, 12 up, 12 in 2024-06-10T00:37:05.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:05 smithi193 ceph-mon[36716]: pgmap v281: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 434 KiB/s wr, 59 op/s 2024-06-10T00:37:05.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:05 smithi121 ceph-mon[32378]: pgmap v281: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 434 KiB/s wr, 59 op/s 2024-06-10T00:37:05.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:05 smithi135 ceph-mon[36424]: pgmap v281: 129 pgs: 8 active+clean+snaptrim, 108 active+clean, 13 active+clean+snaptrim_wait; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 434 KiB/s wr, 59 op/s 2024-06-10T00:37:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:07 smithi193 ceph-mon[36716]: pgmap v282: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 930 KiB/s wr, 118 op/s 2024-06-10T00:37:07.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:07 smithi121 ceph-mon[32378]: pgmap v282: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 930 KiB/s wr, 118 op/s 2024-06-10T00:37:07.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:07 smithi135 ceph-mon[36424]: pgmap v282: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 930 KiB/s wr, 118 op/s 2024-06-10T00:37:09.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:09 smithi193 ceph-mon[36716]: pgmap v283: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 930 KiB/s wr, 118 op/s 2024-06-10T00:37:09.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:09 smithi121 ceph-mon[32378]: pgmap v283: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 930 KiB/s wr, 118 op/s 2024-06-10T00:37:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:09 smithi135 ceph-mon[36424]: pgmap v283: 129 pgs: 20 active+clean+snaptrim, 56 active+clean, 53 active+clean+snaptrim_wait; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 930 KiB/s wr, 118 op/s 2024-06-10T00:37:11.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:11 smithi193 ceph-mon[36716]: pgmap v284: 129 pgs: 6 active+clean+snaptrim, 104 active+clean, 19 active+clean+snaptrim_wait; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 165 op/s 2024-06-10T00:37:11.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:11 smithi121 ceph-mon[32378]: pgmap v284: 129 pgs: 6 active+clean+snaptrim, 104 active+clean, 19 active+clean+snaptrim_wait; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 165 op/s 2024-06-10T00:37:11.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:11 smithi135 ceph-mon[36424]: pgmap v284: 129 pgs: 6 active+clean+snaptrim, 104 active+clean, 19 active+clean+snaptrim_wait; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 165 op/s 2024-06-10T00:37:13.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:13 smithi193 ceph-mon[36716]: pgmap v285: 129 pgs: 129 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 209 op/s 2024-06-10T00:37:13.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:13 smithi121 ceph-mon[32378]: pgmap v285: 129 pgs: 129 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 209 op/s 2024-06-10T00:37:13.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:13 smithi135 ceph-mon[36424]: pgmap v285: 129 pgs: 129 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 209 op/s 2024-06-10T00:37:15.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:15 smithi193 ceph-mon[36716]: pgmap v286: 129 pgs: 129 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 228 op/s 2024-06-10T00:37:15.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:15 smithi193 ceph-mon[36716]: osdmap e107: 12 total, 12 up, 12 in 2024-06-10T00:37:15.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:15 smithi121 ceph-mon[32378]: pgmap v286: 129 pgs: 129 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 228 op/s 2024-06-10T00:37:15.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:15 smithi121 ceph-mon[32378]: osdmap e107: 12 total, 12 up, 12 in 2024-06-10T00:37:15.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:15 smithi135 ceph-mon[36424]: pgmap v286: 129 pgs: 129 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 228 op/s 2024-06-10T00:37:15.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:15 smithi135 ceph-mon[36424]: osdmap e107: 12 total, 12 up, 12 in 2024-06-10T00:37:17.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:17 smithi193 ceph-mon[36716]: pgmap v288: 129 pgs: 129 active+clean; 1.6 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-06-10T00:37:17.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:17 smithi121 ceph-mon[32378]: pgmap v288: 129 pgs: 129 active+clean; 1.6 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-06-10T00:37:17.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:17 smithi135 ceph-mon[36424]: pgmap v288: 129 pgs: 129 active+clean; 1.6 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-06-10T00:37:19.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:19 smithi193 ceph-mon[36716]: pgmap v289: 129 pgs: 129 active+clean; 1.6 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-06-10T00:37:19.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:19 smithi121 ceph-mon[32378]: pgmap v289: 129 pgs: 129 active+clean; 1.6 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-06-10T00:37:19.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:19 smithi135 ceph-mon[36424]: pgmap v289: 129 pgs: 129 active+clean; 1.6 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-06-10T00:37:21.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:21 smithi193 ceph-mon[36716]: pgmap v290: 129 pgs: 129 active+clean; 1.6 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 268 op/s 2024-06-10T00:37:21.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:21 smithi121 ceph-mon[32378]: pgmap v290: 129 pgs: 129 active+clean; 1.6 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 268 op/s 2024-06-10T00:37:21.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:21 smithi135 ceph-mon[36424]: pgmap v290: 129 pgs: 129 active+clean; 1.6 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 268 op/s 2024-06-10T00:37:23.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:23 smithi193 ceph-mon[36716]: pgmap v291: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 280 op/s 2024-06-10T00:37:23.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:23 smithi121 ceph-mon[32378]: pgmap v291: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 280 op/s 2024-06-10T00:37:23.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:23 smithi135 ceph-mon[36424]: pgmap v291: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 280 op/s 2024-06-10T00:37:25.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:25 smithi193 ceph-mon[36716]: pgmap v292: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-06-10T00:37:25.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:25 smithi121 ceph-mon[32378]: pgmap v292: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-06-10T00:37:25.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:25 smithi135 ceph-mon[36424]: pgmap v292: 129 pgs: 129 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 293 op/s 2024-06-10T00:37:27.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:27 smithi193 ceph-mon[36716]: pgmap v293: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 268 op/s 2024-06-10T00:37:27.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:27 smithi121 ceph-mon[32378]: pgmap v293: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 268 op/s 2024-06-10T00:37:27.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:27 smithi135 ceph-mon[36424]: pgmap v293: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 268 op/s 2024-06-10T00:37:29.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:29 smithi193 ceph-mon[36716]: pgmap v294: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 232 op/s 2024-06-10T00:37:29.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:29 smithi121 ceph-mon[32378]: pgmap v294: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 232 op/s 2024-06-10T00:37:29.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:29 smithi135 ceph-mon[36424]: pgmap v294: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 232 op/s 2024-06-10T00:37:30.070 INFO:tasks.quiescer.fs.[cephfs]:Total duration (35.6) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-06-10T00:37:30.070 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.8 2024-06-10T00:37:30.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:37:30.444 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:37:31.365 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:31 smithi121 ceph-mon[32378]: pgmap v295: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 296 op/s 2024-06-10T00:37:31.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:31 smithi193 ceph-mon[36716]: pgmap v295: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 296 op/s 2024-06-10T00:37:31.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:31 smithi135 ceph-mon[36424]: pgmap v295: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 296 op/s 2024-06-10T00:37:32.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.780+0000 7fb03faa4700 1 -- 172.21.15.121:0/3673823813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb03806b100 msgr2=0x7fb03806b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:32.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.780+0000 7fb03faa4700 1 --2- 172.21.15.121:0/3673823813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb03806b100 0x7fb03806b580 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb028009990 tx=0x7fb028009ca0 comp rx=0 tx=0).stop 2024-06-10T00:37:32.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.780+0000 7fb03faa4700 1 -- 172.21.15.121:0/3673823813 shutdown_connections 2024-06-10T00:37:32.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.780+0000 7fb03faa4700 1 --2- 172.21.15.121:0/3673823813 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb038104ec0 0x7fb038105370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:32.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.780+0000 7fb03faa4700 1 --2- 172.21.15.121:0/3673823813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0381073b0 0x7fb03806abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.780+0000 7fb03faa4700 1 --2- 172.21.15.121:0/3673823813 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb03806b100 0x7fb03806b580 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.780+0000 7fb03faa4700 1 -- 172.21.15.121:0/3673823813 >> 172.21.15.121:0/3673823813 conn(0x7fb038078340 msgr2=0x7fb038078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.780+0000 7fb03faa4700 1 -- 172.21.15.121:0/3673823813 shutdown_connections 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.780+0000 7fb03faa4700 1 -- 172.21.15.121:0/3673823813 wait complete. 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03faa4700 1 Processor -- start 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03faa4700 1 -- start start 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03faa4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb03806b100 0x7fb03819f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03faa4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038104ec0 0x7fb03819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03faa4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0381073b0 0x7fb0381a3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03faa4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb038118dc0 con 0x7fb038104ec0 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03faa4700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fb038118c40 con 0x7fb03806b100 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03faa4700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb038118f40 con 0x7fb0381073b0 2024-06-10T00:37:32.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03d03f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038104ec0 0x7fb03819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03d03f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038104ec0 0x7fb03819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44338/0 (socket says 172.21.15.121:44338) 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03d03f700 1 -- 172.21.15.121:0/1905310234 learned_addr learned my addr 172.21.15.121:0/1905310234 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03d840700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb03806b100 0x7fb03819f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03e041700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0381073b0 0x7fb0381a3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03d03f700 1 -- 172.21.15.121:0/1905310234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0381073b0 msgr2=0x7fb0381a3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03d03f700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0381073b0 0x7fb0381a3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03d03f700 1 -- 172.21.15.121:0/1905310234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb03806b100 msgr2=0x7fb03819f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03d03f700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb03806b100 0x7fb03819f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03d03f700 1 -- 172.21.15.121:0/1905310234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0381a4260 con 0x7fb038104ec0 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.781+0000 7fb03d03f700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038104ec0 0x7fb03819f790 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb028000c00 tx=0x7fb02800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.782+0000 7fb02effd700 1 -- 172.21.15.121:0/1905310234 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb028027070 con 0x7fb038104ec0 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.782+0000 7fb03faa4700 1 -- 172.21.15.121:0/1905310234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0381a44f0 con 0x7fb038104ec0 2024-06-10T00:37:32.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.782+0000 7fb02effd700 1 -- 172.21.15.121:0/1905310234 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb028003d70 con 0x7fb038104ec0 2024-06-10T00:37:32.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.782+0000 7fb03faa4700 1 -- 172.21.15.121:0/1905310234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0381abdd0 con 0x7fb038104ec0 2024-06-10T00:37:32.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.782+0000 7fb02effd700 1 -- 172.21.15.121:0/1905310234 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02800f460 con 0x7fb038104ec0 2024-06-10T00:37:32.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.783+0000 7fb03faa4700 1 -- 172.21.15.121:0/1905310234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb038069180 con 0x7fb038104ec0 2024-06-10T00:37:32.785 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.784+0000 7fb02effd700 1 -- 172.21.15.121:0/1905310234 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fb02802a070 con 0x7fb038104ec0 2024-06-10T00:37:32.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.784+0000 7fb02effd700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb0240a2d90 0x7fb0240a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:32.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.785+0000 7fb02effd700 1 -- 172.21.15.121:0/1905310234 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7fb0280d0690 con 0x7fb038104ec0 2024-06-10T00:37:32.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.785+0000 7fb03d840700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb0240a2d90 0x7fb0240a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:32.786 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.785+0000 7fb03d840700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb0240a2d90 0x7fb0240a5250 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb034009be0 tx=0x7fb034009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:32.789 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:32.789+0000 7fb02effd700 1 -- 172.21.15.121:0/1905310234 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb028013070 con 0x7fb038104ec0 2024-06-10T00:37:33.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.170+0000 7fb03faa4700 1 -- 172.21.15.121:0/1905310234 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0381abfa0 con 0x7fb038104ec0 2024-06-10T00:37:33.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.171+0000 7fb02effd700 1 -- 172.21.15.121:0/1905310234 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11895 (secure 0 0 0) 0x7fb02809cf50 con 0x7fb038104ec0 2024-06-10T00:37:33.171 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:37:33.171 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:35:59.291154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":189,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:37:33.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 -- 172.21.15.121:0/1905310234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb0240a2d90 msgr2=0x7fb0240a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:33.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb0240a2d90 0x7fb0240a5250 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb034009be0 tx=0x7fb034009260 comp rx=0 tx=0).stop 2024-06-10T00:37:33.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 -- 172.21.15.121:0/1905310234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038104ec0 msgr2=0x7fb03819f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:33.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038104ec0 0x7fb03819f790 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb028000c00 tx=0x7fb02800bab0 comp rx=0 tx=0).stop 2024-06-10T00:37:33.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 -- 172.21.15.121:0/1905310234 shutdown_connections 2024-06-10T00:37:33.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fb0240a2d90 0x7fb0240a5250 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:33.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb0381073b0 0x7fb0381a3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:33.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb03806b100 0x7fb03819f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:33.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 --2- 172.21.15.121:0/1905310234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb038104ec0 0x7fb03819f790 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:33.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 -- 172.21.15.121:0/1905310234 >> 172.21.15.121:0/1905310234 conn(0x7fb038078340 msgr2=0x7fb038103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:33.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 -- 172.21.15.121:0/1905310234 shutdown_connections 2024-06-10T00:37:33.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:33.173+0000 7fb03faa4700 1 -- 172.21.15.121:0/1905310234 wait complete. 2024-06-10T00:37:33.174 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 33 2024-06-10T00:37:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:33 smithi121 ceph-mon[32378]: pgmap v296: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 284 op/s 2024-06-10T00:37:33.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:33 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1905310234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:37:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:33 smithi193 ceph-mon[36716]: pgmap v296: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 284 op/s 2024-06-10T00:37:33.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:33 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1905310234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:37:33.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:33 smithi135 ceph-mon[36424]: pgmap v296: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.4 MiB/s wr, 284 op/s 2024-06-10T00:37:33.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:33 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1905310234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:37:33.853 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0' 2024-06-10T00:37:33.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0 2024-06-10T00:37:34.190 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:37:35.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:35 smithi193 ceph-mon[36716]: pgmap v297: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-06-10T00:37:35.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:35 smithi121 ceph-mon[32378]: pgmap v297: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-06-10T00:37:35.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:35 smithi135 ceph-mon[36424]: pgmap v297: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 292 op/s 2024-06-10T00:37:37.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.367+0000 7f3970c85700 1 -- 172.21.15.121:0/2162371582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f396c1179d0 msgr2=0x7f396c119dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:37.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.367+0000 7f3970c85700 1 --2- 172.21.15.121:0/2162371582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f396c1179d0 0x7f396c119dc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f396400b0d0 tx=0x7f396400b3e0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.368+0000 7f3970c85700 1 -- 172.21.15.121:0/2162371582 shutdown_connections 2024-06-10T00:37:37.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.368+0000 7f3970c85700 1 --2- 172.21.15.121:0/2162371582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f396c1179d0 0x7f396c119dc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.368+0000 7f3970c85700 1 --2- 172.21.15.121:0/2162371582 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c109490 0x7f396c109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.368+0000 7f3970c85700 1 --2- 172.21.15.121:0/2162371582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396c109db0 0x7f396c117490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.368+0000 7f3970c85700 1 -- 172.21.15.121:0/2162371582 >> 172.21.15.121:0/2162371582 conn(0x7f396c06d600 msgr2=0x7f396c06da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:37.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.370+0000 7f3970c85700 1 -- 172.21.15.121:0/2162371582 shutdown_connections 2024-06-10T00:37:37.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.372+0000 7f3970c85700 1 -- 172.21.15.121:0/2162371582 wait complete. 2024-06-10T00:37:37.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.373+0000 7f3970c85700 1 Processor -- start 2024-06-10T00:37:37.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.373+0000 7f3970c85700 1 -- start start 2024-06-10T00:37:37.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.374+0000 7f3970c85700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c109490 0x7f396c117360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.374+0000 7f3970c85700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396c109db0 0x7f396c123310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.374+0000 7f3970c85700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f396c1179d0 0x7f396c123940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.374+0000 7f3970c85700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f396c10e240 con 0x7f396c109db0 2024-06-10T00:37:37.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.374+0000 7f3970c85700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f396c10e0c0 con 0x7f396c109490 2024-06-10T00:37:37.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.374+0000 7f3970c85700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f396c10e3c0 con 0x7f396c1179d0 2024-06-10T00:37:37.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.374+0000 7f3969d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396c109db0 0x7f396c123310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.374+0000 7f3969d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396c109db0 0x7f396c123310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44354/0 (socket says 172.21.15.121:44354) 2024-06-10T00:37:37.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.374+0000 7f3969d9b700 1 -- 172.21.15.121:0/883223667 learned_addr learned my addr 172.21.15.121:0/883223667 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:37.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.374+0000 7f396ad9d700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f396c1179d0 0x7f396c123940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.374+0000 7f396a59c700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c109490 0x7f396c117360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.375+0000 7f3969d9b700 1 -- 172.21.15.121:0/883223667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f396c1179d0 msgr2=0x7f396c123940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.375+0000 7f3969d9b700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f396c1179d0 0x7f396c123940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.375+0000 7f3969d9b700 1 -- 172.21.15.121:0/883223667 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c109490 msgr2=0x7f396c117360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.375+0000 7f3969d9b700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c109490 0x7f396c117360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.375+0000 7f3969d9b700 1 -- 172.21.15.121:0/883223667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c1bc5b0 con 0x7f396c109db0 2024-06-10T00:37:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.375+0000 7f396a59c700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c109490 0x7f396c117360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:37:37.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.375+0000 7f3969d9b700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396c109db0 0x7f396c123310 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f395c00d950 tx=0x7f395c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:37.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.377+0000 7f395b7fe700 1 -- 172.21.15.121:0/883223667 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c016070 con 0x7f396c109db0 2024-06-10T00:37:37.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.378+0000 7f3970c85700 1 -- 172.21.15.121:0/883223667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c1bc780 con 0x7f396c109db0 2024-06-10T00:37:37.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.378+0000 7f3970c85700 1 -- 172.21.15.121:0/883223667 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c1bcd10 con 0x7f396c109db0 2024-06-10T00:37:37.379 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.378+0000 7f395b7fe700 1 -- 172.21.15.121:0/883223667 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f395c009b40 con 0x7f396c109db0 2024-06-10T00:37:37.380 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.378+0000 7f395b7fe700 1 -- 172.21.15.121:0/883223667 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c00f620 con 0x7f396c109db0 2024-06-10T00:37:37.383 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.383+0000 7f395b7fe700 1 -- 172.21.15.121:0/883223667 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f395c019070 con 0x7f396c109db0 2024-06-10T00:37:37.384 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.383+0000 7f397b508b80 1 Processor -- start 2024-06-10T00:37:37.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.384+0000 7f397b508b80 1 -- start start 2024-06-10T00:37:37.385 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.384+0000 7f395b7fe700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f39540a2f30 0x7f39540a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.385+0000 7f397b508b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b148c590 0x5560b148c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.385+0000 7f397b508b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148ceb0 0x5560b14867a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.385+0000 7f397b508b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 0x5560b1487100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.385+0000 7f397b508b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5560b13790f0 con 0x5560b148ceb0 2024-06-10T00:37:37.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.385+0000 7f397b508b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5560b13d79e0 con 0x5560b1486ce0 2024-06-10T00:37:37.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.385+0000 7f397b508b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5560b133f5f0 con 0x5560b148c590 2024-06-10T00:37:37.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.385+0000 7f396ad9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 0x5560b1487100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.385+0000 7f396ad9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 0x5560b1487100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:45154/0 (socket says 172.21.15.121:45154) 2024-06-10T00:37:37.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.385+0000 7f396ad9d700 1 -- 172.21.15.121:0/2001756834 learned_addr learned my addr 172.21.15.121:0/2001756834 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:37.386 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.385+0000 7f396a59c700 1 --2- 172.21.15.121:0/2001756834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b148c590 0x5560b148c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.385+0000 7f395b7fe700 1 -- 172.21.15.121:0/883223667 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7f395c0cf720 con 0x7f396c109db0 2024-06-10T00:37:37.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.386+0000 7f396ad9d700 1 -- 172.21.15.121:0/2001756834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b148c590 msgr2=0x5560b148c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:37.387 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.386+0000 7f396ad9d700 1 --2- 172.21.15.121:0/2001756834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b148c590 0x5560b148c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.386+0000 7f3969d9b700 1 --2- 172.21.15.121:0/2001756834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148ceb0 0x5560b14867a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.386+0000 7f396ad9d700 1 -- 172.21.15.121:0/2001756834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148ceb0 msgr2=0x5560b14867a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.386+0000 7f396ad9d700 1 --2- 172.21.15.121:0/2001756834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148ceb0 0x5560b14867a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.386+0000 7f396ad9d700 1 -- 172.21.15.121:0/2001756834 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560b1326a60 con 0x5560b1486ce0 2024-06-10T00:37:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.386+0000 7f396ad9d700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f39540a2f30 0x7f39540a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.388 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.386+0000 7f3969d9b700 1 --2- 172.21.15.121:0/2001756834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148ceb0 0x5560b14867a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:37:37.390 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.388+0000 7f396ad9d700 1 --2- 172.21.15.121:0/2001756834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 0x5560b1487100 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f39640060b0 tx=0x7f3964009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce9af21582ae5f21 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.388+0000 7f396ad9d700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f39540a2f30 0x7f39540a53f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3964009850 tx=0x7f396401e000 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.388+0000 7f3958ff9700 1 -- 172.21.15.121:0/2001756834 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396402c020 con 0x5560b1486ce0 2024-06-10T00:37:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.388+0000 7f3958ff9700 1 -- 172.21.15.121:0/2001756834 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3964007d20 con 0x5560b1486ce0 2024-06-10T00:37:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.388+0000 7f3958ff9700 1 -- 172.21.15.121:0/2001756834 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964005040 con 0x5560b1486ce0 2024-06-10T00:37:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.389+0000 7f397b508b80 1 -- 172.21.15.121:0/2001756834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 msgr2=0x5560b1487100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.389+0000 7f397b508b80 1 --2- 172.21.15.121:0/2001756834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 0x5560b1487100 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f39640060b0 tx=0x7f3964009260 comp rx=0 tx=0).stop 2024-06-10T00:37:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.390+0000 7f397b508b80 1 -- 172.21.15.121:0/2001756834 shutdown_connections 2024-06-10T00:37:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.390+0000 7f397b508b80 1 --2- 172.21.15.121:0/2001756834 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 0x5560b1487100 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.390+0000 7f397b508b80 1 --2- 172.21.15.121:0/2001756834 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b148c590 0x5560b148c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.390+0000 7f397b508b80 1 --2- 172.21.15.121:0/2001756834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148ceb0 0x5560b14867a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.391 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.390+0000 7f397b508b80 1 -- 172.21.15.121:0/2001756834 >> 172.21.15.121:0/2001756834 conn(0x5560b13b3210 msgr2=0x5560b1481a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:37.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.391+0000 7f397b508b80 1 -- 172.21.15.121:0/2001756834 shutdown_connections 2024-06-10T00:37:37.392 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.391+0000 7f397b508b80 1 -- 172.21.15.121:0/2001756834 wait complete. 2024-06-10T00:37:37.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.392+0000 7f397b508b80 1 -- start start 2024-06-10T00:37:37.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.392+0000 7f397b508b80 1 Processor -- start 2024-06-10T00:37:37.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.392+0000 7f397b508b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 0x5560b1492c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.392+0000 7f397b508b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b148ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.396 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.392+0000 7f397b508b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b148ceb0 0x5560b1490490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.392+0000 7f397b508b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5560b1378150 con 0x5560b148c590 2024-06-10T00:37:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.392+0000 7f397b508b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5560b13d39a0 con 0x5560b1486ce0 2024-06-10T00:37:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.392+0000 7f397b508b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5560b1493830 con 0x5560b148ceb0 2024-06-10T00:37:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.392+0000 7f397b508b80 10 client.0 _ll_register_callbacks cb 0x5560b136d740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:37:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.393+0000 7f396ad9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b148ceb0 0x5560b1490490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.393+0000 7f396ad9d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b148ceb0 0x5560b1490490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:60162/0 (socket says 172.21.15.121:60162) 2024-06-10T00:37:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.393+0000 7f396ad9d700 1 -- 172.21.15.121:0/379245219 learned_addr learned my addr 172.21.15.121:0/379245219 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.393+0000 7f396a59c700 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 0x5560b1492c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.393+0000 7f3969d9b700 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b148ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.393+0000 7f3969d9b700 1 -- 172.21.15.121:0/379245219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b148ceb0 msgr2=0x5560b1490490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:37.397 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.393+0000 7f3969d9b700 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b148ceb0 0x5560b1490490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.393+0000 7f3969d9b700 1 -- 172.21.15.121:0/379245219 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 msgr2=0x5560b1492c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:37.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.393+0000 7f3969d9b700 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 0x5560b1492c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.393+0000 7f3969d9b700 1 -- 172.21.15.121:0/379245219 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560b139d8e0 con 0x5560b148c590 2024-06-10T00:37:37.398 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.394+0000 7f3969d9b700 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b148ff50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f395c0cca20 tx=0x7f395c09b1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7319356f2bd78f6 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:37.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.396+0000 7f3951ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:37:37.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.396+0000 7f3951ffb700 1 -- 172.21.15.121:0/379245219 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c0ce040 con 0x5560b148c590 2024-06-10T00:37:37.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.397+0000 7f3951ffb700 1 -- 172.21.15.121:0/379245219 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7f395c0860d0 con 0x5560b148c590 2024-06-10T00:37:37.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.398+0000 7f397b508b80 1 -- 172.21.15.121:0/379245219 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3964007d20 con 0x5560b148c590 2024-06-10T00:37:37.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.398+0000 7f3951ffb700 1 -- 172.21.15.121:0/379245219 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f395c020040 con 0x5560b148c590 2024-06-10T00:37:37.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.399+0000 7f397b508b80 10 client.16815 fetch_fsmap learned FSMap version 33 2024-06-10T00:37:37.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.399+0000 7f397b508b80 1 -- 172.21.15.121:0/379245219 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5560b14c0020 con 0x5560b148c590 2024-06-10T00:37:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.402+0000 7f3951ffb700 1 -- 172.21.15.121:0/379245219 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3964007d20 con 0x5560b148c590 2024-06-10T00:37:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.402+0000 7f3951ffb700 1 -- 172.21.15.121:0/379245219 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f395c097e60 con 0x5560b148c590 2024-06-10T00:37:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.403+0000 7f397b508b80 10 client.16815 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:37:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.403+0000 7f397b508b80 10 client.16815 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:37:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.403+0000 7f397b508b80 20 client.16815 populate_metadata read hostname 'smithi121' 2024-06-10T00:37:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.403+0000 7f397b508b80 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560b14c29f0 0x5560b14c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.403+0000 7f397b508b80 4 client.16815 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:37:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.403+0000 7f397b508b80 1 -- 172.21.15.121:0/379245219 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560b13e4ee0 con 0x5560b14c29f0 2024-06-10T00:37:37.404 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.403+0000 7f396a59c700 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560b14c29f0 0x5560b14c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.404+0000 7f396a59c700 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560b14c29f0 0x5560b14c4e90 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fba46f67e2b1623 server_cookie=95984483a9157d80 in_seq=0 out_seq=0 2024-06-10T00:37:37.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.404+0000 7f3951ffb700 10 client.16815 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:37:37.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.405+0000 7f3951ffb700 1 -- 172.21.15.121:0/379245219 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5560b13e4ee0 con 0x5560b14c29f0 2024-06-10T00:37:37.405 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.405+0000 7f3951ffb700 10 client.16815 handle_command_reply: tid=0 2024-06-10T00:37:37.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.405+0000 7f397b508b80 1 client.16815 shutdown 2024-06-10T00:37:37.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.406+0000 7f397b508b80 1 -- 172.21.15.121:0/379245219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 msgr2=0x5560b148ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:37.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.406+0000 7f397b508b80 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b148ff50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f395c0cca20 tx=0x7f395c09b1d0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.407 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.407+0000 7f397b508b80 1 -- 172.21.15.121:0/379245219 shutdown_connections 2024-06-10T00:37:37.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.407+0000 7f397b508b80 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560b14c29f0 0x5560b14c4e90 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.407+0000 7f397b508b80 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b148ceb0 0x5560b1490490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.407+0000 7f397b508b80 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b1486ce0 0x5560b1492c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.407+0000 7f397b508b80 1 --2- 172.21.15.121:0/379245219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b148ff50 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.407+0000 7f397b508b80 1 -- 172.21.15.121:0/379245219 >> 172.21.15.121:0/379245219 conn(0x5560b13b3210 msgr2=0x5560b1480c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:37.408 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.407+0000 7f3951ffb700 0 client.16815 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:37:37.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.408+0000 7f397b508b80 1 -- 172.21.15.121:0/379245219 shutdown_connections 2024-06-10T00:37:37.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.408+0000 7f397b508b80 1 -- 172.21.15.121:0/379245219 wait complete. 2024-06-10T00:37:37.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.409+0000 7f397b508b80 20 client.16815 trim_cache size 0 max 16384 2024-06-10T00:37:37.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f397b508b80 1 Processor -- start 2024-06-10T00:37:37.448 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f397b508b80 1 -- start start 2024-06-10T00:37:37.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f397b508b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b1486ce0 0x5560b1492790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f397b508b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b14813c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f397b508b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b148ceb0 0x5560b1481900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f397b508b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f394c0118f0 con 0x5560b148c590 2024-06-10T00:37:37.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f3969d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b14813c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f397b508b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f394c00f0c0 con 0x5560b148ceb0 2024-06-10T00:37:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f397b508b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f394c00cf30 con 0x5560b1486ce0 2024-06-10T00:37:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f3969d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b14813c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44374/0 (socket says 172.21.15.121:44374) 2024-06-10T00:37:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f3969d9b700 1 -- 172.21.15.121:0/676342305 learned_addr learned my addr 172.21.15.121:0/676342305 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.448+0000 7f396a59c700 1 --2- 172.21.15.121:0/676342305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b1486ce0 0x5560b1492790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.449+0000 7f3969d9b700 1 -- 172.21.15.121:0/676342305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b1486ce0 msgr2=0x5560b1492790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.449+0000 7f3969d9b700 1 --2- 172.21.15.121:0/676342305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b1486ce0 0x5560b1492790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.449+0000 7f3969d9b700 1 -- 172.21.15.121:0/676342305 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b148ceb0 msgr2=0x5560b1481900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.449+0000 7f3969d9b700 1 --2- 172.21.15.121:0/676342305 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b148ceb0 0x5560b1481900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.449+0000 7f3969d9b700 1 -- 172.21.15.121:0/676342305 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560b148f020 con 0x5560b148c590 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.449+0000 7f3969d9b700 1 --2- 172.21.15.121:0/676342305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b14813c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f395c00b2f0 tx=0x7f395c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fcfae2af84c1f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.449+0000 7f3953fff700 1 -- 172.21.15.121:0/676342305 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c0ce040 con 0x5560b148c590 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.449+0000 7f3953fff700 1 -- 172.21.15.121:0/676342305 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f395c0a4460 con 0x5560b148c590 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.449+0000 7f3953fff700 1 -- 172.21.15.121:0/676342305 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c0b4080 con 0x5560b148c590 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.450+0000 7f397b508b80 1 -- 172.21.15.121:0/676342305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 msgr2=0x5560b14813c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.450+0000 7f397b508b80 1 --2- 172.21.15.121:0/676342305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b14813c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f395c00b2f0 tx=0x7f395c000bc0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.451+0000 7f397b508b80 1 -- 172.21.15.121:0/676342305 shutdown_connections 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.451+0000 7f397b508b80 1 --2- 172.21.15.121:0/676342305 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b148ceb0 0x5560b1481900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.451+0000 7f397b508b80 1 --2- 172.21.15.121:0/676342305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b1486ce0 0x5560b1492790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.451+0000 7f397b508b80 1 --2- 172.21.15.121:0/676342305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b14813c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.451+0000 7f397b508b80 1 -- 172.21.15.121:0/676342305 >> 172.21.15.121:0/676342305 conn(0x5560b13b3210 msgr2=0x5560b148b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:37.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.451+0000 7f397b508b80 1 -- 172.21.15.121:0/676342305 shutdown_connections 2024-06-10T00:37:37.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.451+0000 7f397b508b80 1 -- 172.21.15.121:0/676342305 wait complete. 2024-06-10T00:37:37.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.451+0000 7f397b508b80 1 -- start start 2024-06-10T00:37:37.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.452+0000 7f397b508b80 1 Processor -- start 2024-06-10T00:37:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.452+0000 7f397b508b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b1486ce0 0x5560b1499080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.452+0000 7f397b508b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b1495b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.452+0000 7f397b508b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b148ceb0 0x5560b1496060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.452+0000 7f397b508b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f394c00d580 con 0x5560b148c590 2024-06-10T00:37:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.452+0000 7f397b508b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f394c00e9f0 con 0x5560b148ceb0 2024-06-10T00:37:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.452+0000 7f397b508b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f394c00dc50 con 0x5560b1486ce0 2024-06-10T00:37:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.452+0000 7f397b508b80 10 client.0 _ll_register_callbacks cb 0x5560b136d740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:37:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.453+0000 7f3969d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b1495b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.453+0000 7f3969d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b1495b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44384/0 (socket says 172.21.15.121:44384) 2024-06-10T00:37:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.453+0000 7f3969d9b700 1 -- 172.21.15.121:0/3233315435 learned_addr learned my addr 172.21.15.121:0/3233315435 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:37.456 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.453+0000 7f396a59c700 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b1486ce0 0x5560b1499080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.453+0000 7f396ad9d700 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b148ceb0 0x5560b1496060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.453+0000 7f3969d9b700 1 -- 172.21.15.121:0/3233315435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b1486ce0 msgr2=0x5560b1499080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.453+0000 7f3969d9b700 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b1486ce0 0x5560b1499080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.453+0000 7f3969d9b700 1 -- 172.21.15.121:0/3233315435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b148ceb0 msgr2=0x5560b1496060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.453+0000 7f3969d9b700 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b148ceb0 0x5560b1496060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.453+0000 7f3969d9b700 1 -- 172.21.15.121:0/3233315435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560b1496650 con 0x5560b148c590 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.453+0000 7f3969d9b700 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b1495b20 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f395c09b780 tx=0x7f395c097480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20663725abc58ae server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.454+0000 7f3951ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.454+0000 7f3951ffb700 1 -- 172.21.15.121:0/3233315435 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c0c2060 con 0x5560b148c590 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.454+0000 7f3951ffb700 1 -- 172.21.15.121:0/3233315435 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7f395c0ccca0 con 0x5560b148c590 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.454+0000 7f397b508b80 1 -- 172.21.15.121:0/3233315435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f395c0a4460 con 0x5560b148c590 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.454+0000 7f3951ffb700 1 -- 172.21.15.121:0/3233315435 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f395c07f050 con 0x5560b148c590 2024-06-10T00:37:37.457 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.454+0000 7f397b508b80 10 client.16821 fetch_fsmap learned FSMap version 33 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.454+0000 7f397b508b80 1 -- 172.21.15.121:0/3233315435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5560b152e520 con 0x5560b148c590 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.454+0000 7f3951ffb700 1 -- 172.21.15.121:0/3233315435 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f395c0a4460 con 0x5560b148c590 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.455+0000 7f3951ffb700 1 -- 172.21.15.121:0/3233315435 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f395c0cf280 con 0x5560b148c590 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.455+0000 7f397b508b80 10 client.16821 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.455+0000 7f397b508b80 10 client.16821 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.455+0000 7f397b508b80 20 client.16821 populate_metadata read hostname 'smithi121' 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.455+0000 7f397b508b80 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560b152ead0 0x5560b1530ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.455+0000 7f397b508b80 4 client.16821 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}] 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.455+0000 7f397b508b80 1 -- 172.21.15.121:0/3233315435 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}) -- 0x5560b13df460 con 0x5560b152ead0 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.455+0000 7f396a59c700 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560b152ead0 0x5560b1530ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.455+0000 7f3951ffb700 10 client.16821 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:37:37.458 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:37.455+0000 7f396a59c700 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560b152ead0 0x5560b1530ef0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ecda49282a9e147 server_cookie=a9a6a78451e7b35d in_seq=0 out_seq=0 2024-06-10T00:37:37.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:37 smithi193 ceph-mon[36716]: pgmap v298: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 315 op/s 2024-06-10T00:37:37.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:37 smithi121 ceph-mon[32378]: pgmap v298: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 315 op/s 2024-06-10T00:37:37.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:37 smithi135 ceph-mon[36424]: pgmap v298: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 315 op/s 2024-06-10T00:37:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.303+0000 7f3951ffb700 1 -- 172.21.15.121:0/3233315435 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5560b13df460 con 0x5560b152ead0 2024-06-10T00:37:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.303+0000 7f3951ffb700 10 client.16821 handle_command_reply: tid=0 2024-06-10T00:37:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.303+0000 7f397b508b80 1 client.16821 shutdown 2024-06-10T00:37:38.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.303+0000 7f397b508b80 1 -- 172.21.15.121:0/3233315435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 msgr2=0x5560b1495b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.303+0000 7f397b508b80 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b1495b20 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f395c09b780 tx=0x7f395c097480 comp rx=0 tx=0).stop 2024-06-10T00:37:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.304+0000 7f397b508b80 1 -- 172.21.15.121:0/3233315435 shutdown_connections 2024-06-10T00:37:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.304+0000 7f397b508b80 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5560b152ead0 0x5560b1530ef0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.304+0000 7f397b508b80 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5560b148ceb0 0x5560b1496060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.304+0000 7f397b508b80 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5560b1486ce0 0x5560b1499080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.304+0000 7f3951ffb700 0 client.16821 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:37:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.304+0000 7f397b508b80 1 --2- 172.21.15.121:0/3233315435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5560b148c590 0x5560b1495b20 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.304+0000 7f397b508b80 1 -- 172.21.15.121:0/3233315435 >> 172.21.15.121:0/3233315435 conn(0x5560b13b3210 msgr2=0x5560b1485790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.304+0000 7f397b508b80 1 -- 172.21.15.121:0/3233315435 shutdown_connections 2024-06-10T00:37:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.304+0000 7f397b508b80 1 -- 172.21.15.121:0/3233315435 wait complete. 2024-06-10T00:37:38.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.304+0000 7f397b508b80 20 client.16821 trim_cache size 0 max 16384 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 33, 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 26, 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "b5fb53b2": { 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 26, 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:37:38.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 137.8, 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:37:38.307 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:37:38.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 -- 172.21.15.121:0/883223667 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f39540a2f30 msgr2=0x7f39540a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:38.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f39540a2f30 0x7f39540a53f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3964009850 tx=0x7f396401e000 comp rx=0 tx=0).stop 2024-06-10T00:37:38.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 -- 172.21.15.121:0/883223667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396c109db0 msgr2=0x7f396c123310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:38.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396c109db0 0x7f396c123310 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f395c00d950 tx=0x7f395c00dd10 comp rx=0 tx=0).stop 2024-06-10T00:37:38.308 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 -- 172.21.15.121:0/883223667 shutdown_connections 2024-06-10T00:37:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f39540a2f30 0x7f39540a53f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f396c1179d0 0x7f396c123940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f396c109490 0x7f396c117360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 --2- 172.21.15.121:0/883223667 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f396c109db0 0x7f396c123310 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 -- 172.21.15.121:0/883223667 >> 172.21.15.121:0/883223667 conn(0x7f396c06d600 msgr2=0x7f396c117e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 -- 172.21.15.121:0/883223667 shutdown_connections 2024-06-10T00:37:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:38.305+0000 7f3970c85700 1 -- 172.21.15.121:0/883223667 wait complete. 2024-06-10T00:37:39.633 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:39 smithi193 ceph-mon[36716]: pgmap v299: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 247 op/s 2024-06-10T00:37:39.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:39 smithi121 ceph-mon[32378]: pgmap v299: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 247 op/s 2024-06-10T00:37:39.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:39 smithi135 ceph-mon[36424]: pgmap v299: 129 pgs: 129 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 247 op/s 2024-06-10T00:37:41.365 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'b5fb53b2', quiesce took 11.3 seconds. Will release after: 6.5 2024-06-10T00:37:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:41 smithi193 ceph-mon[36716]: pgmap v300: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-06-10T00:37:41.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:41 smithi121 ceph-mon[32378]: pgmap v300: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-06-10T00:37:41.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:41 smithi135 ceph-mon[36424]: pgmap v300: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-06-10T00:37:43.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:43 smithi121 ceph-mon[32378]: pgmap v301: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 284 op/s 2024-06-10T00:37:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:43 smithi135 ceph-mon[36424]: pgmap v301: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 284 op/s 2024-06-10T00:37:43.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:43 smithi193 ceph-mon[36716]: pgmap v301: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 284 op/s 2024-06-10T00:37:45.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:45 smithi121 ceph-mon[32378]: pgmap v302: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 251 op/s 2024-06-10T00:37:45.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:45 smithi135 ceph-mon[36424]: pgmap v302: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 251 op/s 2024-06-10T00:37:45.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:45 smithi193 ceph-mon[36716]: pgmap v302: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 251 op/s 2024-06-10T00:37:47.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:47 smithi121 ceph-mon[32378]: pgmap v303: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 199 op/s 2024-06-10T00:37:47.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:47 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:37:47.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:47 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:47.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:47 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:47.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:47 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:47.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:47 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:47 smithi135 ceph-mon[36424]: pgmap v303: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 199 op/s 2024-06-10T00:37:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:47 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:37:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:47 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:47 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:47 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:47 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:47.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:37:47.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:47 smithi193 ceph-mon[36716]: pgmap v303: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 199 op/s 2024-06-10T00:37:47.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:47 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:37:47.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:47 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:47.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:47 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:47.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:47 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:47.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:47 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:48.106 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:37:48.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:48 smithi121 ceph-mon[32378]: pgmap v304: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 820 KiB/s wr, 124 op/s 2024-06-10T00:37:48.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:48 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:48.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:48 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:48.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:48 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:37:48.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:48 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:37:48.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:48 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:49.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:48 smithi135 ceph-mon[36424]: pgmap v304: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 820 KiB/s wr, 124 op/s 2024-06-10T00:37:49.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:48 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:49.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:48 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:49.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:48 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:37:49.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:48 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:37:49.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:48 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:49.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:48 smithi193 ceph-mon[36716]: pgmap v304: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 820 KiB/s wr, 124 op/s 2024-06-10T00:37:49.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:48 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:49.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:48 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:49.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:48 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:37:49.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:48 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:37:49.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:48 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:37:50.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.056+0000 7f68e0030700 1 -- 172.21.15.121:0/3557780417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d81010b0 msgr2=0x7f68d8101490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:50.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.056+0000 7f68e0030700 1 --2- 172.21.15.121:0/3557780417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d81010b0 0x7f68d8101490 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f68d4009990 tx=0x7f68d4009ca0 comp rx=0 tx=0).stop 2024-06-10T00:37:50.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.056+0000 7f68e0030700 1 -- 172.21.15.121:0/3557780417 shutdown_connections 2024-06-10T00:37:50.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.056+0000 7f68e0030700 1 --2- 172.21.15.121:0/3557780417 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68d8105f50 0x7f68d810da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:50.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.056+0000 7f68e0030700 1 --2- 172.21.15.121:0/3557780417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d81010b0 0x7f68d8101490 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:50.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.056+0000 7f68e0030700 1 --2- 172.21.15.121:0/3557780417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68d8101a60 0x7f68d8105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:50.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.056+0000 7f68e0030700 1 -- 172.21.15.121:0/3557780417 >> 172.21.15.121:0/3557780417 conn(0x7f68d80766b0 msgr2=0x7f68d8076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:50.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.056+0000 7f68e0030700 1 -- 172.21.15.121:0/3557780417 shutdown_connections 2024-06-10T00:37:50.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.056+0000 7f68e0030700 1 -- 172.21.15.121:0/3557780417 wait complete. 2024-06-10T00:37:50.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.056+0000 7f68e0030700 1 Processor -- start 2024-06-10T00:37:50.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68e0030700 1 -- start start 2024-06-10T00:37:50.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68e0030700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d81010b0 0x7f68d819e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:50.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68e0030700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68d8101a60 0x7f68d819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:50.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68e0030700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68d8105f50 0x7f68d8198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:50.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68e0030700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f68d8112870 con 0x7f68d81010b0 2024-06-10T00:37:50.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68e0030700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f68d81126f0 con 0x7f68d8105f50 2024-06-10T00:37:50.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68e0030700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f68d81129f0 con 0x7f68d8101a60 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68dddcc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d81010b0 0x7f68d819e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68dddcc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d81010b0 0x7f68d819e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36296/0 (socket says 172.21.15.121:36296) 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68dddcc700 1 -- 172.21.15.121:0/470730148 learned_addr learned my addr 172.21.15.121:0/470730148 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68dd5cb700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68d8101a60 0x7f68d819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68de5cd700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68d8105f50 0x7f68d8198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68dddcc700 1 -- 172.21.15.121:0/470730148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68d8101a60 msgr2=0x7f68d819edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68dddcc700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68d8101a60 0x7f68d819edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68dddcc700 1 -- 172.21.15.121:0/470730148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68d8105f50 msgr2=0x7f68d8198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68dddcc700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68d8105f50 0x7f68d8198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68dddcc700 1 -- 172.21.15.121:0/470730148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d81991d0 con 0x7f68d81010b0 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68dd5cb700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68d8101a60 0x7f68d819edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68dddcc700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d81010b0 0x7f68d819e8b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f68d40038b0 tx=0x7f68d4005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.057+0000 7f68ceffd700 1 -- 172.21.15.121:0/470730148 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4026070 con 0x7f68d81010b0 2024-06-10T00:37:50.061 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.058+0000 7f68e0030700 1 -- 172.21.15.121:0/470730148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d8199460 con 0x7f68d81010b0 2024-06-10T00:37:50.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.058+0000 7f68ceffd700 1 -- 172.21.15.121:0/470730148 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68d4004030 con 0x7f68d81010b0 2024-06-10T00:37:50.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.058+0000 7f68ceffd700 1 -- 172.21.15.121:0/470730148 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4021410 con 0x7f68d81010b0 2024-06-10T00:37:50.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.058+0000 7f68e0030700 1 -- 172.21.15.121:0/470730148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d81a5780 con 0x7f68d81010b0 2024-06-10T00:37:50.062 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.059+0000 7f68e0030700 1 -- 172.21.15.121:0/470730148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d81090e0 con 0x7f68d81010b0 2024-06-10T00:37:50.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.060+0000 7f68ceffd700 1 -- 172.21.15.121:0/470730148 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f68d4029070 con 0x7f68d81010b0 2024-06-10T00:37:50.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.060+0000 7f68ceffd700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68c40a2d90 0x7f68c40a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:50.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.061+0000 7f68dd5cb700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68c40a2d90 0x7f68c40a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:50.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.061+0000 7f68ceffd700 1 -- 172.21.15.121:0/470730148 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7f68d40d02f0 con 0x7f68d81010b0 2024-06-10T00:37:50.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.061+0000 7f68dd5cb700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68c40a2d90 0x7f68c40a5250 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f68d80ff8d0 tx=0x7f68c8008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:50.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.065+0000 7f68ceffd700 1 -- 172.21.15.121:0/470730148 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f68d405d850 con 0x7f68d81010b0 2024-06-10T00:37:50.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.308+0000 7f68e0030700 1 -- 172.21.15.121:0/470730148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68d8069180 con 0x7f68d81010b0 2024-06-10T00:37:50.309 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.309+0000 7f68ceffd700 1 -- 172.21.15.121:0/470730148 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11895 (secure 0 0 0) 0x7f68d8069180 con 0x7f68d81010b0 2024-06-10T00:37:50.310 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:37:50.310 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:35:59.291154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":189,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:37:50.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 -- 172.21.15.121:0/470730148 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68c40a2d90 msgr2=0x7f68c40a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:50.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68c40a2d90 0x7f68c40a5250 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f68d80ff8d0 tx=0x7f68c8008040 comp rx=0 tx=0).stop 2024-06-10T00:37:50.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 -- 172.21.15.121:0/470730148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d81010b0 msgr2=0x7f68d819e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:50.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d81010b0 0x7f68d819e8b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f68d40038b0 tx=0x7f68d4005590 comp rx=0 tx=0).stop 2024-06-10T00:37:50.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 -- 172.21.15.121:0/470730148 shutdown_connections 2024-06-10T00:37:50.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f68c40a2d90 0x7f68c40a5250 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:50.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f68d8105f50 0x7f68d8198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:50.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d81010b0 0x7f68d819e8b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:50.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 --2- 172.21.15.121:0/470730148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68d8101a60 0x7f68d819edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:50.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 -- 172.21.15.121:0/470730148 >> 172.21.15.121:0/470730148 conn(0x7f68d80766b0 msgr2=0x7f68d8100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:50.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 -- 172.21.15.121:0/470730148 shutdown_connections 2024-06-10T00:37:50.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:50.311+0000 7f68e0030700 1 -- 172.21.15.121:0/470730148 wait complete. 2024-06-10T00:37:50.313 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 33 2024-06-10T00:37:50.951 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db --set-id b5fb53b2 --release --await' 2024-06-10T00:37:50.951 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db --set-id b5fb53b2 --release --await 2024-06-10T00:37:50.974 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:50 smithi121 ceph-mon[32378]: pgmap v305: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 824 KiB/s wr, 127 op/s 2024-06-10T00:37:50.974 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:50 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/470730148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:37:51.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:50 smithi135 ceph-mon[36424]: pgmap v305: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 824 KiB/s wr, 127 op/s 2024-06-10T00:37:51.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:50 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/470730148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:37:51.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:50 smithi193 ceph-mon[36716]: pgmap v305: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 824 KiB/s wr, 127 op/s 2024-06-10T00:37:51.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:50 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/470730148' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:37:51.190 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:37:52.826 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:52 smithi121 ceph-mon[32378]: pgmap v306: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 282 KiB/s wr, 63 op/s 2024-06-10T00:37:53.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:52 smithi135 ceph-mon[36424]: pgmap v306: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 282 KiB/s wr, 63 op/s 2024-06-10T00:37:53.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.077+0000 7f7422d60700 1 -- 172.21.15.121:0/3266390046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c06b3c0 msgr2=0x7f741c10f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:53.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.077+0000 7f7422d60700 1 --2- 172.21.15.121:0/3266390046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c06b3c0 0x7f741c10f310 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f7414009990 tx=0x7f7414009ca0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.077+0000 7f7422d60700 1 -- 172.21.15.121:0/3266390046 shutdown_connections 2024-06-10T00:37:53.080 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.077+0000 7f7422d60700 1 --2- 172.21.15.121:0/3266390046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c06b3c0 0x7f741c10f310 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.077+0000 7f7422d60700 1 --2- 172.21.15.121:0/3266390046 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f741c1084f0 0x7f741c1088d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.077+0000 7f7422d60700 1 --2- 172.21.15.121:0/3266390046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f741c06aa00 0x7f741c06ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.077+0000 7f7422d60700 1 -- 172.21.15.121:0/3266390046 >> 172.21.15.121:0/3266390046 conn(0x7f741c078300 msgr2=0x7f741c078710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.077+0000 7f7422d60700 1 -- 172.21.15.121:0/3266390046 shutdown_connections 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.077+0000 7f7422d60700 1 -- 172.21.15.121:0/3266390046 wait complete. 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7422d60700 1 Processor -- start 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7422d60700 1 -- start start 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7422d60700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f741c06aa00 0x7f741c198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7422d60700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c06b3c0 0x7f741c199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7422d60700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f741c1084f0 0x7f741c19d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7422d60700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f741c112850 con 0x7f741c06b3c0 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7422d60700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f741c1126d0 con 0x7f741c1084f0 2024-06-10T00:37:53.081 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7422d60700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f741c1129d0 con 0x7f741c06aa00 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7413fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c06b3c0 0x7f741c199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7413fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c06b3c0 0x7f741c199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36322/0 (socket says 172.21.15.121:36322) 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7413fff700 1 -- 172.21.15.121:0/70425126 learned_addr learned my addr 172.21.15.121:0/70425126 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7420afc700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f741c06aa00 0x7f741c198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f74212fd700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f741c1084f0 0x7f741c19d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7413fff700 1 -- 172.21.15.121:0/70425126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f741c06aa00 msgr2=0x7f741c198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7413fff700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f741c06aa00 0x7f741c198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7413fff700 1 -- 172.21.15.121:0/70425126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f741c1084f0 msgr2=0x7f741c19d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7413fff700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f741c1084f0 0x7f741c19d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f7413fff700 1 -- 172.21.15.121:0/70425126 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741c19dd50 con 0x7f741c06b3c0 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.078+0000 7f74212fd700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f741c1084f0 0x7f741c19d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.079+0000 7f7413fff700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c06b3c0 0x7f741c199280 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f740c00b530 tx=0x7f740c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:53.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.079+0000 7f7411ffb700 1 -- 172.21.15.121:0/70425126 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c015020 con 0x7f741c06b3c0 2024-06-10T00:37:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.079+0000 7f7411ffb700 1 -- 172.21.15.121:0/70425126 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f740c011d10 con 0x7f741c06b3c0 2024-06-10T00:37:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.079+0000 7f7422d60700 1 -- 172.21.15.121:0/70425126 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741c19e040 con 0x7f741c06b3c0 2024-06-10T00:37:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.079+0000 7f7411ffb700 1 -- 172.21.15.121:0/70425126 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c00f2e0 con 0x7f741c06b3c0 2024-06-10T00:37:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.079+0000 7f7422d60700 1 -- 172.21.15.121:0/70425126 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f741c1a58c0 con 0x7f741c06b3c0 2024-06-10T00:37:53.083 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.081+0000 7f7411ffb700 1 -- 172.21.15.121:0/70425126 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f740c018020 con 0x7f741c06b3c0 2024-06-10T00:37:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.082+0000 7f7411ffb700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74040a2d90 0x7f74040a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.082+0000 7f7420afc700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74040a2d90 0x7f74040a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.082+0000 7f7411ffb700 1 -- 172.21.15.121:0/70425126 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7f740c0ceff0 con 0x7f741c06b3c0 2024-06-10T00:37:53.084 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f7420afc700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74040a2d90 0x7f74040a5250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f741800a400 tx=0x7f741800d040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:53.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f742d5e3b80 1 Processor -- start 2024-06-10T00:37:53.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f742d5e3b80 1 -- start start 2024-06-10T00:37:53.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d68c40 0x5578c4d69020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d62e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d633d0 0x5578c4d63810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5578c4cb4910 con 0x5578c4d69560 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5578c4c5c820 con 0x5578c4d68c40 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5578c4cb08d0 con 0x5578c4d633d0 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f7413fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d62e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f7413fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d62e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36326/0 (socket says 172.21.15.121:36326) 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.083+0000 7f7413fff700 1 -- 172.21.15.121:0/961736195 learned_addr learned my addr 172.21.15.121:0/961736195 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f7420afc700 1 --2- 172.21.15.121:0/961736195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d68c40 0x5578c4d69020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f7413fff700 1 -- 172.21.15.121:0/961736195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d633d0 msgr2=0x5578c4d63810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f74212fd700 1 --2- 172.21.15.121:0/961736195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d633d0 0x5578c4d63810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f7413fff700 1 --2- 172.21.15.121:0/961736195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d633d0 0x5578c4d63810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f7413fff700 1 -- 172.21.15.121:0/961736195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d68c40 msgr2=0x5578c4d69020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f7413fff700 1 --2- 172.21.15.121:0/961736195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d68c40 0x5578c4d69020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f7413fff700 1 -- 172.21.15.121:0/961736195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578c4c53050 con 0x5578c4d69560 2024-06-10T00:37:53.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f7420afc700 1 --2- 172.21.15.121:0/961736195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d68c40 0x5578c4d69020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f7413fff700 1 --2- 172.21.15.121:0/961736195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d62e90 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f740c00bc40 tx=0x7f740c09ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3d0a75e601d3ea server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f7402ffd700 1 -- 172.21.15.121:0/961736195 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c09f5e0 con 0x5578c4d69560 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f7402ffd700 1 -- 172.21.15.121:0/961736195 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f740c0a3ea0 con 0x5578c4d69560 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.084+0000 7f7402ffd700 1 -- 172.21.15.121:0/961736195 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c09f360 con 0x5578c4d69560 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.085+0000 7f742d5e3b80 1 -- 172.21.15.121:0/961736195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 msgr2=0x5578c4d62e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.085+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/961736195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d62e90 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f740c00bc40 tx=0x7f740c09ac10 comp rx=0 tx=0).stop 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.085+0000 7f742d5e3b80 1 -- 172.21.15.121:0/961736195 shutdown_connections 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.085+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/961736195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d633d0 0x5578c4d63810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.085+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/961736195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d68c40 0x5578c4d69020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.085+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/961736195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d62e90 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.085+0000 7f742d5e3b80 1 -- 172.21.15.121:0/961736195 >> 172.21.15.121:0/961736195 conn(0x5578c4c8ddc0 msgr2=0x5578c4d5e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.085+0000 7f742d5e3b80 1 -- 172.21.15.121:0/961736195 shutdown_connections 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.085+0000 7f742d5e3b80 1 -- 172.21.15.121:0/961736195 wait complete. 2024-06-10T00:37:53.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.085+0000 7f742d5e3b80 1 -- start start 2024-06-10T00:37:53.089 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.085+0000 7f742d5e3b80 1 Processor -- start 2024-06-10T00:37:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d633d0 0x5578c4d6f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d68c40 0x5578c4d6c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d6cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5578c4d6ffd0 con 0x5578c4d69560 2024-06-10T00:37:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5578c4c54cf0 con 0x5578c4d633d0 2024-06-10T00:37:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5578c4d70150 con 0x5578c4d68c40 2024-06-10T00:37:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f742d5e3b80 10 client.0 _ll_register_callbacks cb 0x5578c4c4ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:37:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f74212fd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d6cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f74212fd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d6cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36330/0 (socket says 172.21.15.121:36330) 2024-06-10T00:37:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f74212fd700 1 -- 172.21.15.121:0/4248559448 learned_addr learned my addr 172.21.15.121:0/4248559448 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:53.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f7420afc700 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d633d0 0x5578c4d6f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f7413fff700 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d68c40 0x5578c4d6c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f74212fd700 1 -- 172.21.15.121:0/4248559448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d68c40 msgr2=0x5578c4d6c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f74212fd700 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d68c40 0x5578c4d6c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f74212fd700 1 -- 172.21.15.121:0/4248559448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d633d0 msgr2=0x5578c4d6f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f74212fd700 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d633d0 0x5578c4d6f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f74212fd700 1 -- 172.21.15.121:0/4248559448 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578c4c039d0 con 0x5578c4d69560 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f74212fd700 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d6cb70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f741400b5c0 tx=0x7f7414005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebea5d6a9f904469 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f73fbfff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f73fbfff700 1 -- 172.21.15.121:0/4248559448 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7414027060 con 0x5578c4d69560 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.086+0000 7f73fbfff700 1 -- 172.21.15.121:0/4248559448 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7f74140052c0 con 0x5578c4d69560 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f742d5e3b80 1 -- 172.21.15.121:0/4248559448 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f740c0a3ea0 con 0x5578c4d69560 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f73fbfff700 1 -- 172.21.15.121:0/4248559448 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f741402e050 con 0x5578c4d69560 2024-06-10T00:37:53.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f742d5e3b80 10 client.16839 fetch_fsmap learned FSMap version 33 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f742d5e3b80 1 -- 172.21.15.121:0/4248559448 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5578c4d7a850 con 0x5578c4d69560 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f73fbfff700 1 -- 172.21.15.121:0/4248559448 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f740c0a3ea0 con 0x5578c4d69560 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f73fbfff700 1 -- 172.21.15.121:0/4248559448 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f7414021070 con 0x5578c4d69560 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f742d5e3b80 10 client.16839 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f742d5e3b80 10 client.16839 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f742d5e3b80 20 client.16839 populate_metadata read hostname 'smithi121' 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5578c4d7ce90 0x5578c4d7f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f742d5e3b80 4 client.16839 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.087+0000 7f742d5e3b80 1 -- 172.21.15.121:0/4248559448 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578c4cc1e10 con 0x5578c4d7ce90 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.088+0000 7f7420afc700 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5578c4d7ce90 0x5578c4d7f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.088+0000 7f73fbfff700 10 client.16839 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:37:53.094 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.088+0000 7f7420afc700 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5578c4d7ce90 0x5578c4d7f2a0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c44081c7bd35992 server_cookie=893bd145c76366e6 in_seq=0 out_seq=0 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f73fbfff700 1 -- 172.21.15.121:0/4248559448 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5578c4cc1e10 con 0x5578c4d7ce90 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f73fbfff700 10 client.16839 handle_command_reply: tid=0 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 1 client.16839 shutdown 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 1 -- 172.21.15.121:0/4248559448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 msgr2=0x5578c4d6cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d6cb70 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f741400b5c0 tx=0x7f7414005bc0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 1 -- 172.21.15.121:0/4248559448 shutdown_connections 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5578c4d7ce90 0x5578c4d7f2a0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d69560 0x5578c4d6cb70 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d633d0 0x5578c4d6f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/4248559448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d68c40 0x5578c4d6c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 1 -- 172.21.15.121:0/4248559448 >> 172.21.15.121:0/4248559448 conn(0x5578c4c8ddc0 msgr2=0x5578c4d61e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f73fbfff700 0 client.16839 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:37:53.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 1 -- 172.21.15.121:0/4248559448 shutdown_connections 2024-06-10T00:37:53.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 1 -- 172.21.15.121:0/4248559448 wait complete. 2024-06-10T00:37:53.096 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.089+0000 7f742d5e3b80 20 client.16839 trim_cache size 0 max 16384 2024-06-10T00:37:53.111 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.109+0000 7f742d5e3b80 1 Processor -- start 2024-06-10T00:37:53.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.109+0000 7f742d5e3b80 1 -- start start 2024-06-10T00:37:53.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.109+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d633d0 0x5578c4d62e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.109+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d68c40 0x5578c4d66d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.109+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d69560 0x5578c4d6c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.109+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f741c1b1090 con 0x5578c4d633d0 2024-06-10T00:37:53.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.109+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f741c1ae860 con 0x5578c4d68c40 2024-06-10T00:37:53.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.109+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f741c1ac6d0 con 0x5578c4d69560 2024-06-10T00:37:53.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.109+0000 7f7420afc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d633d0 0x5578c4d62e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f7420afc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d633d0 0x5578c4d62e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36342/0 (socket says 172.21.15.121:36342) 2024-06-10T00:37:53.112 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f7420afc700 1 -- 172.21.15.121:0/2752118976 learned_addr learned my addr 172.21.15.121:0/2752118976 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:53.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f7420afc700 1 -- 172.21.15.121:0/2752118976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d69560 msgr2=0x5578c4d6c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:53.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f7420afc700 1 --2- 172.21.15.121:0/2752118976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d69560 0x5578c4d6c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f7413fff700 1 --2- 172.21.15.121:0/2752118976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d68c40 0x5578c4d66d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f7420afc700 1 -- 172.21.15.121:0/2752118976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d68c40 msgr2=0x5578c4d66d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:53.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f7420afc700 1 --2- 172.21.15.121:0/2752118976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d68c40 0x5578c4d66d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f7420afc700 1 -- 172.21.15.121:0/2752118976 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578c4d6aba0 con 0x5578c4d633d0 2024-06-10T00:37:53.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f7420afc700 1 --2- 172.21.15.121:0/2752118976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d633d0 0x5578c4d62e80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7418009d90 tx=0x7f7418011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9708d90aafa1c4a6 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:53.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f74027fc700 1 -- 172.21.15.121:0/2752118976 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741802d030 con 0x5578c4d633d0 2024-06-10T00:37:53.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f74027fc700 1 -- 172.21.15.121:0/2752118976 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7418022990 con 0x5578c4d633d0 2024-06-10T00:37:53.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.110+0000 7f74027fc700 1 -- 172.21.15.121:0/2752118976 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7418031980 con 0x5578c4d633d0 2024-06-10T00:37:53.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.111+0000 7f742d5e3b80 1 -- 172.21.15.121:0/2752118976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d633d0 msgr2=0x5578c4d62e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:53.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.111+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/2752118976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d633d0 0x5578c4d62e80 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7418009d90 tx=0x7f7418011000 comp rx=0 tx=0).stop 2024-06-10T00:37:53.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.111+0000 7f742d5e3b80 1 -- 172.21.15.121:0/2752118976 shutdown_connections 2024-06-10T00:37:53.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.111+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/2752118976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d69560 0x5578c4d6c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.111+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/2752118976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d633d0 0x5578c4d62e80 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.111+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/2752118976 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d68c40 0x5578c4d66d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.111+0000 7f742d5e3b80 1 -- 172.21.15.121:0/2752118976 >> 172.21.15.121:0/2752118976 conn(0x5578c4c8ddc0 msgr2=0x5578c4d62770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:53.116 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.111+0000 7f742d5e3b80 1 -- 172.21.15.121:0/2752118976 shutdown_connections 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.111+0000 7f742d5e3b80 1 -- 172.21.15.121:0/2752118976 wait complete. 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.111+0000 7f742d5e3b80 1 -- start start 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.111+0000 7f742d5e3b80 1 Processor -- start 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d633d0 0x5578c4d75a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d68c40 0x5578c4d723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f742d5e3b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d69560 0x5578c4d728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f741c1acd20 con 0x5578c4d68c40 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f741c1ae190 con 0x5578c4d69560 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f742d5e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f741c1ad3f0 con 0x5578c4d633d0 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f742d5e3b80 10 client.0 _ll_register_callbacks cb 0x5578c4c4ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f7413fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d68c40 0x5578c4d723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f7413fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d68c40 0x5578c4d723b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36346/0 (socket says 172.21.15.121:36346) 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f7413fff700 1 -- 172.21.15.121:0/27248213 learned_addr learned my addr 172.21.15.121:0/27248213 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:53.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f7420afc700 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d633d0 0x5578c4d75a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f7413fff700 1 -- 172.21.15.121:0/27248213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d633d0 msgr2=0x5578c4d75a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f7413fff700 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d633d0 0x5578c4d75a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f7413fff700 1 -- 172.21.15.121:0/27248213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d69560 msgr2=0x5578c4d728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f7413fff700 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d69560 0x5578c4d728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f7413fff700 1 -- 172.21.15.121:0/27248213 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578c4d72fa0 con 0x5578c4d68c40 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f7420afc700 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d633d0 0x5578c4d75a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f7413fff700 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d68c40 0x5578c4d723b0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f740c09af30 tx=0x7f740c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c64e8d4586bdb server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f73fbfff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f73fbfff700 1 -- 172.21.15.121:0/27248213 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f740c0b8070 con 0x5578c4d68c40 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f73fbfff700 1 -- 172.21.15.121:0/27248213 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7f740c010610 con 0x5578c4d68c40 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.112+0000 7f742d5e3b80 1 -- 172.21.15.121:0/27248213 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7418022990 con 0x5578c4d68c40 2024-06-10T00:37:53.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f73fbfff700 1 -- 172.21.15.121:0/27248213 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f740c0a3ea0 con 0x5578c4d68c40 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f742d5e3b80 10 client.16845 fetch_fsmap learned FSMap version 33 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f742d5e3b80 1 -- 172.21.15.121:0/27248213 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5578c4de8ae0 con 0x5578c4d68c40 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f73fbfff700 1 -- 172.21.15.121:0/27248213 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7418022990 con 0x5578c4d68c40 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f73fbfff700 1 -- 172.21.15.121:0/27248213 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f740c09f160 con 0x5578c4d68c40 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f742d5e3b80 10 client.16845 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f742d5e3b80 10 client.16845 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f742d5e3b80 20 client.16845 populate_metadata read hostname 'smithi121' 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5578c4de8d60 0x5578c4deb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f742d5e3b80 4 client.16845 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "b5fb53b2", "release": true, "await": true}] 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f742d5e3b80 1 -- 172.21.15.121:0/27248213 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "set_id": "b5fb53b2", "release": true, "await": true}) -- 0x5578c4cbc390 con 0x5578c4de8d60 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.113+0000 7f7420afc700 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5578c4de8d60 0x5578c4deb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.114+0000 7f73fbfff700 10 client.16845 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:37:53.119 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.114+0000 7f7420afc700 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5578c4de8d60 0x5578c4deb150 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83d298ac5f02787a server_cookie=67323cab8992fe2c in_seq=0 out_seq=0 2024-06-10T00:37:53.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:52 smithi193 ceph-mon[36716]: pgmap v306: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 282 KiB/s wr, 63 op/s 2024-06-10T00:37:53.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f73fbfff700 1 -- 172.21.15.121:0/27248213 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5578c4cbc390 con 0x5578c4de8d60 2024-06-10T00:37:53.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f73fbfff700 10 client.16845 handle_command_reply: tid=0 2024-06-10T00:37:53.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 1 client.16845 shutdown 2024-06-10T00:37:53.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 1 -- 172.21.15.121:0/27248213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d68c40 msgr2=0x5578c4d723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:53.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d68c40 0x5578c4d723b0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f740c09af30 tx=0x7f740c004750 comp rx=0 tx=0).stop 2024-06-10T00:37:53.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 1 -- 172.21.15.121:0/27248213 shutdown_connections 2024-06-10T00:37:53.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5578c4de8d60 0x5578c4deb150 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5578c4d69560 0x5578c4d728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578c4d633d0 0x5578c4d75a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f73fbfff700 0 client.16845 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:37:53.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 1 --2- 172.21.15.121:0/27248213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578c4d68c40 0x5578c4d723b0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 1 -- 172.21.15.121:0/27248213 >> 172.21.15.121:0/27248213 conn(0x5578c4c8ddc0 msgr2=0x5578c4d626a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:53.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 1 -- 172.21.15.121:0/27248213 shutdown_connections 2024-06-10T00:37:53.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 1 -- 172.21.15.121:0/27248213 wait complete. 2024-06-10T00:37:53.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.283+0000 7f742d5e3b80 20 client.16845 trim_cache size 0 max 16384 2024-06-10T00:37:53.286 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:37:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 33, 2024-06-10T00:37:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:37:53.286 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 28, 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "b5fb53b2": { 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 28, 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 137.8, 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:37:53.287 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:37:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:37:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:37:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:37:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:37:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:37:53.288 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:37:53.288 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:37:53.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 -- 172.21.15.121:0/70425126 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74040a2d90 msgr2=0x7f74040a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:53.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74040a2d90 0x7f74040a5250 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f741800a400 tx=0x7f741800d040 comp rx=0 tx=0).stop 2024-06-10T00:37:53.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 -- 172.21.15.121:0/70425126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c06b3c0 msgr2=0x7f741c199280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:53.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c06b3c0 0x7f741c199280 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f740c00b530 tx=0x7f740c00b840 comp rx=0 tx=0).stop 2024-06-10T00:37:53.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 -- 172.21.15.121:0/70425126 shutdown_connections 2024-06-10T00:37:53.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f74040a2d90 0x7f74040a5250 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f741c1084f0 0x7f741c19d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f741c06aa00 0x7f741c198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 --2- 172.21.15.121:0/70425126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c06b3c0 0x7f741c199280 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:53.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 -- 172.21.15.121:0/70425126 >> 172.21.15.121:0/70425126 conn(0x7f741c078300 msgr2=0x7f741c100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:53.289 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 -- 172.21.15.121:0/70425126 shutdown_connections 2024-06-10T00:37:53.290 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:53.284+0000 7f7422d60700 1 -- 172.21.15.121:0/70425126 wait complete. 2024-06-10T00:37:54.118 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'b5fb53b2', total seconds elapsed: 24.0 2024-06-10T00:37:54.118 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.8 2024-06-10T00:37:54.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:37:54.421 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:37:54.905 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:54 smithi121 ceph-mon[32378]: pgmap v307: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 52 KiB/s wr, 29 op/s 2024-06-10T00:37:55.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:54 smithi135 ceph-mon[36424]: pgmap v307: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 52 KiB/s wr, 29 op/s 2024-06-10T00:37:55.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:54 smithi193 ceph-mon[36716]: pgmap v307: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 52 KiB/s wr, 29 op/s 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.288+0000 7f532681d700 1 -- 172.21.15.121:0/592006095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5320101a40 msgr2=0x7f5320105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.288+0000 7f532681d700 1 --2- 172.21.15.121:0/592006095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5320101a40 0x7f5320105800 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f5314009990 tx=0x7f5314009ca0 comp rx=0 tx=0).stop 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.288+0000 7f532681d700 1 -- 172.21.15.121:0/592006095 shutdown_connections 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.288+0000 7f532681d700 1 --2- 172.21.15.121:0/592006095 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5320105f30 0x7f532010da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.288+0000 7f532681d700 1 --2- 172.21.15.121:0/592006095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5320101090 0x7f5320101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.288+0000 7f532681d700 1 --2- 172.21.15.121:0/592006095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5320101a40 0x7f5320105800 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.288+0000 7f532681d700 1 -- 172.21.15.121:0/592006095 >> 172.21.15.121:0/592006095 conn(0x7f53200766b0 msgr2=0x7f5320076ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.288+0000 7f532681d700 1 -- 172.21.15.121:0/592006095 shutdown_connections 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.288+0000 7f532681d700 1 -- 172.21.15.121:0/592006095 wait complete. 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.288+0000 7f532681d700 1 Processor -- start 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.288+0000 7f532681d700 1 -- start start 2024-06-10T00:37:56.292 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f532681d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5320101090 0x7f532019e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f532681d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5320101a40 0x7f532019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f532681d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5320105f30 0x7f5320198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f532681d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5320112850 con 0x7f5320101090 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f532681d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f53201126d0 con 0x7f5320105f30 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f532681d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f53201129d0 con 0x7f5320101a40 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5320101090 0x7f532019e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5320101090 0x7f532019e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36374/0 (socket says 172.21.15.121:36374) 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531ffff700 1 -- 172.21.15.121:0/759029276 learned_addr learned my addr 172.21.15.121:0/759029276 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f5324dba700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5320105f30 0x7f5320198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531f7fe700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5320101a40 0x7f532019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531ffff700 1 -- 172.21.15.121:0/759029276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5320101a40 msgr2=0x7f532019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531ffff700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5320101a40 0x7f532019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:56.293 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531ffff700 1 -- 172.21.15.121:0/759029276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5320105f30 msgr2=0x7f5320198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:56.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531ffff700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5320105f30 0x7f5320198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:56.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531ffff700 1 -- 172.21.15.121:0/759029276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53201991b0 con 0x7f5320101090 2024-06-10T00:37:56.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531ffff700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5320101090 0x7f532019e890 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f531000d980 tx=0x7f531000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:56.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531d7fa700 1 -- 172.21.15.121:0/759029276 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5310017070 con 0x7f5320101090 2024-06-10T00:37:56.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f532681d700 1 -- 172.21.15.121:0/759029276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53201994a0 con 0x7f5320101090 2024-06-10T00:37:56.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531d7fa700 1 -- 172.21.15.121:0/759029276 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5310005020 con 0x7f5320101090 2024-06-10T00:37:56.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.289+0000 7f531d7fa700 1 -- 172.21.15.121:0/759029276 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5310009bc0 con 0x7f5320101090 2024-06-10T00:37:56.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.290+0000 7f532681d700 1 -- 172.21.15.121:0/759029276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53201a5760 con 0x7f5320101090 2024-06-10T00:37:56.294 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.290+0000 7f532681d700 1 -- 172.21.15.121:0/759029276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5320069180 con 0x7f5320101090 2024-06-10T00:37:56.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.292+0000 7f531d7fa700 1 -- 172.21.15.121:0/759029276 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f53100051d0 con 0x7f5320101090 2024-06-10T00:37:56.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.292+0000 7f531d7fa700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f53080a2d90 0x7f53080a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:56.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.292+0000 7f531f7fe700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f53080a2d90 0x7f53080a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:56.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.292+0000 7f531d7fa700 1 -- 172.21.15.121:0/759029276 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7f53100cfb70 con 0x7f5320101090 2024-06-10T00:37:56.295 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.293+0000 7f531f7fe700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f53080a2d90 0x7f53080a5250 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5320100a10 tx=0x7f531401a040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:56.297 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.296+0000 7f531d7fa700 1 -- 172.21.15.121:0/759029276 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5310020020 con 0x7f5320101090 2024-06-10T00:37:56.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.637+0000 7f532681d700 1 -- 172.21.15.121:0/759029276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f532002dca0 con 0x7f5320101090 2024-06-10T00:37:56.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.638+0000 7f531d7fa700 1 -- 172.21.15.121:0/759029276 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11895 (secure 0 0 0) 0x7f531009c3a0 con 0x7f5320101090 2024-06-10T00:37:56.639 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:37:56.640 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:35:59.291154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":189,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:37:56.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.642+0000 7f532681d700 1 -- 172.21.15.121:0/759029276 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f53080a2d90 msgr2=0x7f53080a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:56.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.642+0000 7f532681d700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f53080a2d90 0x7f53080a5250 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5320100a10 tx=0x7f531401a040 comp rx=0 tx=0).stop 2024-06-10T00:37:56.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.643+0000 7f532681d700 1 -- 172.21.15.121:0/759029276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5320101090 msgr2=0x7f532019e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:56.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.643+0000 7f532681d700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5320101090 0x7f532019e890 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f531000d980 tx=0x7f531000dc90 comp rx=0 tx=0).stop 2024-06-10T00:37:56.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.644+0000 7f532681d700 1 -- 172.21.15.121:0/759029276 shutdown_connections 2024-06-10T00:37:56.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.644+0000 7f532681d700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f53080a2d90 0x7f53080a5250 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:56.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.644+0000 7f532681d700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5320105f30 0x7f5320198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:56.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.644+0000 7f532681d700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5320101090 0x7f532019e890 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:56.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.644+0000 7f532681d700 1 --2- 172.21.15.121:0/759029276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5320101a40 0x7f532019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:56.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.644+0000 7f532681d700 1 -- 172.21.15.121:0/759029276 >> 172.21.15.121:0/759029276 conn(0x7f53200766b0 msgr2=0x7f53200ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:56.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.645+0000 7f532681d700 1 -- 172.21.15.121:0/759029276 shutdown_connections 2024-06-10T00:37:56.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:56.646+0000 7f532681d700 1 -- 172.21.15.121:0/759029276 wait complete. 2024-06-10T00:37:56.650 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 33 2024-06-10T00:37:56.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:56 smithi121 ceph-mon[32378]: pgmap v308: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 280 KiB/s wr, 19 op/s 2024-06-10T00:37:56.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:56 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/759029276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:37:57.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:56 smithi135 ceph-mon[36424]: pgmap v308: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 280 KiB/s wr, 19 op/s 2024-06-10T00:37:57.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:56 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/759029276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:37:57.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:56 smithi193 ceph-mon[36716]: pgmap v308: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 280 KiB/s wr, 19 op/s 2024-06-10T00:37:57.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:56 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/759029276' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:37:57.453 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0' 2024-06-10T00:37:57.453 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db / --timeout 90 --expiration 137.8 --await-for 45.0 2024-06-10T00:37:57.700 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:37:58.940 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:37:58 smithi121 ceph-mon[32378]: pgmap v309: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 280 KiB/s wr, 13 op/s 2024-06-10T00:37:59.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:37:58 smithi135 ceph-mon[36424]: pgmap v309: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 280 KiB/s wr, 13 op/s 2024-06-10T00:37:59.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:37:58 smithi193 ceph-mon[36716]: pgmap v309: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 280 KiB/s wr, 13 op/s 2024-06-10T00:37:59.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.552+0000 7f76c1c02700 1 -- 172.21.15.121:0/2060586569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76bc1092d0 msgr2=0x7f76bc115770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:59.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.552+0000 7f76c1c02700 1 --2- 172.21.15.121:0/2060586569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76bc1092d0 0x7f76bc115770 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f76b4009f90 tx=0x7f76b400b330 comp rx=0 tx=0).stop 2024-06-10T00:37:59.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.552+0000 7f76c1c02700 1 -- 172.21.15.121:0/2060586569 shutdown_connections 2024-06-10T00:37:59.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.552+0000 7f76c1c02700 1 --2- 172.21.15.121:0/2060586569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76bc1092d0 0x7f76bc115770 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.552+0000 7f76c1c02700 1 --2- 172.21.15.121:0/2060586569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76bc10d7e0 0x7f76bc10dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.552+0000 7f76c1c02700 1 --2- 172.21.15.121:0/2060586569 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76bc108910 0x7f76bc108d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.552+0000 7f76c1c02700 1 -- 172.21.15.121:0/2060586569 >> 172.21.15.121:0/2060586569 conn(0x7f76bc078160 msgr2=0x7f76bc07a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:59.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.552+0000 7f76c1c02700 1 -- 172.21.15.121:0/2060586569 shutdown_connections 2024-06-10T00:37:59.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.552+0000 7f76c1c02700 1 -- 172.21.15.121:0/2060586569 wait complete. 2024-06-10T00:37:59.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.553+0000 7f76c1c02700 1 Processor -- start 2024-06-10T00:37:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.554+0000 7f76c1c02700 1 -- start start 2024-06-10T00:37:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.554+0000 7f76c1c02700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76bc108910 0x7f76bc073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.554+0000 7f76c1c02700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76bc1092d0 0x7f76bc073ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.554+0000 7f76c1c02700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76bc10d7e0 0x7f76bc06cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.554+0000 7f76c1c02700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc118c50 con 0x7f76bc10d7e0 2024-06-10T00:37:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.554+0000 7f76c1c02700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc118ad0 con 0x7f76bc1092d0 2024-06-10T00:37:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.554+0000 7f76c1c02700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc118dd0 con 0x7f76bc108910 2024-06-10T00:37:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.555+0000 7f76c1401700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76bc10d7e0 0x7f76bc06cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.556 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.555+0000 7f76c1401700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76bc10d7e0 0x7f76bc06cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36550/0 (socket says 172.21.15.121:36550) 2024-06-10T00:37:59.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.555+0000 7f76c1401700 1 -- 172.21.15.121:0/243067460 learned_addr learned my addr 172.21.15.121:0/243067460 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:59.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.555+0000 7f76c1401700 1 -- 172.21.15.121:0/243067460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76bc108910 msgr2=0x7f76bc073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:59.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.555+0000 7f76c1401700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76bc108910 0x7f76bc073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.555+0000 7f76c1401700 1 -- 172.21.15.121:0/243067460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76bc1092d0 msgr2=0x7f76bc073ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:59.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.555+0000 7f76c1401700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76bc1092d0 0x7f76bc073ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.555+0000 7f76c1401700 1 -- 172.21.15.121:0/243067460 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76bc06d4c0 con 0x7f76bc10d7e0 2024-06-10T00:37:59.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.556+0000 7f76c1401700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76bc10d7e0 0x7f76bc06cbc0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f76b4009550 tx=0x7f76b4009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:59.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.560+0000 7f76b9ffb700 1 -- 172.21.15.121:0/243067460 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4021070 con 0x7f76bc10d7e0 2024-06-10T00:37:59.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.561+0000 7f76c1c02700 1 -- 172.21.15.121:0/243067460 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76bc06d750 con 0x7f76bc10d7e0 2024-06-10T00:37:59.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.561+0000 7f76c1c02700 1 -- 172.21.15.121:0/243067460 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76bc0717b0 con 0x7f76bc10d7e0 2024-06-10T00:37:59.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.564+0000 7f76b9ffb700 1 -- 172.21.15.121:0/243067460 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76b4007c70 con 0x7f76bc10d7e0 2024-06-10T00:37:59.565 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.565+0000 7f76b9ffb700 1 -- 172.21.15.121:0/243067460 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b4005240 con 0x7f76bc10d7e0 2024-06-10T00:37:59.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.567+0000 7f76cd6e7b80 1 Processor -- start 2024-06-10T00:37:59.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.568+0000 7f76cd6e7b80 1 -- start start 2024-06-10T00:37:59.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.568+0000 7f76b9ffb700 1 -- 172.21.15.121:0/243067460 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f76b4024070 con 0x7f76bc10d7e0 2024-06-10T00:37:59.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.568+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.568+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae5790 0x55f7ccadf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.568+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccadf5c0 0x55f7ccadf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.568+0000 7f76cd6e7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f7cca309a0 con 0x55f7ccae4e70 2024-06-10T00:37:59.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.568+0000 7f76cd6e7b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55f7cc9d88b0 con 0x55f7ccadf5c0 2024-06-10T00:37:59.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.568+0000 7f76cd6e7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f7cca2c960 con 0x55f7ccae5790 2024-06-10T00:37:59.569 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.569+0000 7f76b9ffb700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f76ac0a3010 0x7f76ac0a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76bbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76bbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae5250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36556/0 (socket says 172.21.15.121:36556) 2024-06-10T00:37:59.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76bbfff700 1 -- 172.21.15.121:0/1352247766 learned_addr learned my addr 172.21.15.121:0/1352247766 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:59.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76bbfff700 1 --2- 172.21.15.121:0/1352247766 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccadf5c0 0x55f7ccadf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76b9ffb700 1 -- 172.21.15.121:0/243067460 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7f76b40d0820 con 0x7f76bc10d7e0 2024-06-10T00:37:59.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76c1401700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f76ac0a3010 0x7f76ac0a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76c0c00700 1 --2- 172.21.15.121:0/1352247766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae5790 0x55f7ccadf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76bbfff700 1 -- 172.21.15.121:0/1352247766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae5790 msgr2=0x55f7ccadf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:59.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76bbfff700 1 --2- 172.21.15.121:0/1352247766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae5790 0x55f7ccadf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76bbfff700 1 -- 172.21.15.121:0/1352247766 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccadf5c0 msgr2=0x55f7ccadf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:59.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76bbfff700 1 --2- 172.21.15.121:0/1352247766 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccadf5c0 0x55f7ccadf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76bbfff700 1 -- 172.21.15.121:0/1352247766 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7cc9cf0e0 con 0x55f7ccae4e70 2024-06-10T00:37:59.571 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.570+0000 7f76c0c00700 1 --2- 172.21.15.121:0/1352247766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae5790 0x55f7ccadf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:37:59.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.571+0000 7f76bbfff700 1 --2- 172.21.15.121:0/1352247766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae5250 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f76a8014880 tx=0x7f76a800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade340ee0ded4e5a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:59.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.571+0000 7f76a6ffd700 1 -- 172.21.15.121:0/1352247766 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a8004a30 con 0x55f7ccae4e70 2024-06-10T00:37:59.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.572+0000 7f76a6ffd700 1 -- 172.21.15.121:0/1352247766 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76a8005500 con 0x55f7ccae4e70 2024-06-10T00:37:59.572 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.572+0000 7f76a6ffd700 1 -- 172.21.15.121:0/1352247766 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a800f900 con 0x55f7ccae4e70 2024-06-10T00:37:59.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.573+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1352247766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 msgr2=0x55f7ccae5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:59.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.573+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/1352247766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae5250 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f76a8014880 tx=0x7f76a800b390 comp rx=0 tx=0).stop 2024-06-10T00:37:59.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.573+0000 7f76c1401700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f76ac0a3010 0x7f76ac0a54d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f76b400b6c0 tx=0x7f76b4009920 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:59.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.573+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1352247766 shutdown_connections 2024-06-10T00:37:59.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.573+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/1352247766 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccadf5c0 0x55f7ccadf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.573+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/1352247766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae5250 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.573+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/1352247766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae5790 0x55f7ccadf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.573+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1352247766 >> 172.21.15.121:0/1352247766 conn(0x55f7cca09e50 msgr2=0x55f7ccada370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:59.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.573+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1352247766 shutdown_connections 2024-06-10T00:37:59.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.573+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1352247766 wait complete. 2024-06-10T00:37:59.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.574+0000 7f76cd6e7b80 1 -- start start 2024-06-10T00:37:59.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.574+0000 7f76cd6e7b80 1 Processor -- start 2024-06-10T00:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.575+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccadf5c0 0x55f7ccaeb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.575+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.575+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccae8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.575+0000 7f76cd6e7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f7ccaec2a0 con 0x55f7ccae4e70 2024-06-10T00:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.575+0000 7f76cd6e7b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55f7cc9d0d80 con 0x55f7ccae5790 2024-06-10T00:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.575+0000 7f76cd6e7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f7ccaec420 con 0x55f7ccadf5c0 2024-06-10T00:37:59.576 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.575+0000 7f76cd6e7b80 10 client.0 _ll_register_callbacks cb 0x55f7cc9caab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.576+0000 7f76c0c00700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.576+0000 7f76c0c00700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:36572/0 (socket says 172.21.15.121:36572) 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.576+0000 7f76c0c00700 1 -- 172.21.15.121:0/2395509416 learned_addr learned my addr 172.21.15.121:0/2395509416 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.576+0000 7f76bbfff700 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccae8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.576+0000 7f76bbfff700 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccadf5c0 0x55f7ccaeb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.577+0000 7f76c0c00700 1 -- 172.21.15.121:0/2395509416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccadf5c0 msgr2=0x55f7ccaeb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.577+0000 7f76c0c00700 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccadf5c0 0x55f7ccaeb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.577+0000 7f76c0c00700 1 -- 172.21.15.121:0/2395509416 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 msgr2=0x55f7ccae8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.577+0000 7f76c0c00700 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccae8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.577+0000 7f76c0c00700 1 -- 172.21.15.121:0/2395509416 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7cc97fa60 con 0x55f7ccae4e70 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.578+0000 7f76c0c00700 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae87d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f76b00105f0 tx=0x7f76b0010900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db70a532931c4b53 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:59.578 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.578+0000 7f769bfff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:37:59.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.578+0000 7f769bfff700 1 -- 172.21.15.121:0/2395509416 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0012860 con 0x55f7ccae4e70 2024-06-10T00:37:59.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.578+0000 7f769bfff700 1 -- 172.21.15.121:0/2395509416 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7f76b0012a00 con 0x55f7ccae4e70 2024-06-10T00:37:59.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.578+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2395509416 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76a8005500 con 0x55f7ccae4e70 2024-06-10T00:37:59.579 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.579+0000 7f769bfff700 1 -- 172.21.15.121:0/2395509416 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f76b0024050 con 0x55f7ccae4e70 2024-06-10T00:37:59.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.579+0000 7f76cd6e7b80 10 client.16863 fetch_fsmap learned FSMap version 33 2024-06-10T00:37:59.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.579+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2395509416 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f7ccaf6b90 con 0x55f7ccae4e70 2024-06-10T00:37:59.580 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.580+0000 7f769bfff700 1 -- 172.21.15.121:0/2395509416 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76a8005500 con 0x55f7ccae4e70 2024-06-10T00:37:59.581 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.581+0000 7f769bfff700 1 -- 172.21.15.121:0/2395509416 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f76b000f040 con 0x55f7ccae4e70 2024-06-10T00:37:59.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.582+0000 7f76cd6e7b80 10 client.16863 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:37:59.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.582+0000 7f76cd6e7b80 10 client.16863 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:37:59.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.582+0000 7f76cd6e7b80 20 client.16863 populate_metadata read hostname 'smithi121' 2024-06-10T00:37:59.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.582+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f7ccaf9630 0x55f7ccafba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.582+0000 7f76cd6e7b80 4 client.16863 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:37:59.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.582+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2395509416 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7cca3dea0 con 0x55f7ccaf9630 2024-06-10T00:37:59.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.582+0000 7f76bbfff700 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f7ccaf9630 0x55f7ccafba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.583+0000 7f76bbfff700 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f7ccaf9630 0x55f7ccafba70 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4297d720f2fb1bcc server_cookie=e8b61a26f409451d in_seq=0 out_seq=0 2024-06-10T00:37:59.583 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.583+0000 7f769bfff700 10 client.16863 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:37:59.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.585+0000 7f769bfff700 1 -- 172.21.15.121:0/2395509416 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55f7cca3dea0 con 0x55f7ccaf9630 2024-06-10T00:37:59.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.585+0000 7f769bfff700 10 client.16863 handle_command_reply: tid=0 2024-06-10T00:37:59.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.585+0000 7f76cd6e7b80 1 client.16863 shutdown 2024-06-10T00:37:59.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.587+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2395509416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 msgr2=0x55f7ccae87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:59.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.587+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae87d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f76b00105f0 tx=0x7f76b0010900 comp rx=0 tx=0).stop 2024-06-10T00:37:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.587+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2395509416 shutdown_connections 2024-06-10T00:37:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.587+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f7ccaf9630 0x55f7ccafba70 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.587+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccae8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.587+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccadf5c0 0x55f7ccaeb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.587+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2395509416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccae87d0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.587+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2395509416 >> 172.21.15.121:0/2395509416 conn(0x55f7cca09e50 msgr2=0x55f7ccade070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:59.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.587+0000 7f769bfff700 0 client.16863 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:37:59.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.590+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2395509416 shutdown_connections 2024-06-10T00:37:59.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.590+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2395509416 wait complete. 2024-06-10T00:37:59.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.590+0000 7f76cd6e7b80 20 client.16863 trim_cache size 0 max 16384 2024-06-10T00:37:59.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.634+0000 7f76cd6e7b80 1 Processor -- start 2024-06-10T00:37:59.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.634+0000 7f76cd6e7b80 1 -- start start 2024-06-10T00:37:59.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.635+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccadf5c0 0x55f7ccadf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.635+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae4e70 0x55f7ccada410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.635+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccb01340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.635+0000 7f76cd6e7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f76a0011780 con 0x55f7ccadf5c0 2024-06-10T00:37:59.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.635+0000 7f76c0c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae4e70 0x55f7ccada410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.635+0000 7f76c0c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae4e70 0x55f7ccada410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:43102/0 (socket says 172.21.15.121:43102) 2024-06-10T00:37:59.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.635+0000 7f76c0c00700 1 -- 172.21.15.121:0/1259939921 learned_addr learned my addr 172.21.15.121:0/1259939921 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:59.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.635+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1259939921 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f76a000ef50 con 0x55f7ccae5790 2024-06-10T00:37:59.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.635+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1259939921 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76a000ce50 con 0x55f7ccae4e70 2024-06-10T00:37:59.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.636+0000 7f76bbfff700 1 --2- 172.21.15.121:0/1259939921 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccb01340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.636+0000 7f76bbfff700 1 --2- 172.21.15.121:0/1259939921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccadf5c0 0x55f7ccadf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.636+0000 7f76bbfff700 1 -- 172.21.15.121:0/1259939921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae4e70 msgr2=0x55f7ccada410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:59.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.636+0000 7f76bbfff700 1 --2- 172.21.15.121:0/1259939921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae4e70 0x55f7ccada410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.636+0000 7f76bbfff700 1 -- 172.21.15.121:0/1259939921 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 msgr2=0x55f7ccb01340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:59.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.636+0000 7f76bbfff700 1 --2- 172.21.15.121:0/1259939921 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccb01340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.636+0000 7f76bbfff700 1 -- 172.21.15.121:0/1259939921 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7ccae7790 con 0x55f7ccadf5c0 2024-06-10T00:37:59.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.637+0000 7f76bbfff700 1 --2- 172.21.15.121:0/1259939921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccadf5c0 0x55f7ccadf080 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x55f7ccafc2b0 tx=0x7f76a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73373bf111a56517 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:59.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.638+0000 7f76a67fc700 1 -- 172.21.15.121:0/1259939921 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a800fe80 con 0x55f7ccadf5c0 2024-06-10T00:37:59.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.638+0000 7f76a67fc700 1 -- 172.21.15.121:0/1259939921 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76a8014070 con 0x55f7ccadf5c0 2024-06-10T00:37:59.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.638+0000 7f76a67fc700 1 -- 172.21.15.121:0/1259939921 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76a8004f50 con 0x55f7ccadf5c0 2024-06-10T00:37:59.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.640+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1259939921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccadf5c0 msgr2=0x55f7ccadf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:37:59.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.640+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/1259939921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccadf5c0 0x55f7ccadf080 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x55f7ccafc2b0 tx=0x7f76a8004750 comp rx=0 tx=0).stop 2024-06-10T00:37:59.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.640+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1259939921 shutdown_connections 2024-06-10T00:37:59.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.641+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/1259939921 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccb01340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.641+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/1259939921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccadf5c0 0x55f7ccadf080 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.641+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/1259939921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccae4e70 0x55f7ccada410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.641+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1259939921 >> 172.21.15.121:0/1259939921 conn(0x55f7cca09e50 msgr2=0x55f7ccad9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:37:59.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.641+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1259939921 shutdown_connections 2024-06-10T00:37:59.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.641+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/1259939921 wait complete. 2024-06-10T00:37:59.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.642+0000 7f76cd6e7b80 1 -- start start 2024-06-10T00:37:59.642 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.642+0000 7f76cd6e7b80 1 Processor -- start 2024-06-10T00:37:59.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.642+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccadf5c0 0x55f7ccaf1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.642+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccb00c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76cd6e7b80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccb01150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.643 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76cd6e7b80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f76a000d410 con 0x55f7ccae4e70 2024-06-10T00:37:59.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76cd6e7b80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f76a000e880 con 0x55f7ccae5790 2024-06-10T00:37:59.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76cd6e7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f76a000dae0 con 0x55f7ccadf5c0 2024-06-10T00:37:59.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76cd6e7b80 10 client.0 _ll_register_callbacks cb 0x55f7cc9caab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:37:59.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76bbfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccb01150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76bbfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccadf5c0 0x55f7ccaf1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76bbfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccadf5c0 0x55f7ccaf1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:43118/0 (socket says 172.21.15.121:43118) 2024-06-10T00:37:59.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76bbfff700 1 -- 172.21.15.121:0/2428203835 learned_addr learned my addr 172.21.15.121:0/2428203835 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:37:59.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76c0c00700 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccb00c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76c0c00700 1 -- 172.21.15.121:0/2428203835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccadf5c0 msgr2=0x55f7ccaf1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:59.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.643+0000 7f76c0c00700 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccadf5c0 0x55f7ccaf1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.644+0000 7f76c0c00700 1 -- 172.21.15.121:0/2428203835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 msgr2=0x55f7ccb01150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:37:59.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.644+0000 7f76c0c00700 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccb01150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:37:59.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.644+0000 7f76c0c00700 1 -- 172.21.15.121:0/2428203835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7ccb01890 con 0x55f7ccae4e70 2024-06-10T00:37:59.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.644+0000 7f76c0c00700 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccb00c10 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f76b0006010 tx=0x7f76b0011850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96091d8608c0802d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:37:59.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.644+0000 7f769bfff700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:37:59.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.649+0000 7f769bfff700 1 -- 172.21.15.121:0/2428203835 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0021040 con 0x55f7ccae4e70 2024-06-10T00:37:59.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.649+0000 7f769bfff700 1 -- 172.21.15.121:0/2428203835 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9525+0+0 (secure 0 0 0) 0x7f76b0011ac0 con 0x55f7ccae4e70 2024-06-10T00:37:59.649 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.649+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2428203835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76a8014070 con 0x55f7ccae4e70 2024-06-10T00:37:59.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.649+0000 7f769bfff700 1 -- 172.21.15.121:0/2428203835 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f76b00097b0 con 0x55f7ccae4e70 2024-06-10T00:37:59.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.650+0000 7f76cd6e7b80 10 client.16869 fetch_fsmap learned FSMap version 33 2024-06-10T00:37:59.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.650+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2428203835 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f7ccb64240 con 0x55f7ccae4e70 2024-06-10T00:37:59.650 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.650+0000 7f769bfff700 1 -- 172.21.15.121:0/2428203835 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76a8014070 con 0x55f7ccae4e70 2024-06-10T00:37:59.651 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.651+0000 7f769bfff700 1 -- 172.21.15.121:0/2428203835 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f76b0003680 con 0x55f7ccae4e70 2024-06-10T00:37:59.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.651+0000 7f76cd6e7b80 10 client.16869 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:37:59.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.651+0000 7f76cd6e7b80 10 client.16869 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:37:59.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.651+0000 7f76cd6e7b80 20 client.16869 populate_metadata read hostname 'smithi121' 2024-06-10T00:37:59.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.651+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f7ccb64a50 0x55f7ccb66e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:37:59.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.651+0000 7f76cd6e7b80 4 client.16869 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}] 2024-06-10T00:37:59.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.651+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2428203835 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.8, "await_for": 45.0}) -- 0x55f7cca38420 con 0x55f7ccb64a50 2024-06-10T00:37:59.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.651+0000 7f76bbfff700 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f7ccb64a50 0x55f7ccb66e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:37:59.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.652+0000 7f76bbfff700 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f7ccb64a50 0x55f7ccb66e70 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2300a85ef459efaa server_cookie=b19817eeaa38b9cc in_seq=0 out_seq=0 2024-06-10T00:37:59.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:37:59.652+0000 7f769bfff700 10 client.16869 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:38:00.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f769bfff700 1 -- 172.21.15.121:0/2428203835 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f7cca38420 con 0x55f7ccb64a50 2024-06-10T00:38:00.113 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f769bfff700 10 client.16869 handle_command_reply: tid=0 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 1 client.16869 shutdown 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2428203835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 msgr2=0x55f7ccb00c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccb00c10 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f76b0006010 tx=0x7f76b0011850 comp rx=0 tx=0).stop 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2428203835 shutdown_connections 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55f7ccb64a50 0x55f7ccb66e70 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55f7ccae5790 0x55f7ccb01150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7ccadf5c0 0x55f7ccaf1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f769bfff700 0 client.16869 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 1 --2- 172.21.15.121:0/2428203835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f7ccae4e70 0x55f7ccb00c10 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2428203835 >> 172.21.15.121:0/2428203835 conn(0x55f7cca09e50 msgr2=0x55f7ccad34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2428203835 shutdown_connections 2024-06-10T00:38:00.114 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 1 -- 172.21.15.121:0/2428203835 wait complete. 2024-06-10T00:38:00.115 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.113+0000 7f76cd6e7b80 20 client.16869 trim_cache size 0 max 16384 2024-06-10T00:38:00.115 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:38:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 33, 2024-06-10T00:38:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:38:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 30, 2024-06-10T00:38:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:38:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: "6073a0a1": { 2024-06-10T00:38:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 30, 2024-06-10T00:38:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:38:00.115 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 137.8, 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "QUIESCED", 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:00.116 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:00.117 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:00.117 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:38:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 -- 172.21.15.121:0/243067460 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f76ac0a3010 msgr2=0x7f76ac0a54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f76ac0a3010 0x7f76ac0a54d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f76b400b6c0 tx=0x7f76b4009920 comp rx=0 tx=0).stop 2024-06-10T00:38:00.117 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 -- 172.21.15.121:0/243067460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76bc10d7e0 msgr2=0x7f76bc06cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76bc10d7e0 0x7f76bc06cbc0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f76b4009550 tx=0x7f76b4009390 comp rx=0 tx=0).stop 2024-06-10T00:38:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 -- 172.21.15.121:0/243067460 shutdown_connections 2024-06-10T00:38:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f76ac0a3010 0x7f76ac0a54d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76bc10d7e0 0x7f76bc06cbc0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76bc108910 0x7f76bc073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 --2- 172.21.15.121:0/243067460 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f76bc1092d0 0x7f76bc073ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 -- 172.21.15.121:0/243067460 >> 172.21.15.121:0/243067460 conn(0x7f76bc078160 msgr2=0x7f76bc0785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 -- 172.21.15.121:0/243067460 shutdown_connections 2024-06-10T00:38:00.118 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:00.114+0000 7f76c1c02700 1 -- 172.21.15.121:0/243067460 wait complete. 2024-06-10T00:38:00.803 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '6073a0a1', quiesce took 6.7 seconds. Will release after: 11.100000000000001 2024-06-10T00:38:01.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:00 smithi135 ceph-mon[36424]: pgmap v310: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 651 KiB/s wr, 74 op/s 2024-06-10T00:38:01.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:00 smithi193 ceph-mon[36716]: pgmap v310: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 651 KiB/s wr, 74 op/s 2024-06-10T00:38:01.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:00 smithi121 ceph-mon[32378]: pgmap v310: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 651 KiB/s wr, 74 op/s 2024-06-10T00:38:03.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:02 smithi135 ceph-mon[36424]: pgmap v311: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-06-10T00:38:03.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:02 smithi135 ceph-mon[36424]: osdmap e108: 12 total, 12 up, 12 in 2024-06-10T00:38:03.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:02 smithi121 ceph-mon[32378]: pgmap v311: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-06-10T00:38:03.011 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:02 smithi121 ceph-mon[32378]: osdmap e108: 12 total, 12 up, 12 in 2024-06-10T00:38:03.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:02 smithi193 ceph-mon[36716]: pgmap v311: 129 pgs: 129 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 118 op/s 2024-06-10T00:38:03.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:02 smithi193 ceph-mon[36716]: osdmap e108: 12 total, 12 up, 12 in 2024-06-10T00:38:05.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:04 smithi193 ceph-mon[36716]: pgmap v313: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-06-10T00:38:05.201 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:04 smithi121 ceph-mon[32378]: pgmap v313: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-06-10T00:38:05.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:04 smithi135 ceph-mon[36424]: pgmap v313: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 170 op/s 2024-06-10T00:38:07.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:06 smithi193 ceph-mon[36716]: pgmap v314: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 206 op/s 2024-06-10T00:38:07.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:06 smithi121 ceph-mon[32378]: pgmap v314: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 206 op/s 2024-06-10T00:38:07.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:06 smithi135 ceph-mon[36424]: pgmap v314: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 206 op/s 2024-06-10T00:38:09.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:08 smithi193 ceph-mon[36716]: pgmap v315: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 206 op/s 2024-06-10T00:38:09.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:08 smithi121 ceph-mon[32378]: pgmap v315: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 206 op/s 2024-06-10T00:38:09.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:08 smithi135 ceph-mon[36424]: pgmap v315: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 206 op/s 2024-06-10T00:38:11.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:10 smithi193 ceph-mon[36716]: pgmap v316: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 967 KiB/s wr, 136 op/s 2024-06-10T00:38:11.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:10 smithi121 ceph-mon[32378]: pgmap v316: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 967 KiB/s wr, 136 op/s 2024-06-10T00:38:11.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:10 smithi135 ceph-mon[36424]: pgmap v316: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 967 KiB/s wr, 136 op/s 2024-06-10T00:38:11.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:38:12.410 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:38:12.950 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:12 smithi121 ceph-mon[32378]: pgmap v317: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 103 KiB/s wr, 82 op/s 2024-06-10T00:38:13.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:12 smithi193 ceph-mon[36716]: pgmap v317: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 103 KiB/s wr, 82 op/s 2024-06-10T00:38:13.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:12 smithi135 ceph-mon[36424]: pgmap v317: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 103 KiB/s wr, 82 op/s 2024-06-10T00:38:15.098 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:14 smithi121 ceph-mon[32378]: pgmap v318: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 87 KiB/s wr, 70 op/s 2024-06-10T00:38:15.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:14 smithi193 ceph-mon[36716]: pgmap v318: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 87 KiB/s wr, 70 op/s 2024-06-10T00:38:15.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:14 smithi135 ceph-mon[36424]: pgmap v318: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 87 KiB/s wr, 70 op/s 2024-06-10T00:38:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.479+0000 7f8ed0bb2700 1 -- 172.21.15.121:0/3242216252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ecc109db0 msgr2=0x7f8ecc1150a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.479+0000 7f8ed0bb2700 1 --2- 172.21.15.121:0/3242216252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ecc109db0 0x7f8ecc1150a0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc009990 tx=0x7f8ebc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:38:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.480+0000 7f8ed0bb2700 1 -- 172.21.15.121:0/3242216252 shutdown_connections 2024-06-10T00:38:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.480+0000 7f8ed0bb2700 1 --2- 172.21.15.121:0/3242216252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ecc1155e0 0x7f8ecc1179d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.480+0000 7f8ed0bb2700 1 --2- 172.21.15.121:0/3242216252 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ecc109490 0x7f8ecc109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.480+0000 7f8ed0bb2700 1 --2- 172.21.15.121:0/3242216252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ecc109db0 0x7f8ecc1150a0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:16.770 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.480+0000 7f8ed0bb2700 1 -- 172.21.15.121:0/3242216252 >> 172.21.15.121:0/3242216252 conn(0x7f8ecc06d600 msgr2=0x7f8ecc06da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.482+0000 7f8ed0bb2700 1 -- 172.21.15.121:0/3242216252 shutdown_connections 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.482+0000 7f8ed0bb2700 1 -- 172.21.15.121:0/3242216252 wait complete. 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.483+0000 7f8ed0bb2700 1 Processor -- start 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.485+0000 7f8ed0bb2700 1 -- start start 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.485+0000 7f8ed0bb2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ecc109490 0x7f8ecc114ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.485+0000 7f8ed0bb2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ecc109db0 0x7f8ecc111f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.485+0000 7f8eca59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ecc109490 0x7f8ecc114ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.486+0000 7f8eca59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ecc109490 0x7f8ecc114ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:35776/0 (socket says 172.21.15.121:35776) 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.486+0000 7f8eca59c700 1 -- 172.21.15.121:0/2496191664 learned_addr learned my addr 172.21.15.121:0/2496191664 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.486+0000 7f8ed0bb2700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ecc1155e0 0x7f8ecc112490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.486+0000 7f8ed0bb2700 1 -- 172.21.15.121:0/2496191664 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8ecc11d330 con 0x7f8ecc109db0 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.486+0000 7f8ec9d9b700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ecc109db0 0x7f8ecc111f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:16.771 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.486+0000 7f8ecad9d700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ecc1155e0 0x7f8ecc112490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.486+0000 7f8ed0bb2700 1 -- 172.21.15.121:0/2496191664 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f8ecc11d1b0 con 0x7f8ecc109490 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.486+0000 7f8ed0bb2700 1 -- 172.21.15.121:0/2496191664 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8ecc11d4b0 con 0x7f8ecc1155e0 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.487+0000 7f8eca59c700 1 -- 172.21.15.121:0/2496191664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ecc1155e0 msgr2=0x7f8ecc112490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.487+0000 7f8eca59c700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ecc1155e0 0x7f8ecc112490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.487+0000 7f8eca59c700 1 -- 172.21.15.121:0/2496191664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ecc109db0 msgr2=0x7f8ecc111f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.487+0000 7f8eca59c700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ecc109db0 0x7f8ecc111f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.487+0000 7f8eca59c700 1 -- 172.21.15.121:0/2496191664 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ecc112d70 con 0x7f8ecc109490 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.487+0000 7f8ec9d9b700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ecc109db0 0x7f8ecc111f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.487+0000 7f8eca59c700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ecc109490 0x7f8ecc114ea0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400ef10 tx=0x7f8ec400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.487+0000 7f8ebb7fe700 1 -- 172.21.15.121:0/2496191664 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec400cad0 con 0x7f8ecc109490 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.488+0000 7f8ed0bb2700 1 -- 172.21.15.121:0/2496191664 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ecc1b4130 con 0x7f8ecc109490 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.488+0000 7f8ed0bb2700 1 -- 172.21.15.121:0/2496191664 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ecc1b4600 con 0x7f8ecc109490 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.488+0000 7f8ebb7fe700 1 -- 172.21.15.121:0/2496191664 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ec4019070 con 0x7f8ecc109490 2024-06-10T00:38:16.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.489+0000 7f8ebb7fe700 1 -- 172.21.15.121:0/2496191664 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec40146e0 con 0x7f8ecc109490 2024-06-10T00:38:16.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.489+0000 7f8eb97fa700 1 -- 172.21.15.121:0/2496191664 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ecc06eb90 con 0x7f8ecc109490 2024-06-10T00:38:16.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.491+0000 7f8ebb7fe700 1 -- 172.21.15.121:0/2496191664 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f8ec4012050 con 0x7f8ecc109490 2024-06-10T00:38:16.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.492+0000 7f8ebb7fe700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8eb40a2d30 0x7f8eb40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:16.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.493+0000 7f8ebb7fe700 1 -- 172.21.15.121:0/2496191664 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9613+0+0 (secure 0 0 0) 0x7f8ec40cfad0 con 0x7f8ecc109490 2024-06-10T00:38:16.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.493+0000 7f8ec9d9b700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8eb40a2d30 0x7f8eb40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:16.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.494+0000 7f8ec9d9b700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8eb40a2d30 0x7f8eb40a51f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00b2e0 tx=0x7f8ebc01a040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:16.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.496+0000 7f8ebb7fe700 1 -- 172.21.15.121:0/2496191664 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8ec4010040 con 0x7f8ecc109490 2024-06-10T00:38:16.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.955+0000 7f8eb97fa700 1 -- 172.21.15.121:0/2496191664 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ecc113480 con 0x7f8ecc109490 2024-06-10T00:38:16.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.957+0000 7f8ebb7fe700 1 -- 172.21.15.121:0/2496191664 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11895 (secure 0 0 0) 0x7f8ec409c250 con 0x7f8ecc109490 2024-06-10T00:38:16.957 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:38:16.957 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:35:59.291154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":189,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:38:16.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 -- 172.21.15.121:0/2496191664 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8eb40a2d30 msgr2=0x7f8eb40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:16.963 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8eb40a2d30 0x7f8eb40a51f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00b2e0 tx=0x7f8ebc01a040 comp rx=0 tx=0).stop 2024-06-10T00:38:16.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 -- 172.21.15.121:0/2496191664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ecc109490 msgr2=0x7f8ecc114ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:16.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ecc109490 0x7f8ecc114ea0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400ef10 tx=0x7f8ec400c3a0 comp rx=0 tx=0).stop 2024-06-10T00:38:16.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 -- 172.21.15.121:0/2496191664 shutdown_connections 2024-06-10T00:38:16.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f8eb40a2d30 0x7f8eb40a51f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:16.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ecc1155e0 0x7f8ecc112490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:16.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ecc109490 0x7f8ecc114ea0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:16.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 --2- 172.21.15.121:0/2496191664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ecc109db0 0x7f8ecc111f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:16.964 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 -- 172.21.15.121:0/2496191664 >> 172.21.15.121:0/2496191664 conn(0x7f8ecc06d600 msgr2=0x7f8ecc071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:16.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 -- 172.21.15.121:0/2496191664 shutdown_connections 2024-06-10T00:38:16.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:16.963+0000 7f8eb97fa700 1 -- 172.21.15.121:0/2496191664 wait complete. 2024-06-10T00:38:16.965 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 33 2024-06-10T00:38:17.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:16 smithi193 ceph-mon[36716]: pgmap v319: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 71 KiB/s wr, 45 op/s 2024-06-10T00:38:17.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:16 smithi121 ceph-mon[32378]: pgmap v319: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 71 KiB/s wr, 45 op/s 2024-06-10T00:38:17.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:16 smithi135 ceph-mon[36424]: pgmap v319: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 71 KiB/s wr, 45 op/s 2024-06-10T00:38:17.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:17 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2496191664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:38:18.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:17 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2496191664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:38:18.195 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db --set-id 6073a0a1 --release --await' 2024-06-10T00:38:18.195 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db --set-id 6073a0a1 --release --await 2024-06-10T00:38:18.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:17 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2496191664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:38:18.642 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:38:19.067 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:18 smithi121 ceph-mon[32378]: pgmap v320: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-06-10T00:38:19.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:18 smithi193 ceph-mon[36716]: pgmap v320: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-06-10T00:38:19.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:18 smithi135 ceph-mon[36424]: pgmap v320: 129 pgs: 16 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 68 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-06-10T00:38:21.090 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:20 smithi121 ceph-mon[32378]: pgmap v321: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.2 KiB/s wr, 10 op/s 2024-06-10T00:38:21.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:20 smithi193 ceph-mon[36716]: pgmap v321: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.2 KiB/s wr, 10 op/s 2024-06-10T00:38:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:20 smithi135 ceph-mon[36424]: pgmap v321: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.2 KiB/s wr, 10 op/s 2024-06-10T00:38:21.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.531+0000 7f3d962b7700 1 -- 172.21.15.121:0/328880874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d90105a80 msgr2=0x7f3d90105e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:21.532 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.531+0000 7f3d962b7700 1 --2- 172.21.15.121:0/328880874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d90105a80 0x7f3d90105e60 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f3d8001a9a0 tx=0x7f3d8001acb0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.531+0000 7f3d962b7700 1 -- 172.21.15.121:0/328880874 shutdown_connections 2024-06-10T00:38:21.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.531+0000 7f3d962b7700 1 --2- 172.21.15.121:0/328880874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d901016e0 0x7f3d9010d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.531+0000 7f3d962b7700 1 --2- 172.21.15.121:0/328880874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d90105a80 0x7f3d90105e60 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.531+0000 7f3d962b7700 1 --2- 172.21.15.121:0/328880874 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d90100d20 0x7f3d901011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.531+0000 7f3d962b7700 1 -- 172.21.15.121:0/328880874 >> 172.21.15.121:0/328880874 conn(0x7f3d900765f0 msgr2=0x7f3d90076a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:21.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.531+0000 7f3d962b7700 1 -- 172.21.15.121:0/328880874 shutdown_connections 2024-06-10T00:38:21.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.531+0000 7f3d962b7700 1 -- 172.21.15.121:0/328880874 wait complete. 2024-06-10T00:38:21.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.533+0000 7f3d962b7700 1 Processor -- start 2024-06-10T00:38:21.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.533+0000 7f3d962b7700 1 -- start start 2024-06-10T00:38:21.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.533+0000 7f3d962b7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d90100d20 0x7f3d90107960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.533+0000 7f3d962b7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d901016e0 0x7f3d90107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.533+0000 7f3d962b7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d90105a80 0x7f3d901083e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.533+0000 7f3d962b7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9006e790 con 0x7f3d901016e0 2024-06-10T00:38:21.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.533+0000 7f3d962b7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9006e610 con 0x7f3d90105a80 2024-06-10T00:38:21.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.533+0000 7f3d962b7700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9006e910 con 0x7f3d90100d20 2024-06-10T00:38:21.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.534+0000 7f3d94854700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d90105a80 0x7f3d901083e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.534+0000 7f3d8f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d901016e0 0x7f3d90107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.534+0000 7f3d8f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d901016e0 0x7f3d90107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33890/0 (socket says 172.21.15.121:33890) 2024-06-10T00:38:21.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.534+0000 7f3d94854700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d90105a80 0x7f3d901083e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:49376/0 (socket says 172.21.15.121:49376) 2024-06-10T00:38:21.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.534+0000 7f3d8f7fe700 1 -- 172.21.15.121:0/31876239 learned_addr learned my addr 172.21.15.121:0/31876239 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:21.535 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.534+0000 7f3d8ffff700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d90100d20 0x7f3d90107960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.534+0000 7f3d8f7fe700 1 -- 172.21.15.121:0/31876239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d90100d20 msgr2=0x7f3d90107960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:21.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.534+0000 7f3d8f7fe700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d90100d20 0x7f3d90107960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.534+0000 7f3d8f7fe700 1 -- 172.21.15.121:0/31876239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d90105a80 msgr2=0x7f3d901083e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:21.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.534+0000 7f3d8f7fe700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d90105a80 0x7f3d901083e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.534+0000 7f3d8f7fe700 1 -- 172.21.15.121:0/31876239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d901b53d0 con 0x7f3d901016e0 2024-06-10T00:38:21.536 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.536+0000 7f3d8f7fe700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d901016e0 0x7f3d90107ea0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800d950 tx=0x7f3d7800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:21.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.539+0000 7f3d8d7fa700 1 -- 172.21.15.121:0/31876239 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78016070 con 0x7f3d901016e0 2024-06-10T00:38:21.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.539+0000 7f3d962b7700 1 -- 172.21.15.121:0/31876239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d901b56c0 con 0x7f3d901016e0 2024-06-10T00:38:21.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.539+0000 7f3d962b7700 1 -- 172.21.15.121:0/31876239 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d901b5c00 con 0x7f3d901016e0 2024-06-10T00:38:21.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.539+0000 7f3d8d7fa700 1 -- 172.21.15.121:0/31876239 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d78009e20 con 0x7f3d901016e0 2024-06-10T00:38:21.539 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.539+0000 7f3d8d7fa700 1 -- 172.21.15.121:0/31876239 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7800f620 con 0x7f3d901016e0 2024-06-10T00:38:21.545 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.543+0000 7f3d8d7fa700 1 -- 172.21.15.121:0/31876239 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3d78019070 con 0x7f3d901016e0 2024-06-10T00:38:21.548 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.548+0000 7f3d8d7fa700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3d7c0a2f90 0x7f3d7c0a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.548+0000 7f3da0b3ab80 1 Processor -- start 2024-06-10T00:38:21.549 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.549+0000 7f3d8d7fa700 1 -- 172.21.15.121:0/31876239 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9613+0+0 (secure 0 0 0) 0x7f3d780cf730 con 0x7f3d901016e0 2024-06-10T00:38:21.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.549+0000 7f3d8ffff700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3d7c0a2f90 0x7f3d7c0a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.550 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.549+0000 7f3da0b3ab80 1 -- start start 2024-06-10T00:38:21.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.549+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dbde0 0x5565f03dc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.550+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 0x5565f03d5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.550+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03d6530 0x5565f03d6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.550+0000 7f3da0b3ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5565f0327910 con 0x5565f03dbde0 2024-06-10T00:38:21.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.550+0000 7f3da0b3ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5565f02cf820 con 0x5565f03d6530 2024-06-10T00:38:21.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.550+0000 7f3da0b3ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5565f03238d0 con 0x5565f03dc700 2024-06-10T00:38:21.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.550+0000 7f3d94854700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03d6530 0x5565f03d6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.550+0000 7f3d8f7fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 0x5565f03d5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.550+0000 7f3d8f7fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 0x5565f03d5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:57806/0 (socket says 172.21.15.121:57806) 2024-06-10T00:38:21.551 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.550+0000 7f3d8f7fe700 1 -- 172.21.15.121:0/2360998200 learned_addr learned my addr 172.21.15.121:0/2360998200 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:21.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.551+0000 7f3d94854700 1 -- 172.21.15.121:0/2360998200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 msgr2=0x5565f03d5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:21.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.551+0000 7f3d94854700 1 --2- 172.21.15.121:0/2360998200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 0x5565f03d5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.551+0000 7f3d94854700 1 -- 172.21.15.121:0/2360998200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dbde0 msgr2=0x5565f03dc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:21.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.551+0000 7f3d94854700 1 --2- 172.21.15.121:0/2360998200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dbde0 0x5565f03dc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.552 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.551+0000 7f3d94854700 1 -- 172.21.15.121:0/2360998200 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565f02c6050 con 0x5565f03d6530 2024-06-10T00:38:21.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.551+0000 7f3d94854700 1 --2- 172.21.15.121:0/2360998200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03d6530 0x5565f03d6950 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009f70 tx=0x7f3d88009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f71615b7b9a55a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:21.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.552+0000 7f3d8ffff700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3d7c0a2f90 0x7f3d7c0a5450 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3d80006010 tx=0x7f3d80003760 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:21.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.552+0000 7f3d867fc700 1 -- 172.21.15.121:0/2360998200 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d88014620 con 0x5565f03d6530 2024-06-10T00:38:21.553 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.552+0000 7f3d867fc700 1 -- 172.21.15.121:0/2360998200 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d88019070 con 0x5565f03d6530 2024-06-10T00:38:21.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.553+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/2360998200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03d6530 msgr2=0x5565f03d6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:21.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.553+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/2360998200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03d6530 0x5565f03d6950 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009f70 tx=0x7f3d88009650 comp rx=0 tx=0).stop 2024-06-10T00:38:21.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.553+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/2360998200 shutdown_connections 2024-06-10T00:38:21.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.553+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/2360998200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03d6530 0x5565f03d6950 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.553+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/2360998200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dbde0 0x5565f03dc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.553+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/2360998200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 0x5565f03d5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.553+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/2360998200 >> 172.21.15.121:0/2360998200 conn(0x5565f0300dc0 msgr2=0x5565f03d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:21.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.554+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/2360998200 shutdown_connections 2024-06-10T00:38:21.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.554+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/2360998200 wait complete. 2024-06-10T00:38:21.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.555+0000 7f3da0b3ab80 1 -- start start 2024-06-10T00:38:21.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.555+0000 7f3da0b3ab80 1 Processor -- start 2024-06-10T00:38:21.557 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03d6530 0x5565f03e2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03df740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 0x5565f03dfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3d8ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03d6530 0x5565f03e2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3d8ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03d6530 0x5565f03e2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33912/0 (socket says 172.21.15.121:33912) 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3d8ffff700 1 -- 172.21.15.121:0/3367363712 learned_addr learned my addr 172.21.15.121:0/3367363712 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/3367363712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5565f03e3210 con 0x5565f03d6530 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/3367363712 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x5565f02c7cf0 con 0x5565f03dbde0 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/3367363712 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5565f03e3390 con 0x5565f03dc700 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3d94854700 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 0x5565f03dfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3da0b3ab80 10 client.0 _ll_register_callbacks cb 0x5565f02c1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3d8f7fe700 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03df740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.558 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3d8ffff700 1 -- 172.21.15.121:0/3367363712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 msgr2=0x5565f03dfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:21.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3d8ffff700 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 0x5565f03dfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3d8ffff700 1 -- 172.21.15.121:0/3367363712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 msgr2=0x5565f03df740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:21.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3d8ffff700 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03df740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3d8ffff700 1 -- 172.21.15.121:0/3367363712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565f02769d0 con 0x5565f03d6530 2024-06-10T00:38:21.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.556+0000 7f3d94854700 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 0x5565f03dfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:38:21.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.557+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/3367363712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5565f0334e10 con 0x5565f03d6530 2024-06-10T00:38:21.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.557+0000 7f3d8ffff700 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03d6530 0x5565f03e2440 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3d8001c590 tx=0x7f3d8001a320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e755ef15d836734 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:21.559 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.557+0000 7f3d6f7fe700 10 client.16884 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:38:21.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.558+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3367363712 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d800052d0 con 0x5565f03d6530 2024-06-10T00:38:21.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.558+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3367363712 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9613+0+0 (secure 0 0 0) 0x7f3d80005470 con 0x5565f03d6530 2024-06-10T00:38:21.560 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.558+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3367363712 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3d80005690 con 0x5565f03d6530 2024-06-10T00:38:21.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.558+0000 7f3da0b3ab80 10 client.16884 fetch_fsmap learned FSMap version 33 2024-06-10T00:38:21.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.558+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/3367363712 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({fsmap=33}) -- 0x5565f03edb10 con 0x5565f03d6530 2024-06-10T00:38:21.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.558+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3367363712 <== mon.0 v2:172.21.15.121:3300/0 4 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f3d8002f020 con 0x5565f03d6530 2024-06-10T00:38:21.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.559+0000 7f3da0b3ab80 10 client.16884 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:38:21.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.559+0000 7f3da0b3ab80 10 client.16884 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:38:21.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.559+0000 7f3da0b3ab80 20 client.16884 populate_metadata read hostname 'smithi121' 2024-06-10T00:38:21.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.559+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5565f03f0040 0x5565f03f2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.559+0000 7f3da0b3ab80 4 client.16884 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:38:21.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.559+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/3367363712 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565f032f390 con 0x5565f03f0040 2024-06-10T00:38:21.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.560+0000 7f3d8f7fe700 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5565f03f0040 0x5565f03f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.561+0000 7f3d6f7fe700 10 client.16884 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:38:21.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.561+0000 7f3d8f7fe700 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5565f03f0040 0x5565f03f2490 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc4d98e023ace755 server_cookie=ce8da3c2afc14215 in_seq=0 out_seq=0 2024-06-10T00:38:21.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.566+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/3367363712 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5565f032f390 con 0x5565f03f0040 2024-06-10T00:38:21.566 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.566+0000 7f3d6f7fe700 10 client.16884 handle_command_reply: tid=0 2024-06-10T00:38:21.570 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.570+0000 7f3da0b3ab80 1 client.16884 shutdown 2024-06-10T00:38:21.573 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.573+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/3367363712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03d6530 msgr2=0x5565f03e2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:21.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.573+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03d6530 0x5565f03e2440 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3d8001c590 tx=0x7f3d8001a320 comp rx=0 tx=0).stop 2024-06-10T00:38:21.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.573+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/3367363712 shutdown_connections 2024-06-10T00:38:21.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.573+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5565f03f0040 0x5565f03f2490 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.573+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03dc700 0x5565f03dfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.573+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03d6530 0x5565f03e2440 secure :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3d8001c590 tx=0x7f3d8001a320 comp rx=0 tx=0).stop 2024-06-10T00:38:21.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.573+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/3367363712 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03df740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.573+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/3367363712 >> 172.21.15.121:0/3367363712 conn(0x5565f0300dc0 msgr2=0x5565f03d4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:21.574 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.573+0000 7f3d6f7fe700 0 client.16884 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:38:21.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.574+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/3367363712 shutdown_connections 2024-06-10T00:38:21.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.574+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/3367363712 wait complete. 2024-06-10T00:38:21.575 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.575+0000 7f3da0b3ab80 20 client.16884 trim_cache size 0 max 16384 2024-06-10T00:38:21.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.613+0000 7f3da0b3ab80 1 Processor -- start 2024-06-10T00:38:21.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.613+0000 7f3da0b3ab80 1 -- start start 2024-06-10T00:38:21.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.613+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03d5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.613+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 0x5565f03dd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03f7da0 0x5565f03f8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3da0b3ab80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d700119b0 con 0x5565f03dc700 2024-06-10T00:38:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3da0b3ab80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7000f180 con 0x5565f03dbde0 2024-06-10T00:38:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3da0b3ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7000cff0 con 0x5565f03f7da0 2024-06-10T00:38:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3d8f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 0x5565f03dd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3d8f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 0x5565f03dd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:33920/0 (socket says 172.21.15.121:33920) 2024-06-10T00:38:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3d8f7fe700 1 -- 172.21.15.121:0/91080009 learned_addr learned my addr 172.21.15.121:0/91080009 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:21.615 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3d8ffff700 1 --2- 172.21.15.121:0/91080009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03d5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3d8f7fe700 1 -- 172.21.15.121:0/91080009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03f7da0 msgr2=0x5565f03f8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3d8f7fe700 1 --2- 172.21.15.121:0/91080009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03f7da0 0x5565f03f8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3d8f7fe700 1 -- 172.21.15.121:0/91080009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 msgr2=0x5565f03d5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3d8f7fe700 1 --2- 172.21.15.121:0/91080009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03d5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.614+0000 7f3d8f7fe700 1 -- 172.21.15.121:0/91080009 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565f03cb9c0 con 0x5565f03dc700 2024-06-10T00:38:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.615+0000 7f3d8f7fe700 1 --2- 172.21.15.121:0/91080009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 0x5565f03dd150 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009520 tx=0x7f3d7809bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44636373209fe4fd server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.615+0000 7f3d85ffb700 1 -- 172.21.15.121:0/91080009 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d780b4020 con 0x5565f03dc700 2024-06-10T00:38:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.615+0000 7f3d85ffb700 1 -- 172.21.15.121:0/91080009 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d780a5060 con 0x5565f03dc700 2024-06-10T00:38:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.615+0000 7f3d85ffb700 1 -- 172.21.15.121:0/91080009 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78020040 con 0x5565f03dc700 2024-06-10T00:38:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.616+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/91080009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 msgr2=0x5565f03dd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.616+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/91080009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 0x5565f03dd150 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009520 tx=0x7f3d7809bfd0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.618+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/91080009 shutdown_connections 2024-06-10T00:38:21.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.618+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/91080009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03f7da0 0x5565f03f8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.618 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.618+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/91080009 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03d5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.618+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/91080009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 0x5565f03dd150 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.619 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.618+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/91080009 >> 172.21.15.121:0/91080009 conn(0x5565f0300dc0 msgr2=0x5565f03caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.621+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/91080009 shutdown_connections 2024-06-10T00:38:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.621+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/91080009 wait complete. 2024-06-10T00:38:21.621 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.621+0000 7f3da0b3ab80 1 -- start start 2024-06-10T00:38:21.622 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.622+0000 7f3da0b3ab80 1 Processor -- start 2024-06-10T00:38:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.622+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03e8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 0x5565f03e8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3da0b3ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03f7da0 0x5565f03e5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3d8f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 0x5565f03e8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3d8ffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03e8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.623 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3d8ffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03e8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:49428/0 (socket says 172.21.15.121:49428) 2024-06-10T00:38:21.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3d8ffff700 1 -- 172.21.15.121:0/26499088 learned_addr learned my addr 172.21.15.121:0/26499088 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:21.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/26499088 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7000d640 con 0x5565f03dc700 2024-06-10T00:38:21.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/26499088 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7000eab0 con 0x5565f03dbde0 2024-06-10T00:38:21.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/26499088 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7000dd10 con 0x5565f03f7da0 2024-06-10T00:38:21.624 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3d94854700 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03f7da0 0x5565f03e5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3da0b3ab80 10 client.0 _ll_register_callbacks cb 0x5565f02c1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:38:21.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3d8ffff700 1 -- 172.21.15.121:0/26499088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03f7da0 msgr2=0x5565f03e5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:21.626 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.623+0000 7f3d8ffff700 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03f7da0 0x5565f03e5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.624+0000 7f3d8ffff700 1 -- 172.21.15.121:0/26499088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 msgr2=0x5565f03e8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:21.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.624+0000 7f3d8ffff700 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 0x5565f03e8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.624+0000 7f3d8ffff700 1 -- 172.21.15.121:0/26499088 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565f03e6360 con 0x5565f03dbde0 2024-06-10T00:38:21.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.624+0000 7f3d8ffff700 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03e8760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x5565f03deff0 tx=0x7f3d8000d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6e38f5f502818e5 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:21.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.624+0000 7f3d6f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.135:3300/0 2024-06-10T00:38:21.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.624+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/26499088 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d80005c00 con 0x5565f03dbde0 2024-06-10T00:38:21.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.624+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/26499088 <== mon.2 v2:172.21.15.135:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9613+0+0 (secure 0 0 0) 0x7f3d80031440 con 0x5565f03dbde0 2024-06-10T00:38:21.627 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.624+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/26499088 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d780a5060 con 0x5565f03dbde0 2024-06-10T00:38:21.628 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.627+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/26499088 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3d800054d0 con 0x5565f03dbde0 2024-06-10T00:38:21.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.631+0000 7f3da0b3ab80 10 client.26399 fetch_fsmap learned FSMap version 33 2024-06-10T00:38:21.634 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.631+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/26499088 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5565f045c7f0 con 0x5565f03dbde0 2024-06-10T00:38:21.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.632+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/26499088 <== mon.2 v2:172.21.15.135:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5565f0334e10 con 0x5565f03dbde0 2024-06-10T00:38:21.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.632+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/26499088 <== mon.2 v2:172.21.15.135:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f3d8002f070 con 0x5565f03dbde0 2024-06-10T00:38:21.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.632+0000 7f3da0b3ab80 10 client.26399 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:38:21.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.632+0000 7f3da0b3ab80 10 client.26399 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:38:21.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.632+0000 7f3da0b3ab80 20 client.26399 populate_metadata read hostname 'smithi121' 2024-06-10T00:38:21.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.632+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5565f045ce70 0x5565f045f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:21.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.632+0000 7f3da0b3ab80 4 client.26399 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "6073a0a1", "release": true, "await": true}] 2024-06-10T00:38:21.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.632+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/26499088 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "set_id": "6073a0a1", "release": true, "await": true}) -- 0x7f3d780a5060 con 0x5565f045ce70 2024-06-10T00:38:21.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.633+0000 7f3d8f7fe700 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5565f045ce70 0x5565f045f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:21.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.634+0000 7f3d8f7fe700 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5565f045ce70 0x5565f045f260 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab6197317a96c5cb server_cookie=3fc8fe5d4a442441 in_seq=0 out_seq=0 2024-06-10T00:38:21.636 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.634+0000 7f3d6f7fe700 10 client.26399 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:38:21.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.770+0000 7f3d6f7fe700 1 -- 172.21.15.121:0/26499088 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f3d780a5060 con 0x5565f045ce70 2024-06-10T00:38:21.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.770+0000 7f3d6f7fe700 10 client.26399 handle_command_reply: tid=0 2024-06-10T00:38:21.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.770+0000 7f3da0b3ab80 1 client.26399 shutdown 2024-06-10T00:38:21.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.771+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/26499088 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 msgr2=0x5565f03e8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:21.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.771+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03e8760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x5565f03deff0 tx=0x7f3d8000d000 comp rx=0 tx=0).stop 2024-06-10T00:38:21.772 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.771+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/26499088 shutdown_connections 2024-06-10T00:38:21.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.771+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x5565f045ce70 0x5565f045f260 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.771+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5565f03f7da0 0x5565f03e5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.771+0000 7f3d6f7fe700 0 client.26399 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:38:21.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.771+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x5565f03dbde0 0x5565f03e8760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.771+0000 7f3da0b3ab80 1 --2- 172.21.15.121:0/26499088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5565f03dc700 0x5565f03e8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.771+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/26499088 >> 172.21.15.121:0/26499088 conn(0x5565f0300dc0 msgr2=0x5565f03ca410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:21.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.771+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/26499088 shutdown_connections 2024-06-10T00:38:21.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.772+0000 7f3da0b3ab80 1 -- 172.21.15.121:0/26499088 wait complete. 2024-06-10T00:38:21.773 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.772+0000 7f3da0b3ab80 20 client.26399 trim_cache size 0 max 16384 2024-06-10T00:38:21.773 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 33, 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 32, 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": { 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: "6073a0a1": { 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: "version": 32, 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: "age_ref": 0.0, 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:21.774 INFO:teuthology.orchestra.run.smithi121.stdout: "timeout": 90.0, 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: "expiration": 137.8, 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: "members": { 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: "file:/": { 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: "excluded": false, 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: "state": { 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RELEASED", 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: "age": 0.0 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:21.775 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:38:21.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 -- 172.21.15.121:0/31876239 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3d7c0a2f90 msgr2=0x7f3d7c0a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:21.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3d7c0a2f90 0x7f3d7c0a5450 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3d80006010 tx=0x7f3d80003760 comp rx=0 tx=0).stop 2024-06-10T00:38:21.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 -- 172.21.15.121:0/31876239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d901016e0 msgr2=0x7f3d90107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:21.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d901016e0 0x7f3d90107ea0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800d950 tx=0x7f3d7800dc60 comp rx=0 tx=0).stop 2024-06-10T00:38:21.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 -- 172.21.15.121:0/31876239 shutdown_connections 2024-06-10T00:38:21.776 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3d7c0a2f90 0x7f3d7c0a5450 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d90105a80 0x7f3d901083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d90100d20 0x7f3d90107960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:21.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 --2- 172.21.15.121:0/31876239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d901016e0 0x7f3d90107ea0 secure :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800d950 tx=0x7f3d7800dc60 comp rx=0 tx=0).stop 2024-06-10T00:38:21.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 -- 172.21.15.121:0/31876239 >> 172.21.15.121:0/31876239 conn(0x7f3d900765f0 msgr2=0x7f3d9010c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:21.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 -- 172.21.15.121:0/31876239 shutdown_connections 2024-06-10T00:38:21.777 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:21.773+0000 7f3d962b7700 1 -- 172.21.15.121:0/31876239 wait complete. 2024-06-10T00:38:22.612 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '6073a0a1', total seconds elapsed: 28.5 2024-06-10T00:38:22.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:22 smithi121 ceph-mon[32378]: pgmap v322: 129 pgs: 8 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 109 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-06-10T00:38:23.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:22 smithi193 ceph-mon[36716]: pgmap v322: 129 pgs: 8 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 109 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-06-10T00:38:23.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:22 smithi135 ceph-mon[36424]: pgmap v322: 129 pgs: 8 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 109 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 682 B/s wr, 7 op/s 2024-06-10T00:38:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:24 smithi193 ceph-mon[36716]: pgmap v323: 129 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 110 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 214 KiB/s wr, 9 op/s 2024-06-10T00:38:25.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:24 smithi121 ceph-mon[32378]: pgmap v323: 129 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 110 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 214 KiB/s wr, 9 op/s 2024-06-10T00:38:25.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:24 smithi135 ceph-mon[36424]: pgmap v323: 129 pgs: 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 110 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 214 KiB/s wr, 9 op/s 2024-06-10T00:38:27.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:26 smithi193 ceph-mon[36716]: pgmap v324: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 291 KiB/s wr, 38 op/s 2024-06-10T00:38:27.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:26 smithi121 ceph-mon[32378]: pgmap v324: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 291 KiB/s wr, 38 op/s 2024-06-10T00:38:27.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:26 smithi135 ceph-mon[36424]: pgmap v324: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 291 KiB/s wr, 38 op/s 2024-06-10T00:38:29.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:28 smithi193 ceph-mon[36716]: pgmap v325: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 291 KiB/s wr, 35 op/s 2024-06-10T00:38:29.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:28 smithi121 ceph-mon[32378]: pgmap v325: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 291 KiB/s wr, 35 op/s 2024-06-10T00:38:29.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:28 smithi135 ceph-mon[36424]: pgmap v325: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 291 KiB/s wr, 35 op/s 2024-06-10T00:38:30.433 INFO:tasks.workunit:Stopping ['suites/blogbench.sh'] on client.0... 2024-06-10T00:38:30.433 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-06-10T00:38:30.879 DEBUG:teuthology.parallel:result is None 2024-06-10T00:38:30.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-06-10T00:38:30.936 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-06-10T00:38:30.936 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-06-10T00:38:30.948 INFO:tasks.quiescer:joining Quiescers 2024-06-10T00:38:30.949 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-06-10T00:38:30.949 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:38:31.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:30 smithi193 ceph-mon[36716]: pgmap v326: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 1008 KiB/s wr, 104 op/s 2024-06-10T00:38:31.186 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:38:31.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:30 smithi121 ceph-mon[32378]: pgmap v326: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 1008 KiB/s wr, 104 op/s 2024-06-10T00:38:31.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:30 smithi135 ceph-mon[36424]: pgmap v326: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 1008 KiB/s wr, 104 op/s 2024-06-10T00:38:32.988 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:32 smithi121 ceph-mon[32378]: pgmap v327: 129 pgs: 129 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 141 op/s 2024-06-10T00:38:33.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:32 smithi193 ceph-mon[36716]: pgmap v327: 129 pgs: 129 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 141 op/s 2024-06-10T00:38:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:32 smithi135 ceph-mon[36424]: pgmap v327: 129 pgs: 129 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 141 op/s 2024-06-10T00:38:33.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.344+0000 7f0fc7234700 1 -- 172.21.15.121:0/4288763651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fc0100b50 msgr2=0x7f0fc0100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:33.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.344+0000 7f0fc7234700 1 --2- 172.21.15.121:0/4288763651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fc0100b50 0x7f0fc0100fd0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0fac009990 tx=0x7f0fac009ca0 comp rx=0 tx=0).stop 2024-06-10T00:38:33.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.344+0000 7f0fc7234700 1 -- 172.21.15.121:0/4288763651 shutdown_connections 2024-06-10T00:38:33.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.344+0000 7f0fc7234700 1 --2- 172.21.15.121:0/4288763651 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fc0101510 0x7f0fc010d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:33.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.344+0000 7f0fc7234700 1 --2- 172.21.15.121:0/4288763651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fc0106b90 0x7f0fc0106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:33.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.344+0000 7f0fc7234700 1 --2- 172.21.15.121:0/4288763651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fc0100b50 0x7f0fc0100fd0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:33.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.344+0000 7f0fc7234700 1 -- 172.21.15.121:0/4288763651 >> 172.21.15.121:0/4288763651 conn(0x7f0fc0077a00 msgr2=0x7f0fc00fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:33.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.344+0000 7f0fc7234700 1 -- 172.21.15.121:0/4288763651 shutdown_connections 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.344+0000 7f0fc7234700 1 -- 172.21.15.121:0/4288763651 wait complete. 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc7234700 1 Processor -- start 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc7234700 1 -- start start 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc7234700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fc0100b50 0x7f0fc00fce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc7234700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fc0101510 0x7f0fc00fd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc7234700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fc0106b90 0x7f0fc00fd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc7234700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0110ef0 con 0x7f0fc0106b90 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc7234700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0110d70 con 0x7f0fc0101510 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc7234700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0111070 con 0x7f0fc0100b50 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc57d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fc0106b90 0x7f0fc00fd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc57d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fc0106b90 0x7f0fc00fd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59442/0 (socket says 172.21.15.121:59442) 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc57d1700 1 -- 172.21.15.121:0/2668284689 learned_addr learned my addr 172.21.15.121:0/2668284689 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fc4fd0700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fc0100b50 0x7f0fc00fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:33.376 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.345+0000 7f0fbffff700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fc0101510 0x7f0fc00fd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fc57d1700 1 -- 172.21.15.121:0/2668284689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fc0100b50 msgr2=0x7f0fc00fce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fc57d1700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fc0100b50 0x7f0fc00fce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fc57d1700 1 -- 172.21.15.121:0/2668284689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fc0101510 msgr2=0x7f0fc00fd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fc57d1700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fc0101510 0x7f0fc00fd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fc57d1700 1 -- 172.21.15.121:0/2668284689 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc01a7280 con 0x7f0fc0106b90 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fbffff700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fc0101510 0x7f0fc00fd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fc4fd0700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fc0100b50 0x7f0fc00fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fc57d1700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fc0106b90 0x7f0fc00fd8f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800b970 tx=0x7f0fb800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fbdffb700 1 -- 172.21.15.121:0/2668284689 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8019070 con 0x7f0fc0106b90 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fbdffb700 1 -- 172.21.15.121:0/2668284689 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb8014440 con 0x7f0fc0106b90 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fc7234700 1 -- 172.21.15.121:0/2668284689 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc01a7570 con 0x7f0fc0106b90 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fbdffb700 1 -- 172.21.15.121:0/2668284689 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb80075d0 con 0x7f0fc0106b90 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.346+0000 7f0fc7234700 1 -- 172.21.15.121:0/2668284689 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc01a7ab0 con 0x7f0fc0106b90 2024-06-10T00:38:33.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.347+0000 7f0fc7234700 1 -- 172.21.15.121:0/2668284689 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc0069180 con 0x7f0fc0106b90 2024-06-10T00:38:33.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.348+0000 7f0fbdffb700 1 -- 172.21.15.121:0/2668284689 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0fb801c070 con 0x7f0fc0106b90 2024-06-10T00:38:33.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.349+0000 7f0fbdffb700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0fb00a2d90 0x7f0fb00a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:33.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.349+0000 7f0fc4fd0700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0fb00a2d90 0x7f0fb00a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:33.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.349+0000 7f0fbdffb700 1 -- 172.21.15.121:0/2668284689 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9613+0+0 (secure 0 0 0) 0x7f0fb80cf050 con 0x7f0fc0106b90 2024-06-10T00:38:33.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.349+0000 7f0fc4fd0700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0fb00a2d90 0x7f0fb00a5250 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40099d0 tx=0x7f0fb4008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:33.378 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.352+0000 7f0fbdffb700 1 -- 172.21.15.121:0/2668284689 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0fb809b940 con 0x7f0fc0106b90 2024-06-10T00:38:33.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.594+0000 7f0fc7234700 1 -- 172.21.15.121:0/2668284689 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fc01a7d70 con 0x7f0fc0106b90 2024-06-10T00:38:33.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.594+0000 7f0fbdffb700 1 -- 172.21.15.121:0/2668284689 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11895 (secure 0 0 0) 0x7f0fb809b240 con 0x7f0fc0106b90 2024-06-10T00:38:33.595 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:38:33.595 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:35:59.291154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":189,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:38:33.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 -- 172.21.15.121:0/2668284689 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0fb00a2d90 msgr2=0x7f0fb00a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:33.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0fb00a2d90 0x7f0fb00a5250 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40099d0 tx=0x7f0fb4008040 comp rx=0 tx=0).stop 2024-06-10T00:38:33.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 -- 172.21.15.121:0/2668284689 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fc0106b90 msgr2=0x7f0fc00fd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:33.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fc0106b90 0x7f0fc00fd8f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800b970 tx=0x7f0fb800bd30 comp rx=0 tx=0).stop 2024-06-10T00:38:33.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 -- 172.21.15.121:0/2668284689 shutdown_connections 2024-06-10T00:38:33.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0fb00a2d90 0x7f0fb00a5250 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:33.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fc0106b90 0x7f0fc00fd8f0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:33.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0fc0100b50 0x7f0fc00fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:33.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 --2- 172.21.15.121:0/2668284689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0fc0101510 0x7f0fc00fd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:33.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 -- 172.21.15.121:0/2668284689 >> 172.21.15.121:0/2668284689 conn(0x7f0fc0077a00 msgr2=0x7f0fc00fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:33.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 -- 172.21.15.121:0/2668284689 shutdown_connections 2024-06-10T00:38:33.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:33.597+0000 7f0fc7234700 1 -- 172.21.15.121:0/2668284689 wait complete. 2024-06-10T00:38:33.598 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 33 2024-06-10T00:38:33.885 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:33 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2668284689' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:38:34.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:33 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2668284689' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:38:34.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:33 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2668284689' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:38:34.293 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24479 quiesce db --cancel --all' 2024-06-10T00:38:34.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- timeout 105 ceph tell mds.24479 quiesce db --cancel --all 2024-06-10T00:38:34.540 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:38:34.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:34 smithi121 ceph-mon[32378]: pgmap v328: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-06-10T00:38:35.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:34 smithi193 ceph-mon[36716]: pgmap v328: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-06-10T00:38:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:34 smithi135 ceph-mon[36424]: pgmap v328: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 163 op/s 2024-06-10T00:38:36.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.320+0000 7f2ef8f69700 1 -- 172.21.15.121:0/2471087928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ef406b5c0 msgr2=0x7f2ef410f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:36.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.320+0000 7f2ef8f69700 1 --2- 172.21.15.121:0/2471087928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ef406b5c0 0x7f2ef410f320 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00b0a0 tx=0x7f2eec00b3b0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.321 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:36 smithi121 ceph-mon[32378]: osdmap e109: 12 total, 12 up, 12 in 2024-06-10T00:38:36.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.320+0000 7f2ef8f69700 1 -- 172.21.15.121:0/2471087928 shutdown_connections 2024-06-10T00:38:36.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.320+0000 7f2ef8f69700 1 --2- 172.21.15.121:0/2471087928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ef406b5c0 0x7f2ef410f320 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.320+0000 7f2ef8f69700 1 --2- 172.21.15.121:0/2471087928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ef4108520 0x7f2ef4108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.320+0000 7f2ef8f69700 1 --2- 172.21.15.121:0/2471087928 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ef406ac00 0x7f2ef406b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.320+0000 7f2ef8f69700 1 -- 172.21.15.121:0/2471087928 >> 172.21.15.121:0/2471087928 conn(0x7f2ef40773b0 msgr2=0x7f2ef40777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:36.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.320+0000 7f2ef8f69700 1 -- 172.21.15.121:0/2471087928 shutdown_connections 2024-06-10T00:38:36.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.320+0000 7f2ef8f69700 1 -- 172.21.15.121:0/2471087928 wait complete. 2024-06-10T00:38:36.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef8f69700 1 Processor -- start 2024-06-10T00:38:36.326 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef8f69700 1 -- start start 2024-06-10T00:38:36.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef8f69700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ef406ac00 0x7f2ef4101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef8f69700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ef406b5c0 0x7f2ef41019d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef8f69700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ef4108520 0x7f2ef4101f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef8f69700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef4112890 con 0x7f2ef4108520 2024-06-10T00:38:36.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef8f69700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef4112710 con 0x7f2ef406b5c0 2024-06-10T00:38:36.327 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef8f69700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef4112a10 con 0x7f2ef406ac00 2024-06-10T00:38:36.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef2d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ef4108520 0x7f2ef4101f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef2d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ef4108520 0x7f2ef4101f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59452/0 (socket says 172.21.15.121:59452) 2024-06-10T00:38:36.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef2d9d700 1 -- 172.21.15.121:0/1020194621 learned_addr learned my addr 172.21.15.121:0/1020194621 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:36.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.321+0000 7f2ef259c700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ef406ac00 0x7f2ef4101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.322+0000 7f2ef2d9d700 1 -- 172.21.15.121:0/1020194621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ef406ac00 msgr2=0x7f2ef4101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:36.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.322+0000 7f2ef2d9d700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ef406ac00 0x7f2ef4101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.328 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.322+0000 7f2ef2d9d700 1 -- 172.21.15.121:0/1020194621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ef406b5c0 msgr2=0x7f2ef41019d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.322+0000 7f2ef2d9d700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ef406b5c0 0x7f2ef41019d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.322+0000 7f2ef2d9d700 1 -- 172.21.15.121:0/1020194621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef41a72b0 con 0x7f2ef4108520 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.322+0000 7f2ef2d9d700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ef4108520 0x7f2ef4101f10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f2eec004e40 tx=0x7f2eec004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.322+0000 7f2ee37fe700 1 -- 172.21.15.121:0/1020194621 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec021070 con 0x7f2ef4108520 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.322+0000 7f2ee37fe700 1 -- 172.21.15.121:0/1020194621 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eec007de0 con 0x7f2ef4108520 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.322+0000 7f2ef8f69700 1 -- 172.21.15.121:0/1020194621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef41a7540 con 0x7f2ef4108520 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.322+0000 7f2ee37fe700 1 -- 172.21.15.121:0/1020194621 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec005440 con 0x7f2ef4108520 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.322+0000 7f2ef8f69700 1 -- 172.21.15.121:0/1020194621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef41a79a0 con 0x7f2ef4108520 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.324+0000 7f2ee37fe700 1 -- 172.21.15.121:0/1020194621 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f2eec024070 con 0x7f2ef4108520 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.325+0000 7f2ee37fe700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2edc0a2d90 0x7f2edc0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.325+0000 7f2ef259c700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2edc0a2d90 0x7f2edc0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.329 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.325+0000 7f2ee37fe700 1 -- 172.21.15.121:0/1020194621 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9525+0+0 (secure 0 0 0) 0x7f2eec00f020 con 0x7f2ef4108520 2024-06-10T00:38:36.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.325+0000 7f2ef259c700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2edc0a2d90 0x7f2edc0a5250 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009910 tx=0x7f2ee8008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:36.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.326+0000 7f2f037ecb80 1 Processor -- start 2024-06-10T00:38:36.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.326+0000 7f2f037ecb80 1 -- start start 2024-06-10T00:38:36.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.326+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b739850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.339 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.326+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739d90 0x55a51b733680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.326+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b733bc0 0x55a51b733fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a51b625fd0 con 0x55a51b739470 2024-06-10T00:38:36.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55a51b6848c0 con 0x55a51b739d90 2024-06-10T00:38:36.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a51b5ec4d0 con 0x55a51b733bc0 2024-06-10T00:38:36.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ef259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b739850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ef259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b739850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59458/0 (socket says 172.21.15.121:59458) 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ef259c700 1 -- 172.21.15.121:0/3776308451 learned_addr learned my addr 172.21.15.121:0/3776308451 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ef1d9b700 1 --2- 172.21.15.121:0/3776308451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739d90 0x55a51b733680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ef259c700 1 -- 172.21.15.121:0/3776308451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b733bc0 msgr2=0x55a51b733fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ef259c700 1 --2- 172.21.15.121:0/3776308451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b733bc0 0x55a51b733fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ef259c700 1 -- 172.21.15.121:0/3776308451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739d90 msgr2=0x55a51b733680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ef259c700 1 --2- 172.21.15.121:0/3776308451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739d90 0x55a51b733680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ef259c700 1 -- 172.21.15.121:0/3776308451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a51b5d3940 con 0x55a51b739470 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ef259c700 1 --2- 172.21.15.121:0/3776308451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b739850 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800d6d0 tx=0x7f2ee8010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6b39a9c183758d server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ee0ff9700 1 -- 172.21.15.121:0/3776308451 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8024070 con 0x55a51b739470 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ee0ff9700 1 -- 172.21.15.121:0/3776308451 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ee8014b20 con 0x55a51b739470 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.327+0000 7f2ee0ff9700 1 -- 172.21.15.121:0/3776308451 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee801fb60 con 0x55a51b739470 2024-06-10T00:38:36.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.328+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3776308451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 msgr2=0x55a51b739850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:36.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.328+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/3776308451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b739850 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2ee800d6d0 tx=0x7f2ee8010000 comp rx=0 tx=0).stop 2024-06-10T00:38:36.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.328+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3776308451 shutdown_connections 2024-06-10T00:38:36.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.328+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/3776308451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b733bc0 0x55a51b733fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.328+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/3776308451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b739850 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.342 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.328+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/3776308451 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739d90 0x55a51b733680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.328+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3776308451 >> 172.21.15.121:0/3776308451 conn(0x55a51b6600f0 msgr2=0x55a51b72e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.328+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3776308451 shutdown_connections 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.328+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3776308451 wait complete. 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.328+0000 7f2f037ecb80 1 -- start start 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2f037ecb80 1 Processor -- start 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b73fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739470 0x55a51b73ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b739d90 0x55a51b73d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a51b625030 con 0x55a51b733bc0 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x55a51b680880 con 0x55a51b739470 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a51b740770 con 0x55a51b739d90 2024-06-10T00:38:36.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2f037ecb80 10 client.0 _ll_register_callbacks cb 0x55a51b61a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:38:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ef259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b73fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ef259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b73fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59462/0 (socket says 172.21.15.121:59462) 2024-06-10T00:38:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ef259c700 1 -- 172.21.15.121:0/3824323211 learned_addr learned my addr 172.21.15.121:0/3824323211 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ef1d9b700 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739470 0x55a51b73ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ef2d9d700 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b739d90 0x55a51b73d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.344 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ef259c700 1 -- 172.21.15.121:0/3824323211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b739d90 msgr2=0x55a51b73d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ef259c700 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b739d90 0x55a51b73d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ef259c700 1 -- 172.21.15.121:0/3824323211 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739470 msgr2=0x55a51b73ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ef259c700 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739470 0x55a51b73ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ef259c700 1 -- 172.21.15.121:0/3824323211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a51b64a7c0 con 0x55a51b733bc0 2024-06-10T00:38:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ef259c700 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b73fb60 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8007fd0 tx=0x7f2ee80146f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69df5c83e6eac40 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.329+0000 7f2ed9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:38:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/3824323211 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee802c070 con 0x55a51b733bc0 2024-06-10T00:38:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/3824323211 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9525+0+0 (secure 0 0 0) 0x7f2ee801f7c0 con 0x55a51b733bc0 2024-06-10T00:38:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3824323211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ee8014b20 con 0x55a51b733bc0 2024-06-10T00:38:36.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/3824323211 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2ee8038050 con 0x55a51b733bc0 2024-06-10T00:38:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2f037ecb80 10 client.16905 fetch_fsmap learned FSMap version 33 2024-06-10T00:38:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3824323211 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a51b74af50 con 0x55a51b733bc0 2024-06-10T00:38:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/3824323211 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ee8014b20 con 0x55a51b733bc0 2024-06-10T00:38:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/3824323211 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f2ee8027050 con 0x55a51b733bc0 2024-06-10T00:38:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2f037ecb80 10 client.16905 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:38:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2f037ecb80 10 client.16905 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:38:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2f037ecb80 20 client.16905 populate_metadata read hostname 'smithi121' 2024-06-10T00:38:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a51b74d420 0x55a51b74f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2f037ecb80 4 client.16905 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-06-10T00:38:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.330+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3824323211 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a51b691dc0 con 0x55a51b74d420 2024-06-10T00:38:36.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.331+0000 7f2ef1d9b700 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a51b74d420 0x55a51b74f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.331+0000 7f2ef1d9b700 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a51b74d420 0x55a51b74f810 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27e796f70b32b4b server_cookie=d206e258bbd460a7 in_seq=0 out_seq=0 2024-06-10T00:38:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.331+0000 7f2ed9ffb700 10 client.16905 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:38:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/3824323211 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55a51b691dc0 con 0x55a51b74d420 2024-06-10T00:38:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2ed9ffb700 10 client.16905 handle_command_reply: tid=0 2024-06-10T00:38:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 1 client.16905 shutdown 2024-06-10T00:38:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3824323211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 msgr2=0x55a51b73fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b73fb60 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8007fd0 tx=0x7f2ee80146f0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3824323211 shutdown_connections 2024-06-10T00:38:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a51b74d420 0x55a51b74f810 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b739d90 0x55a51b73d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b73fb60 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/3824323211 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739470 0x55a51b73ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3824323211 >> 172.21.15.121:0/3824323211 conn(0x55a51b6600f0 msgr2=0x55a51b72db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:36.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2ed9ffb700 0 client.16905 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:38:36.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3824323211 shutdown_connections 2024-06-10T00:38:36.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 1 -- 172.21.15.121:0/3824323211 wait complete. 2024-06-10T00:38:36.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.332+0000 7f2f037ecb80 20 client.16905 trim_cache size 0 max 16384 2024-06-10T00:38:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2f037ecb80 1 Processor -- start 2024-06-10T00:38:36.359 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2f037ecb80 1 -- start start 2024-06-10T00:38:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b733bc0 0x55a51b738710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b72e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739d90 0x55a51b72e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41b32e0 con 0x55a51b739470 2024-06-10T00:38:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41b0ab0 con 0x55a51b739d90 2024-06-10T00:38:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41ae920 con 0x55a51b733bc0 2024-06-10T00:38:36.360 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2ef1d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b72e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2ef1d9b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b72e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59478/0 (socket says 172.21.15.121:59478) 2024-06-10T00:38:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2ef1d9b700 1 -- 172.21.15.121:0/2071542990 learned_addr learned my addr 172.21.15.121:0/2071542990 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2ef1d9b700 1 -- 172.21.15.121:0/2071542990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b733bc0 msgr2=0x55a51b738710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2ef1d9b700 1 --2- 172.21.15.121:0/2071542990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b733bc0 0x55a51b738710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2ef1d9b700 1 -- 172.21.15.121:0/2071542990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739d90 msgr2=0x55a51b72e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:36.361 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2ef1d9b700 1 --2- 172.21.15.121:0/2071542990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739d90 0x55a51b72e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.352+0000 7f2ef1d9b700 1 -- 172.21.15.121:0/2071542990 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a51b726410 con 0x55a51b739470 2024-06-10T00:38:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2ef1d9b700 1 --2- 172.21.15.121:0/2071542990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b72e2a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4006fd0 tx=0x7f2ee400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c330b8b45ac2497 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2edbfff700 1 -- 172.21.15.121:0/2071542990 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40168b0 con 0x55a51b739470 2024-06-10T00:38:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2edbfff700 1 -- 172.21.15.121:0/2071542990 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ee4022070 con 0x55a51b739470 2024-06-10T00:38:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2edbfff700 1 -- 172.21.15.121:0/2071542990 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee401d400 con 0x55a51b739470 2024-06-10T00:38:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2071542990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 msgr2=0x55a51b72e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:36.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/2071542990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b72e2a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4006fd0 tx=0x7f2ee400bda0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2071542990 shutdown_connections 2024-06-10T00:38:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/2071542990 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739d90 0x55a51b72e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/2071542990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b733bc0 0x55a51b738710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.363 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/2071542990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b739470 0x55a51b72e2a0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2071542990 >> 172.21.15.121:0/2071542990 conn(0x55a51b6600f0 msgr2=0x55a51b738000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2071542990 shutdown_connections 2024-06-10T00:38:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.353+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2071542990 wait complete. 2024-06-10T00:38:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2f037ecb80 1 -- start start 2024-06-10T00:38:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2f037ecb80 1 Processor -- start 2024-06-10T00:38:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b745a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.364 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739470 0x55a51b745f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2f037ecb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b739d90 0x55a51b742a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41aef70 con 0x55a51b733bc0 2024-06-10T00:38:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41b03e0 con 0x55a51b739470 2024-06-10T00:38:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2f037ecb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41af640 con 0x55a51b739d90 2024-06-10T00:38:36.365 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2f037ecb80 10 client.0 _ll_register_callbacks cb 0x55a51b61a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-06-10T00:38:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2ef259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b745a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2ef259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b745a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59488/0 (socket says 172.21.15.121:59488) 2024-06-10T00:38:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2ef259c700 1 -- 172.21.15.121:0/2433091522 learned_addr learned my addr 172.21.15.121:0/2433091522 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.354+0000 7f2ef1d9b700 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739470 0x55a51b745f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ef259c700 1 -- 172.21.15.121:0/2433091522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b739d90 msgr2=0x55a51b742a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:36.366 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ef259c700 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b739d90 0x55a51b742a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ef259c700 1 -- 172.21.15.121:0/2433091522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739470 msgr2=0x55a51b745f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-10T00:38:36.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ef259c700 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739470 0x55a51b745f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ef259c700 1 -- 172.21.15.121:0/2433091522 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a51b742ff0 con 0x55a51b733bc0 2024-06-10T00:38:36.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ef1d9b700 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739470 0x55a51b745f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:38:36.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ef259c700 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b745a30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8012620 tx=0x7f2ee80128c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60daaa1be442ea0a server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:36.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ed9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-06-10T00:38:36.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/2433091522 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8035070 con 0x55a51b733bc0 2024-06-10T00:38:36.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/2433091522 <== mon.0 v2:172.21.15.121:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 9525+0+0 (secure 0 0 0) 0x7f2ee8012c90 con 0x55a51b733bc0 2024-06-10T00:38:36.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2433091522 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ee4022070 con 0x55a51b733bc0 2024-06-10T00:38:36.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/2433091522 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2ee8044050 con 0x55a51b733bc0 2024-06-10T00:38:36.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2f037ecb80 10 client.16911 fetch_fsmap learned FSMap version 33 2024-06-10T00:38:36.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2433091522 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a51b7b8800 con 0x55a51b733bc0 2024-06-10T00:38:36.368 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/2433091522 <== mon.0 v2:172.21.15.121:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ee4022070 con 0x55a51b733bc0 2024-06-10T00:38:36.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/2433091522 <== mon.0 v2:172.21.15.121:3300/0 5 ==== fsmap(e 33) ==== 8370+0+0 (secure 0 0 0) 0x7f2ee8027030 con 0x55a51b733bc0 2024-06-10T00:38:36.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2f037ecb80 10 client.16911 fetch_fsmap finished waiting for FSMap version 33 2024-06-10T00:38:36.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2f037ecb80 10 client.16911 resolve_mds: validated gid 24479 aka daemon mds.b 2024-06-10T00:38:36.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2f037ecb80 20 client.16911 populate_metadata read hostname 'smithi121' 2024-06-10T00:38:36.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a51b7b8fa0 0x55a51b7bb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:36.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2f037ecb80 4 client.16911 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-06-10T00:38:36.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.355+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2433091522 --> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x55a51b68c340 con 0x55a51b7b8fa0 2024-06-10T00:38:36.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.356+0000 7f2ef1d9b700 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a51b7b8fa0 0x55a51b7bb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:36.369 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2ef1d9b700 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a51b7b8fa0 0x55a51b7bb3c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c484cca8c322049 server_cookie=f3361bc2fddc596 in_seq=0 out_seq=0 2024-06-10T00:38:36.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2ed9ffb700 10 client.16911 ms_handle_connect on v2:172.21.15.135:6838/1599631648 2024-06-10T00:38:36.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2ed9ffb700 1 -- 172.21.15.121:0/2433091522 <== mds.0 v2:172.21.15.135:6838/1599631648 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x55a51b68c340 con 0x55a51b7b8fa0 2024-06-10T00:38:36.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2ed9ffb700 10 client.16911 handle_command_reply: tid=0 2024-06-10T00:38:36.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2f037ecb80 1 client.16911 shutdown 2024-06-10T00:38:36.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2433091522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 msgr2=0x55a51b745a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:36.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b745a30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8012620 tx=0x7f2ee80128c0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2433091522 shutdown_connections 2024-06-10T00:38:36.370 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.135:6838/1599631648,v1:172.21.15.135:6839/1599631648] conn(0x55a51b7b8fa0 0x55a51b7bb3c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a51b739d90 0x55a51b742a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2ed9ffb700 0 client.16911 ms_handle_reset on v2:172.21.15.135:6838/1599631648 2024-06-10T00:38:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a51b733bc0 0x55a51b745a30 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.357+0000 7f2f037ecb80 1 --2- 172.21.15.121:0/2433091522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x55a51b739470 0x55a51b745f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2433091522 >> 172.21.15.121:0/2433091522 conn(0x55a51b6600f0 msgr2=0x55a51b738890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2433091522 shutdown_connections 2024-06-10T00:38:36.371 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2f037ecb80 1 -- 172.21.15.121:0/2433091522 wait complete. 2024-06-10T00:38:36.372 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2f037ecb80 20 client.16911 trim_cache size 0 max 16384 2024-06-10T00:38:36.372 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:38:36.372 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 33, 2024-06-10T00:38:36.373 INFO:teuthology.orchestra.run.smithi121.stdout: "leader": 24479, 2024-06-10T00:38:36.373 INFO:teuthology.orchestra.run.smithi121.stdout: "set_version": 32, 2024-06-10T00:38:36.373 INFO:teuthology.orchestra.run.smithi121.stdout: "sets": {} 2024-06-10T00:38:36.373 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:38:36.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 -- 172.21.15.121:0/1020194621 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2edc0a2d90 msgr2=0x7f2edc0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:36.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2edc0a2d90 0x7f2edc0a5250 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009910 tx=0x7f2ee8008040 comp rx=0 tx=0).stop 2024-06-10T00:38:36.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 -- 172.21.15.121:0/1020194621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ef4108520 msgr2=0x7f2ef4101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:36.374 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ef4108520 0x7f2ef4101f10 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f2eec004e40 tx=0x7f2eec004f20 comp rx=0 tx=0).stop 2024-06-10T00:38:36.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 -- 172.21.15.121:0/1020194621 shutdown_connections 2024-06-10T00:38:36.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f2edc0a2d90 0x7f2edc0a5250 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ef4108520 0x7f2ef4101f10 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ef406ac00 0x7f2ef4101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 --2- 172.21.15.121:0/1020194621 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2ef406b5c0 0x7f2ef41019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:36.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 -- 172.21.15.121:0/1020194621 >> 172.21.15.121:0/1020194621 conn(0x7f2ef40773b0 msgr2=0x7f2ef4100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:36.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 -- 172.21.15.121:0/1020194621 shutdown_connections 2024-06-10T00:38:36.375 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:36.358+0000 7f2ef8f69700 1 -- 172.21.15.121:0/1020194621 wait complete. 2024-06-10T00:38:36.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:36 smithi135 ceph-mon[36424]: osdmap e109: 12 total, 12 up, 12 in 2024-06-10T00:38:36.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:36 smithi193 ceph-mon[36716]: osdmap e109: 12 total, 12 up, 12 in 2024-06-10T00:38:37.095 INFO:tasks.quiescer:done joining Quiescers 2024-06-10T00:38:37.096 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-06-10T00:38:37.106 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-06-10T00:38:37.106 INFO:teuthology.task.exec:Executing custom commands... 2024-06-10T00:38:37.107 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi121.front.sepia.ceph.com 2024-06-10T00:38:37.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-06-10T00:38:37.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-06-10T00:38:37.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-06-10T00:38:37.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.349+0000 7fe130c53640 1 -- 172.21.15.121:0/3633903458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe12c1008d0 msgr2=0x7fe12c100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.349+0000 7fe130c53640 1 --2- 172.21.15.121:0/3633903458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe12c1008d0 0x7fe12c100d50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe11c009cd0 tx=0x7fe11c02a8e0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.349+0000 7fe130c53640 1 -- 172.21.15.121:0/3633903458 shutdown_connections 2024-06-10T00:38:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.349+0000 7fe130c53640 1 --2- 172.21.15.121:0/3633903458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe12c101290 0x7fe12c10d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.349+0000 7fe130c53640 1 --2- 172.21.15.121:0/3633903458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe12c1008d0 0x7fe12c100d50 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.349+0000 7fe130c53640 1 --2- 172.21.15.121:0/3633903458 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe12c106890 0x7fe12c106c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.349+0000 7fe130c53640 1 -- 172.21.15.121:0/3633903458 >> 172.21.15.121:0/3633903458 conn(0x7fe12c077a20 msgr2=0x7fe12c077e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.349+0000 7fe130c53640 1 -- 172.21.15.121:0/3633903458 shutdown_connections 2024-06-10T00:38:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.349+0000 7fe130c53640 1 -- 172.21.15.121:0/3633903458 wait complete. 2024-06-10T00:38:37.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe130c53640 1 Processor -- start 2024-06-10T00:38:37.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe130c53640 1 -- start start 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe130c53640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe12c1008d0 0x7fe12c1970c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe130c53640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe12c101290 0x7fe12c197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe130c53640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe12c106890 0x7fe12c19b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe130c53640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c110ca0 con 0x7fe12c106890 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe130c53640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c110b20 con 0x7fe12c101290 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe130c53640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe12c110e20 con 0x7fe12c1008d0 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe12ad9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe12c106890 0x7fe12c19b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe12ad9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe12c106890 0x7fe12c19b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59504/0 (socket says 172.21.15.121:59504) 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe12ad9d640 1 -- 172.21.15.121:0/1972459741 learned_addr learned my addr 172.21.15.121:0/1972459741 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe129d9b640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe12c101290 0x7fe12c197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe12a59c640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe12c1008d0 0x7fe12c1970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe12ad9d640 1 -- 172.21.15.121:0/1972459741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe12c1008d0 msgr2=0x7fe12c1970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.350+0000 7fe12ad9d640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe12c1008d0 0x7fe12c1970c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.351+0000 7fe12ad9d640 1 -- 172.21.15.121:0/1972459741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe12c101290 msgr2=0x7fe12c197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.351+0000 7fe12ad9d640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe12c101290 0x7fe12c197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.351+0000 7fe12ad9d640 1 -- 172.21.15.121:0/1972459741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12c19c0b0 con 0x7fe12c106890 2024-06-10T00:38:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.351+0000 7fe129d9b640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe12c101290 0x7fe12c197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:38:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.351+0000 7fe12ad9d640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe12c106890 0x7fe12c19b980 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe124009fd0 tx=0x7fe12400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.351+0000 7fe11b7fe640 1 -- 172.21.15.121:0/1972459741 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe124011830 con 0x7fe12c106890 2024-06-10T00:38:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.351+0000 7fe130c53640 1 -- 172.21.15.121:0/1972459741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe12c19c3a0 con 0x7fe12c106890 2024-06-10T00:38:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.351+0000 7fe11b7fe640 1 -- 172.21.15.121:0/1972459741 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe124016070 con 0x7fe12c106890 2024-06-10T00:38:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.351+0000 7fe11b7fe640 1 -- 172.21.15.121:0/1972459741 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12400f2e0 con 0x7fe12c106890 2024-06-10T00:38:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.351+0000 7fe130c53640 1 -- 172.21.15.121:0/1972459741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe12c1a3c30 con 0x7fe12c106890 2024-06-10T00:38:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.352+0000 7fe130c53640 1 -- 172.21.15.121:0/1972459741 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe12c04f760 con 0x7fe12c106890 2024-06-10T00:38:37.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.353+0000 7fe11b7fe640 1 -- 172.21.15.121:0/1972459741 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe1240119d0 con 0x7fe12c106890 2024-06-10T00:38:37.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.354+0000 7fe11b7fe640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe1140a2da0 0x7fe1140a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:37.355 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.354+0000 7fe12a59c640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe1140a2da0 0x7fe1140a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:37.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.354+0000 7fe11b7fe640 1 -- 172.21.15.121:0/1972459741 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9525+0+0 (secure 0 0 0) 0x7fe1240cf4c0 con 0x7fe12c106890 2024-06-10T00:38:37.356 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.354+0000 7fe12a59c640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe1140a2da0 0x7fe1140a5260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe120005e10 tx=0x7fe120005da0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:37.357 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.357+0000 7fe11b7fe640 1 -- 172.21.15.121:0/1972459741 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe12409be00 con 0x7fe12c106890 2024-06-10T00:38:37.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:37 smithi135 ceph-mon[36424]: pgmap v330: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-06-10T00:38:37.554 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.554+0000 7fe130c53640 1 -- 172.21.15.121:0/1972459741 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fe12c103a20 con 0x7fe1140a2da0 2024-06-10T00:38:37.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.564+0000 7fe11b7fe640 1 -- 172.21.15.121:0/1972459741 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7fe12c103a20 con 0x7fe1140a2da0 2024-06-10T00:38:37.565 INFO:teuthology.orchestra.run.smithi121.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-06-10T00:00:00", "created": "2024-06-10T00:28:09", "first": "2024-06-10T00:29:00", "last": "2024-06-10T00:38:00", "last_pruned": "2024-06-10T00:38:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-06-10T00:38:37.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 -- 172.21.15.121:0/1972459741 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe1140a2da0 msgr2=0x7fe1140a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:37.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe1140a2da0 0x7fe1140a5260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe120005e10 tx=0x7fe120005da0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 -- 172.21.15.121:0/1972459741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe12c106890 msgr2=0x7fe12c19b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:37.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe12c106890 0x7fe12c19b980 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe124009fd0 tx=0x7fe12400b8e0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 -- 172.21.15.121:0/1972459741 shutdown_connections 2024-06-10T00:38:37.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe12c106890 0x7fe12c19b980 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.567 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe12c101290 0x7fe12c197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe1140a2da0 0x7fe1140a5260 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 --2- 172.21.15.121:0/1972459741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe12c1008d0 0x7fe12c1970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 -- 172.21.15.121:0/1972459741 >> 172.21.15.121:0/1972459741 conn(0x7fe12c077a20 msgr2=0x7fe12c103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:37.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 -- 172.21.15.121:0/1972459741 shutdown_connections 2024-06-10T00:38:37.568 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.566+0000 7fe130c53640 1 -- 172.21.15.121:0/1972459741 wait complete. 2024-06-10T00:38:37.577 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:37 smithi121 ceph-mon[32378]: pgmap v330: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-06-10T00:38:37.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-06-10T00:38:37.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:37 smithi193 ceph-mon[36716]: pgmap v330: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-06-10T00:38:37.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.683+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2586219951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb30100af0 msgr2=0x7ffb30100f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:37.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.683+0000 7ffb34fd5640 1 --2- 172.21.15.121:0/2586219951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb30100af0 0x7ffb30100f70 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ffb24009930 tx=0x7ffb2402a420 comp rx=0 tx=0).stop 2024-06-10T00:38:37.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.683+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2586219951 shutdown_connections 2024-06-10T00:38:37.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.683+0000 7ffb34fd5640 1 --2- 172.21.15.121:0/2586219951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb301014b0 0x7ffb3010d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.683+0000 7ffb34fd5640 1 --2- 172.21.15.121:0/2586219951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb30100af0 0x7ffb30100f70 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.683+0000 7ffb34fd5640 1 --2- 172.21.15.121:0/2586219951 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb301058d0 0x7ffb30105cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.683+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2586219951 >> 172.21.15.121:0/2586219951 conn(0x7ffb30077720 msgr2=0x7ffb30077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.683+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2586219951 shutdown_connections 2024-06-10T00:38:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.683+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2586219951 wait complete. 2024-06-10T00:38:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb34fd5640 1 Processor -- start 2024-06-10T00:38:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb34fd5640 1 -- start start 2024-06-10T00:38:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb34fd5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb30100af0 0x7ffb30198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb34fd5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb301014b0 0x7ffb30198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:37.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb34fd5640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb301058d0 0x7ffb3019cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb34fd5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffb30110da0 con 0x7ffb30100af0 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb34fd5640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ffb30110c20 con 0x7ffb301058d0 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb34fd5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffb30110f20 con 0x7ffb301014b0 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb2e59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb30100af0 0x7ffb30198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb2e59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb30100af0 0x7ffb30198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59528/0 (socket says 172.21.15.121:59528) 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb2e59c640 1 -- 172.21.15.121:0/2262032131 learned_addr learned my addr 172.21.15.121:0/2262032131 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.684+0000 7ffb2ed9d640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb301058d0 0x7ffb3019cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb2dd9b640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb301014b0 0x7ffb30198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb2e59c640 1 -- 172.21.15.121:0/2262032131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb301014b0 msgr2=0x7ffb30198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb2e59c640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb301014b0 0x7ffb30198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb2e59c640 1 -- 172.21.15.121:0/2262032131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb301058d0 msgr2=0x7ffb3019cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:37.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb2e59c640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb301058d0 0x7ffb3019cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb2e59c640 1 -- 172.21.15.121:0/2262032131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb3019d320 con 0x7ffb30100af0 2024-06-10T00:38:37.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb2ed9d640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb301058d0 0x7ffb3019cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:38:37.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb2e59c640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb30100af0 0x7ffb30198330 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000b4a0 tx=0x7ffb2000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:37.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb1f7fe640 1 -- 172.21.15.121:0/2262032131 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20014070 con 0x7ffb30100af0 2024-06-10T00:38:37.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb1f7fe640 1 -- 172.21.15.121:0/2262032131 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb20010af0 con 0x7ffb30100af0 2024-06-10T00:38:37.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2262032131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb3019d610 con 0x7ffb30100af0 2024-06-10T00:38:37.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb1f7fe640 1 -- 172.21.15.121:0/2262032131 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20017d10 con 0x7ffb30100af0 2024-06-10T00:38:37.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.685+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2262032131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb301a4ea0 con 0x7ffb30100af0 2024-06-10T00:38:37.689 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.686+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2262032131 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb3004f760 con 0x7ffb30100af0 2024-06-10T00:38:37.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.687+0000 7ffb1f7fe640 1 -- 172.21.15.121:0/2262032131 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ffb20017070 con 0x7ffb30100af0 2024-06-10T00:38:37.690 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.688+0000 7ffb1f7fe640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffb180a2da0 0x7ffb180a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:37.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.688+0000 7ffb2dd9b640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffb180a2da0 0x7ffb180a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:37.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.688+0000 7ffb1f7fe640 1 -- 172.21.15.121:0/2262032131 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9525+0+0 (secure 0 0 0) 0x7ffb200ceb50 con 0x7ffb30100af0 2024-06-10T00:38:37.691 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.688+0000 7ffb2dd9b640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffb180a2da0 0x7ffb180a5260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffb240096c0 tx=0x7ffb24000bc0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:37.693 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.693+0000 7ffb1f7fe640 1 -- 172.21.15.121:0/2262032131 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ffb2009b490 con 0x7ffb30100af0 2024-06-10T00:38:37.892 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.892+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2262032131 --> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7ffb3006a1f0 con 0x7ffb180a2da0 2024-06-10T00:38:37.900 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.900+0000 7ffb1f7fe640 1 -- 172.21.15.121:0/2262032131 <== mgr.15216 v2:172.21.15.121:6800/2387911981 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7ffb3006a1f0 con 0x7ffb180a2da0 2024-06-10T00:38:37.900 INFO:teuthology.orchestra.run.smithi121.stdout:/ 1m 6m3h 2024-06-10T00:38:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2262032131 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffb180a2da0 msgr2=0x7ffb180a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffb180a2da0 0x7ffb180a5260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffb240096c0 tx=0x7ffb24000bc0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2262032131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb30100af0 msgr2=0x7ffb30198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb30100af0 0x7ffb30198330 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000b4a0 tx=0x7ffb2000b960 comp rx=0 tx=0).stop 2024-06-10T00:38:37.903 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2262032131 shutdown_connections 2024-06-10T00:38:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb301058d0 0x7ffb3019cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ffb180a2da0 0x7ffb180a5260 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb301014b0 0x7ffb30198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 --2- 172.21.15.121:0/2262032131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb30100af0 0x7ffb30198330 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2262032131 >> 172.21.15.121:0/2262032131 conn(0x7ffb30077720 msgr2=0x7ffb30104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2262032131 shutdown_connections 2024-06-10T00:38:37.904 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:37.902+0000 7ffb34fd5640 1 -- 172.21.15.121:0/2262032131 wait complete. 2024-06-10T00:38:37.914 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-06-10T00:38:37.924 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-06-10T00:38:37.924 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/fwd_scrub.py", line 155, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/fwd_scrub.py", line 90, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-06-10T00:38:37.925 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-06-10T00:38:37.935 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-10T00:38:38.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.038+0000 7fea0f789640 1 -- 172.21.15.121:0/3989542307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea081082b0 msgr2=0x7fea0806a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.038+0000 7fea0f789640 1 --2- 172.21.15.121:0/3989542307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea081082b0 0x7fea0806a040 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009900 tx=0x7fe9f802a480 comp rx=0 tx=0).stop 2024-06-10T00:38:38.040 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.038+0000 7fea0f789640 1 -- 172.21.15.121:0/3989542307 shutdown_connections 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.038+0000 7fea0f789640 1 --2- 172.21.15.121:0/3989542307 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea0806a770 0x7fea0810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.038+0000 7fea0f789640 1 --2- 172.21.15.121:0/3989542307 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea081082b0 0x7fea0806a040 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.038+0000 7fea0f789640 1 --2- 172.21.15.121:0/3989542307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea08107990 0x7fea08107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.038+0000 7fea0f789640 1 -- 172.21.15.121:0/3989542307 >> 172.21.15.121:0/3989542307 conn(0x7fea080766a0 msgr2=0x7fea08076ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.038+0000 7fea0f789640 1 -- 172.21.15.121:0/3989542307 shutdown_connections 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.038+0000 7fea0f789640 1 -- 172.21.15.121:0/3989542307 wait complete. 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.038+0000 7fea0f789640 1 Processor -- start 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.038+0000 7fea0f789640 1 -- start start 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0f789640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea0806a770 0x7fea081982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0f789640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea08107990 0x7fea081987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0f789640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea081082b0 0x7fea0819cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0f789640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fea08111da0 con 0x7fea08107990 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0f789640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fea08111c20 con 0x7fea0806a770 2024-06-10T00:38:38.041 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0f789640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea08111f20 con 0x7fea081082b0 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0cd24640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea08107990 0x7fea081987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0cd24640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea08107990 0x7fea081987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59542/0 (socket says 172.21.15.121:59542) 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0cd24640 1 -- 172.21.15.121:0/2580772860 learned_addr learned my addr 172.21.15.121:0/2580772860 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0d525640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea0806a770 0x7fea081982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0dd26640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea081082b0 0x7fea0819cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0cd24640 1 -- 172.21.15.121:0/2580772860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea081082b0 msgr2=0x7fea0819cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0cd24640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea081082b0 0x7fea0819cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0cd24640 1 -- 172.21.15.121:0/2580772860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea0806a770 msgr2=0x7fea081982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0cd24640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea0806a770 0x7fea081982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0cd24640 1 -- 172.21.15.121:0/2580772860 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea0819d290 con 0x7fea08107990 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0d525640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea0806a770 0x7fea081982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0cd24640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea08107990 0x7fea081987e0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8006010 tx=0x7fe9f8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:38.042 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fe9fe7fc640 1 -- 172.21.15.121:0/2580772860 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f803f070 con 0x7fea08107990 2024-06-10T00:38:38.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fe9fe7fc640 1 -- 172.21.15.121:0/2580772860 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9f8031e30 con 0x7fea08107990 2024-06-10T00:38:38.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0f789640 1 -- 172.21.15.121:0/2580772860 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea0819d520 con 0x7fea08107990 2024-06-10T00:38:38.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fe9fe7fc640 1 -- 172.21.15.121:0/2580772860 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8035030 con 0x7fea08107990 2024-06-10T00:38:38.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.039+0000 7fea0f789640 1 -- 172.21.15.121:0/2580772860 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea081a4e10 con 0x7fea08107990 2024-06-10T00:38:38.043 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.040+0000 7fea0f789640 1 -- 172.21.15.121:0/2580772860 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea081030b0 con 0x7fea08107990 2024-06-10T00:38:38.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.042+0000 7fe9fe7fc640 1 -- 172.21.15.121:0/2580772860 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe9f8042070 con 0x7fea08107990 2024-06-10T00:38:38.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.043+0000 7fe9fe7fc640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe9f40a2d00 0x7fe9f40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.043+0000 7fe9fe7fc640 1 -- 172.21.15.121:0/2580772860 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9525+0+0 (secure 0 0 0) 0x7fe9f80f0860 con 0x7fea08107990 2024-06-10T00:38:38.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.043+0000 7fea0d525640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe9f40a2d00 0x7fe9f40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.044 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.043+0000 7fea0d525640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe9f40a2d00 0x7fe9f40a51c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fea04009930 tx=0x7fea04008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:38.046 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.046+0000 7fe9fe7fc640 1 -- 172.21.15.121:0/2580772860 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe9f80bd090 con 0x7fea08107990 2024-06-10T00:38:38.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.225+0000 7fea0f789640 1 -- 172.21.15.121:0/2580772860 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea08068c10 con 0x7fea08107990 2024-06-10T00:38:38.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.226+0000 7fe9fe7fc640 1 -- 172.21.15.121:0/2580772860 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+25486 (secure 0 0 0) 0x7fe9f80bc990 con 0x7fea08107990 2024-06-10T00:38:38.227 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:38:38.227 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":109,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:38:35.234268+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":108,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":108,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/4266501776":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/4078460254":"2024-06-11T00:27:57.451553+0000","172.21.15.121:6801/3506007074":"2024-06-11T00:27:57.451553+0000","172.21.15.121:6800/3506007074":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/58990727":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/1002791322":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:38:38.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 -- 172.21.15.121:0/2580772860 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe9f40a2d00 msgr2=0x7fe9f40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe9f40a2d00 0x7fe9f40a51c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fea04009930 tx=0x7fea04008040 comp rx=0 tx=0).stop 2024-06-10T00:38:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 -- 172.21.15.121:0/2580772860 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea08107990 msgr2=0x7fea081987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea08107990 0x7fea081987e0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8006010 tx=0x7fe9f8003ba0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 -- 172.21.15.121:0/2580772860 shutdown_connections 2024-06-10T00:38:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe9f40a2d00 0x7fe9f40a51c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea081082b0 0x7fea0819cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea08107990 0x7fea081987e0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 --2- 172.21.15.121:0/2580772860 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fea0806a770 0x7fea081982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 -- 172.21.15.121:0/2580772860 >> 172.21.15.121:0/2580772860 conn(0x7fea080766a0 msgr2=0x7fea080ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 -- 172.21.15.121:0/2580772860 shutdown_connections 2024-06-10T00:38:38.229 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.228+0000 7fea0f789640 1 -- 172.21.15.121:0/2580772860 wait complete. 2024-06-10T00:38:38.240 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-10T00:19:04.337527+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-06-10T00:23:53.357210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-06-10T00:23:56.483170+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 108, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-06-10T00:24:07.390280+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 108, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-06-10T00:38:38.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-06-10T00:38:38.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.344+0000 7f489f488640 1 -- 172.21.15.121:0/3086483713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4898107980 msgr2=0x7f4898107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.344+0000 7f489f488640 1 --2- 172.21.15.121:0/3086483713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4898107980 0x7f4898107d60 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f4888009930 tx=0x7f488802a420 comp rx=0 tx=0).stop 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.344+0000 7f489f488640 1 -- 172.21.15.121:0/3086483713 shutdown_connections 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.344+0000 7f489f488640 1 --2- 172.21.15.121:0/3086483713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f489806a960 0x7f489810e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.344+0000 7f489f488640 1 --2- 172.21.15.121:0/3086483713 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4898069fa0 0x7f489806a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.344+0000 7f489f488640 1 --2- 172.21.15.121:0/3086483713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4898107980 0x7f4898107d60 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.344+0000 7f489f488640 1 -- 172.21.15.121:0/3086483713 >> 172.21.15.121:0/3086483713 conn(0x7f48980766a0 msgr2=0x7f4898076ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.344+0000 7f489f488640 1 -- 172.21.15.121:0/3086483713 shutdown_connections 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.344+0000 7f489f488640 1 -- 172.21.15.121:0/3086483713 wait complete. 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489f488640 1 Processor -- start 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489f488640 1 -- start start 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489f488640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4898069fa0 0x7f48981982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489f488640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f489806a960 0x7f48981987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489f488640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4898107980 0x7f489819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489f488640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4898111d90 con 0x7f4898069fa0 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489f488640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f4898111c10 con 0x7f4898107980 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489f488640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4898111f10 con 0x7f489806a960 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489d224640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4898069fa0 0x7f48981982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489d224640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4898069fa0 0x7f48981982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59568/0 (socket says 172.21.15.121:59568) 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489d224640 1 -- 172.21.15.121:0/1788022707 learned_addr learned my addr 172.21.15.121:0/1788022707 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489da25640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4898107980 0x7f489819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489d224640 1 -- 172.21.15.121:0/1788022707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f489806a960 msgr2=0x7f48981987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489d224640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f489806a960 0x7f48981987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489d224640 1 -- 172.21.15.121:0/1788022707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4898107980 msgr2=0x7f489819cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489d224640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4898107980 0x7f489819cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489d224640 1 -- 172.21.15.121:0/1788022707 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f489819d250 con 0x7f4898069fa0 2024-06-10T00:38:38.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.345+0000 7f489da25640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4898107980 0x7f489819cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:38:38.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.346+0000 7f489d224640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4898069fa0 0x7f48981982b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f4888009230 tx=0x7f4888004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:38.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.346+0000 7f488e7fc640 1 -- 172.21.15.121:0/1788022707 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488803f030 con 0x7f4898069fa0 2024-06-10T00:38:38.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.346+0000 7f489f488640 1 -- 172.21.15.121:0/1788022707 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f489819d4e0 con 0x7f4898069fa0 2024-06-10T00:38:38.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.346+0000 7f488e7fc640 1 -- 172.21.15.121:0/1788022707 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4888003ad0 con 0x7f4898069fa0 2024-06-10T00:38:38.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.346+0000 7f488e7fc640 1 -- 172.21.15.121:0/1788022707 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888035070 con 0x7f4898069fa0 2024-06-10T00:38:38.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.346+0000 7f489f488640 1 -- 172.21.15.121:0/1788022707 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48981a4dd0 con 0x7f4898069fa0 2024-06-10T00:38:38.349 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.347+0000 7f489f488640 1 -- 172.21.15.121:0/1788022707 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48981030d0 con 0x7f4898069fa0 2024-06-10T00:38:38.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.348+0000 7f488e7fc640 1 -- 172.21.15.121:0/1788022707 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f4888042030 con 0x7f4898069fa0 2024-06-10T00:38:38.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.349+0000 7f488e7fc640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f48840a2d80 0x7f48840a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.349+0000 7f489ca23640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f48840a2d80 0x7f48840a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.349+0000 7f488e7fc640 1 -- 172.21.15.121:0/1788022707 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9525+0+0 (secure 0 0 0) 0x7f48880f0240 con 0x7f4898069fa0 2024-06-10T00:38:38.350 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.349+0000 7f489ca23640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f48840a2d80 0x7f48840a5240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4894005fd0 tx=0x7f4894005850 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:38.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.352+0000 7f488e7fc640 1 -- 172.21.15.121:0/1788022707 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f48880bca10 con 0x7f4898069fa0 2024-06-10T00:38:38.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:38 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2580772860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:38:38.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.530+0000 7f489f488640 1 -- 172.21.15.121:0/1788022707 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4898109ee0 con 0x7f4898069fa0 2024-06-10T00:38:38.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.531+0000 7f488e7fc640 1 -- 172.21.15.121:0/1788022707 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) ==== 93+0+10 (secure 0 0 0) 0x7f48880bc310 con 0x7f4898069fa0 2024-06-10T00:38:38.531 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 1 2024-06-10T00:38:38.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 -- 172.21.15.121:0/1788022707 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f48840a2d80 msgr2=0x7f48840a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f48840a2d80 0x7f48840a5240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4894005fd0 tx=0x7f4894005850 comp rx=0 tx=0).stop 2024-06-10T00:38:38.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 -- 172.21.15.121:0/1788022707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4898069fa0 msgr2=0x7f48981982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4898069fa0 0x7f48981982b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f4888009230 tx=0x7f4888004270 comp rx=0 tx=0).stop 2024-06-10T00:38:38.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 -- 172.21.15.121:0/1788022707 shutdown_connections 2024-06-10T00:38:38.533 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4898107980 0x7f489819cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f48840a2d80 0x7f48840a5240 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f489806a960 0x7f48981987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 --2- 172.21.15.121:0/1788022707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4898069fa0 0x7f48981982b0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 -- 172.21.15.121:0/1788022707 >> 172.21.15.121:0/1788022707 conn(0x7f48980766a0 msgr2=0x7f48980fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:38.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 -- 172.21.15.121:0/1788022707 shutdown_connections 2024-06-10T00:38:38.534 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.533+0000 7f489f488640 1 -- 172.21.15.121:0/1788022707 wait complete. 2024-06-10T00:38:38.543 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:38 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2580772860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:38:38.544 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-06-10T00:38:38.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:38 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2580772860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:38:38.652 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.647+0000 7fc3aaa76640 1 -- 172.21.15.121:0/2846975972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a4100ef0 msgr2=0x7fc3a4106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.647+0000 7fc3aaa76640 1 --2- 172.21.15.121:0/2846975972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a4100ef0 0x7fc3a4106620 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fc390009900 tx=0x7fc39002a410 comp rx=0 tx=0).stop 2024-06-10T00:38:38.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.647+0000 7fc3aaa76640 1 -- 172.21.15.121:0/2846975972 shutdown_connections 2024-06-10T00:38:38.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.647+0000 7fc3aaa76640 1 --2- 172.21.15.121:0/2846975972 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3a4106d00 0x7fc3a410e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.647+0000 7fc3aaa76640 1 --2- 172.21.15.121:0/2846975972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a4100ef0 0x7fc3a4106620 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.653 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.647+0000 7fc3aaa76640 1 --2- 172.21.15.121:0/2846975972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3a4100540 0x7fc3a4100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.647+0000 7fc3aaa76640 1 -- 172.21.15.121:0/2846975972 >> 172.21.15.121:0/2846975972 conn(0x7fc3a4075b30 msgr2=0x7fc3a4075f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:38.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.647+0000 7fc3aaa76640 1 -- 172.21.15.121:0/2846975972 shutdown_connections 2024-06-10T00:38:38.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.647+0000 7fc3aaa76640 1 -- 172.21.15.121:0/2846975972 wait complete. 2024-06-10T00:38:38.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3aaa76640 1 Processor -- start 2024-06-10T00:38:38.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3aaa76640 1 -- start start 2024-06-10T00:38:38.654 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3aaa76640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3a4100540 0x7fc3a4198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3aaa76640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3a4106d00 0x7fc3a4198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3aaa76640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a419cbf0 0x7fc3a419d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3aaa76640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a4111df0 con 0x7fc3a419cbf0 2024-06-10T00:38:38.655 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3aaa76640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a4111c70 con 0x7fc3a4100540 2024-06-10T00:38:38.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3aaa76640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a4111f70 con 0x7fc3a4106d00 2024-06-10T00:38:38.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3a9013640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a419cbf0 0x7fc3a419d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3a9013640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a419cbf0 0x7fc3a419d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59582/0 (socket says 172.21.15.121:59582) 2024-06-10T00:38:38.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3a9013640 1 -- 172.21.15.121:0/580890176 learned_addr learned my addr 172.21.15.121:0/580890176 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:38.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3a8812640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3a4100540 0x7fc3a4198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.656 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3a3fff640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3a4106d00 0x7fc3a4198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3a9013640 1 -- 172.21.15.121:0/580890176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3a4106d00 msgr2=0x7fc3a4198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3a9013640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3a4106d00 0x7fc3a4198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3a9013640 1 -- 172.21.15.121:0/580890176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3a4100540 msgr2=0x7fc3a4198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3a9013640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3a4100540 0x7fc3a4198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.657 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.648+0000 7fc3a9013640 1 -- 172.21.15.121:0/580890176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a419d6e0 con 0x7fc3a419cbf0 2024-06-10T00:38:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.649+0000 7fc3a9013640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a419cbf0 0x7fc3a419d0a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00eaa0 tx=0x7fc39c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.649+0000 7fc3a1ffb640 1 -- 172.21.15.121:0/580890176 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39c00cdb0 con 0x7fc3a419cbf0 2024-06-10T00:38:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.649+0000 7fc3aaa76640 1 -- 172.21.15.121:0/580890176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a419d9d0 con 0x7fc3a419cbf0 2024-06-10T00:38:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.649+0000 7fc3a1ffb640 1 -- 172.21.15.121:0/580890176 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc39c005020 con 0x7fc3a419cbf0 2024-06-10T00:38:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.649+0000 7fc3a1ffb640 1 -- 172.21.15.121:0/580890176 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39c0116b0 con 0x7fc3a419cbf0 2024-06-10T00:38:38.658 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.649+0000 7fc3aaa76640 1 -- 172.21.15.121:0/580890176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a41a5140 con 0x7fc3a419cbf0 2024-06-10T00:38:38.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.650+0000 7fc3aaa76640 1 -- 172.21.15.121:0/580890176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3a4068c10 con 0x7fc3a419cbf0 2024-06-10T00:38:38.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.651+0000 7fc3a1ffb640 1 -- 172.21.15.121:0/580890176 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fc39c005560 con 0x7fc3a419cbf0 2024-06-10T00:38:38.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.652+0000 7fc3a1ffb640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc3940a2d00 0x7fc3940a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.652+0000 7fc3a8812640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc3940a2d00 0x7fc3940a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.659 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.652+0000 7fc3a1ffb640 1 -- 172.21.15.121:0/580890176 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9525+0+0 (secure 0 0 0) 0x7fc39c0cf9a0 con 0x7fc3a419cbf0 2024-06-10T00:38:38.660 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.652+0000 7fc3a8812640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc3940a2d00 0x7fc3940a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc398005fd0 tx=0x7fc39800b390 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:38.661 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.656+0000 7fc3a1ffb640 1 -- 172.21.15.121:0/580890176 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc39c016030 con 0x7fc3a419cbf0 2024-06-10T00:38:38.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.834+0000 7fc3aaa76640 1 -- 172.21.15.121:0/580890176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc3a4100ef0 con 0x7fc3a419cbf0 2024-06-10T00:38:38.835 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.835+0000 7fc3a1ffb640 1 -- 172.21.15.121:0/580890176 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) ==== 104+0+11 (secure 0 0 0) 0x7fc39c09bb80 con 0x7fc3a419cbf0 2024-06-10T00:38:38.835 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-06-10T00:38:38.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 -- 172.21.15.121:0/580890176 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc3940a2d00 msgr2=0x7fc3940a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.837 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc3940a2d00 0x7fc3940a51c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc398005fd0 tx=0x7fc39800b390 comp rx=0 tx=0).stop 2024-06-10T00:38:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 -- 172.21.15.121:0/580890176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a419cbf0 msgr2=0x7fc3a419d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a419cbf0 0x7fc3a419d0a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00eaa0 tx=0x7fc39c00ef60 comp rx=0 tx=0).stop 2024-06-10T00:38:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 -- 172.21.15.121:0/580890176 shutdown_connections 2024-06-10T00:38:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3a419cbf0 0x7fc3a419d0a0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fc3940a2d00 0x7fc3940a51c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3a4106d00 0x7fc3a4198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 --2- 172.21.15.121:0/580890176 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc3a4100540 0x7fc3a4198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 -- 172.21.15.121:0/580890176 >> 172.21.15.121:0/580890176 conn(0x7fc3a4075b30 msgr2=0x7fc3a40fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 -- 172.21.15.121:0/580890176 shutdown_connections 2024-06-10T00:38:38.838 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.837+0000 7fc3aaa76640 1 -- 172.21.15.121:0/580890176 wait complete. 2024-06-10T00:38:38.848 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-06-10T00:38:38.954 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.952+0000 7f0ea84a5640 1 -- 172.21.15.121:0/4102768234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ea006a020 msgr2=0x7f0ea006a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.952+0000 7f0ea84a5640 1 --2- 172.21.15.121:0/4102768234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ea006a020 0x7f0ea006a4a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009930 tx=0x7f0e9002a420 comp rx=0 tx=0).stop 2024-06-10T00:38:38.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.952+0000 7f0ea84a5640 1 -- 172.21.15.121:0/4102768234 shutdown_connections 2024-06-10T00:38:38.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.952+0000 7f0ea84a5640 1 --2- 172.21.15.121:0/4102768234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ea006a9e0 0x7f0ea010e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.952+0000 7f0ea84a5640 1 --2- 172.21.15.121:0/4102768234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ea006a020 0x7f0ea006a4a0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.952+0000 7f0ea84a5640 1 --2- 172.21.15.121:0/4102768234 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ea0107990 0x7f0ea0107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.952+0000 7f0ea84a5640 1 -- 172.21.15.121:0/4102768234 >> 172.21.15.121:0/4102768234 conn(0x7f0ea0077900 msgr2=0x7f0ea0077d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:38.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.952+0000 7f0ea84a5640 1 -- 172.21.15.121:0/4102768234 shutdown_connections 2024-06-10T00:38:38.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.952+0000 7f0ea84a5640 1 -- 172.21.15.121:0/4102768234 wait complete. 2024-06-10T00:38:38.955 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea84a5640 1 Processor -- start 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea84a5640 1 -- start start 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea84a5640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ea006a020 0x7f0ea01982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea84a5640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ea006a9e0 0x7f0ea0198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea84a5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ea0107990 0x7f0ea019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea84a5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea0111da0 con 0x7f0ea0107990 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea84a5640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea0111c20 con 0x7f0ea006a9e0 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea84a5640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea0111f20 con 0x7f0ea006a020 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea6a42640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ea0107990 0x7f0ea019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea6a42640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ea0107990 0x7f0ea019cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59596/0 (socket says 172.21.15.121:59596) 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea6a42640 1 -- 172.21.15.121:0/369973920 learned_addr learned my addr 172.21.15.121:0/369973920 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea5a40640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ea006a9e0 0x7f0ea0198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea6241640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ea006a020 0x7f0ea01982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.956 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea6a42640 1 -- 172.21.15.121:0/369973920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ea006a020 msgr2=0x7f0ea01982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea6a42640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ea006a020 0x7f0ea01982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea6a42640 1 -- 172.21.15.121:0/369973920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ea006a9e0 msgr2=0x7f0ea0198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea6a42640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ea006a9e0 0x7f0ea0198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.953+0000 7f0ea6a42640 1 -- 172.21.15.121:0/369973920 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea019d260 con 0x7f0ea0107990 2024-06-10T00:38:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.954+0000 7f0ea6a42640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ea0107990 0x7f0ea019cb30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800eaa0 tx=0x7f0e9800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.954+0000 7f0e977fe640 1 -- 172.21.15.121:0/369973920 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9800cdb0 con 0x7f0ea0107990 2024-06-10T00:38:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.954+0000 7f0e977fe640 1 -- 172.21.15.121:0/369973920 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e98005020 con 0x7f0ea0107990 2024-06-10T00:38:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.954+0000 7f0ea84a5640 1 -- 172.21.15.121:0/369973920 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea019d550 con 0x7f0ea0107990 2024-06-10T00:38:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.954+0000 7f0e977fe640 1 -- 172.21.15.121:0/369973920 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e980116d0 con 0x7f0ea0107990 2024-06-10T00:38:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.954+0000 7f0ea84a5640 1 -- 172.21.15.121:0/369973920 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea01a4de0 con 0x7f0ea0107990 2024-06-10T00:38:38.957 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.955+0000 7f0ea84a5640 1 -- 172.21.15.121:0/369973920 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ea0068c10 con 0x7f0ea0107990 2024-06-10T00:38:38.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.956+0000 7f0e977fe640 1 -- 172.21.15.121:0/369973920 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0e980054d0 con 0x7f0ea0107990 2024-06-10T00:38:38.958 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.956+0000 7f0e977fe640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0e8c0a2d00 0x7f0e8c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:38.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.957+0000 7f0ea6241640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0e8c0a2d00 0x7f0e8c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:38.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.957+0000 7f0e977fe640 1 -- 172.21.15.121:0/369973920 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9525+0+0 (secure 0 0 0) 0x7f0e980cfb10 con 0x7f0ea0107990 2024-06-10T00:38:38.959 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.957+0000 7f0ea6241640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0e8c0a2d00 0x7f0e8c0a51c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c005e00 tx=0x7f0e9c005c50 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:38.961 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:38.961+0000 7f0e977fe640 1 -- 172.21.15.121:0/369973920 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0e98016030 con 0x7f0ea0107990 2024-06-10T00:38:39.135 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.135+0000 7f0ea84a5640 1 -- 172.21.15.121:0/369973920 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0ea002d2e0 con 0x7f0ea0107990 2024-06-10T00:38:39.136 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.135+0000 7f0e977fe640 1 -- 172.21.15.121:0/369973920 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) ==== 100+0+11 (secure 0 0 0) 0x7f0e9809c340 con 0x7f0ea0107990 2024-06-10T00:38:39.136 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 64 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 -- 172.21.15.121:0/369973920 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0e8c0a2d00 msgr2=0x7f0e8c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0e8c0a2d00 0x7f0e8c0a51c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c005e00 tx=0x7f0e9c005c50 comp rx=0 tx=0).stop 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 -- 172.21.15.121:0/369973920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ea0107990 msgr2=0x7f0ea019cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ea0107990 0x7f0ea019cb30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800eaa0 tx=0x7f0e9800ef60 comp rx=0 tx=0).stop 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 -- 172.21.15.121:0/369973920 shutdown_connections 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ea0107990 0x7f0ea019cb30 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0ea006a9e0 0x7f0ea0198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0e8c0a2d00 0x7f0e8c0a51c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 --2- 172.21.15.121:0/369973920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ea006a020 0x7f0ea01982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 -- 172.21.15.121:0/369973920 >> 172.21.15.121:0/369973920 conn(0x7f0ea0077900 msgr2=0x7f0ea00fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 -- 172.21.15.121:0/369973920 shutdown_connections 2024-06-10T00:38:39.138 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.137+0000 7f0ea84a5640 1 -- 172.21.15.121:0/369973920 wait complete. 2024-06-10T00:38:39.149 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-06-10T00:38:39.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.254+0000 7f3f44c3f640 1 -- 172.21.15.121:0/714804966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f40101960 msgr2=0x7f3f40101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:39.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.254+0000 7f3f44c3f640 1 --2- 172.21.15.121:0/714804966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f40101960 0x7f3f40101de0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009930 tx=0x7f3f3402a420 comp rx=0 tx=0).stop 2024-06-10T00:38:39.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.254+0000 7f3f44c3f640 1 -- 172.21.15.121:0/714804966 shutdown_connections 2024-06-10T00:38:39.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.254+0000 7f3f44c3f640 1 --2- 172.21.15.121:0/714804966 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40102320 0x7f3f4010e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.254+0000 7f3f44c3f640 1 --2- 172.21.15.121:0/714804966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f40101960 0x7f3f40101de0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.254+0000 7f3f44c3f640 1 --2- 172.21.15.121:0/714804966 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f40107970 0x7f3f40107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.254+0000 7f3f44c3f640 1 -- 172.21.15.121:0/714804966 >> 172.21.15.121:0/714804966 conn(0x7f3f400fd410 msgr2=0x7f3f400ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:39.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.254+0000 7f3f44c3f640 1 -- 172.21.15.121:0/714804966 shutdown_connections 2024-06-10T00:38:39.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.254+0000 7f3f44c3f640 1 -- 172.21.15.121:0/714804966 wait complete. 2024-06-10T00:38:39.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.255+0000 7f3f44c3f640 1 Processor -- start 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.255+0000 7f3f44c3f640 1 -- start start 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.255+0000 7f3f44c3f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f40101960 0x7f3f401982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.255+0000 7f3f44c3f640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40102320 0x7f3f40198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.255+0000 7f3f44c3f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f40107970 0x7f3f4019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.255+0000 7f3f44c3f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3f40111d50 con 0x7f3f40101960 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.255+0000 7f3f44c3f640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f3f40111bd0 con 0x7f3f40102320 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.255+0000 7f3f44c3f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3f40111ed0 con 0x7f3f40107970 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.255+0000 7f3f3e59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f40101960 0x7f3f401982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.255+0000 7f3f3e59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f40101960 0x7f3f401982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40020/0 (socket says 172.21.15.121:40020) 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.255+0000 7f3f3e59c640 1 -- 172.21.15.121:0/707340812 learned_addr learned my addr 172.21.15.121:0/707340812 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f3dd9b640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40102320 0x7f3f40198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:39.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f3ed9d640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f40107970 0x7f3f4019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f3e59c640 1 -- 172.21.15.121:0/707340812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f40107970 msgr2=0x7f3f4019cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f3e59c640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f40107970 0x7f3f4019cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f3e59c640 1 -- 172.21.15.121:0/707340812 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40102320 msgr2=0x7f3f40198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f3e59c640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40102320 0x7f3f40198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f3e59c640 1 -- 172.21.15.121:0/707340812 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4019d2e0 con 0x7f3f40101960 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f3e59c640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f40101960 0x7f3f401982f0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000d950 tx=0x7f3f3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f2f7fe640 1 -- 172.21.15.121:0/707340812 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f30017030 con 0x7f3f40101960 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f44c3f640 1 -- 172.21.15.121:0/707340812 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4019d5d0 con 0x7f3f40101960 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f2f7fe640 1 -- 172.21.15.121:0/707340812 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f30004020 con 0x7f3f40101960 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f2f7fe640 1 -- 172.21.15.121:0/707340812 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f300047a0 con 0x7f3f40101960 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.256+0000 7f3f44c3f640 1 -- 172.21.15.121:0/707340812 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f401a4e60 con 0x7f3f40101960 2024-06-10T00:38:39.259 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.257+0000 7f3f44c3f640 1 -- 172.21.15.121:0/707340812 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f400687c0 con 0x7f3f40101960 2024-06-10T00:38:39.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.258+0000 7f3f2f7fe640 1 -- 172.21.15.121:0/707340812 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3f300041d0 con 0x7f3f40101960 2024-06-10T00:38:39.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.259+0000 7f3f2f7fe640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3f280a2d00 0x7f3f280a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:39.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.259+0000 7f3f3dd9b640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3f280a2d00 0x7f3f280a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:39.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.259+0000 7f3f2f7fe640 1 -- 172.21.15.121:0/707340812 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9525+0+0 (secure 0 0 0) 0x7f3f300cf770 con 0x7f3f40101960 2024-06-10T00:38:39.260 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.259+0000 7f3f3dd9b640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3f280a2d00 0x7f3f280a51c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009900 tx=0x7f3f340058d0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:39.262 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.262+0000 7f3f2f7fe640 1 -- 172.21.15.121:0/707340812 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3f3009bfa0 con 0x7f3f40101960 2024-06-10T00:38:39.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.438+0000 7f3f44c3f640 1 -- 172.21.15.121:0/707340812 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3f40199640 con 0x7f3f40101960 2024-06-10T00:38:39.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.438+0000 7f3f2f7fe640 1 -- 172.21.15.121:0/707340812 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v109) ==== 103+0+11 (secure 0 0 0) 0x7f3f3009b8a0 con 0x7f3f40101960 2024-06-10T00:38:39.439 INFO:teuthology.orchestra.run.smithi121.stdout:pg_num: 32 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 -- 172.21.15.121:0/707340812 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3f280a2d00 msgr2=0x7f3f280a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3f280a2d00 0x7f3f280a51c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009900 tx=0x7f3f340058d0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 -- 172.21.15.121:0/707340812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f40101960 msgr2=0x7f3f401982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f40101960 0x7f3f401982f0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000d950 tx=0x7f3f3000de10 comp rx=0 tx=0).stop 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 -- 172.21.15.121:0/707340812 shutdown_connections 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f3f280a2d00 0x7f3f280a51c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f40107970 0x7f3f4019cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3f40102320 0x7f3f40198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 --2- 172.21.15.121:0/707340812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f40101960 0x7f3f401982f0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 -- 172.21.15.121:0/707340812 >> 172.21.15.121:0/707340812 conn(0x7f3f400fd410 msgr2=0x7f3f400fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 -- 172.21.15.121:0/707340812 shutdown_connections 2024-06-10T00:38:39.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.440+0000 7f3f44c3f640 1 -- 172.21.15.121:0/707340812 wait complete. 2024-06-10T00:38:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:39 smithi121 ceph-mon[32378]: from='client.16917 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:38:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:39 smithi121 ceph-mon[32378]: pgmap v331: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-06-10T00:38:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:39 smithi121 ceph-mon[32378]: from='client.16923 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:38:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:39 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1788022707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-10T00:38:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:39 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/580890176' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-10T00:38:39.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:39 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/369973920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-10T00:38:39.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-06-10T00:38:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:39 smithi135 ceph-mon[36424]: from='client.16917 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:38:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:39 smithi135 ceph-mon[36424]: pgmap v331: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-06-10T00:38:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:39 smithi135 ceph-mon[36424]: from='client.16923 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:38:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:39 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1788022707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-10T00:38:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:39 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/580890176' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-10T00:38:39.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:39 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/369973920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-10T00:38:39.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.593+0000 7f35a67c1640 1 -- 172.21.15.121:0/2535895256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a0100870 msgr2=0x7f35a0100cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:39.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.593+0000 7f35a67c1640 1 --2- 172.21.15.121:0/2535895256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a0100870 0x7f35a0100cf0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f3594009930 tx=0x7f359402a420 comp rx=0 tx=0).stop 2024-06-10T00:38:39.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.594+0000 7f35a67c1640 1 -- 172.21.15.121:0/2535895256 shutdown_connections 2024-06-10T00:38:39.597 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.594+0000 7f35a67c1640 1 --2- 172.21.15.121:0/2535895256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35a0101230 0x7f35a010d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.594+0000 7f35a67c1640 1 --2- 172.21.15.121:0/2535895256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a0100870 0x7f35a0100cf0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.594+0000 7f35a67c1640 1 --2- 172.21.15.121:0/2535895256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35a0106880 0x7f35a0106c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.594+0000 7f35a67c1640 1 -- 172.21.15.121:0/2535895256 >> 172.21.15.121:0/2535895256 conn(0x7f35a0076c20 msgr2=0x7f35a0077030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.594+0000 7f35a67c1640 1 -- 172.21.15.121:0/2535895256 shutdown_connections 2024-06-10T00:38:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.594+0000 7f35a67c1640 1 -- 172.21.15.121:0/2535895256 wait complete. 2024-06-10T00:38:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.594+0000 7f35a67c1640 1 Processor -- start 2024-06-10T00:38:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.594+0000 7f35a67c1640 1 -- start start 2024-06-10T00:38:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f35a67c1640 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35a0100870 0x7f35a01991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f35a67c1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a0101230 0x7f35a01996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f35a67c1640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35a0106880 0x7f35a01a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:39.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f35a67c1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f35a0110c90 con 0x7f35a0101230 2024-06-10T00:38:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f35a67c1640 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f35a0110b10 con 0x7f35a0100870 2024-06-10T00:38:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f35a67c1640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f35a0110e10 con 0x7f35a0106880 2024-06-10T00:38:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a0101230 0x7f35a01996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359f7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a0101230 0x7f35a01996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40034/0 (socket says 172.21.15.121:40034) 2024-06-10T00:38:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359f7fe640 1 -- 172.21.15.121:0/969643897 learned_addr learned my addr 172.21.15.121:0/969643897 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:38:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359ffff640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35a0100870 0x7f35a01991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f35a4d5e640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35a0106880 0x7f35a01a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359f7fe640 1 -- 172.21.15.121:0/969643897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35a0106880 msgr2=0x7f35a01a4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359f7fe640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35a0106880 0x7f35a01a4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359f7fe640 1 -- 172.21.15.121:0/969643897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35a0100870 msgr2=0x7f35a01991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:39.599 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359f7fe640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35a0100870 0x7f35a01991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359f7fe640 1 -- 172.21.15.121:0/969643897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a01a4770 con 0x7f35a0101230 2024-06-10T00:38:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359f7fe640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a0101230 0x7f35a01996f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3594009a50 tx=0x7f3594004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359d7fa640 1 -- 172.21.15.121:0/969643897 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359403f030 con 0x7f35a0101230 2024-06-10T00:38:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359d7fa640 1 -- 172.21.15.121:0/969643897 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3594003ce0 con 0x7f35a0101230 2024-06-10T00:38:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f35a67c1640 1 -- 172.21.15.121:0/969643897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a01a4940 con 0x7f35a0101230 2024-06-10T00:38:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f359d7fa640 1 -- 172.21.15.121:0/969643897 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594035070 con 0x7f35a0101230 2024-06-10T00:38:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.595+0000 7f35a67c1640 1 -- 172.21.15.121:0/969643897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a01a4da0 con 0x7f35a0101230 2024-06-10T00:38:39.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.596+0000 7f35a67c1640 1 -- 172.21.15.121:0/969643897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35a0101fb0 con 0x7f35a0101230 2024-06-10T00:38:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.598+0000 7f359d7fa640 1 -- 172.21.15.121:0/969643897 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f3594042030 con 0x7f35a0101230 2024-06-10T00:38:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.598+0000 7f359d7fa640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35880a2d00 0x7f35880a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:38:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.598+0000 7f359ffff640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35880a2d00 0x7f35880a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:38:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.598+0000 7f359d7fa640 1 -- 172.21.15.121:0/969643897 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 9525+0+0 (secure 0 0 0) 0x7f35940efe70 con 0x7f35a0101230 2024-06-10T00:38:39.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.599+0000 7f359ffff640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35880a2d00 0x7f35880a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3590009890 tx=0x7f3590008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:38:39.602 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.603+0000 7f359d7fa640 1 -- 172.21.15.121:0/969643897 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f35940bc6a0 con 0x7f35a0101230 2024-06-10T00:38:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:39 smithi193 ceph-mon[36716]: from='client.16917 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:38:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:39 smithi193 ceph-mon[36716]: pgmap v331: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-06-10T00:38:39.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:39 smithi193 ceph-mon[36716]: from='client.16923 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-10T00:38:39.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:39 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1788022707' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-10T00:38:39.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:39 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/580890176' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-06-10T00:38:39.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:39 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/369973920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-06-10T00:38:39.827 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.826+0000 7f35a67c1640 1 -- 172.21.15.121:0/969643897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f35a0108870 con 0x7f35a0101230 2024-06-10T00:38:39.834 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.834+0000 7f359d7fa640 1 -- 172.21.15.121:0/969643897 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+740913 (secure 0 0 0) 0x7f35940bbfa0 con 0x7f35a0101230 2024-06-10T00:38:39.845 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:38:39.845 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: "epoch": 26, 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: "active_gid": 15216, 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: "active_name": "x", 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: "active_addrs": { 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6800", 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2387911981 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.846 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:6801", 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2387911981 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: "active_addr": "172.21.15.121:6801/2387911981", 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: "active_change": "2024-06-10T00:27:57.451777+0000", 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: "active_mgr_features": 4540701547738038271, 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: "available": true, 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: "standbys": [ 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:39.847 INFO:teuthology.orchestra.run.smithi121.stdout: "gid": 24862, 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "z", 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_features": 4540701547738038271, 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "available_modules": [ 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "alerts", 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.848 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How frequently to reexamine health status", 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.849 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.850 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.851 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:39.852 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.853 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_destination": { 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_destination", 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.854 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Email address to send alerts to", 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_from_name": { 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_from_name", 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.855 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "Ceph", 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Email From: name", 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_host": { 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_host", 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.856 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SMTP server", 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_password": { 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_password", 2024-06-10T00:38:39.857 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Password to authenticate with", 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_port": { 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_port", 2024-06-10T00:38:39.858 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "465", 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SMTP port", 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_sender": { 2024-06-10T00:38:39.859 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_sender", 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SMTP envelope sender", 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.860 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.861 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_ssl": { 2024-06-10T00:38:39.861 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_ssl", 2024-06-10T00:38:39.861 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.861 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.861 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.861 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.861 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.861 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.861 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.861 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use SSL to connect to SMTP server", 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_user": { 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_user", 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.862 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.863 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "User to authenticate as", 2024-06-10T00:38:39.863 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.863 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.863 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.863 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:39.863 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:39.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.863 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:39.863 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "balancer", 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "active": { 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "active", 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.864 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "automatically balance PGs across cluster", 2024-06-10T00:38:39.865 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.865 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.865 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.865 INFO:teuthology.orchestra.run.smithi121.stdout: "begin_time": { 2024-06-10T00:38:39.865 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "begin_time", 2024-06-10T00:38:39.865 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.865 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.865 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.865 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0000", 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "beginning time of day to automatically balance", 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: "begin_weekday": { 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "begin_weekday", 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:39.866 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "0", 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "6", 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_compat_max_iterations": { 2024-06-10T00:38:39.867 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crush_compat_max_iterations", 2024-06-10T00:38:39.868 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:39.868 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.868 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.868 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "25", 2024-06-10T00:38:39.868 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:39.868 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "250", 2024-06-10T00:38:39.868 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.868 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-06-10T00:38:39.868 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.868 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_compat_metrics": { 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crush_compat_metrics", 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "pgs,objects,bytes", 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-06-10T00:38:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-06-10T00:38:39.870 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.870 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.870 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.870 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_compat_step": { 2024-06-10T00:38:39.870 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crush_compat_step", 2024-06-10T00:38:39.870 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:39.870 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.870 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.871 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0.5", 2024-06-10T00:38:39.871 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "0.001", 2024-06-10T00:38:39.871 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "0.999", 2024-06-10T00:38:39.871 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.871 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "aggressiveness of optimization", 2024-06-10T00:38:39.871 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-06-10T00:38:39.871 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.871 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.871 INFO:teuthology.orchestra.run.smithi121.stdout: "end_time": { 2024-06-10T00:38:39.872 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "end_time", 2024-06-10T00:38:39.872 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.872 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.872 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.872 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "2359", 2024-06-10T00:38:39.872 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.872 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.872 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.872 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "ending time of day to automatically balance", 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "end_weekday": { 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "end_weekday", 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "0", 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "6", 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.873 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:39.874 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.875 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:39.876 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:39.877 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.878 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "min_score": { 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "min_score", 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.879 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": { 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mode", 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "upmap", 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.880 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "crush-compat", 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "none", 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "read", 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap", 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap-read" 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Balancer mode", 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_ids": { 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pool_ids", 2024-06-10T00:38:39.881 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:39.882 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap_max_deviation": { 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "upmap_max_deviation", 2024-06-10T00:38:39.883 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "deviation below which no optimization is attempted", 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap_max_optimizations": { 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "upmap_max_optimizations", 2024-06-10T00:38:39.884 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:39.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cephadm", 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "agent_down_multiplier": { 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "agent_down_multiplier", 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "3.0", 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.886 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "agent_refresh_rate": { 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "agent_refresh_rate", 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "20", 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.887 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "agent_starting_port": { 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "agent_starting_port", 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "4721", 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.888 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-06-10T00:38:39.889 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.889 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.889 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.889 INFO:teuthology.orchestra.run.smithi121.stdout: "allow_ptrace": { 2024-06-10T00:38:39.889 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "allow_ptrace", 2024-06-10T00:38:39.889 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.889 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.889 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.889 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.889 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: "autotune_interval": { 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "autotune_interval", 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.890 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to autotune daemon memory", 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "autotune_memory_target_ratio": { 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "autotune_memory_target_ratio", 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.891 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0.7", 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "cephadm_log_destination": { 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cephadm_log_destination", 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.892 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.893 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.893 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.893 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:39.893 INFO:teuthology.orchestra.run.smithi121.stdout: "file", 2024-06-10T00:38:39.893 INFO:teuthology.orchestra.run.smithi121.stdout: "file,syslog", 2024-06-10T00:38:39.893 INFO:teuthology.orchestra.run.smithi121.stdout: "syslog" 2024-06-10T00:38:39.893 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:39.893 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-06-10T00:38:39.893 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "cgroups_split": { 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cgroups_split", 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.894 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-06-10T00:38:39.895 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.895 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.895 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.895 INFO:teuthology.orchestra.run.smithi121.stdout: "config_checks_enabled": { 2024-06-10T00:38:39.895 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "config_checks_enabled", 2024-06-10T00:38:39.895 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.895 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.895 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.895 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.895 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: "config_dashboard": { 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "config_dashboard", 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.896 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-06-10T00:38:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_alertmanager": { 2024-06-10T00:38:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_alertmanager", 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.899 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_base": { 2024-06-10T00:38:39.899 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_base", 2024-06-10T00:38:39.899 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.899 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.899 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.899 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/ceph", 2024-06-10T00:38:39.899 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.899 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.899 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Container image name, without the tag", 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_elasticsearch": { 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_elasticsearch", 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.900 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.901 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "elasticsearch container image", 2024-06-10T00:38:39.901 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.901 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.901 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.901 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_grafana": { 2024-06-10T00:38:39.901 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_grafana", 2024-06-10T00:38:39.901 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.901 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.901 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_haproxy": { 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_haproxy", 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.902 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "HAproxy container image", 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_jaeger_agent": { 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_jaeger_agent", 2024-06-10T00:38:39.903 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Jaeger agent container image", 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_jaeger_collector": { 2024-06-10T00:38:39.904 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_jaeger_collector", 2024-06-10T00:38:39.905 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.905 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.905 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.905 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-06-10T00:38:39.905 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.905 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.905 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.905 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Jaeger collector container image", 2024-06-10T00:38:39.905 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.905 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.906 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.906 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_jaeger_query": { 2024-06-10T00:38:39.906 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_jaeger_query", 2024-06-10T00:38:39.906 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.906 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.906 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.906 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-06-10T00:38:39.906 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.907 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.907 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.907 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Jaeger query container image", 2024-06-10T00:38:39.907 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.907 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.907 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.907 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_keepalived": { 2024-06-10T00:38:39.907 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_keepalived", 2024-06-10T00:38:39.907 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.908 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.908 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.908 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-06-10T00:38:39.908 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.908 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.908 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.908 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Keepalived container image", 2024-06-10T00:38:39.908 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.908 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_loki": { 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_loki", 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Loki container image", 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.909 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_node_exporter": { 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_node_exporter", 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.910 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_nvmeof": { 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_nvmeof", 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Nvme-of container image", 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.911 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_prometheus": { 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_prometheus", 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.912 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_promtail": { 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_promtail", 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Promtail container image", 2024-06-10T00:38:39.913 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_samba": { 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_samba", 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.914 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Samba/SMB container image", 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_snmp_gateway": { 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_snmp_gateway", 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.915 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SNMP Gateway container image", 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "container_init": { 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_init", 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.916 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Run podman/docker with `--init`", 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "daemon_cache_timeout": { 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "daemon_cache_timeout", 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.917 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: "default_cephadm_command_timeout": { 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "default_cephadm_command_timeout", 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "900", 2024-06-10T00:38:39.918 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "default_registry": { 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "default_registry", 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io", 2024-06-10T00:38:39.919 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "device_cache_timeout": { 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "device_cache_timeout", 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "1800", 2024-06-10T00:38:39.920 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "seconds to cache device inventory", 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "device_enhanced_scan": { 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "device_enhanced_scan", 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.921 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use libstoragemgmt during device scans", 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "facts_cache_timeout": { 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "facts_cache_timeout", 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:39.922 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "seconds to cache host facts data", 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "grafana_dashboards_path": { 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "grafana_dashboards_path", 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.923 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "host_check_interval": { 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "host_check_interval", 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:39.924 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to perform a host check", 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "hw_monitoring": { 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "hw_monitoring", 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.925 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "inventory_list_all": { 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "inventory_list_all", 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.926 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.927 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.928 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.929 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.929 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.929 INFO:teuthology.orchestra.run.smithi121.stdout: "log_refresh_metadata": { 2024-06-10T00:38:39.929 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_refresh_metadata", 2024-06-10T00:38:39.929 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.929 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.929 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.929 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.930 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.931 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.931 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.931 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.931 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-06-10T00:38:39.931 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.931 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.931 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.931 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:39.932 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:39.933 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:39.933 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.933 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.933 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.933 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.933 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:39.933 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:39.933 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "manage_etc_ceph_ceph_conf": { 2024-06-10T00:38:39.934 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-06-10T00:38:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-06-10T00:38:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.935 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.936 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.936 INFO:teuthology.orchestra.run.smithi121.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-06-10T00:38:39.936 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-06-10T00:38:39.936 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.936 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.936 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.936 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "*", 2024-06-10T00:38:39.936 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.936 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.936 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.937 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-06-10T00:38:39.937 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.937 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.937 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.937 INFO:teuthology.orchestra.run.smithi121.stdout: "max_count_per_host": { 2024-06-10T00:38:39.937 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_count_per_host", 2024-06-10T00:38:39.937 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:39.937 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.937 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.937 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:39.938 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.938 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.938 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.938 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "max number of daemons per service per host", 2024-06-10T00:38:39.938 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.938 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.938 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.938 INFO:teuthology.orchestra.run.smithi121.stdout: "max_osd_draining_count": { 2024-06-10T00:38:39.938 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_osd_draining_count", 2024-06-10T00:38:39.939 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:39.939 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.939 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.939 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:39.939 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.939 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.939 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.939 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-06-10T00:38:39.939 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.939 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.940 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.940 INFO:teuthology.orchestra.run.smithi121.stdout: "migration_current": { 2024-06-10T00:38:39.940 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "migration_current", 2024-06-10T00:38:39.940 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:39.940 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.940 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.940 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.940 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.940 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "internal - do not modify", 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": { 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mode", 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "root", 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.941 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: "cephadm-package", 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: "root" 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "mode for remote execution of cephadm", 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: "oob_default_addr": { 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "oob_default_addr", 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.942 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "169.254.1.1", 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Default address for RedFish API (oob management).", 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "prometheus_alerts_path": { 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "prometheus_alerts_path", 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.943 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_insecure": { 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_insecure", 2024-06-10T00:38:39.944 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_password": { 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_password", 2024-06-10T00:38:39.945 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_url": { 2024-06-10T00:38:39.946 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_url", 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_username": { 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_username", 2024-06-10T00:38:39.947 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.948 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.948 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.948 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.948 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.948 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.948 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.948 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-06-10T00:38:39.948 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.948 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "secure_monitoring_stack": { 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "secure_monitoring_stack", 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.949 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "service_discovery_port": { 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "service_discovery_port", 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8765", 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "cephadm service discovery port", 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.950 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "ssh_config_file": { 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssh_config_file", 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.951 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "use_agent": { 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "use_agent", 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.952 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "use_repo_digest": { 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "use_repo_digest", 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.953 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_on_failed_host_check": { 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_on_failed_host_check", 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise a health warning if the host check fails", 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.954 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_on_stray_daemons": { 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_on_stray_daemons", 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-06-10T00:38:39.955 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_on_stray_hosts": { 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_on_stray_hosts", 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.956 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-06-10T00:38:39.957 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.957 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.957 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.957 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:39.957 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:39.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.957 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:39.957 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crash", 2024-06-10T00:38:39.957 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:39.957 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:39.957 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:39.958 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.959 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.960 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.961 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:39.961 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.961 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.961 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:39.961 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:39.961 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:39.961 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:39.961 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:39.961 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.962 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout: "retain_interval": { 2024-06-10T00:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "retain_interval", 2024-06-10T00:38:39.963 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:39.964 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.964 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.964 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "31536000", 2024-06-10T00:38:39.964 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.964 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.964 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.964 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how long to retain crashes before pruning them", 2024-06-10T00:38:39.964 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.964 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.964 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.965 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_recent_interval": { 2024-06-10T00:38:39.965 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_recent_interval", 2024-06-10T00:38:39.965 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:39.965 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.965 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:39.965 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "1209600", 2024-06-10T00:38:39.965 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.965 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.965 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.965 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "time interval in which to warn about recent crashes", 2024-06-10T00:38:39.966 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.966 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.966 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.966 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:39.966 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:39.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.966 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:39.966 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "dashboard", 2024-06-10T00:38:39.966 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:39.966 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:39.966 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:39.967 INFO:teuthology.orchestra.run.smithi121.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-06-10T00:38:39.967 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-06-10T00:38:39.967 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:39.967 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.967 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.967 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:39.967 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.967 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.967 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.967 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: "ALERTMANAGER_API_HOST": { 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ALERTMANAGER_API_HOST", 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.968 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.969 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.969 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.969 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.969 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.969 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.969 INFO:teuthology.orchestra.run.smithi121.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-06-10T00:38:39.969 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-06-10T00:38:39.969 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.969 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.970 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.970 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.970 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.970 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.970 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.970 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.970 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.970 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.970 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.971 INFO:teuthology.orchestra.run.smithi121.stdout: "AUDIT_API_ENABLED": { 2024-06-10T00:38:39.971 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "AUDIT_API_ENABLED", 2024-06-10T00:38:39.971 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.971 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.971 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.971 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.971 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.971 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.971 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.971 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.972 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.972 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.972 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.972 INFO:teuthology.orchestra.run.smithi121.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-06-10T00:38:39.972 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-06-10T00:38:39.972 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.972 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.972 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.972 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "ENABLE_BROWSABLE_API": { 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ENABLE_BROWSABLE_API", 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.973 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.974 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.974 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.974 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.974 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.974 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.974 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.974 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.974 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.974 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-06-10T00:38:39.975 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-06-10T00:38:39.975 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.975 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.975 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.975 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.975 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.975 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.975 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.976 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.977 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.978 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_NFS": { 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.979 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_RBD": { 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.980 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_RGW": { 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.981 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.982 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_PASSWORD": { 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_PASSWORD", 2024-06-10T00:38:39.983 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "admin", 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-06-10T00:38:39.984 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_URL": { 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_URL", 2024-06-10T00:38:39.985 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_USERNAME": { 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_USERNAME", 2024-06-10T00:38:39.986 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "admin", 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-06-10T00:38:39.987 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.988 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.990 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "ISSUE_TRACKER_API_KEY": { 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.991 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "PROMETHEUS_API_HOST": { 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PROMETHEUS_API_HOST", 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.992 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.993 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.994 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.995 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.996 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.997 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.997 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.997 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.997 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-06-10T00:38:39.997 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-06-10T00:38:39.997 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.997 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.997 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:39.998 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:39.999 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:39.999 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:39.999 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:39.999 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:39.999 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:39.999 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:39.999 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:39.999 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:39.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:39.999 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.000 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-06-10T00:38:40.001 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-06-10T00:38:40.001 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.001 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.001 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.001 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.001 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.001 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.001 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.001 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.001 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.001 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_ENABLED": { 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_ENABLED", 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.002 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.003 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.003 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.003 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-06-10T00:38:40.003 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-06-10T00:38:40.003 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.003 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.003 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.004 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-06-10T00:38:40.005 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-06-10T00:38:40.006 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.006 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.006 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.006 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8", 2024-06-10T00:38:40.006 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.006 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.006 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.006 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.006 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.006 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: "REST_REQUESTS_TIMEOUT": { 2024-06-10T00:38:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-06-10T00:38:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "45", 2024-06-10T00:38:40.007 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.008 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.008 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.008 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.008 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.008 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.008 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.008 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_ACCESS_KEY": { 2024-06-10T00:38:40.008 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_ACCESS_KEY", 2024-06-10T00:38:40.008 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.008 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.009 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.009 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.009 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.009 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.009 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.009 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.009 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.009 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.009 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.009 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.009 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "admin", 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_SECRET_KEY": { 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_SECRET_KEY", 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_SSL_VERIFY": { 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_SSL_VERIFY", 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.012 INFO:teuthology.orchestra.run.smithi121.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.013 INFO:teuthology.orchestra.run.smithi121.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-06-10T00:38:40.014 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-06-10T00:38:40.014 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.014 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.014 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.014 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.014 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.014 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.014 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.014 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.014 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.014 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.015 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "cross_origin_url": { 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cross_origin_url", 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.016 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "crt_file": { 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crt_file", 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.017 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: "debug": { 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "debug", 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.018 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Enable/disable debug options", 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "jwt_token_ttl": { 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "jwt_token_ttl", 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "28800", 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.019 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "key_file": { 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "key_file", 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.020 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.021 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.024 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.025 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "motd": { 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "motd", 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.026 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "The message of the day", 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "redirect_resolve_ip_addr": { 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "redirect_resolve_ip_addr", 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.027 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "server_addr": { 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_addr", 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.028 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "::", 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "server_port": { 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_port", 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.029 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.030 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8080", 2024-06-10T00:38:40.030 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.030 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.030 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.030 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.030 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.030 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.030 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.030 INFO:teuthology.orchestra.run.smithi121.stdout: "ssl": { 2024-06-10T00:38:40.031 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssl", 2024-06-10T00:38:40.031 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.031 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.031 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.031 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.031 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.031 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.031 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.031 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.031 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.031 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "ssl_server_port": { 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssl_server_port", 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8443", 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.032 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.033 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.033 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.033 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_behaviour": { 2024-06-10T00:38:40.033 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_behaviour", 2024-06-10T00:38:40.033 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.033 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.033 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "redirect", 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: "redirect" 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.034 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_error_status_code": { 2024-06-10T00:38:40.035 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_error_status_code", 2024-06-10T00:38:40.035 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.035 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.035 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.035 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "500", 2024-06-10T00:38:40.035 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "400", 2024-06-10T00:38:40.035 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "599", 2024-06-10T00:38:40.035 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.035 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.035 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: "url_prefix": { 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "url_prefix", 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.036 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.037 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.037 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.037 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.037 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.037 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.037 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.037 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "devicehealth", 2024-06-10T00:38:40.037 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.037 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.038 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.038 INFO:teuthology.orchestra.run.smithi121.stdout: "enable_monitoring": { 2024-06-10T00:38:40.038 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enable_monitoring", 2024-06-10T00:38:40.038 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.038 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.038 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.038 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.038 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.038 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.038 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "monitor device health metrics", 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.039 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.040 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.040 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.040 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.040 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.040 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.040 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.040 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.040 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.040 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.041 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.041 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.041 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.041 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.041 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.041 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.041 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.041 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.041 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.042 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.042 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.042 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.042 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.042 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.042 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.042 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.042 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.042 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.042 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.043 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.044 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.045 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.045 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.045 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.045 INFO:teuthology.orchestra.run.smithi121.stdout: "mark_out_threshold": { 2024-06-10T00:38:40.045 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mark_out_threshold", 2024-06-10T00:38:40.045 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.045 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.045 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.045 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "2419200", 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": { 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pool_name", 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.046 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "device_health_metrics", 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "name of pool in which to store device health metrics", 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "retention_period": { 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "retention_period", 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.047 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "15552000", 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how long to retain device health metrics", 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "scrape_frequency": { 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "scrape_frequency", 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.048 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "86400", 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to scrape device health metrics", 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "self_heal": { 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "self_heal", 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.049 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.050 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to wake up and check device health", 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_threshold": { 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_threshold", 2024-06-10T00:38:40.051 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "7257600", 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.052 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "diskprediction_local", 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.053 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.054 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.055 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.056 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.057 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "predict_interval": { 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "predict_interval", 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "86400", 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.058 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "predictor_model": { 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "predictor_model", 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "prophetstor", 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.059 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.060 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "influx", 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": false, 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "influxdb python module not found", 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: "batch_size": { 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "batch_size", 2024-06-10T00:38:40.061 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5000", 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "database": { 2024-06-10T00:38:40.062 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "database", 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "ceph", 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.063 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": { 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "hostname", 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "InfluxDB server hostname", 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.064 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "30", 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "5", 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.065 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.066 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.067 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.067 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.067 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.067 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.067 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.067 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.067 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.067 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.067 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.068 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.069 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.069 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.069 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.069 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.069 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.069 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.069 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.069 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.069 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.070 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.071 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.071 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.071 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.071 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.071 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.071 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.071 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.071 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.071 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "password": { 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "password", 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "password of InfluxDB server user", 2024-06-10T00:38:40.072 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: "port": { 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "port", 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8086", 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.073 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "InfluxDB server port", 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: "ssl": { 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssl", 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "false", 2024-06-10T00:38:40.074 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.075 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.075 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.075 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-06-10T00:38:40.075 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.075 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.075 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.075 INFO:teuthology.orchestra.run.smithi121.stdout: "threads": { 2024-06-10T00:38:40.075 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "threads", 2024-06-10T00:38:40.075 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.076 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.076 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.076 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:40.076 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:40.076 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "32", 2024-06-10T00:38:40.076 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.076 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-06-10T00:38:40.076 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.076 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.076 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.077 INFO:teuthology.orchestra.run.smithi121.stdout: "username": { 2024-06-10T00:38:40.077 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "username", 2024-06-10T00:38:40.077 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.077 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.077 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.077 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.077 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.077 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.077 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.078 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "username of InfluxDB server user", 2024-06-10T00:38:40.078 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.078 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.078 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.078 INFO:teuthology.orchestra.run.smithi121.stdout: "verify_ssl": { 2024-06-10T00:38:40.078 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "verify_ssl", 2024-06-10T00:38:40.078 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "true", 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.079 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "insights", 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.080 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.081 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.082 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.083 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.084 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "iostat", 2024-06-10T00:38:40.085 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.086 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.087 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.088 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.089 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.090 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "k8sevents", 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_event_retention_days": { 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ceph_event_retention_days", 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.091 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "7", 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Days to hold ceph event information within local cache", 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "config_check_secs": { 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "config_check_secs", 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.092 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "10", 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.093 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.094 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.095 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.096 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.097 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "localpool", 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: "failure_domain": { 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "failure_domain", 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "host", 2024-06-10T00:38:40.099 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.100 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.100 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.100 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "failure domain for any created local pool", 2024-06-10T00:38:40.100 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-06-10T00:38:40.100 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.100 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.100 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.100 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.100 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.100 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.101 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.102 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.102 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.102 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.102 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.102 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.102 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.102 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.103 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.104 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.104 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.104 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.104 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.104 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.104 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.104 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.104 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.104 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.104 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.105 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.106 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.106 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.106 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.106 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.106 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.106 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.106 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.106 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.106 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": { 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "min_size", 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "default min_size for any created local pool", 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.108 INFO:teuthology.orchestra.run.smithi121.stdout: "num_rep": { 2024-06-10T00:38:40.108 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "num_rep", 2024-06-10T00:38:40.108 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.108 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.108 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.108 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "3", 2024-06-10T00:38:40.108 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.108 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "default replica count for any created local pool", 2024-06-10T00:38:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": { 2024-06-10T00:38:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pg_num", 2024-06-10T00:38:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.109 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.110 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.110 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "128", 2024-06-10T00:38:40.110 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.110 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.110 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.110 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "default pg_num for any created local pool", 2024-06-10T00:38:40.110 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.110 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.110 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.111 INFO:teuthology.orchestra.run.smithi121.stdout: "prefix": { 2024-06-10T00:38:40.111 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "prefix", 2024-06-10T00:38:40.111 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.111 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.111 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.111 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.111 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "name prefix for any created local pool", 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: "subtree": { 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "subtree", 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.112 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "rack", 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "CRUSH level for which to create a local pool", 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mds_autoscaler", 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.113 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.114 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.115 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.116 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.117 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.118 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mirroring", 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.119 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.120 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.121 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.122 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.123 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "nfs", 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.125 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.126 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.127 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.128 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.129 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.130 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "orchestrator", 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "fail_fs": { 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "fail_fs", 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.131 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.132 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.133 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.133 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.133 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.133 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.133 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.133 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.133 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.133 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.133 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.133 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.134 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.135 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.136 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.137 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.137 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.137 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.137 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.137 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.137 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.137 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.137 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.137 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.137 INFO:teuthology.orchestra.run.smithi121.stdout: "orchestrator": { 2024-06-10T00:38:40.138 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "orchestrator", 2024-06-10T00:38:40.138 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.138 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.138 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.138 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.138 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.138 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.138 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.138 INFO:teuthology.orchestra.run.smithi121.stdout: "cephadm", 2024-06-10T00:38:40.138 INFO:teuthology.orchestra.run.smithi121.stdout: "rook", 2024-06-10T00:38:40.139 INFO:teuthology.orchestra.run.smithi121.stdout: "test_orchestrator" 2024-06-10T00:38:40.139 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.139 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Orchestrator backend", 2024-06-10T00:38:40.139 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.139 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.139 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.139 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.139 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.139 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.139 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "osd_perf_query", 2024-06-10T00:38:40.140 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.140 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.140 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.140 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.140 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.140 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.140 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.140 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.140 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.140 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.141 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.141 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.141 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.141 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.141 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.141 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.141 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.141 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.141 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.141 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.141 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.142 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.143 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.143 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.143 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.143 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.143 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.143 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.143 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.143 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.143 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.144 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.144 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.144 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.144 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.144 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.144 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.144 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.144 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.144 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.144 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.145 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.145 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.145 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.145 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.145 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.145 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.145 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.145 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.146 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.146 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.146 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.146 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.146 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.146 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.146 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.147 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.147 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.147 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.147 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.147 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.147 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.147 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "osd_support", 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.148 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.149 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.150 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.151 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.152 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pg_autoscaler", 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.153 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.154 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.155 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.156 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.157 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:40.158 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.159 INFO:teuthology.orchestra.run.smithi121.stdout: "threshold": { 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "threshold", 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "3.0", 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1.0", 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "scaling threshold", 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.160 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "progress", 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "allow_pg_recovery_event": { 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "allow_pg_recovery_event", 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.161 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.162 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.162 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "allow the module to show pg recovery progress", 2024-06-10T00:38:40.162 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.162 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.162 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.162 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.162 INFO:teuthology.orchestra.run.smithi121.stdout: "enabled": { 2024-06-10T00:38:40.162 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enabled", 2024-06-10T00:38:40.162 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.162 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.162 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.163 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.163 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.163 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.163 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.163 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.163 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.163 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.163 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.163 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.163 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.164 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.165 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.166 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.167 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.168 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.169 INFO:teuthology.orchestra.run.smithi121.stdout: "max_completed_events": { 2024-06-10T00:38:40.169 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_completed_events", 2024-06-10T00:38:40.169 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.169 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.169 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.169 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "50", 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "number of past completed events to remember", 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.170 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.171 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:40.171 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.171 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.171 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.171 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how long the module is going to sleep", 2024-06-10T00:38:40.171 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.171 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.171 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.171 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "prometheus", 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: "cache": { 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cache", 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.173 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.173 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.173 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.173 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.173 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.173 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.173 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.173 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.173 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.173 INFO:teuthology.orchestra.run.smithi121.stdout: "exclude_perf_counters": { 2024-06-10T00:38:40.173 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "exclude_perf_counters", 2024-06-10T00:38:40.174 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.174 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.174 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.174 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.174 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.174 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.174 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.174 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-06-10T00:38:40.174 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-06-10T00:38:40.174 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.175 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.175 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.175 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.175 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.175 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.175 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.175 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.175 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.175 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.176 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.176 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.176 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.176 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.176 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.176 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.176 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.176 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.176 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.176 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.177 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.178 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.178 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.178 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.178 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.178 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.178 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.178 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.178 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.179 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.179 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.179 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.179 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.179 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.179 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.179 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.179 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.179 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.179 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.179 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.180 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.180 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.180 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.180 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.180 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.180 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.180 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.180 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.181 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.181 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.181 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.181 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.181 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.181 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.181 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.181 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.181 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.181 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_stats_pools": { 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rbd_stats_pools", 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.182 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_stats_pools_refresh_interval": { 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "300", 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.183 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "scrape_interval": { 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "scrape_interval", 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "15.0", 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.184 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "server_addr": { 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_addr", 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "::", 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.185 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "server_port": { 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_port", 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "9283", 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-06-10T00:38:40.186 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "stale_cache_strategy": { 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "stale_cache_strategy", 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "log", 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.187 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_behaviour": { 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_behaviour", 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "default", 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.188 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: "default", 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: "error" 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_error_status_code": { 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_error_status_code", 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.189 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "500", 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "400", 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "599", 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.190 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rbd_support", 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.191 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.192 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.193 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.193 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.193 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.193 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.193 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.193 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.193 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.193 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.193 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.193 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.194 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.194 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.194 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.194 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.194 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.194 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.194 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.194 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.194 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.194 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.194 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.195 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "max_concurrent_snap_create": { 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_concurrent_snap_create", 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.196 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "mirror_snapshot_schedule": { 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mirror_snapshot_schedule", 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.197 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "trash_purge_schedule": { 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "trash_purge_schedule", 2024-06-10T00:38:40.198 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.199 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "restful", 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "enable_auth": { 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enable_auth", 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.200 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "key_file": { 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "key_file", 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.201 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.202 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.202 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.202 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.202 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.203 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.204 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.205 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.205 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.205 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.205 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.205 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.205 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.205 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.205 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.205 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.206 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.207 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "server_addr": { 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_addr", 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.208 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.209 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.209 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.209 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.209 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.209 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.209 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.209 INFO:teuthology.orchestra.run.smithi121.stdout: "server_port": { 2024-06-10T00:38:40.209 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_port", 2024-06-10T00:38:40.209 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.209 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.211 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.211 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rgw", 2024-06-10T00:38:40.211 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.211 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.211 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.211 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.211 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.211 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.211 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.212 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.213 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.213 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.213 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.213 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.213 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.213 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.213 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.213 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.213 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.213 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.214 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.214 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.214 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.214 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.214 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.214 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.214 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.214 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.215 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.215 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.215 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.215 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.215 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.215 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.215 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.215 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.217 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.218 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.218 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.218 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.218 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.218 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.218 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.219 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.219 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.219 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.219 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.219 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.219 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.219 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.219 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.220 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.220 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.220 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.220 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.220 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.220 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.220 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.220 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.220 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.221 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.221 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.221 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rook", 2024-06-10T00:38:40.221 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.221 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.221 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.221 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.221 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.221 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.222 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.223 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.224 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.226 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "storage_class": { 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "storage_class", 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "local", 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.227 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "selftest", 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.228 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.229 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.230 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.231 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.232 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.233 INFO:teuthology.orchestra.run.smithi121.stdout: "roption1": { 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "roption1", 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "roption2": { 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "roption2", 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "xyz", 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption1": { 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption1", 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.236 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption2": { 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption2", 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption3": { 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption3", 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.238 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption4": { 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption4", 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.239 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption5": { 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption5", 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.240 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.241 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.241 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.241 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.241 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption6": { 2024-06-10T00:38:40.241 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption6", 2024-06-10T00:38:40.241 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.241 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.241 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.241 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.241 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.241 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption7": { 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption7", 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.242 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "42", 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "testkey": { 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "testkey", 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.243 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.244 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.244 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.244 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.244 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.244 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.244 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.244 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.244 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.244 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.244 INFO:teuthology.orchestra.run.smithi121.stdout: "testlkey": { 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "testlkey", 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.246 INFO:teuthology.orchestra.run.smithi121.stdout: "testnewline": { 2024-06-10T00:38:40.246 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "testnewline", 2024-06-10T00:38:40.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.246 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.246 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.246 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.246 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.246 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.246 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.246 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.246 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "snap_schedule", 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: "allow_m_granularity": { 2024-06-10T00:38:40.247 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "allow_m_granularity", 2024-06-10T00:38:40.248 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.248 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.248 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.248 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.248 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.248 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.248 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.248 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "allow minute scheduled snapshots", 2024-06-10T00:38:40.248 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.248 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.249 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.249 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.249 INFO:teuthology.orchestra.run.smithi121.stdout: "dump_on_update": { 2024-06-10T00:38:40.249 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "dump_on_update", 2024-06-10T00:38:40.249 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.249 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.249 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.249 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.249 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.249 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.250 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.250 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "dump database to debug log on update", 2024-06-10T00:38:40.250 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.250 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.250 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.250 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.250 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.250 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.250 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.250 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.251 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.251 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.251 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.251 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.251 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.251 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.251 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.251 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.251 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.251 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.252 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.252 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.252 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.252 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.252 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.252 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.252 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.252 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.252 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.253 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.253 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.253 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.253 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.253 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.253 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.253 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.253 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.254 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.255 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.256 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "stats", 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.257 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.258 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.259 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.260 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.261 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "status", 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.262 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.263 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.264 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.265 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.266 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.267 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "telegraf", 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "address": { 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "address", 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.268 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "15", 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.269 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.270 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.271 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.272 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.273 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.273 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.273 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.273 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.273 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.273 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.273 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.273 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.273 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.273 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.274 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.274 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.274 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.274 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.274 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.274 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.274 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.274 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.274 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.275 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.276 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.276 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.276 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.276 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.276 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "telemetry", 2024-06-10T00:38:40.276 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.276 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.276 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_basic": { 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_basic", 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share basic cluster information (size, version)", 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.278 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_crash": { 2024-06-10T00:38:40.278 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_crash", 2024-06-10T00:38:40.278 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.278 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.278 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.278 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.278 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.278 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.278 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_device": { 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_device", 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.279 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.280 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.280 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-06-10T00:38:40.280 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.280 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.280 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.280 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_ident": { 2024-06-10T00:38:40.280 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_ident", 2024-06-10T00:38:40.280 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.280 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_perf": { 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_perf", 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.281 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share various performance metrics of a cluster", 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "contact": { 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "contact", 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.282 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "description": { 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "description", 2024-06-10T00:38:40.283 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.284 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.284 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.284 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.284 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.284 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.284 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.284 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.284 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.284 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.285 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.285 INFO:teuthology.orchestra.run.smithi121.stdout: "device_url": { 2024-06-10T00:38:40.285 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "device_url", 2024-06-10T00:38:40.285 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.285 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.285 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.285 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-06-10T00:38:40.285 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.285 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: "enabled": { 2024-06-10T00:38:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enabled", 2024-06-10T00:38:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.286 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.287 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.288 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:40.288 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:40.288 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.288 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.288 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.288 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "24", 2024-06-10T00:38:40.288 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "8", 2024-06-10T00:38:40.288 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.288 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "last_opt_revision": { 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "last_opt_revision", 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "1", 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.289 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "leaderboard": { 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "leaderboard", 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.290 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "leaderboard_description": { 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "leaderboard_description", 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.291 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.292 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.295 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.296 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "organization": { 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "organization", 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.297 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "proxy": { 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "proxy", 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.298 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: "url": { 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "url", 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.299 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "test_orchestrator", 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.300 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.301 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.302 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.303 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.303 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.303 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.303 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.303 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.303 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.303 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.303 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.303 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.304 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.305 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "volumes", 2024-06-10T00:38:40.306 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.307 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.308 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.309 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.310 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.310 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.310 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.310 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.310 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.310 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.310 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.310 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.310 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.311 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.312 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.312 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.312 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.312 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.312 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.312 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.312 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.312 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.312 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "max_concurrent_clones": { 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_concurrent_clones", 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "4", 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Number of asynchronous cloner threads", 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.313 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.314 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.314 INFO:teuthology.orchestra.run.smithi121.stdout: "periodic_async_work": { 2024-06-10T00:38:40.314 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "periodic_async_work", 2024-06-10T00:38:40.314 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.314 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.314 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.314 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.314 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.314 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.315 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.315 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Periodically check for async work", 2024-06-10T00:38:40.315 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.315 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.315 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.315 INFO:teuthology.orchestra.run.smithi121.stdout: "snapshot_clone_delay": { 2024-06-10T00:38:40.315 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "snapshot_clone_delay", 2024-06-10T00:38:40.315 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.315 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.315 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.316 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:40.316 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.316 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.316 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.316 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-06-10T00:38:40.316 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.316 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.316 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.316 INFO:teuthology.orchestra.run.smithi121.stdout: "snapshot_clone_no_wait": { 2024-06-10T00:38:40.316 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "snapshot_clone_no_wait", 2024-06-10T00:38:40.317 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.317 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.317 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.317 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.317 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.317 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.317 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.317 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-06-10T00:38:40.317 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.317 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.317 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.318 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix", 2024-06-10T00:38:40.318 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.318 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.318 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.318 INFO:teuthology.orchestra.run.smithi121.stdout: "discovery_interval": { 2024-06-10T00:38:40.318 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "discovery_interval", 2024-06-10T00:38:40.318 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:40.319 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.319 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.319 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "100", 2024-06-10T00:38:40.319 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.319 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.319 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.319 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.319 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.319 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.319 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.320 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.320 INFO:teuthology.orchestra.run.smithi121.stdout: "identifier": { 2024-06-10T00:38:40.320 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "identifier", 2024-06-10T00:38:40.320 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.320 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.320 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.320 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.320 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.320 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.321 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.321 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.321 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.321 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.321 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.321 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.321 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:40.321 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:40.321 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.322 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.323 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.324 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.325 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.326 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.327 INFO:teuthology.orchestra.run.smithi121.stdout: "zabbix_host": { 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix_host", 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.328 INFO:teuthology.orchestra.run.smithi121.stdout: "zabbix_port": { 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix_port", 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10051", 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "zabbix_sender": { 2024-06-10T00:38:40.329 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix_sender", 2024-06-10T00:38:40.330 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.330 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.330 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.330 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-06-10T00:38:40.330 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.330 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.330 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.330 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.330 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.330 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.330 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: "gid": 24875, 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "y", 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: "mgr_features": 4540701547738038271, 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: "available_modules": [ 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "alerts", 2024-06-10T00:38:40.331 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How frequently to reexamine health status", 2024-06-10T00:38:40.332 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.333 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.334 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.335 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.336 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.337 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_destination": { 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_destination", 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.338 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Email address to send alerts to", 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_from_name": { 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_from_name", 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "Ceph", 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.339 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Email From: name", 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_host": { 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_host", 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.340 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SMTP server", 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_password": { 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_password", 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.341 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Password to authenticate with", 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_port": { 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_port", 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "465", 2024-06-10T00:38:40.342 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.343 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.343 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.343 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SMTP port", 2024-06-10T00:38:40.343 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.343 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.343 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.343 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.343 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_sender": { 2024-06-10T00:38:40.343 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_sender", 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SMTP envelope sender", 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_ssl": { 2024-06-10T00:38:40.344 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_ssl", 2024-06-10T00:38:40.345 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.345 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.345 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.345 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.345 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.345 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.345 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.345 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use SSL to connect to SMTP server", 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_user": { 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_user", 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.346 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "User to authenticate as", 2024-06-10T00:38:40.347 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.347 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.347 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.347 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.347 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.347 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.347 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "balancer", 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "active": { 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "active", 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.348 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "automatically balance PGs across cluster", 2024-06-10T00:38:40.349 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.349 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.349 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.349 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.349 INFO:teuthology.orchestra.run.smithi121.stdout: "begin_time": { 2024-06-10T00:38:40.349 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "begin_time", 2024-06-10T00:38:40.349 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.349 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.349 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.349 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0000", 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "beginning time of day to automatically balance", 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "begin_weekday": { 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "begin_weekday", 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.350 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.351 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:40.351 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "0", 2024-06-10T00:38:40.351 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "6", 2024-06-10T00:38:40.351 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.351 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-06-10T00:38:40.351 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-10T00:38:40.351 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.351 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_compat_max_iterations": { 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crush_compat_max_iterations", 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "25", 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "250", 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_compat_metrics": { 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crush_compat_metrics", 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "pgs,objects,bytes", 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.354 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_compat_step": { 2024-06-10T00:38:40.354 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crush_compat_step", 2024-06-10T00:38:40.354 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.354 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.354 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.354 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0.5", 2024-06-10T00:38:40.354 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "0.001", 2024-06-10T00:38:40.354 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "0.999", 2024-06-10T00:38:40.354 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.354 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "aggressiveness of optimization", 2024-06-10T00:38:40.354 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-06-10T00:38:40.355 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.355 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.355 INFO:teuthology.orchestra.run.smithi121.stdout: "end_time": { 2024-06-10T00:38:40.355 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "end_time", 2024-06-10T00:38:40.355 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.355 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.355 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.356 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "2359", 2024-06-10T00:38:40.356 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.356 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.356 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.356 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "ending time of day to automatically balance", 2024-06-10T00:38:40.356 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-10T00:38:40.356 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.356 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.357 INFO:teuthology.orchestra.run.smithi121.stdout: "end_weekday": { 2024-06-10T00:38:40.357 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "end_weekday", 2024-06-10T00:38:40.357 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:40.357 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.357 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.357 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:40.357 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "0", 2024-06-10T00:38:40.357 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "6", 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.358 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.359 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.360 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.361 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.362 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "min_score": { 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "min_score", 2024-06-10T00:38:40.363 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": { 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mode", 2024-06-10T00:38:40.364 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "upmap", 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "crush-compat", 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "none", 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "read", 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap", 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap-read" 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Balancer mode", 2024-06-10T00:38:40.365 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_ids": { 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pool_ids", 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-06-10T00:38:40.366 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-06-10T00:38:40.367 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap_max_deviation": { 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "upmap_max_deviation", 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "deviation below which no optimization is attempted", 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap_max_optimizations": { 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "upmap_max_optimizations", 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.369 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cephadm", 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: "agent_down_multiplier": { 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "agent_down_multiplier", 2024-06-10T00:38:40.370 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "3.0", 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "agent_refresh_rate": { 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "agent_refresh_rate", 2024-06-10T00:38:40.371 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "20", 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "agent_starting_port": { 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "agent_starting_port", 2024-06-10T00:38:40.372 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "4721", 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "allow_ptrace": { 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "allow_ptrace", 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.373 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: "autotune_interval": { 2024-06-10T00:38:40.374 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "autotune_interval", 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to autotune daemon memory", 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "autotune_memory_target_ratio": { 2024-06-10T00:38:40.375 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "autotune_memory_target_ratio", 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0.7", 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.376 INFO:teuthology.orchestra.run.smithi121.stdout: "cephadm_log_destination": { 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cephadm_log_destination", 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "file", 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "file,syslog", 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "syslog" 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.377 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "cgroups_split": { 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cgroups_split", 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.378 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.379 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-06-10T00:38:40.379 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.379 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.379 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.379 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.379 INFO:teuthology.orchestra.run.smithi121.stdout: "config_checks_enabled": { 2024-06-10T00:38:40.379 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "config_checks_enabled", 2024-06-10T00:38:40.379 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "config_dashboard": { 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "config_dashboard", 2024-06-10T00:38:40.380 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.381 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.381 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.381 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.381 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.381 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.381 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.381 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-06-10T00:38:40.381 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.381 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.381 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_alertmanager": { 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_alertmanager", 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.382 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.383 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.383 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_base": { 2024-06-10T00:38:40.383 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_base", 2024-06-10T00:38:40.383 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.383 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.383 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.383 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/ceph", 2024-06-10T00:38:40.383 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.383 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.383 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.384 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Container image name, without the tag", 2024-06-10T00:38:40.384 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.384 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.384 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.384 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.384 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_elasticsearch": { 2024-06-10T00:38:40.384 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_elasticsearch", 2024-06-10T00:38:40.384 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.384 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.384 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.384 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-06-10T00:38:40.385 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.385 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.385 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.385 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "elasticsearch container image", 2024-06-10T00:38:40.385 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.385 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.385 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.385 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.385 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_grafana": { 2024-06-10T00:38:40.385 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_grafana", 2024-06-10T00:38:40.385 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.386 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.386 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.386 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-06-10T00:38:40.386 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.386 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.386 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.386 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:40.386 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.386 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.386 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.386 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_haproxy": { 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_haproxy", 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "HAproxy container image", 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.387 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.388 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.388 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.388 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_jaeger_agent": { 2024-06-10T00:38:40.388 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_jaeger_agent", 2024-06-10T00:38:40.388 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.388 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.388 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.388 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-06-10T00:38:40.388 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.389 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.389 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.389 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Jaeger agent container image", 2024-06-10T00:38:40.389 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.389 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.389 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.389 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_jaeger_collector": { 2024-06-10T00:38:40.389 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_jaeger_collector", 2024-06-10T00:38:40.390 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.390 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.390 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.390 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-06-10T00:38:40.390 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.390 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.390 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.390 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Jaeger collector container image", 2024-06-10T00:38:40.391 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.391 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.391 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.391 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_jaeger_query": { 2024-06-10T00:38:40.391 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_jaeger_query", 2024-06-10T00:38:40.391 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.391 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.391 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.391 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-06-10T00:38:40.391 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Jaeger query container image", 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_keepalived": { 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_keepalived", 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.392 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Keepalived container image", 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_loki": { 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_loki", 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.393 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Loki container image", 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_node_exporter": { 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_node_exporter", 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.394 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_nvmeof": { 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_nvmeof", 2024-06-10T00:38:40.395 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Nvme-of container image", 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_prometheus": { 2024-06-10T00:38:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_prometheus", 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.397 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_promtail": { 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_promtail", 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Promtail container image", 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_samba": { 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_samba", 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Samba/SMB container image", 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.399 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_snmp_gateway": { 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_snmp_gateway", 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SNMP Gateway container image", 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.400 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "container_init": { 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_init", 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Run podman/docker with `--init`", 2024-06-10T00:38:40.401 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "daemon_cache_timeout": { 2024-06-10T00:38:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "daemon_cache_timeout", 2024-06-10T00:38:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.402 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "default_cephadm_command_timeout": { 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "default_cephadm_command_timeout", 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "900", 2024-06-10T00:38:40.403 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "default_registry": { 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "default_registry", 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io", 2024-06-10T00:38:40.404 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "device_cache_timeout": { 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "device_cache_timeout", 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.405 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "1800", 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "seconds to cache device inventory", 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "device_enhanced_scan": { 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "device_enhanced_scan", 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.406 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use libstoragemgmt during device scans", 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "facts_cache_timeout": { 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "facts_cache_timeout", 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.407 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "seconds to cache host facts data", 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "grafana_dashboards_path": { 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "grafana_dashboards_path", 2024-06-10T00:38:40.408 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "host_check_interval": { 2024-06-10T00:38:40.409 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "host_check_interval", 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to perform a host check", 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.410 INFO:teuthology.orchestra.run.smithi121.stdout: "hw_monitoring": { 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "hw_monitoring", 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.411 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "inventory_list_all": { 2024-06-10T00:38:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "inventory_list_all", 2024-06-10T00:38:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.412 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.413 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.414 INFO:teuthology.orchestra.run.smithi121.stdout: "log_refresh_metadata": { 2024-06-10T00:38:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_refresh_metadata", 2024-06-10T00:38:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.415 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.416 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-06-10T00:38:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.417 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.418 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.419 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: "manage_etc_ceph_ceph_conf": { 2024-06-10T00:38:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-06-10T00:38:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-06-10T00:38:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.421 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-06-10T00:38:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-06-10T00:38:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "*", 2024-06-10T00:38:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.422 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-06-10T00:38:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "max_count_per_host": { 2024-06-10T00:38:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_count_per_host", 2024-06-10T00:38:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.423 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "max number of daemons per service per host", 2024-06-10T00:38:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.424 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "max_osd_draining_count": { 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_osd_draining_count", 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.426 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "migration_current": { 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "migration_current", 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "internal - do not modify", 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.427 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": { 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mode", 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "root", 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "cephadm-package", 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: "root" 2024-06-10T00:38:40.428 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "mode for remote execution of cephadm", 2024-06-10T00:38:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "oob_default_addr": { 2024-06-10T00:38:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "oob_default_addr", 2024-06-10T00:38:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "169.254.1.1", 2024-06-10T00:38:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Default address for RedFish API (oob management).", 2024-06-10T00:38:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "prometheus_alerts_path": { 2024-06-10T00:38:40.430 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "prometheus_alerts_path", 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_insecure": { 2024-06-10T00:38:40.431 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_insecure", 2024-06-10T00:38:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-06-10T00:38:40.432 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_password": { 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_password", 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.433 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_url": { 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_url", 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.434 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_username": { 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_username", 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.435 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "secure_monitoring_stack": { 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "secure_monitoring_stack", 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.436 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "service_discovery_port": { 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "service_discovery_port", 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8765", 2024-06-10T00:38:40.437 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "cephadm service discovery port", 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "ssh_config_file": { 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssh_config_file", 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.438 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-06-10T00:38:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "use_agent": { 2024-06-10T00:38:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "use_agent", 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "use_repo_digest": { 2024-06-10T00:38:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "use_repo_digest", 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_on_failed_host_check": { 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_on_failed_host_check", 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise a health warning if the host check fails", 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_on_stray_daemons": { 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_on_stray_daemons", 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_on_stray_hosts": { 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_on_stray_hosts", 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crash", 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "retain_interval": { 2024-06-10T00:38:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "retain_interval", 2024-06-10T00:38:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "31536000", 2024-06-10T00:38:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how long to retain crashes before pruning them", 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_recent_interval": { 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_recent_interval", 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "1209600", 2024-06-10T00:38:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "time interval in which to warn about recent crashes", 2024-06-10T00:38:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "dashboard", 2024-06-10T00:38:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "ALERTMANAGER_API_HOST": { 2024-06-10T00:38:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ALERTMANAGER_API_HOST", 2024-06-10T00:38:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-06-10T00:38:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-06-10T00:38:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "AUDIT_API_ENABLED": { 2024-06-10T00:38:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "AUDIT_API_ENABLED", 2024-06-10T00:38:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.460 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-06-10T00:38:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-06-10T00:38:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.461 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "ENABLE_BROWSABLE_API": { 2024-06-10T00:38:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ENABLE_BROWSABLE_API", 2024-06-10T00:38:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.463 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.464 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.465 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.466 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.467 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_NFS": { 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.468 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_RBD": { 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.469 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_RGW": { 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.470 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-06-10T00:38:40.471 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_PASSWORD": { 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_PASSWORD", 2024-06-10T00:38:40.472 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "admin", 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-06-10T00:38:40.473 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_URL": { 2024-06-10T00:38:40.474 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_URL", 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.475 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_USERNAME": { 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_USERNAME", 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "admin", 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.476 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.477 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.478 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-06-10T00:38:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-06-10T00:38:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.479 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "ISSUE_TRACKER_API_KEY": { 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.480 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "PROMETHEUS_API_HOST": { 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PROMETHEUS_API_HOST", 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.481 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.482 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.483 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.484 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-06-10T00:38:40.485 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.486 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-06-10T00:38:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-06-10T00:38:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.487 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.488 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-06-10T00:38:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-06-10T00:38:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.489 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-06-10T00:38:40.490 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_ENABLED": { 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_ENABLED", 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.492 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.493 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.493 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.493 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-06-10T00:38:40.493 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-06-10T00:38:40.493 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.493 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.493 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.493 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-06-10T00:38:40.493 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.493 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.493 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.494 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.494 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.494 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.494 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.494 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.494 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-06-10T00:38:40.494 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-06-10T00:38:40.494 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.495 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.495 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.495 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.495 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.495 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.495 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.495 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.495 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.495 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.495 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.495 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8", 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.496 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "REST_REQUESTS_TIMEOUT": { 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "45", 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.497 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_ACCESS_KEY": { 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_ACCESS_KEY", 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.498 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.499 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.499 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.499 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-06-10T00:38:40.499 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-06-10T00:38:40.499 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.499 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.499 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.499 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "admin", 2024-06-10T00:38:40.499 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.499 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.499 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_SECRET_KEY": { 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_SECRET_KEY", 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.500 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_SSL_VERIFY": { 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_SSL_VERIFY", 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.501 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:40.502 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.503 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.504 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "cross_origin_url": { 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cross_origin_url", 2024-06-10T00:38:40.505 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "crt_file": { 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crt_file", 2024-06-10T00:38:40.506 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.507 INFO:teuthology.orchestra.run.smithi121.stdout: "debug": { 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "debug", 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Enable/disable debug options", 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.508 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "jwt_token_ttl": { 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "jwt_token_ttl", 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "28800", 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.509 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "key_file": { 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "key_file", 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.510 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.511 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.511 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.511 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.511 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.511 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.511 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.511 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.511 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.511 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.511 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.511 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.512 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.513 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.514 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.514 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.514 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.514 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.514 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.514 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.514 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.514 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.514 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.514 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.514 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.515 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.515 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.515 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.515 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.515 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.515 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.515 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.515 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.515 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.515 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.516 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.516 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.516 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.516 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.516 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.516 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.516 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.516 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.516 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.516 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.516 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.517 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.517 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.517 INFO:teuthology.orchestra.run.smithi121.stdout: "motd": { 2024-06-10T00:38:40.517 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "motd", 2024-06-10T00:38:40.517 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.517 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.517 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.517 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.517 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.517 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.517 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "The message of the day", 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: "redirect_resolve_ip_addr": { 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "redirect_resolve_ip_addr", 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.518 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "server_addr": { 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_addr", 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.519 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "::", 2024-06-10T00:38:40.520 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.520 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.520 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.520 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.520 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.520 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.520 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.520 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.520 INFO:teuthology.orchestra.run.smithi121.stdout: "server_port": { 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_port", 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8080", 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.521 INFO:teuthology.orchestra.run.smithi121.stdout: "ssl": { 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssl", 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.522 INFO:teuthology.orchestra.run.smithi121.stdout: "ssl_server_port": { 2024-06-10T00:38:40.523 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssl_server_port", 2024-06-10T00:38:40.523 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.523 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.523 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.523 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8443", 2024-06-10T00:38:40.523 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.523 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.523 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.523 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.523 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.524 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.524 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.524 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.524 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_behaviour": { 2024-06-10T00:38:40.524 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_behaviour", 2024-06-10T00:38:40.524 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.524 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.524 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.524 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "redirect", 2024-06-10T00:38:40.524 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: "redirect" 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_error_status_code": { 2024-06-10T00:38:40.525 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_error_status_code", 2024-06-10T00:38:40.526 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.526 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.526 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.526 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "500", 2024-06-10T00:38:40.526 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "400", 2024-06-10T00:38:40.526 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "599", 2024-06-10T00:38:40.526 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.526 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.526 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.526 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.527 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.527 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.527 INFO:teuthology.orchestra.run.smithi121.stdout: "url_prefix": { 2024-06-10T00:38:40.527 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "url_prefix", 2024-06-10T00:38:40.527 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.527 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.527 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.527 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.527 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.527 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.527 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.528 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.528 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.528 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.528 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.528 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.528 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.528 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.528 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "devicehealth", 2024-06-10T00:38:40.528 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.529 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.529 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.529 INFO:teuthology.orchestra.run.smithi121.stdout: "enable_monitoring": { 2024-06-10T00:38:40.529 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enable_monitoring", 2024-06-10T00:38:40.529 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.529 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.529 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.529 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.529 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.529 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.530 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.530 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "monitor device health metrics", 2024-06-10T00:38:40.530 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.530 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.530 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.530 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.530 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.530 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.530 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.530 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.531 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.531 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.531 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.531 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.531 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.531 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.531 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.531 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.531 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.531 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.532 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.532 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.532 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.532 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.532 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.532 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.532 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.532 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.532 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.532 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.532 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.533 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.534 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.535 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.536 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.536 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.536 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.536 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.536 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.536 INFO:teuthology.orchestra.run.smithi121.stdout: "mark_out_threshold": { 2024-06-10T00:38:40.536 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mark_out_threshold", 2024-06-10T00:38:40.536 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.536 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.536 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.536 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "2419200", 2024-06-10T00:38:40.537 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.537 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.537 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.537 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-06-10T00:38:40.537 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.537 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.537 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.537 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.537 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": { 2024-06-10T00:38:40.538 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pool_name", 2024-06-10T00:38:40.538 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.538 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.538 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.538 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "device_health_metrics", 2024-06-10T00:38:40.538 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.538 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.538 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.538 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "name of pool in which to store device health metrics", 2024-06-10T00:38:40.538 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: "retention_period": { 2024-06-10T00:38:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "retention_period", 2024-06-10T00:38:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "15552000", 2024-06-10T00:38:40.539 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how long to retain device health metrics", 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: "scrape_frequency": { 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "scrape_frequency", 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.540 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "86400", 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to scrape device health metrics", 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "self_heal": { 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "self_heal", 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.541 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.542 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.542 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.542 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.542 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.542 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-06-10T00:38:40.542 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.542 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.542 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.542 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to wake up and check device health", 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.543 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_threshold": { 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_threshold", 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "7257600", 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.544 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.545 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.545 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.545 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.545 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "diskprediction_local", 2024-06-10T00:38:40.545 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.545 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.545 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.545 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.545 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.545 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.546 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.547 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.547 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.547 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.547 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.547 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.547 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.547 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.547 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.547 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.547 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.547 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.548 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.549 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.549 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.549 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.549 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.549 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.549 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.549 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.549 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.549 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.550 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "predict_interval": { 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "predict_interval", 2024-06-10T00:38:40.551 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "86400", 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "predictor_model": { 2024-06-10T00:38:40.552 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "predictor_model", 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "prophetstor", 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.553 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.554 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:40.554 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:40.554 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.554 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.554 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.554 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:40.554 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.554 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.556 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "influx", 2024-06-10T00:38:40.556 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": false, 2024-06-10T00:38:40.556 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "influxdb python module not found", 2024-06-10T00:38:40.556 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.556 INFO:teuthology.orchestra.run.smithi121.stdout: "batch_size": { 2024-06-10T00:38:40.556 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "batch_size", 2024-06-10T00:38:40.556 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.556 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.556 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.556 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5000", 2024-06-10T00:38:40.556 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "database": { 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "database", 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.557 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "ceph", 2024-06-10T00:38:40.558 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.558 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.558 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.558 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-06-10T00:38:40.558 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.558 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.559 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.559 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.559 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": { 2024-06-10T00:38:40.559 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "hostname", 2024-06-10T00:38:40.559 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.559 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.559 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.559 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.559 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.559 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "InfluxDB server hostname", 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.560 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "30", 2024-06-10T00:38:40.561 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "5", 2024-06-10T00:38:40.561 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.561 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.561 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-06-10T00:38:40.561 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.561 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.561 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.561 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.561 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.561 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.562 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.562 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.562 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.562 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.562 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.562 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.562 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.562 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.562 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.563 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.563 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.563 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.563 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.563 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.563 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.563 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.563 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.563 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.563 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.563 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.564 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.564 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.564 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.564 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.564 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.564 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.564 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.564 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.564 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.565 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.565 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.565 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.565 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.565 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.565 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.565 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.565 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.566 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.567 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "password": { 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "password", 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "password of InfluxDB server user", 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "port": { 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "port", 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8086", 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.569 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "InfluxDB server port", 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: "ssl": { 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssl", 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "false", 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.570 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.571 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.571 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-06-10T00:38:40.571 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.571 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.571 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.571 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.571 INFO:teuthology.orchestra.run.smithi121.stdout: "threads": { 2024-06-10T00:38:40.571 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "threads", 2024-06-10T00:38:40.571 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.571 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.571 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "32", 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "username": { 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "username", 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.572 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "username of InfluxDB server user", 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "verify_ssl": { 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "verify_ssl", 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.573 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "true", 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "insights", 2024-06-10T00:38:40.574 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.575 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.575 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.575 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.575 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.575 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.575 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.575 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.575 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.575 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.575 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.575 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.576 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.577 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.578 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.579 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "iostat", 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.581 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.582 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.583 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.584 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.584 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.584 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.584 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.584 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.584 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.584 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.584 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.584 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.584 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.584 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.585 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.586 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.586 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.586 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.586 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.586 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.586 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.586 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.586 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "k8sevents", 2024-06-10T00:38:40.586 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_event_retention_days": { 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ceph_event_retention_days", 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "7", 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Days to hold ceph event information within local cache", 2024-06-10T00:38:40.587 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "config_check_secs": { 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "config_check_secs", 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "10", 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-06-10T00:38:40.588 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.589 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.590 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.591 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.592 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.593 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.594 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.594 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.594 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.594 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.594 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.594 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.594 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.594 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.594 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.594 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "localpool", 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: "failure_domain": { 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "failure_domain", 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "host", 2024-06-10T00:38:40.595 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "failure domain for any created local pool", 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.596 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.597 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.598 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.598 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.598 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.598 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.598 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.598 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.598 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.598 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.598 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.598 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.598 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.599 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.599 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.599 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.599 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.599 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.599 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.599 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.599 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.599 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.599 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.599 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.600 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.600 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.600 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.600 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.600 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.600 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.600 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.600 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.600 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.600 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.601 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.601 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.601 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.601 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.601 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.601 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.601 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.601 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.601 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.601 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": { 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "min_size", 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.602 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "default min_size for any created local pool", 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "num_rep": { 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "num_rep", 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.603 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "3", 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "default replica count for any created local pool", 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": { 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pg_num", 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.604 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.605 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "128", 2024-06-10T00:38:40.605 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.605 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.605 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.605 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "default pg_num for any created local pool", 2024-06-10T00:38:40.605 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.605 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.605 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.606 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.606 INFO:teuthology.orchestra.run.smithi121.stdout: "prefix": { 2024-06-10T00:38:40.606 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "prefix", 2024-06-10T00:38:40.606 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.606 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.606 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.606 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.606 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.606 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.606 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.606 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "name prefix for any created local pool", 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "subtree": { 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "subtree", 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "rack", 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.607 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "CRUSH level for which to create a local pool", 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mds_autoscaler", 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.608 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.609 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.610 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.611 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.612 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.613 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mirroring", 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.614 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.616 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.617 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.618 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.620 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.620 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "nfs", 2024-06-10T00:38:40.620 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.620 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.620 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.620 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.620 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.620 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.620 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.620 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.620 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.621 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.622 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.623 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.624 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.624 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.624 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.624 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.624 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.624 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.624 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.624 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.624 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.624 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.624 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.625 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.625 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.625 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.625 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.625 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.625 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.625 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.625 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.625 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.625 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.626 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.626 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.626 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.626 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.626 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.626 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.626 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.626 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.626 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "orchestrator", 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: "fail_fs": { 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "fail_fs", 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.630 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.631 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.631 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.631 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.631 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.631 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-06-10T00:38:40.631 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.631 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.631 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.631 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.632 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.632 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.632 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.632 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.632 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.632 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.632 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.632 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.632 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.632 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.632 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.633 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.634 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.635 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.636 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.637 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.637 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.637 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.637 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.637 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.637 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.637 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.637 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.637 INFO:teuthology.orchestra.run.smithi121.stdout: "orchestrator": { 2024-06-10T00:38:40.638 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "orchestrator", 2024-06-10T00:38:40.638 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.638 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.638 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.638 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.638 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.638 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.638 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.638 INFO:teuthology.orchestra.run.smithi121.stdout: "cephadm", 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: "rook", 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: "test_orchestrator" 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Orchestrator backend", 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "osd_perf_query", 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.639 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.640 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.640 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.640 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.640 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.640 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.640 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.640 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.640 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.640 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.640 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.641 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.642 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "osd_support", 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.645 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.646 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.648 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.649 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.650 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pg_autoscaler", 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.651 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.652 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.653 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.654 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.655 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:40.656 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.657 INFO:teuthology.orchestra.run.smithi121.stdout: "threshold": { 2024-06-10T00:38:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "threshold", 2024-06-10T00:38:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "3.0", 2024-06-10T00:38:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1.0", 2024-06-10T00:38:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "scaling threshold", 2024-06-10T00:38:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-06-10T00:38:40.658 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "progress", 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "allow_pg_recovery_event": { 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "allow_pg_recovery_event", 2024-06-10T00:38:40.659 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "allow the module to show pg recovery progress", 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.660 INFO:teuthology.orchestra.run.smithi121.stdout: "enabled": { 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enabled", 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.661 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.662 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.663 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.664 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.665 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.666 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "max_completed_events": { 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_completed_events", 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.667 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "50", 2024-06-10T00:38:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "number of past completed events to remember", 2024-06-10T00:38:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:40.668 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how long the module is going to sleep", 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.669 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "prometheus", 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "cache": { 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cache", 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.670 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "exclude_perf_counters": { 2024-06-10T00:38:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "exclude_perf_counters", 2024-06-10T00:38:40.671 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-06-10T00:38:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-06-10T00:38:40.672 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.673 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.674 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.675 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.676 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.677 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.678 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_stats_pools": { 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rbd_stats_pools", 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.679 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_stats_pools_refresh_interval": { 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "300", 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.680 INFO:teuthology.orchestra.run.smithi121.stdout: "scrape_interval": { 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "scrape_interval", 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "15.0", 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.681 INFO:teuthology.orchestra.run.smithi121.stdout: "server_addr": { 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_addr", 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "::", 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.682 INFO:teuthology.orchestra.run.smithi121.stdout: "server_port": { 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_port", 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "9283", 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "stale_cache_strategy": { 2024-06-10T00:38:40.683 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "stale_cache_strategy", 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "log", 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.684 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_behaviour": { 2024-06-10T00:38:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_behaviour", 2024-06-10T00:38:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "default", 2024-06-10T00:38:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.685 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "default", 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "error" 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_error_status_code": { 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_error_status_code", 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.686 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "500", 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "400", 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "599", 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rbd_support", 2024-06-10T00:38:40.687 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.688 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.689 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.690 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.691 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.692 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "max_concurrent_snap_create": { 2024-06-10T00:38:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_concurrent_snap_create", 2024-06-10T00:38:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.693 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "mirror_snapshot_schedule": { 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mirror_snapshot_schedule", 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.694 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "trash_purge_schedule": { 2024-06-10T00:38:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "trash_purge_schedule", 2024-06-10T00:38:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.695 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.696 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "restful", 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "enable_auth": { 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enable_auth", 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.697 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "key_file": { 2024-06-10T00:38:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "key_file", 2024-06-10T00:38:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.698 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.699 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.700 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.701 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.702 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.703 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.704 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.705 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "server_addr": { 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_addr", 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.706 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "server_port": { 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_port", 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.707 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rgw", 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.708 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.709 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.710 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.711 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.712 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.713 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rook", 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.714 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.715 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.716 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.717 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.718 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "storage_class": { 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "storage_class", 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "local", 2024-06-10T00:38:40.719 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "selftest", 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.720 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.721 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.722 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.723 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.724 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "roption1": { 2024-06-10T00:38:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "roption1", 2024-06-10T00:38:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.726 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "roption2": { 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "roption2", 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "xyz", 2024-06-10T00:38:40.727 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption1": { 2024-06-10T00:38:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption1", 2024-06-10T00:38:40.728 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption2": { 2024-06-10T00:38:40.729 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption2", 2024-06-10T00:38:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.730 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption3": { 2024-06-10T00:38:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption3", 2024-06-10T00:38:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.732 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption4": { 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption4", 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.733 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption5": { 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption5", 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.734 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption6": { 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption6", 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.735 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption7": { 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption7", 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "42", 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.736 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "testkey": { 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "testkey", 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.737 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "testlkey": { 2024-06-10T00:38:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "testlkey", 2024-06-10T00:38:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.738 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.739 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "testnewline": { 2024-06-10T00:38:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "testnewline", 2024-06-10T00:38:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.740 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.741 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "snap_schedule", 2024-06-10T00:38:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "allow_m_granularity": { 2024-06-10T00:38:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "allow_m_granularity", 2024-06-10T00:38:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.742 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "allow minute scheduled snapshots", 2024-06-10T00:38:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.743 INFO:teuthology.orchestra.run.smithi121.stdout: "dump_on_update": { 2024-06-10T00:38:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "dump_on_update", 2024-06-10T00:38:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "dump database to debug log on update", 2024-06-10T00:38:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.744 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.745 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.746 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.747 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.748 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.749 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.750 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "stats", 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.751 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.752 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.753 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.754 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.755 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.756 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "status", 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.757 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.758 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.759 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.760 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.761 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.762 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "telegraf", 2024-06-10T00:38:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "address": { 2024-06-10T00:38:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "address", 2024-06-10T00:38:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.763 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.764 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "15", 2024-06-10T00:38:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.765 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.766 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.767 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.768 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.770 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.771 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.772 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "telemetry", 2024-06-10T00:38:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.773 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_basic": { 2024-06-10T00:38:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_basic", 2024-06-10T00:38:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.774 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share basic cluster information (size, version)", 2024-06-10T00:38:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_crash": { 2024-06-10T00:38:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_crash", 2024-06-10T00:38:40.775 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.776 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_device": { 2024-06-10T00:38:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_device", 2024-06-10T00:38:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-06-10T00:38:40.777 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_ident": { 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_ident", 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.778 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_perf": { 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_perf", 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.779 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share various performance metrics of a cluster", 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "contact": { 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "contact", 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.780 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "description": { 2024-06-10T00:38:40.781 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "description", 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.782 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "device_url": { 2024-06-10T00:38:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "device_url", 2024-06-10T00:38:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-06-10T00:38:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.783 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "enabled": { 2024-06-10T00:38:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enabled", 2024-06-10T00:38:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.784 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:40.785 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "24", 2024-06-10T00:38:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "8", 2024-06-10T00:38:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.786 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "last_opt_revision": { 2024-06-10T00:38:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "last_opt_revision", 2024-06-10T00:38:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "1", 2024-06-10T00:38:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.787 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "leaderboard": { 2024-06-10T00:38:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "leaderboard", 2024-06-10T00:38:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.788 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "leaderboard_description": { 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "leaderboard_description", 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.789 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.790 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.791 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.792 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.793 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.794 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.795 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "organization": { 2024-06-10T00:38:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "organization", 2024-06-10T00:38:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.796 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "proxy": { 2024-06-10T00:38:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "proxy", 2024-06-10T00:38:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.797 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.798 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "url": { 2024-06-10T00:38:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "url", 2024-06-10T00:38:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-06-10T00:38:40.799 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.800 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "test_orchestrator", 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.801 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.802 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.803 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.804 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.805 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.806 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "volumes", 2024-06-10T00:38:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.807 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.808 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.809 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.810 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.811 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.812 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.813 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "max_concurrent_clones": { 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_concurrent_clones", 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.814 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "4", 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Number of asynchronous cloner threads", 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "periodic_async_work": { 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "periodic_async_work", 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.815 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Periodically check for async work", 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "snapshot_clone_delay": { 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "snapshot_clone_delay", 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.816 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "snapshot_clone_no_wait": { 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "snapshot_clone_no_wait", 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.817 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.818 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix", 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "discovery_interval": { 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "discovery_interval", 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "100", 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.819 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "identifier": { 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "identifier", 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.820 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.821 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.822 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.823 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.824 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.825 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.826 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "zabbix_host": { 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix_host", 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.827 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "zabbix_port": { 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix_port", 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10051", 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.828 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: "zabbix_sender": { 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix_sender", 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.829 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: "modules": [ 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: "cephadm", 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: "dashboard", 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: "iostat", 2024-06-10T00:38:40.830 INFO:teuthology.orchestra.run.smithi121.stdout: "nfs", 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: "restful", 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: "snap_schedule" 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: "available_modules": [ 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "alerts", 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.831 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.833 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.833 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:40.833 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.833 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.834 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.834 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How frequently to reexamine health status", 2024-06-10T00:38:40.834 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.834 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.834 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.834 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.834 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.834 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.834 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.835 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.836 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.836 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.836 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.836 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.836 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.836 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.836 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.836 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.836 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.837 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.837 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.837 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.837 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.837 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.837 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.837 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.837 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.837 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.837 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.837 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.838 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.838 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.838 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.838 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.838 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.838 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.838 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.838 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.838 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.838 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.838 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.839 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.839 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.839 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.839 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.839 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.839 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.839 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.839 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.839 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.839 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.840 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.840 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.840 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.840 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.840 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.840 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.840 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.840 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.840 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.840 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.840 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_destination": { 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_destination", 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Email address to send alerts to", 2024-06-10T00:38:40.841 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.842 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.842 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.842 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.842 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_from_name": { 2024-06-10T00:38:40.842 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_from_name", 2024-06-10T00:38:40.842 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.842 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.842 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.842 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "Ceph", 2024-06-10T00:38:40.842 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.843 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.843 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.843 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Email From: name", 2024-06-10T00:38:40.843 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.843 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.843 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.843 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.843 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_host": { 2024-06-10T00:38:40.843 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_host", 2024-06-10T00:38:40.844 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.844 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.844 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.844 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.844 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.844 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.844 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.844 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SMTP server", 2024-06-10T00:38:40.845 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.845 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.845 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.845 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.845 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_password": { 2024-06-10T00:38:40.845 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_password", 2024-06-10T00:38:40.845 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.845 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.845 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.846 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.846 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.846 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.846 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.846 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Password to authenticate with", 2024-06-10T00:38:40.846 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.846 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.846 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.846 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.846 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_port": { 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_port", 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "465", 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SMTP port", 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.847 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_sender": { 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_sender", 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SMTP envelope sender", 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.848 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_ssl": { 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_ssl", 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use SSL to connect to SMTP server", 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.849 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "smtp_user": { 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "smtp_user", 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "User to authenticate as", 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.850 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "balancer", 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: "active": { 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "active", 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.851 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "automatically balance PGs across cluster", 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "begin_time": { 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "begin_time", 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.852 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0000", 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "beginning time of day to automatically balance", 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "begin_weekday": { 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "begin_weekday", 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.853 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "0", 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "6", 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_compat_max_iterations": { 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crush_compat_max_iterations", 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.854 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "25", 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "250", 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_compat_metrics": { 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crush_compat_metrics", 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.855 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "pgs,objects,bytes", 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "crush_compat_step": { 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crush_compat_step", 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.856 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0.5", 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "0.001", 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "0.999", 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "aggressiveness of optimization", 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "end_time": { 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "end_time", 2024-06-10T00:38:40.857 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "2359", 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "ending time of day to automatically balance", 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "end_weekday": { 2024-06-10T00:38:40.858 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "end_weekday", 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "0", 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "6", 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.859 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.860 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.861 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.862 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.863 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "min_score": { 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "min_score", 2024-06-10T00:38:40.864 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.865 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.865 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.865 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:40.865 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.865 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.865 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.865 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-06-10T00:38:40.865 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.865 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.865 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.865 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": { 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mode", 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "upmap", 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "crush-compat", 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "none", 2024-06-10T00:38:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: "read", 2024-06-10T00:38:40.867 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap", 2024-06-10T00:38:40.867 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap-read" 2024-06-10T00:38:40.867 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.867 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Balancer mode", 2024-06-10T00:38:40.867 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.867 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.867 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.867 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.867 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_ids": { 2024-06-10T00:38:40.867 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pool_ids", 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:40.868 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.869 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap_max_deviation": { 2024-06-10T00:38:40.870 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "upmap_max_deviation", 2024-06-10T00:38:40.870 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.870 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.870 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.870 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:40.870 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:40.870 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.870 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.870 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "deviation below which no optimization is attempted", 2024-06-10T00:38:40.871 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-06-10T00:38:40.871 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.871 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.871 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.871 INFO:teuthology.orchestra.run.smithi121.stdout: "upmap_max_optimizations": { 2024-06-10T00:38:40.871 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "upmap_max_optimizations", 2024-06-10T00:38:40.871 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:40.871 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.871 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.871 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.871 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cephadm", 2024-06-10T00:38:40.872 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "agent_down_multiplier": { 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "agent_down_multiplier", 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "3.0", 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.873 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-06-10T00:38:40.874 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.874 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.874 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.874 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.874 INFO:teuthology.orchestra.run.smithi121.stdout: "agent_refresh_rate": { 2024-06-10T00:38:40.874 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "agent_refresh_rate", 2024-06-10T00:38:40.874 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.874 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.874 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.874 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "20", 2024-06-10T00:38:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-06-10T00:38:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: "agent_starting_port": { 2024-06-10T00:38:40.875 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "agent_starting_port", 2024-06-10T00:38:40.876 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.876 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.876 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.876 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "4721", 2024-06-10T00:38:40.876 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.876 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.876 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.876 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: "allow_ptrace": { 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "allow_ptrace", 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.877 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.878 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.878 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-06-10T00:38:40.878 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-06-10T00:38:40.878 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.878 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.878 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.878 INFO:teuthology.orchestra.run.smithi121.stdout: "autotune_interval": { 2024-06-10T00:38:40.878 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "autotune_interval", 2024-06-10T00:38:40.878 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.878 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.878 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to autotune daemon memory", 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: "autotune_memory_target_ratio": { 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "autotune_memory_target_ratio", 2024-06-10T00:38:40.879 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0.7", 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.880 INFO:teuthology.orchestra.run.smithi121.stdout: "cephadm_log_destination": { 2024-06-10T00:38:40.881 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cephadm_log_destination", 2024-06-10T00:38:40.881 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.881 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.881 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.881 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.881 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.881 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.881 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.881 INFO:teuthology.orchestra.run.smithi121.stdout: "file", 2024-06-10T00:38:40.881 INFO:teuthology.orchestra.run.smithi121.stdout: "file,syslog", 2024-06-10T00:38:40.881 INFO:teuthology.orchestra.run.smithi121.stdout: "syslog" 2024-06-10T00:38:40.882 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.882 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-06-10T00:38:40.882 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.882 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.882 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.882 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.882 INFO:teuthology.orchestra.run.smithi121.stdout: "cgroups_split": { 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cgroups_split", 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.883 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "config_checks_enabled": { 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "config_checks_enabled", 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.884 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "config_dashboard": { 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "config_dashboard", 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.885 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_alertmanager": { 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_alertmanager", 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.886 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_base": { 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_base", 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/ceph", 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Container image name, without the tag", 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.887 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_elasticsearch": { 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_elasticsearch", 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "elasticsearch container image", 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.888 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_grafana": { 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_grafana", 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.889 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_haproxy": { 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_haproxy", 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "HAproxy container image", 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.890 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_jaeger_agent": { 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_jaeger_agent", 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Jaeger agent container image", 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.891 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_jaeger_collector": { 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_jaeger_collector", 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Jaeger collector container image", 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.892 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_jaeger_query": { 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_jaeger_query", 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Jaeger query container image", 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.893 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_keepalived": { 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_keepalived", 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Keepalived container image", 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.894 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_loki": { 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_loki", 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Loki container image", 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.895 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_node_exporter": { 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_node_exporter", 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.896 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_nvmeof": { 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_nvmeof", 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Nvme-of container image", 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.897 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_prometheus": { 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_prometheus", 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Prometheus container image", 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.898 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_promtail": { 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_promtail", 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Promtail container image", 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.899 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_samba": { 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_samba", 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Samba/SMB container image", 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.900 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "container_image_snmp_gateway": { 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_image_snmp_gateway", 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "SNMP Gateway container image", 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.901 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "container_init": { 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "container_init", 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Run podman/docker with `--init`", 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.902 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "daemon_cache_timeout": { 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "daemon_cache_timeout", 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-06-10T00:38:40.903 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: "default_cephadm_command_timeout": { 2024-06-10T00:38:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "default_cephadm_command_timeout", 2024-06-10T00:38:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.904 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "900", 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "default_registry": { 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "default_registry", 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.905 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "docker.io", 2024-06-10T00:38:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-06-10T00:38:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.906 INFO:teuthology.orchestra.run.smithi121.stdout: "device_cache_timeout": { 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "device_cache_timeout", 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "1800", 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "seconds to cache device inventory", 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.907 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "device_enhanced_scan": { 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "device_enhanced_scan", 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use libstoragemgmt during device scans", 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.908 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.909 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.909 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.909 INFO:teuthology.orchestra.run.smithi121.stdout: "facts_cache_timeout": { 2024-06-10T00:38:40.909 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "facts_cache_timeout", 2024-06-10T00:38:40.909 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.909 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.909 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.909 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:40.909 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.909 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.909 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "seconds to cache host facts data", 2024-06-10T00:38:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: "grafana_dashboards_path": { 2024-06-10T00:38:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "grafana_dashboards_path", 2024-06-10T00:38:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "host_check_interval": { 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "host_check_interval", 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.911 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.912 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:40.912 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.912 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.912 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.912 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to perform a host check", 2024-06-10T00:38:40.912 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.912 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.912 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.912 INFO:teuthology.orchestra.run.smithi121.stdout: "hw_monitoring": { 2024-06-10T00:38:40.912 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "hw_monitoring", 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.913 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.914 INFO:teuthology.orchestra.run.smithi121.stdout: "inventory_list_all": { 2024-06-10T00:38:40.914 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "inventory_list_all", 2024-06-10T00:38:40.914 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.914 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.914 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.914 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.914 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.914 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.914 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.915 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-06-10T00:38:40.915 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.915 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.915 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.915 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.915 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.915 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.915 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.916 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.916 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.916 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.916 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.916 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.916 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.916 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.916 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.916 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.916 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "log_refresh_metadata": { 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_refresh_metadata", 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.917 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.918 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.919 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.920 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "manage_etc_ceph_ceph_conf": { 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.922 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "*", 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.923 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "max_count_per_host": { 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_count_per_host", 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "max number of daemons per service per host", 2024-06-10T00:38:40.924 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "max_osd_draining_count": { 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_osd_draining_count", 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-06-10T00:38:40.925 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "migration_current": { 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "migration_current", 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.926 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "internal - do not modify", 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "mode": { 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mode", 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "root", 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "cephadm-package", 2024-06-10T00:38:40.927 INFO:teuthology.orchestra.run.smithi121.stdout: "root" 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "mode for remote execution of cephadm", 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "oob_default_addr": { 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "oob_default_addr", 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "169.254.1.1", 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.928 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Default address for RedFish API (oob management).", 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "prometheus_alerts_path": { 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "prometheus_alerts_path", 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.929 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_insecure": { 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_insecure", 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.930 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_password": { 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_password", 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.931 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_url": { 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_url", 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.932 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "registry_username": { 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "registry_username", 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.933 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "secure_monitoring_stack": { 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "secure_monitoring_stack", 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.934 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "service_discovery_port": { 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "service_discovery_port", 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8765", 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.935 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "cephadm service discovery port", 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "ssh_config_file": { 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssh_config_file", 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.936 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.937 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.937 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.937 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-06-10T00:38:40.937 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.937 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.937 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.937 INFO:teuthology.orchestra.run.smithi121.stdout: "use_agent": { 2024-06-10T00:38:40.937 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "use_agent", 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.938 INFO:teuthology.orchestra.run.smithi121.stdout: "use_repo_digest": { 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "use_repo_digest", 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.940 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_on_failed_host_check": { 2024-06-10T00:38:40.940 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_on_failed_host_check", 2024-06-10T00:38:40.940 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.940 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.940 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.940 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.940 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.940 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.940 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.940 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise a health warning if the host check fails", 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_on_stray_daemons": { 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_on_stray_daemons", 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.941 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.942 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-06-10T00:38:40.942 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.942 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.942 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.942 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_on_stray_hosts": { 2024-06-10T00:38:40.942 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_on_stray_hosts", 2024-06-10T00:38:40.942 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.942 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.942 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.943 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.944 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crash", 2024-06-10T00:38:40.944 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.944 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.944 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.944 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:40.944 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:40.944 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.944 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.944 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.944 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.944 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.945 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.946 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.946 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:40.946 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:40.946 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.946 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.946 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.946 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.946 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.947 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.947 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.947 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.947 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.947 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.947 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.947 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:40.947 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:40.947 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.948 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.948 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.948 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:40.948 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.948 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.948 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:40.948 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:40.948 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:40.948 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:40.948 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.949 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "retain_interval": { 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "retain_interval", 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.950 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "31536000", 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how long to retain crashes before pruning them", 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_recent_interval": { 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_recent_interval", 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:40.951 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "1209600", 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "time interval in which to warn about recent crashes", 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "dashboard", 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:40.952 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:40.953 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:40.953 INFO:teuthology.orchestra.run.smithi121.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-06-10T00:38:40.953 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-06-10T00:38:40.953 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.953 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.953 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.953 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.953 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.953 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: "ALERTMANAGER_API_HOST": { 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ALERTMANAGER_API_HOST", 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.954 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.955 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "AUDIT_API_ENABLED": { 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "AUDIT_API_ENABLED", 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.956 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.957 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "ENABLE_BROWSABLE_API": { 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ENABLE_BROWSABLE_API", 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.958 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.959 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.960 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.961 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.962 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_NFS": { 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.963 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_RBD": { 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.964 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "FEATURE_TOGGLE_RGW": { 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.965 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.966 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_PASSWORD": { 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_PASSWORD", 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "admin", 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.967 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.968 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_URL": { 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_URL", 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.969 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_API_USERNAME": { 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_API_USERNAME", 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "admin", 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.970 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.971 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.972 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.973 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "ISSUE_TRACKER_API_KEY": { 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.974 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.975 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.975 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.975 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.975 INFO:teuthology.orchestra.run.smithi121.stdout: "PROMETHEUS_API_HOST": { 2024-06-10T00:38:40.975 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PROMETHEUS_API_HOST", 2024-06-10T00:38:40.975 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.975 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-06-10T00:38:40.976 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.977 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.977 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.977 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.977 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.977 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.977 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.977 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.977 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.977 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.978 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.979 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.979 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.979 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-06-10T00:38:40.979 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-06-10T00:38:40.979 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.979 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.979 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.979 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.979 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.981 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.981 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.981 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.981 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.981 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.981 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.981 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.982 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.983 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.983 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-06-10T00:38:40.983 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-06-10T00:38:40.983 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.983 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.983 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.983 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.984 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-06-10T00:38:40.985 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-06-10T00:38:40.985 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.985 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.985 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.985 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.985 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.985 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.985 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.985 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.985 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.986 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.986 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.986 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.986 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-06-10T00:38:40.986 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-06-10T00:38:40.986 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.986 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.986 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.986 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:40.987 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.987 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.987 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.987 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.987 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.987 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.987 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.987 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.988 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_ENABLED": { 2024-06-10T00:38:40.988 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_ENABLED", 2024-06-10T00:38:40.988 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.988 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.988 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.988 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.988 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.989 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.990 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8", 2024-06-10T00:38:40.991 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "REST_REQUESTS_TIMEOUT": { 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "45", 2024-06-10T00:38:40.992 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_ACCESS_KEY": { 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_ACCESS_KEY", 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.993 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "admin", 2024-06-10T00:38:40.994 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_SECRET_KEY": { 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_SECRET_KEY", 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.995 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "RGW_API_SSL_VERIFY": { 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "RGW_API_SSL_VERIFY", 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.996 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.997 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-06-10T00:38:40.998 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:40.999 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "cross_origin_url": { 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cross_origin_url", 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.000 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "crt_file": { 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "crt_file", 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.001 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "debug": { 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "debug", 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.002 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Enable/disable debug options", 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "jwt_token_ttl": { 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "jwt_token_ttl", 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.003 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "28800", 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "key_file": { 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "key_file", 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.004 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.005 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.006 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.007 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.008 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.009 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.009 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.009 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.009 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.009 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.009 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.009 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.009 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.009 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.009 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.009 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.010 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "motd": { 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "motd", 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.011 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.012 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.012 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "The message of the day", 2024-06-10T00:38:41.012 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.012 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.012 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.012 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.012 INFO:teuthology.orchestra.run.smithi121.stdout: "redirect_resolve_ip_addr": { 2024-06-10T00:38:41.012 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "redirect_resolve_ip_addr", 2024-06-10T00:38:41.012 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "server_addr": { 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_addr", 2024-06-10T00:38:41.013 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.014 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.014 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.014 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "::", 2024-06-10T00:38:41.014 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.014 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.014 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.014 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.014 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.014 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "server_port": { 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_port", 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8080", 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.015 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.016 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.016 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.016 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.016 INFO:teuthology.orchestra.run.smithi121.stdout: "ssl": { 2024-06-10T00:38:41.016 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssl", 2024-06-10T00:38:41.016 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.016 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.016 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.016 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.017 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.017 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.017 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.017 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.017 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.017 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.017 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.017 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.017 INFO:teuthology.orchestra.run.smithi121.stdout: "ssl_server_port": { 2024-06-10T00:38:41.017 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssl_server_port", 2024-06-10T00:38:41.017 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8443", 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.018 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_behaviour": { 2024-06-10T00:38:41.019 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_behaviour", 2024-06-10T00:38:41.019 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.019 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.019 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.019 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "redirect", 2024-06-10T00:38:41.019 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.019 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.019 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.019 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.019 INFO:teuthology.orchestra.run.smithi121.stdout: "redirect" 2024-06-10T00:38:41.019 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.020 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.020 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.020 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.020 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.020 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.020 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_error_status_code": { 2024-06-10T00:38:41.020 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_error_status_code", 2024-06-10T00:38:41.020 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.020 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.020 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.021 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "500", 2024-06-10T00:38:41.021 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "400", 2024-06-10T00:38:41.021 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "599", 2024-06-10T00:38:41.021 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.021 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.021 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.021 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.021 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "url_prefix": { 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "url_prefix", 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.022 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "devicehealth", 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: "enable_monitoring": { 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enable_monitoring", 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.023 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "monitor device health metrics", 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.024 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.025 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.026 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.027 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.028 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "mark_out_threshold": { 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mark_out_threshold", 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:41.029 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "2419200", 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "pool_name": { 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pool_name", 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.030 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "device_health_metrics", 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "name of pool in which to store device health metrics", 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "retention_period": { 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "retention_period", 2024-06-10T00:38:41.031 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "15552000", 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how long to retain device health metrics", 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "scrape_frequency": { 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "scrape_frequency", 2024-06-10T00:38:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "86400", 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to scrape device health metrics", 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "self_heal": { 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "self_heal", 2024-06-10T00:38:41.033 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:41.034 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how frequently to wake up and check device health", 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "warn_threshold": { 2024-06-10T00:38:41.035 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "warn_threshold", 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "7257600", 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.036 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "diskprediction_local", 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.037 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.038 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.039 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.040 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.041 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.042 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.042 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.042 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.042 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.042 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.042 INFO:teuthology.orchestra.run.smithi121.stdout: "predict_interval": { 2024-06-10T00:38:41.042 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "predict_interval", 2024-06-10T00:38:41.042 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.042 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.042 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.043 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "86400", 2024-06-10T00:38:41.043 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.043 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.043 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.043 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.043 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.043 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.043 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.043 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "predictor_model": { 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "predictor_model", 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "prophetstor", 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.044 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "600", 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.045 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "influx", 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": false, 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "influxdb python module not found", 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: "batch_size": { 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "batch_size", 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.046 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5000", 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "database": { 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "database", 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.047 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.048 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.048 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "ceph", 2024-06-10T00:38:41.048 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.048 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.048 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.048 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-06-10T00:38:41.048 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.048 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.048 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.048 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.048 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": { 2024-06-10T00:38:41.049 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "hostname", 2024-06-10T00:38:41.049 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.049 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.049 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.049 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.049 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.049 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.049 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.049 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "InfluxDB server hostname", 2024-06-10T00:38:41.049 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.049 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "30", 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "5", 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.050 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-06-10T00:38:41.051 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.051 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.051 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.051 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.051 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.051 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.051 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.051 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.051 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.051 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.051 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.052 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.052 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.052 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.052 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.052 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.052 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.052 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.052 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.052 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.053 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.054 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.054 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.054 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.054 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.054 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.054 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.054 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.054 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.054 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.054 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.055 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.056 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.056 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.056 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.056 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.056 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.056 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.056 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.056 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.056 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.057 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.057 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.057 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.057 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.057 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.057 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.057 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.057 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.057 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.058 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.058 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.058 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.058 INFO:teuthology.orchestra.run.smithi121.stdout: "password": { 2024-06-10T00:38:41.058 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "password", 2024-06-10T00:38:41.058 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.058 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.058 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.058 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.058 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.058 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "password of InfluxDB server user", 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "port": { 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "port", 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "8086", 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.059 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "InfluxDB server port", 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: "ssl": { 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ssl", 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.060 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "false", 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "threads": { 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "threads", 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.061 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "32", 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: "username": { 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "username", 2024-06-10T00:38:41.062 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "username of InfluxDB server user", 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "verify_ssl": { 2024-06-10T00:38:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "verify_ssl", 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "true", 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.064 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "insights", 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.065 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.066 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.067 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.068 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.069 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.070 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "iostat", 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.071 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.072 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.073 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.074 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.075 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "k8sevents", 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_event_retention_days": { 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "ceph_event_retention_days", 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "7", 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Days to hold ceph event information within local cache", 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.077 INFO:teuthology.orchestra.run.smithi121.stdout: "config_check_secs": { 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "config_check_secs", 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "10", 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.078 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.079 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.080 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.080 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.080 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.080 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.080 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.080 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.080 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.080 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.080 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.081 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.082 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.082 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.082 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.082 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.082 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.082 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.082 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.082 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.082 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.082 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.083 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.084 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "localpool", 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: "failure_domain": { 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "failure_domain", 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "host", 2024-06-10T00:38:41.085 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.086 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.086 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.086 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "failure domain for any created local pool", 2024-06-10T00:38:41.086 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-06-10T00:38:41.086 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.086 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.086 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.086 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.086 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.086 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.087 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.088 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.088 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.088 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.088 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.088 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.088 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.088 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.088 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.088 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.088 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.088 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.089 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.089 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.089 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.089 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.089 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.089 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.089 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.089 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.089 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.089 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.090 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.090 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.090 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.090 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.090 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.090 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.090 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.090 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.090 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.090 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.091 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.092 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.092 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.092 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.092 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.092 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.092 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.092 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.092 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.092 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.092 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.092 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "min_size": { 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "min_size", 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "default min_size for any created local pool", 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-06-10T00:38:41.093 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "num_rep": { 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "num_rep", 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "3", 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "default replica count for any created local pool", 2024-06-10T00:38:41.094 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_num": { 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pg_num", 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "128", 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.095 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "default pg_num for any created local pool", 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "prefix": { 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "prefix", 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.096 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "name prefix for any created local pool", 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "subtree": { 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "subtree", 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "rack", 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.097 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "CRUSH level for which to create a local pool", 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mds_autoscaler", 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.098 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.099 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.100 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.101 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.102 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.103 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mirroring", 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.104 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.105 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.106 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.108 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.109 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "nfs", 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.110 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.111 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.112 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.113 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.114 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.114 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.114 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.114 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.114 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.114 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.114 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.114 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.114 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "orchestrator", 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.115 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "fail_fs": { 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "fail_fs", 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-06-10T00:38:41.116 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.117 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.118 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.119 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.120 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.121 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.121 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.121 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.121 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.121 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.121 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.121 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.121 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.121 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.121 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.121 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.122 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.122 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.122 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.122 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.122 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.122 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.122 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.122 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.122 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.122 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.122 INFO:teuthology.orchestra.run.smithi121.stdout: "orchestrator": { 2024-06-10T00:38:41.123 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "orchestrator", 2024-06-10T00:38:41.123 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.123 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.123 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.123 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.123 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.123 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.123 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.123 INFO:teuthology.orchestra.run.smithi121.stdout: "cephadm", 2024-06-10T00:38:41.123 INFO:teuthology.orchestra.run.smithi121.stdout: "rook", 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: "test_orchestrator" 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Orchestrator backend", 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "osd_perf_query", 2024-06-10T00:38:41.124 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.125 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.125 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.125 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.125 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.125 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.125 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.125 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.125 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.125 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.125 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.126 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.126 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.126 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.126 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.126 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.126 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.126 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.126 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.126 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.126 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.126 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.127 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.127 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.127 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.127 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.127 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.127 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.127 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.127 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.128 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.128 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.128 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.128 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.128 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.128 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.128 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.128 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.128 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.128 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.128 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.129 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.129 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.129 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.129 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.129 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.129 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.129 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.129 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.129 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.129 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.130 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "osd_support", 2024-06-10T00:38:41.131 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.132 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.133 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.134 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.135 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.136 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "pg_autoscaler", 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.137 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.138 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.139 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.140 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.141 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.142 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.143 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "threshold": { 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "threshold", 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "3.0", 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1.0", 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "scaling threshold", 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.144 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "progress", 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "allow_pg_recovery_event": { 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "allow_pg_recovery_event", 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.145 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "allow the module to show pg recovery progress", 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "enabled": { 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enabled", 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.146 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.148 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.149 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.150 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.151 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.152 INFO:teuthology.orchestra.run.smithi121.stdout: "max_completed_events": { 2024-06-10T00:38:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_completed_events", 2024-06-10T00:38:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "50", 2024-06-10T00:38:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.153 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "number of past completed events to remember", 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "sleep_interval": { 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "sleep_interval", 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "5", 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.154 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "how long the module is going to sleep", 2024-06-10T00:38:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "prometheus", 2024-06-10T00:38:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.155 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "cache": { 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "cache", 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.156 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "exclude_perf_counters": { 2024-06-10T00:38:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "exclude_perf_counters", 2024-06-10T00:38:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.157 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-06-10T00:38:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-06-10T00:38:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.158 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.159 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.160 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.161 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.162 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.163 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.164 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_stats_pools": { 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rbd_stats_pools", 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.165 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_stats_pools_refresh_interval": { 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "300", 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.166 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "scrape_interval": { 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "scrape_interval", 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "15.0", 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.167 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "server_addr": { 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_addr", 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.168 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "::", 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "server_port": { 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_port", 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.169 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "9283", 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "stale_cache_strategy": { 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "stale_cache_strategy", 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.170 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "log", 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_behaviour": { 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_behaviour", 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.171 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "default", 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "default", 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "error" 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.172 INFO:teuthology.orchestra.run.smithi121.stdout: "standby_error_status_code": { 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "standby_error_status_code", 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "500", 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "400", 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "599", 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.173 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rbd_support", 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.174 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.175 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.176 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.177 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.178 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "max_concurrent_snap_create": { 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_concurrent_snap_create", 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.179 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10", 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "mirror_snapshot_schedule": { 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "mirror_snapshot_schedule", 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.180 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "trash_purge_schedule": { 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "trash_purge_schedule", 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.181 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.182 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "restful", 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "enable_auth": { 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enable_auth", 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.183 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "key_file": { 2024-06-10T00:38:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "key_file", 2024-06-10T00:38:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.184 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.185 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.186 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.187 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.188 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.189 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.190 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.191 INFO:teuthology.orchestra.run.smithi121.stdout: "server_addr": { 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_addr", 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "server_port": { 2024-06-10T00:38:41.192 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "server_port", 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.193 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rgw", 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.194 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.195 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.196 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.197 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.198 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.199 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.200 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rook", 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.201 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.202 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.203 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.204 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.205 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "storage_class": { 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "storage_class", 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.206 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "local", 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.207 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "selftest", 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.208 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.209 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.210 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.211 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "roption1": { 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "roption1", 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.213 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "roption2": { 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "roption2", 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "xyz", 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.214 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption1": { 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption1", 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.215 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption2": { 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption2", 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.216 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption3": { 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption3", 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "float", 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.217 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption4": { 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption4", 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.218 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption5": { 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption5", 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.219 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption6": { 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption6", 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.220 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "rwoption7": { 2024-06-10T00:38:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rwoption7", 2024-06-10T00:38:41.221 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "1", 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "42", 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "testkey": { 2024-06-10T00:38:41.222 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "testkey", 2024-06-10T00:38:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.223 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "testlkey": { 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "testlkey", 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.224 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "testnewline": { 2024-06-10T00:38:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "testnewline", 2024-06-10T00:38:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.225 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "snap_schedule", 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.226 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "allow_m_granularity": { 2024-06-10T00:38:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "allow_m_granularity", 2024-06-10T00:38:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.227 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "allow minute scheduled snapshots", 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "dump_on_update": { 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "dump_on_update", 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.228 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "dump database to debug log on update", 2024-06-10T00:38:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.229 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.230 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.231 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.232 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.233 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.234 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.235 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "stats", 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.236 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.237 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.238 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.239 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.242 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "status", 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.243 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.244 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.245 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.246 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.247 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.248 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "telegraf", 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "address": { 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "address", 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.249 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "15", 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.250 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.251 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.252 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.253 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.255 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "telemetry", 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_basic": { 2024-06-10T00:38:41.256 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_basic", 2024-06-10T00:38:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share basic cluster information (size, version)", 2024-06-10T00:38:41.257 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_crash": { 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_crash", 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.258 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-06-10T00:38:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_device": { 2024-06-10T00:38:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_device", 2024-06-10T00:38:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.259 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_ident": { 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_ident", 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.260 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "channel_perf": { 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "channel_perf", 2024-06-10T00:38:41.261 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Share various performance metrics of a cluster", 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.262 INFO:teuthology.orchestra.run.smithi121.stdout: "contact": { 2024-06-10T00:38:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "contact", 2024-06-10T00:38:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.263 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "description": { 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "description", 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.264 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "device_url": { 2024-06-10T00:38:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "device_url", 2024-06-10T00:38:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-06-10T00:38:41.265 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "enabled": { 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "enabled", 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.266 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.267 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "24", 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "8", 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "last_opt_revision": { 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "last_opt_revision", 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.268 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "1", 2024-06-10T00:38:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.269 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "leaderboard": { 2024-06-10T00:38:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "leaderboard", 2024-06-10T00:38:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.270 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "leaderboard_description": { 2024-06-10T00:38:41.271 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "leaderboard_description", 2024-06-10T00:38:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.272 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.273 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.274 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.275 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.276 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.277 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "organization": { 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "organization", 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.278 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "proxy": { 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "proxy", 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.279 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "url": { 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "url", 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.280 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "test_orchestrator", 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.281 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.283 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.284 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.285 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "volumes", 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.286 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.287 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.288 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.289 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.290 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.291 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "max_concurrent_clones": { 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "max_concurrent_clones", 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "4", 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Number of asynchronous cloner threads", 2024-06-10T00:38:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "periodic_async_work": { 2024-06-10T00:38:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "periodic_async_work", 2024-06-10T00:38:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Periodically check for async work", 2024-06-10T00:38:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "snapshot_clone_delay": { 2024-06-10T00:38:41.294 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "snapshot_clone_delay", 2024-06-10T00:38:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "0", 2024-06-10T00:38:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-06-10T00:38:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.295 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "snapshot_clone_no_wait": { 2024-06-10T00:38:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "snapshot_clone_no_wait", 2024-06-10T00:38:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "True", 2024-06-10T00:38:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.296 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-06-10T00:38:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix", 2024-06-10T00:38:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "can_run": true, 2024-06-10T00:38:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "error_string": "", 2024-06-10T00:38:41.297 INFO:teuthology.orchestra.run.smithi121.stdout: "module_options": { 2024-06-10T00:38:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "discovery_interval": { 2024-06-10T00:38:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "discovery_interval", 2024-06-10T00:38:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "uint", 2024-06-10T00:38:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "100", 2024-06-10T00:38:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.298 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "identifier": { 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "identifier", 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.299 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "interval": { 2024-06-10T00:38:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "interval", 2024-06-10T00:38:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "secs", 2024-06-10T00:38:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.300 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "60", 2024-06-10T00:38:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.301 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "log_level": { 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_level", 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.302 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.303 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster": { 2024-06-10T00:38:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster", 2024-06-10T00:38:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.304 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_cluster_level": { 2024-06-10T00:38:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_cluster_level", 2024-06-10T00:38:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.305 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "info", 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [ 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "", 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "critical", 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "debug", 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "error", 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "info", 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "warning" 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.306 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "log_to_file": { 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "log_to_file", 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "bool", 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 1, 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "False", 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.307 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "zabbix_host": { 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix_host", 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "", 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.308 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "zabbix_port": { 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix_port", 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "int", 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "10051", 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.309 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "zabbix_sender": { 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "zabbix_sender", 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "str", 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "level": "advanced", 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "flags": 0, 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "min": "", 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "max": "", 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "enum_allowed": [], 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "desc": "", 2024-06-10T00:38:41.310 INFO:teuthology.orchestra.run.smithi121.stdout: "long_desc": "", 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "tags": [], 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "see_also": [] 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "services": { 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "dashboard": "https://172.21.15.121:8443/" 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "always_on_modules": { 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "octopus": [ 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "balancer", 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-06-10T00:38:41.311 INFO:teuthology.orchestra.run.smithi121.stdout: "devicehealth", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "orchestrator", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscaler", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "progress", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_support", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "status", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "telemetry", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "volumes" 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "pacific": [ 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "balancer", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "devicehealth", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "orchestrator", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscaler", 2024-06-10T00:38:41.312 INFO:teuthology.orchestra.run.smithi121.stdout: "progress", 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_support", 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "status", 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "telemetry", 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "volumes" 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "quincy": [ 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "balancer", 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "devicehealth", 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "orchestrator", 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscaler", 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "progress", 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_support", 2024-06-10T00:38:41.313 INFO:teuthology.orchestra.run.smithi121.stdout: "status", 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "telemetry", 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "volumes" 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "reef": [ 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "balancer", 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "devicehealth", 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "orchestrator", 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscaler", 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "progress", 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_support", 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "status", 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "telemetry", 2024-06-10T00:38:41.314 INFO:teuthology.orchestra.run.smithi121.stdout: "volumes" 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: ], 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "squid": [ 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "balancer", 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "crash", 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "devicehealth", 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "orchestrator", 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "pg_autoscaler", 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "progress", 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "rbd_support", 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "status", 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "telemetry", 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: "volumes" 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:38:41.315 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "last_failure_osd_epoch": 101, 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "active_clients": [ 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "devicehealth", 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2509575671 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "libcephsqlite", 2024-06-10T00:38:41.316 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 213426499 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "rbd_support", 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2024-06-10T00:38:41.317 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1269721860 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "snap_schedule", 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2580777449 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.318 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "snap_schedule", 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2580102032 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "name": "volumes", 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "addrvec": [ 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: { 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v2", 2024-06-10T00:38:41.319 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "172.21.15.121:0", 2024-06-10T00:38:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2442403653 2024-06-10T00:38:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:38:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:41.320 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-06-10T00:38:41.320 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:38:41.321 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:40 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/707340812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-10T00:38:41.321 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:40 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/969643897' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-06-10T00:38:41.321 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:40 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/707340812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-10T00:38:41.321 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:40 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/969643897' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-06-10T00:38:41.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 -- 172.21.15.121:0/969643897 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35880a2d00 msgr2=0x7f35880a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:41.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35880a2d00 0x7f35880a51c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3590009890 tx=0x7f3590008040 comp rx=0 tx=0).stop 2024-06-10T00:38:41.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 -- 172.21.15.121:0/969643897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a0101230 msgr2=0x7f35a01996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:38:41.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a0101230 0x7f35a01996f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f3594009a50 tx=0x7f3594004270 comp rx=0 tx=0).stop 2024-06-10T00:38:41.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 -- 172.21.15.121:0/969643897 shutdown_connections 2024-06-10T00:38:41.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f35880a2d00 0x7f35880a51c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:41.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35a0106880 0x7f35a01a4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:41.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35a0101230 0x7f35a01996f0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:41.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 --2- 172.21.15.121:0/969643897 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35a0100870 0x7f35a01991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:38:41.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 -- 172.21.15.121:0/969643897 >> 172.21.15.121:0/969643897 conn(0x7f35a0076c20 msgr2=0x7f35a00fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:38:41.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 -- 172.21.15.121:0/969643897 shutdown_connections 2024-06-10T00:38:41.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:38:39.840+0000 7f35a67c1640 1 -- 172.21.15.121:0/969643897 wait complete. 2024-06-10T00:38:41.382 DEBUG:tasks.check_counter:Getting stats from a 2024-06-10T00:38:41.382 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump 2024-06-10T00:38:41.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:40 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/707340812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-06-10T00:38:41.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:40 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/969643897' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-06-10T00:38:41.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:41 smithi193 ceph-mon[36716]: pgmap v332: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 830 KiB/s wr, 129 op/s 2024-06-10T00:38:41.634 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:38:41.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:41 smithi121 ceph-mon[32378]: pgmap v332: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 830 KiB/s wr, 129 op/s 2024-06-10T00:38:41.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:41 smithi135 ceph-mon[36424]: pgmap v332: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 830 KiB/s wr, 129 op/s 2024-06-10T00:38:43.569 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:43 smithi121 ceph-mon[32378]: pgmap v333: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 350 KiB/s wr, 88 op/s 2024-06-10T00:38:43.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:43 smithi193 ceph-mon[36716]: pgmap v333: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 350 KiB/s wr, 88 op/s 2024-06-10T00:38:43.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:43 smithi135 ceph-mon[36424]: pgmap v333: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 350 KiB/s wr, 88 op/s 2024-06-10T00:38:43.962 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:38:43.962 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:38:43.962 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 254, 2024-06-10T00:38:43.962 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 243, 2024-06-10T00:38:43.962 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 1590176, 2024-06-10T00:38:43.962 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 149520, 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 4, 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 0, 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.044292145, 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.014238211, 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.033652010, 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.004343712, 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 242, 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.009950990, 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000041119 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:43.963 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1590176, 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 149520 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 2, 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 186, 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 320, 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 188560, 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 5, 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 3, 2024-06-10T00:38:43.964 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.013148348, 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.008643066, 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.001738776, 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 183, 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.015087598, 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000082445 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:43.965 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 188560 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 4, 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 4, 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 3024, 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 448, 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 2, 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 1, 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.000605675, 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.000036911, 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.000533105, 2024-06-10T00:38:43.966 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 3, 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000687760, 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000229253 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 448 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:43.967 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 0, 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 0, 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 0, 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:38:43.968 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:38:43.969 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:38:43.970 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 148714, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 30, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 0, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 0, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 0, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 0, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:38:43.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 928, 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 2, 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2024-06-10T00:38:43.972 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 256, 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 1554369, 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 256, 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 256, 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 1554369, 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:43.973 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:38:44.959 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-06-10T00:38:44.959 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-06-10T00:38:44.959 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-06-10T00:38:44.959 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-06-10T00:38:44.959 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-06-10T00:38:44.959 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-06-10T00:38:44.960 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-06-10T00:38:44.960 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-06-10T00:38:44.960 DEBUG:tasks.check_counter:Getting stats from d 2024-06-10T00:38:44.960 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.d perf dump 2024-06-10T00:38:45.200 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:38:45.573 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:45 smithi121 ceph-mon[32378]: pgmap v334: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 117 KiB/s wr, 61 op/s 2024-06-10T00:38:45.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:45 smithi193 ceph-mon[36716]: pgmap v334: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 117 KiB/s wr, 61 op/s 2024-06-10T00:38:45.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:45 smithi135 ceph-mon[36424]: pgmap v334: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 117 KiB/s wr, 61 op/s 2024-06-10T00:38:46.979 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:38:46.979 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:38:46.979 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 4, 2024-06-10T00:38:46.979 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 4, 2024-06-10T00:38:46.979 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 3024, 2024-06-10T00:38:46.979 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 448, 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 4, 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 0, 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.001444908, 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.000031548, 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.000520020, 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 3, 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000676344, 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000225448 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:46.980 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 448 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 260, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 249, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 1591808, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 153360, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 2, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 1, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.045427179, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.016276109, 2024-06-10T00:38:46.981 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.033013412, 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.004919038, 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 248, 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.014234353, 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000057396 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1591808, 2024-06-10T00:38:46.982 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 153360 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 2, 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 190, 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 320, 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 189264, 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 5, 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 3, 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.012847226, 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.008832053, 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.001703017, 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:46.983 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 187, 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.011972326, 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000064023 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 189264 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 0, 2024-06-10T00:38:46.984 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:38:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2024-06-10T00:38:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 148714, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 30, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 0, 2024-06-10T00:38:46.987 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 928, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 2, 2024-06-10T00:38:46.988 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 262, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 1555149, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 262, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-06-10T00:38:46.989 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 262, 2024-06-10T00:38:46.990 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 1555149, 2024-06-10T00:38:46.990 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-06-10T00:38:46.990 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:46.990 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:46.990 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:46.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:46.990 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:46.990 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:38:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:47 smithi193 ceph-mon[36716]: pgmap v335: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.5 KiB/s wr, 11 op/s 2024-06-10T00:38:47.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:47 smithi121 ceph-mon[32378]: pgmap v335: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.5 KiB/s wr, 11 op/s 2024-06-10T00:38:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:47 smithi135 ceph-mon[36424]: pgmap v335: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.5 KiB/s wr, 11 op/s 2024-06-10T00:38:48.585 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-06-10T00:38:48.585 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-06-10T00:38:48.586 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-06-10T00:38:48.586 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-06-10T00:38:48.586 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-06-10T00:38:48.586 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-06-10T00:38:48.586 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-06-10T00:38:48.586 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-06-10T00:38:48.586 DEBUG:tasks.check_counter:Getting stats from g 2024-06-10T00:38:48.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.g perf dump 2024-06-10T00:38:48.835 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:38:49.531 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:49 smithi193 ceph-mon[36716]: pgmap v336: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 10 op/s 2024-06-10T00:38:49.531 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:49 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:38:49.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:49 smithi121 ceph-mon[32378]: pgmap v336: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 10 op/s 2024-06-10T00:38:49.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:49 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:38:49.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:49 smithi135 ceph-mon[36424]: pgmap v336: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 10 op/s 2024-06-10T00:38:49.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:49 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:38:50.710 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 4, 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 4, 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 3024, 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 448, 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 4, 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 0, 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.001425890, 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.000027742, 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.000520574, 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 3, 2024-06-10T00:38:50.711 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000743142, 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000247714 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 448 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 2, 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 188, 2024-06-10T00:38:50.712 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 320, 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 188912, 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 5, 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 3, 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.013239916, 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.009270830, 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.001681878, 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 185, 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.011709168, 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000063292 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:50.713 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 188912 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 257, 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 246, 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 1590992, 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 151440, 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 2, 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 1, 2024-06-10T00:38:50.714 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.045628138, 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.015045049, 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.034866747, 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.004722155, 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 245, 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.011013134, 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000044951 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:50.715 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1590992, 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 151440 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 0, 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 0, 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 0, 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:38:50.716 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:38:50.717 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:38:50.718 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 148692, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 30, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 0, 2024-06-10T00:38:50.719 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 0, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 928, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 2, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2024-06-10T00:38:50.720 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 259, 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 1554759, 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 259, 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 259, 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 1554759, 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:50.721 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:50.722 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:38:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: pgmap v337: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.2 KiB/s wr, 12 op/s 2024-06-10T00:38:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:51 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: pgmap v337: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.2 KiB/s wr, 12 op/s 2024-06-10T00:38:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:51 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: pgmap v337: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.2 KiB/s wr, 12 op/s 2024-06-10T00:38:51.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:51.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:51.759 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:51 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:52.228 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-06-10T00:38:52.228 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-06-10T00:38:52.228 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-06-10T00:38:52.228 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-06-10T00:38:52.228 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-06-10T00:38:52.228 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-06-10T00:38:52.228 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-06-10T00:38:52.228 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-06-10T00:38:52.228 DEBUG:tasks.check_counter:Getting stats from j 2024-06-10T00:38:52.229 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.j perf dump 2024-06-10T00:38:52.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:52 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-06-10T00:38:52.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:52 smithi121 ceph-mon[32378]: Unable to set osd_memory_target on smithi193 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-06-10T00:38:52.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:52 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi135 to 176.2M 2024-06-10T00:38:52.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:52 smithi121 ceph-mon[32378]: Unable to set osd_memory_target on smithi135 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-06-10T00:38:52.498 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:38:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:52 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-06-10T00:38:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:52 smithi193 ceph-mon[36716]: Unable to set osd_memory_target on smithi193 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-06-10T00:38:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:52 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi135 to 176.2M 2024-06-10T00:38:52.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:52 smithi193 ceph-mon[36716]: Unable to set osd_memory_target on smithi135 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-06-10T00:38:52.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:52 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-06-10T00:38:52.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:52 smithi135 ceph-mon[36424]: Unable to set osd_memory_target on smithi193 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-06-10T00:38:52.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:52 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi135 to 176.2M 2024-06-10T00:38:52.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:52 smithi135 ceph-mon[36424]: Unable to set osd_memory_target on smithi135 to 184768102: error parsing value: Value '184768102' is below minimum 939524096 2024-06-10T00:38:53.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:53 smithi193 ceph-mon[36716]: pgmap v338: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-06-10T00:38:53.644 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:53 smithi121 ceph-mon[32378]: pgmap v338: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-06-10T00:38:53.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:53 smithi135 ceph-mon[36424]: pgmap v338: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-06-10T00:38:54.352 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-06-10T00:38:54.352 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:38:54.352 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 4, 2024-06-10T00:38:54.352 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 4, 2024-06-10T00:38:54.352 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 3024, 2024-06-10T00:38:54.352 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 448, 2024-06-10T00:38:54.352 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 4, 2024-06-10T00:38:54.352 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 0, 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.001261525, 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.000025385, 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.000468903, 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 3, 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000676560, 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000225520 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:54.353 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 448 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 263, 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 252, 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 1592624, 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 155280, 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 2, 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 1, 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.048308240, 2024-06-10T00:38:54.354 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.015416095, 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.037462541, 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.004758512, 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 251, 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.012609736, 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000050237 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:54.355 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1592624, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 155280 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 2, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 193, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 320, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 189792, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 5, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 3, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.014191445, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.009824958, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.001866779, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:38:54.356 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 190, 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.013781490, 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000072534 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 189792 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:54.357 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2024-06-10T00:38:54.358 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 0, 2024-06-10T00:38:54.358 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2024-06-10T00:38:54.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:54.358 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2024-06-10T00:38:54.358 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 0, 2024-06-10T00:38:54.358 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 0, 2024-06-10T00:38:54.358 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:38:54.358 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:38:54.358 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:38:54.358 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:38:54.358 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:38:54.359 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:38:54.359 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:38:54.359 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:38:54.359 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:38:54.359 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:38:54.359 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:38:54.359 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:38:54.359 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:38:54.359 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2024-06-10T00:38:54.359 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:38:54.359 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:38:54.360 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 148732, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 31, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 0, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 0, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2024-06-10T00:38:54.361 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 0, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 0, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 928, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 2, 2024-06-10T00:38:54.362 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 265, 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 1555539, 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-06-10T00:38:54.363 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 265, 2024-06-10T00:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-06-10T00:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-06-10T00:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 265, 2024-06-10T00:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 1555539, 2024-06-10T00:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-06-10T00:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-06-10T00:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-06-10T00:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-06-10T00:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-06-10T00:38:54.364 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-06-10T00:38:55.144 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-06-10T00:38:55.145 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-06-10T00:38:55.145 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-06-10T00:38:55.145 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-06-10T00:38:55.145 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-06-10T00:38:55.145 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-06-10T00:38:55.145 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-06-10T00:38:55.145 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-06-10T00:38:55.145 DEBUG:tasks.check_counter:Getting stats from b 2024-06-10T00:38:55.145 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.b perf dump 2024-06-10T00:38:55.379 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:55 smithi121 ceph-mon[32378]: pgmap v339: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-06-10T00:38:55.379 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:55 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:55.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:55 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:55.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:55 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:55 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:55 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:55 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:55 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:38:55.382 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:55 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:38:55.383 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:55 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:55.483 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:38:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:55 smithi135 ceph-mon[36424]: pgmap v339: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-06-10T00:38:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:55 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:55 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:55 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:55 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:55 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:55 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:55 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:38:55.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:55 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:38:55.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:55 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:55 smithi193 ceph-mon[36716]: pgmap v339: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-06-10T00:38:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:55 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:55 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:55.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:55 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:55 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:55 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:55 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-06-10T00:38:55.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:55 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:38:55.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:55 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:38:55.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:55 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:38:56.557 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:56 smithi193 ceph-mon[36716]: Adjusting osd_memory_target on smithi121 to 176.1M 2024-06-10T00:38:56.557 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:56 smithi193 ceph-mon[36716]: Unable to set osd_memory_target on smithi121 to 184674918: error parsing value: Value '184674918' is below minimum 939524096 2024-06-10T00:38:56.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:56 smithi121 ceph-mon[32378]: Adjusting osd_memory_target on smithi121 to 176.1M 2024-06-10T00:38:56.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:56 smithi121 ceph-mon[32378]: Unable to set osd_memory_target on smithi121 to 184674918: error parsing value: Value '184674918' is below minimum 939524096 2024-06-10T00:38:56.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:56 smithi135 ceph-mon[36424]: Adjusting osd_memory_target on smithi121 to 176.1M 2024-06-10T00:38:56.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:56 smithi135 ceph-mon[36424]: Unable to set osd_memory_target on smithi121 to 184674918: error parsing value: Value '184674918' is below minimum 939524096 2024-06-10T00:38:57.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:57 smithi135 ceph-mon[36424]: pgmap v340: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1023 B/s wr, 6 op/s 2024-06-10T00:38:57.559 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-06-10T00:38:57.559 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:38:57.559 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 75341, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 81650, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 60545602, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 317829114, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 107, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 5, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 6.092846103, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 2.796038754, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 2.379716314, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 1.163214738, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 81649, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 5.507064846, 2024-06-10T00:38:57.560 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000067448 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 97758, 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.016205260, 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000165 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 46960, 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 4144 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 317259, 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 502872, 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 117185612, 2024-06-10T00:38:57.561 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 434738392, 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 23, 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 5, 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 19.368544007, 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 8.974954643, 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 16.884392167, 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 1.208437022, 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 502869, 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 22.375657486, 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000044495 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.562 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 680875, 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.107473812, 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000157 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 107174032, 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 281002864 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 65878, 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 60689, 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 40217277, 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 128794545, 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 21, 2024-06-10T00:38:57.563 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 7, 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 4.537699295, 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 1.785132634, 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 1.690543745, 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 1.169874816, 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 60688, 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 5.954242604, 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000098112 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 55451, 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.008335186, 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000150 2024-06-10T00:38:57.564 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 2001984, 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 262608 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: "cct": { 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: "total_workers": 1, 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: "unhealthy_workers": 0 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: "finisher-MDSRank": { 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: "queue_len": 0, 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: "complete_latency": { 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 32557, 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 140.406924478, 2024-06-10T00:38:57.565 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.004312649 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: "finisher-PurgeQueue": { 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: "queue_len": 0, 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: "complete_latency": { 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 32285, 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 10.067808015, 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000311841 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: "mds": { 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: "request": 197429, 2024-06-10T00:38:57.566 INFO:teuthology.orchestra.run.smithi135.stdout: "reply": 176732, 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: "reply_latency": { 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 176732, 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 3721.897091224, 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.021059553 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: "slow_reply": 0, 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: "forward": 20689, 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_fetch_complete": 112, 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_fetch_keys": 2, 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_commit": 1063, 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_split": 110, 2024-06-10T00:38:57.567 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_merge": 3, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes": 41380, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_top": 215, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_bottom": 5870, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_pin_tail": 35295, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_pinned": 41072, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_expired": 111082, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_with_caps": 2, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "caps": 2, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "subtrees": 4, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse": 323532, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_hit": 262458, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_forward": 20450, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_discover": 515, 2024-06-10T00:38:57.568 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_dir_fetch": 35, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_remote_ino": 0, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_lock": 378, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "load_cent": 35, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "q": 0, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "exported": 310, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "exported_inodes": 93602, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "imported": 180, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "imported_inodes": 81516, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "openino_dir_fetch": 0, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "openino_backtrace_fetch": 38711, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "openino_peer_discover": 0, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "root_rfiles": 4, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "root_rbytes": 234, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "root_rsnaps": 6, 2024-06-10T00:38:57.569 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_backtrace_fetch": 38711, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_set_tag": 0, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_backtrace_repaired": 0, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_inotable_repaired": 0, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_dir_inodes": 496, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_dir_base_inodes": 10, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_dirfrag_rstats": 486, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_file_inodes": 38215, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_inode_file_caps": 576, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_revoke": 49829, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_grant": 65147, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_trunc": 5, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-10T00:38:57.570 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_caps": 82207, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_caps_dirty": 26599, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_cap_release": 3360, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "process_request_cap_release": 51122 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_cache": { 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_update": 2358, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_update_receipt": 3550, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_try_discover": 1895, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_send_discover": 2438, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_handle_discover": 5384, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "num_strays": 113, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "num_strays_delayed": 0, 2024-06-10T00:38:57.571 INFO:teuthology.orchestra.run.smithi135.stdout: "num_strays_enqueuing": 0, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_created": 18763, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_enqueued": 18650, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_reintegrated": 0, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_migrated": 0, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "num_recovering_processing": 0, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "num_recovering_enqueued": 0, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "num_recovering_prioritized": 0, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "recovery_started": 0, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "recovery_completed": 0, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_quiesce_path": 8, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_quiesce_inode": 31383, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_enqueue_scrub": 10, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_exportdir": 311, 2024-06-10T00:38:57.572 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_flush": 0, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_fragmentdir": 113, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_fragstats": 0, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_inodestats": 0 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_log": { 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "evlrg": 6, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "evadd": 144520, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "evex": 57905, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "evtrm": 49161, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "ev": 95359, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "evexg": 0, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "evexd": 8744, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "segadd": 206, 2024-06-10T00:38:57.573 INFO:teuthology.orchestra.run.smithi135.stdout: "segex": 78, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "segtrm": 67, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "seg": 139, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "segmjr": 8, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "segexg": 0, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "segexd": 11, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "expos": 134920736, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "wrpos": 487445017, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "rdpos": 0, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "jlat": { 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 8998, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 630.576705100, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.070079651 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.574 INFO:teuthology.orchestra.run.smithi135.stdout: "replayed": 0 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_mem": { 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "ino": 22536, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "ino+": 105186, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "ino-": 82650, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "dir": 760, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "dir+": 4114, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "dir-": 3354, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "dn": 41380, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "dn+": 174543, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "dn-": 133163, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "cap": 2, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "cap+": 63657, 2024-06-10T00:38:57.575 INFO:teuthology.orchestra.run.smithi135.stdout: "cap-": 63655, 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "rss": 255096, 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "heap": 231804 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_server": { 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "dispatch_client_request": 209275, 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "dispatch_server_request": 8850, 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_request": 197429, 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_session": 370, 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_peer_request": 19065, 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "req_create_latency": { 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 27410, 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 316.709062607, 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.011554507 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.576 INFO:teuthology.orchestra.run.smithi135.stdout: "req_getattr_latency": { 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 5738, 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 289.997577947, 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.050539835 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "req_getfilelock_latency": { 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "req_link_latency": { 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.577 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookup_latency": { 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 89848, 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 2417.963480036, 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.026911711 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookuphash_latency": { 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupino_latency": { 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.578 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.579 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.579 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupname_latency": { 2024-06-10T00:38:57.579 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.579 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.579 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.579 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.579 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupparent_latency": { 2024-06-10T00:38:57.579 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.579 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.579 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.579 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.580 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupsnap_latency": { 2024-06-10T00:38:57.580 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.580 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.580 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.580 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.580 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lssnap_latency": { 2024-06-10T00:38:57.580 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 10, 2024-06-10T00:38:57.580 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.005274830, 2024-06-10T00:38:57.580 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000527483 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: "req_mkdir_latency": { 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 203, 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 31.892520489, 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.157106012 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: "req_mknod_latency": { 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: "req_mksnap_latency": { 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 10, 2024-06-10T00:38:57.581 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 2.705596889, 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.270559688 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: "req_open_latency": { 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 10711, 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 191.911305223, 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.017917216 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: "req_readdir_latency": { 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 784, 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 2.787880143, 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.003555969 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rename_latency": { 2024-06-10T00:38:57.582 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 27031, 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 415.499535278, 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.015371223 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: "req_renamesnap_latency": { 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: "req_snapdiff_latency": { 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.583 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rmdir_latency": { 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 117, 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 23.790456425, 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.203337234 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rmsnap_latency": { 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 4, 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.175249152, 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.043812288 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rmxattr_latency": { 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.584 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setattr_latency": { 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 7, 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.030356626, 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.004336660 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setdirlayout_latency": { 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setfilelock_latency": { 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.585 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setlayout_latency": { 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setxattr_latency": { 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 8, 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.023667977, 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.002958497 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: "req_symlink_latency": { 2024-06-10T00:38:57.586 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 3, 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.001865397, 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000621799 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "req_unlink_latency": { 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 14848, 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 28.403262205, 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.001912935 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "cap_revoke_eviction": 0, 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "cap_acquisition_throttle": 0, 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "req_getvxattr_latency": { 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.587 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_sessions": { 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: "session_count": 2, 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: "session_add": 6, 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: "session_remove": 4, 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: "sessions_open": 2, 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: "sessions_stale": 0, 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: "total_load": 10641, 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: "average_load": 5320, 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: "avg_session_uptime": 628, 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: "mdthresh_evicted": 0 2024-06-10T00:38:57.588 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "mempool": { 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bloom_filter_bytes": 360, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bloom_filter_items": 360, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:38:57.589 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_extent_items": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_blob_items": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_txc_items": 0, 2024-06-10T00:38:57.590 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_items": 0, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_bytes": 0, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_items": 0, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_anon_bytes": 2080994, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_anon_items": 22600, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_meta_bytes": 0, 2024-06-10T00:38:57.591 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_meta_items": 0, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_bytes": 0, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_items": 0, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_mapbl_items": 0, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_pglog_items": 0, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_bytes": 9608, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_items": 184, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "pgmap_bytes": 0, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "pgmap_items": 0, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_co_bytes": 83748614, 2024-06-10T00:38:57.592 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_co_items": 902195, 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_1_bytes": 0, 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_1_items": 0, 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_2_bytes": 0, 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_2_items": 0 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "objecter": { 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "op_active": 0, 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "op_laggy": 0, 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "op_send": 90462, 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "op_send_bytes": 511424141, 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "op_resend": 31, 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "op_reply": 90462, 2024-06-10T00:38:57.593 INFO:teuthology.orchestra.run.smithi135.stdout: "op_latency": { 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 90462, 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 724.668209735, 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.008010747 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "op_inflight": 0, 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "oplen_avg": { 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 90462, 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 95085 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "op": 90462, 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "op_r": 39366, 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "op_w": 51096, 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "op_rmw": 0, 2024-06-10T00:38:57.594 INFO:teuthology.orchestra.run.smithi135.stdout: "op_pg": 0, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_stat": 47, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_create": 1724, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_read": 531, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_write": 9521, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_writefull": 302, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_writesame": 0, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_append": 0, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_zero": 0, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_truncate": 110, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_delete": 37975, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_mapext": 0, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_sparse_read": 0, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_clonerange": 0, 2024-06-10T00:38:57.595 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_getxattr": 38734, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_setxattr": 2611, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_cmpxattr": 0, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_rmxattr": 0, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_resetxattrs": 0, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_call": 0, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_watch": 0, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_notify": 0, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_src_cmpxattr": 0, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_pgls": 0, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_pgls_filter": 0, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_other": 5, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_active": 0, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_send": 0, 2024-06-10T00:38:57.596 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_resend": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_ping": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "poolop_active": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "poolop_send": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "poolop_resend": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "poolstat_active": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "poolstat_send": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "poolstat_resend": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "statfs_active": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "statfs_send": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "statfs_resend": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "command_active": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "command_send": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "command_resend": 0, 2024-06-10T00:38:57.597 INFO:teuthology.orchestra.run.smithi135.stdout: "map_epoch": 109, 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "map_full": 0, 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "map_inc": 20, 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_sessions": 12, 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_session_open": 12, 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_session_close": 0, 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_laggy": 0, 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_wr": 2855, 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_rd": 228, 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_del": 442 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "oft": { 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_objs": 1, 2024-06-10T00:38:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_kv_pairs": 0, 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_updates": 0, 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_removes": 0 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "osd-slow-ops": { 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "slow_ops_count": 27871 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "purge_queue": { 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing_ops": 0, 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing_ops_high_water": 17, 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing": 0, 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing_high_water": 9, 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executed_ops": 37293, 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executed": 18650, 2024-06-10T00:38:57.599 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_item_in_journal": 0 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 104857600, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 458474, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 162044764, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 458474, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 0, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 0, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 458474, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 162044764, 2024-06-10T00:38:57.600 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-objecter_bytes": { 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 104857600, 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 0, 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 0, 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:38:57.601 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 0, 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 90462, 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 513969335, 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 90462, 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 513969335, 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-objecter_ops": { 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 1024, 2024-06-10T00:38:57.602 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 0, 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 0, 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 0, 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 90462, 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 90462, 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 90462, 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 90462, 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.603 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-write_buf_throttle": { 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 3758096384, 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 18650, 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 2546994, 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 18650, 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 0, 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 0, 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 521, 2024-06-10T00:38:57.604 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 2546994, 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-write_buf_throttle-0x55abd75ae500": { 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 3758096384, 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 144520, 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 483250713, 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:38:57.605 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 144520, 2024-06-10T00:38:57.606 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 0, 2024-06-10T00:38:57.606 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 0, 2024-06-10T00:38:57.606 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 8998, 2024-06-10T00:38:57.606 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 483250713, 2024-06-10T00:38:57.606 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:38:57.606 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:38:57.606 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:38:57.606 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:38:57.606 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:38:57.606 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:38:57.606 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-06-10T00:38:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:57 smithi193 ceph-mon[36716]: pgmap v340: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1023 B/s wr, 6 op/s 2024-06-10T00:38:57.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:57 smithi121 ceph-mon[32378]: pgmap v340: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1023 B/s wr, 6 op/s 2024-06-10T00:38:58.427 INFO:tasks.check_counter:Daemon mds.b mds.exported=310 2024-06-10T00:38:58.427 INFO:tasks.check_counter:Daemon mds.b mds.imported=180 2024-06-10T00:38:58.427 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=2358 2024-06-10T00:38:58.427 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=3550 2024-06-10T00:38:58.428 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-06-10T00:38:58.428 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-06-10T00:38:58.428 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-06-10T00:38:58.428 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=110 2024-06-10T00:38:58.428 DEBUG:tasks.check_counter:Getting stats from e 2024-06-10T00:38:58.428 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.e perf dump 2024-06-10T00:38:58.673 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:38:59.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:38:59 smithi121 ceph-mon[32378]: pgmap v341: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1023 B/s wr, 5 op/s 2024-06-10T00:38:59.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:38:59 smithi135 ceph-mon[36424]: pgmap v341: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1023 B/s wr, 5 op/s 2024-06-10T00:38:59.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:38:59 smithi193 ceph-mon[36716]: pgmap v341: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1023 B/s wr, 5 op/s 2024-06-10T00:39:00.698 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 775, 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 755, 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 98261469, 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 312920, 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 7, 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 3, 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 0.132420913, 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 0.023624165, 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 0.100201684, 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 0.016394775, 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:00.699 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 754, 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.025840459, 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000034271 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 1654912, 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 158048 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:39:00.700 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 736, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 927, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 113447272, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 710025, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 9, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 7, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 0.127275246, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 0.020868210, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 0.080533458, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 0.018894604, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 924, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.039495666, 2024-06-10T00:39:00.701 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000042744 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 476608 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 1786, 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 1786, 2024-06-10T00:39:00.702 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 110727383, 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 567182, 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 6, 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 5, 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 0.211094131, 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 0.046910314, 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 0.101857878, 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 0.049897283, 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 1785, 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.070496287, 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000039493 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.703 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:00.704 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.704 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.704 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.704 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.704 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-06-10T00:39:00.704 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 448 2024-06-10T00:39:00.704 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.704 INFO:teuthology.orchestra.run.smithi135.stdout: "cct": { 2024-06-10T00:39:00.704 INFO:teuthology.orchestra.run.smithi135.stdout: "total_workers": 1, 2024-06-10T00:39:00.704 INFO:teuthology.orchestra.run.smithi135.stdout: "unhealthy_workers": 0 2024-06-10T00:39:00.704 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: "finisher-MDSRank": { 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: "queue_len": 0, 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: "complete_latency": { 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 4885, 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.118125060, 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000024181 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: "finisher-PurgeQueue": { 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: "queue_len": 0, 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: "complete_latency": { 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.705 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: "mds": { 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: "request": 0, 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: "reply": 0, 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: "reply_latency": { 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: "slow_reply": 0, 2024-06-10T00:39:00.706 INFO:teuthology.orchestra.run.smithi135.stdout: "forward": 0, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_fetch_complete": 0, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_fetch_keys": 0, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_commit": 0, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_split": 0, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_merge": 0, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes": 61279, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_top": 1340, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_bottom": 59939, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_pin_tail": 0, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_pinned": 59364, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_expired": 0, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_with_caps": 0, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "caps": 0, 2024-06-10T00:39:00.707 INFO:teuthology.orchestra.run.smithi135.stdout: "subtrees": 1, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_hit": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_forward": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_discover": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_dir_fetch": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_remote_ino": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_lock": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "load_cent": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "q": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "exported": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "exported_inodes": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "imported": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "imported_inodes": 0, 2024-06-10T00:39:00.708 INFO:teuthology.orchestra.run.smithi135.stdout: "openino_dir_fetch": 0, 2024-06-10T00:39:00.709 INFO:teuthology.orchestra.run.smithi135.stdout: "openino_backtrace_fetch": 0, 2024-06-10T00:39:00.709 INFO:teuthology.orchestra.run.smithi135.stdout: "openino_peer_discover": 0, 2024-06-10T00:39:00.709 INFO:teuthology.orchestra.run.smithi135.stdout: "root_rfiles": 0, 2024-06-10T00:39:00.709 INFO:teuthology.orchestra.run.smithi135.stdout: "root_rbytes": 0, 2024-06-10T00:39:00.709 INFO:teuthology.orchestra.run.smithi135.stdout: "root_rsnaps": 0, 2024-06-10T00:39:00.709 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_backtrace_fetch": 0, 2024-06-10T00:39:00.709 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_set_tag": 0, 2024-06-10T00:39:00.709 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_backtrace_repaired": 0, 2024-06-10T00:39:00.709 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_inotable_repaired": 0, 2024-06-10T00:39:00.709 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_dir_inodes": 0, 2024-06-10T00:39:00.709 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_dir_base_inodes": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_dirfrag_rstats": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_file_inodes": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_inode_file_caps": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_revoke": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_grant": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_trunc": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_caps": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_caps_dirty": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_cap_release": 0, 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: "process_request_cap_release": 0 2024-06-10T00:39:00.710 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_cache": { 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_update": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_update_receipt": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_try_discover": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_send_discover": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_handle_discover": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "num_strays": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "num_strays_delayed": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "num_strays_enqueuing": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_created": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_enqueued": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_reintegrated": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_migrated": 0, 2024-06-10T00:39:00.711 INFO:teuthology.orchestra.run.smithi135.stdout: "num_recovering_processing": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "num_recovering_enqueued": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "num_recovering_prioritized": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "recovery_started": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "recovery_completed": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_quiesce_path": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_quiesce_inode": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_enqueue_scrub": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_exportdir": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_flush": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_fragmentdir": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_fragstats": 0, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_inodestats": 0 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.712 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_log": { 2024-06-10T00:39:00.713 INFO:teuthology.orchestra.run.smithi135.stdout: "evlrg": 0, 2024-06-10T00:39:00.713 INFO:teuthology.orchestra.run.smithi135.stdout: "evadd": 0, 2024-06-10T00:39:00.713 INFO:teuthology.orchestra.run.smithi135.stdout: "evex": 0, 2024-06-10T00:39:00.713 INFO:teuthology.orchestra.run.smithi135.stdout: "evtrm": 0, 2024-06-10T00:39:00.713 INFO:teuthology.orchestra.run.smithi135.stdout: "ev": 101432, 2024-06-10T00:39:00.713 INFO:teuthology.orchestra.run.smithi135.stdout: "evexg": 0, 2024-06-10T00:39:00.713 INFO:teuthology.orchestra.run.smithi135.stdout: "evexd": 0, 2024-06-10T00:39:00.713 INFO:teuthology.orchestra.run.smithi135.stdout: "segadd": 0, 2024-06-10T00:39:00.713 INFO:teuthology.orchestra.run.smithi135.stdout: "segex": 0, 2024-06-10T00:39:00.713 INFO:teuthology.orchestra.run.smithi135.stdout: "segtrm": 0, 2024-06-10T00:39:00.713 INFO:teuthology.orchestra.run.smithi135.stdout: "seg": 143, 2024-06-10T00:39:00.714 INFO:teuthology.orchestra.run.smithi135.stdout: "segmjr": 10, 2024-06-10T00:39:00.714 INFO:teuthology.orchestra.run.smithi135.stdout: "segexg": 0, 2024-06-10T00:39:00.714 INFO:teuthology.orchestra.run.smithi135.stdout: "segexd": 0, 2024-06-10T00:39:00.714 INFO:teuthology.orchestra.run.smithi135.stdout: "expos": 4194352, 2024-06-10T00:39:00.714 INFO:teuthology.orchestra.run.smithi135.stdout: "wrpos": 324096715, 2024-06-10T00:39:00.714 INFO:teuthology.orchestra.run.smithi135.stdout: "rdpos": 324091316, 2024-06-10T00:39:00.714 INFO:teuthology.orchestra.run.smithi135.stdout: "jlat": { 2024-06-10T00:39:00.714 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.714 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.715 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.715 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.715 INFO:teuthology.orchestra.run.smithi135.stdout: "replayed": 101432 2024-06-10T00:39:00.715 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.715 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_mem": { 2024-06-10T00:39:00.715 INFO:teuthology.orchestra.run.smithi135.stdout: "ino": 23125, 2024-06-10T00:39:00.716 INFO:teuthology.orchestra.run.smithi135.stdout: "ino+": 78693, 2024-06-10T00:39:00.716 INFO:teuthology.orchestra.run.smithi135.stdout: "ino-": 55568, 2024-06-10T00:39:00.716 INFO:teuthology.orchestra.run.smithi135.stdout: "dir": 640, 2024-06-10T00:39:00.716 INFO:teuthology.orchestra.run.smithi135.stdout: "dir+": 1877, 2024-06-10T00:39:00.716 INFO:teuthology.orchestra.run.smithi135.stdout: "dir-": 1237, 2024-06-10T00:39:00.716 INFO:teuthology.orchestra.run.smithi135.stdout: "dn": 61279, 2024-06-10T00:39:00.716 INFO:teuthology.orchestra.run.smithi135.stdout: "dn+": 121538, 2024-06-10T00:39:00.716 INFO:teuthology.orchestra.run.smithi135.stdout: "dn-": 60259, 2024-06-10T00:39:00.717 INFO:teuthology.orchestra.run.smithi135.stdout: "cap": 0, 2024-06-10T00:39:00.717 INFO:teuthology.orchestra.run.smithi135.stdout: "cap+": 0, 2024-06-10T00:39:00.717 INFO:teuthology.orchestra.run.smithi135.stdout: "cap-": 0, 2024-06-10T00:39:00.717 INFO:teuthology.orchestra.run.smithi135.stdout: "rss": 195156, 2024-06-10T00:39:00.717 INFO:teuthology.orchestra.run.smithi135.stdout: "heap": 215420 2024-06-10T00:39:00.717 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.717 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_server": { 2024-06-10T00:39:00.717 INFO:teuthology.orchestra.run.smithi135.stdout: "dispatch_client_request": 0, 2024-06-10T00:39:00.717 INFO:teuthology.orchestra.run.smithi135.stdout: "dispatch_server_request": 0, 2024-06-10T00:39:00.717 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_request": 0, 2024-06-10T00:39:00.718 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_session": 0, 2024-06-10T00:39:00.718 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_peer_request": 0, 2024-06-10T00:39:00.718 INFO:teuthology.orchestra.run.smithi135.stdout: "req_create_latency": { 2024-06-10T00:39:00.718 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.718 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.718 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.718 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "req_getattr_latency": { 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "req_getfilelock_latency": { 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "req_link_latency": { 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.719 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookup_latency": { 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookuphash_latency": { 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupino_latency": { 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.720 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupname_latency": { 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupparent_latency": { 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupsnap_latency": { 2024-06-10T00:39:00.721 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.722 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.722 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.722 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.722 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lssnap_latency": { 2024-06-10T00:39:00.722 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.722 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.722 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.722 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.723 INFO:teuthology.orchestra.run.smithi135.stdout: "req_mkdir_latency": { 2024-06-10T00:39:00.723 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.723 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.723 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.723 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.723 INFO:teuthology.orchestra.run.smithi135.stdout: "req_mknod_latency": { 2024-06-10T00:39:00.723 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.723 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: "req_mksnap_latency": { 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: "req_open_latency": { 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.724 INFO:teuthology.orchestra.run.smithi135.stdout: "req_readdir_latency": { 2024-06-10T00:39:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rename_latency": { 2024-06-10T00:39:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: "req_renamesnap_latency": { 2024-06-10T00:39:00.725 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: "req_snapdiff_latency": { 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rmdir_latency": { 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.726 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rmsnap_latency": { 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rmxattr_latency": { 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setattr_latency": { 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.727 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setdirlayout_latency": { 2024-06-10T00:39:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setfilelock_latency": { 2024-06-10T00:39:00.728 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setlayout_latency": { 2024-06-10T00:39:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.729 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setxattr_latency": { 2024-06-10T00:39:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: "req_symlink_latency": { 2024-06-10T00:39:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.730 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: "req_unlink_latency": { 2024-06-10T00:39:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: "cap_revoke_eviction": 0, 2024-06-10T00:39:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: "cap_acquisition_throttle": 0, 2024-06-10T00:39:00.731 INFO:teuthology.orchestra.run.smithi135.stdout: "req_getvxattr_latency": { 2024-06-10T00:39:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_sessions": { 2024-06-10T00:39:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: "session_count": 2, 2024-06-10T00:39:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: "session_add": 4, 2024-06-10T00:39:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: "session_remove": 2, 2024-06-10T00:39:00.732 INFO:teuthology.orchestra.run.smithi135.stdout: "sessions_open": 4, 2024-06-10T00:39:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: "sessions_stale": 0, 2024-06-10T00:39:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: "total_load": 0, 2024-06-10T00:39:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: "average_load": 0, 2024-06-10T00:39:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: "avg_session_uptime": 3531, 2024-06-10T00:39:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: "mdthresh_evicted": 0 2024-06-10T00:39:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: "mempool": { 2024-06-10T00:39:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: "bloom_filter_bytes": 0, 2024-06-10T00:39:00.733 INFO:teuthology.orchestra.run.smithi135.stdout: "bloom_filter_items": 0, 2024-06-10T00:39:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:39:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:39:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:39:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:39:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:39:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:39:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:39:00.734 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:39:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:39:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:39:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:39:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:39:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:39:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_extent_items": 0, 2024-06-10T00:39:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:39:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_blob_items": 0, 2024-06-10T00:39:00.735 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:39:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:39:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:39:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:39:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:39:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:39:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:39:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_txc_items": 0, 2024-06-10T00:39:00.736 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:39:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:39:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:39:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_items": 0, 2024-06-10T00:39:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_bytes": 0, 2024-06-10T00:39:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_items": 0, 2024-06-10T00:39:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:39:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:39:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:39:00.737 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_anon_bytes": 1105359, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_anon_items": 23463, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_meta_bytes": 0, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_meta_items": 0, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_bytes": 0, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_items": 0, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_mapbl_items": 0, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_pglog_items": 0, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_bytes": 9608, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_items": 184, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:39:00.738 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "pgmap_bytes": 0, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "pgmap_items": 0, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_co_bytes": 97918349, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_co_items": 952876, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_1_bytes": 0, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_1_items": 0, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_2_bytes": 0, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_2_items": 0 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "objecter": { 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "op_active": 0, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "op_laggy": 0, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "op_send": 3003, 2024-06-10T00:39:00.739 INFO:teuthology.orchestra.run.smithi135.stdout: "op_send_bytes": 21, 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "op_resend": 3, 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "op_reply": 3003, 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "op_latency": { 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 3003, 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 5.219886028, 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.001738223 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "op_inflight": 0, 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "oplen_avg": { 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 3003, 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 3004 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "op": 3003, 2024-06-10T00:39:00.740 INFO:teuthology.orchestra.run.smithi135.stdout: "op_r": 3003, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "op_w": 0, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "op_rmw": 0, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "op_pg": 0, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_stat": 1806, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_create": 0, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_read": 1195, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_write": 0, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_writefull": 0, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_writesame": 0, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_append": 0, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_zero": 0, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_truncate": 0, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_delete": 0, 2024-06-10T00:39:00.741 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_mapext": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_sparse_read": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_clonerange": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_getxattr": 1, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_setxattr": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_cmpxattr": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_rmxattr": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_resetxattrs": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_call": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_watch": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_notify": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_src_cmpxattr": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_pgls": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_pgls_filter": 0, 2024-06-10T00:39:00.742 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_other": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_active": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_send": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_resend": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_ping": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "poolop_active": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "poolop_send": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "poolop_resend": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "poolstat_active": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "poolstat_send": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "poolstat_resend": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "statfs_active": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "statfs_send": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "statfs_resend": 0, 2024-06-10T00:39:00.743 INFO:teuthology.orchestra.run.smithi135.stdout: "command_active": 0, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "command_send": 0, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "command_resend": 0, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "map_epoch": 109, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "map_full": 0, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "map_inc": 11, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_sessions": 11, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_session_open": 11, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_session_close": 0, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_laggy": 0, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_wr": 0, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_rd": 2, 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_del": 0 2024-06-10T00:39:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: "oft": { 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_objs": 0, 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_kv_pairs": 0, 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_updates": 0, 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_removes": 0 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: "osd-slow-ops": { 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: "slow_ops_count": 0 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: "purge_queue": { 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing_ops": 0, 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing_ops_high_water": 0, 2024-06-10T00:39:00.745 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing": 0, 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing_high_water": 0, 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executed_ops": 0, 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executed": 0, 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_item_in_journal": 0 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 104857600, 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 3293, 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 322121996, 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:00.746 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 3293, 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 0, 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 0, 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 3293, 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 322121996, 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-objecter_bytes": { 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 104857600, 2024-06-10T00:39:00.747 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 0, 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 0, 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 3003, 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 319942570, 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 3003, 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 319942570, 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:00.748 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-objecter_ops": { 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 1024, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 0, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 0, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 3003, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 3003, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 3003, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 3003, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.749 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-write_buf_throttle": { 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 3758096384, 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 0, 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 0, 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 0, 2024-06-10T00:39:00.750 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 0, 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 0, 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 0, 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-write_buf_throttle-0x55b8db1b86e0": { 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 3758096384, 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 0, 2024-06-10T00:39:00.751 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 0, 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 0, 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 0, 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 0, 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 0, 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:00.752 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-06-10T00:39:01.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:01 smithi135 ceph-mon[36424]: pgmap v342: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-06-10T00:39:01.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:01 smithi121 ceph-mon[32378]: pgmap v342: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-06-10T00:39:01.701 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-06-10T00:39:01.702 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-06-10T00:39:01.702 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-06-10T00:39:01.702 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-06-10T00:39:01.702 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-06-10T00:39:01.702 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-06-10T00:39:01.703 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-06-10T00:39:01.703 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-06-10T00:39:01.703 DEBUG:tasks.check_counter:Getting stats from h 2024-06-10T00:39:01.703 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.h perf dump 2024-06-10T00:39:01.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:01 smithi193 ceph-mon[36716]: pgmap v342: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-06-10T00:39:01.956 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:39:03.451 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:03 smithi135 ceph-mon[36424]: pgmap v343: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-06-10T00:39:03.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:03 smithi121 ceph-mon[32378]: pgmap v343: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-06-10T00:39:03.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:03 smithi193 ceph-mon[36716]: pgmap v343: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-06-10T00:39:03.910 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-06-10T00:39:03.910 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:39:03.910 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 365948, 2024-06-10T00:39:03.910 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 478685, 2024-06-10T00:39:03.910 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 176927735, 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 403384737, 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 10, 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 4, 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 20.056701317, 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 9.325481121, 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 14.841146737, 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 1.208262933, 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 478686, 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 25.498033809, 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000053266 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.911 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 651812, 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.103864855, 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000159 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 98742880, 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 233020208 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 63273, 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 52798, 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 25338960, 2024-06-10T00:39:03.912 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 105248303, 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 25, 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 8, 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 3.345675218, 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 1.225759755, 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 1.624991143, 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 0.684058470, 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 52799, 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 3.481178749, 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000065932 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.913 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 67436, 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.009802704, 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000145 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 1673680, 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 237376 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 38143, 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 38340, 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 13062027, 2024-06-10T00:39:03.914 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 138656893, 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 22, 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 8, 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 2.828734058, 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 1.127740468, 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 0.559465782, 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 0.952273530, 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 38333, 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 2.761144668, 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000072030 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 35, 2024-06-10T00:39:03.915 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000005052, 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000144 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 3904, 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 484848 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: "cct": { 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: "total_workers": 1, 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: "unhealthy_workers": 0 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: "finisher-MDSRank": { 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: "queue_len": 0, 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: "complete_latency": { 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 54704, 2024-06-10T00:39:03.916 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 92.093407206, 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.001683485 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: "finisher-PurgeQueue": { 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: "queue_len": 0, 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: "complete_latency": { 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 26669, 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 8.690068064, 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000325849 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: "mds": { 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: "request": 156888, 2024-06-10T00:39:03.917 INFO:teuthology.orchestra.run.smithi135.stdout: "reply": 144237, 2024-06-10T00:39:03.918 INFO:teuthology.orchestra.run.smithi135.stdout: "reply_latency": { 2024-06-10T00:39:03.918 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 144237, 2024-06-10T00:39:03.918 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 4905.733478064, 2024-06-10T00:39:03.918 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.034011616 2024-06-10T00:39:03.918 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.918 INFO:teuthology.orchestra.run.smithi135.stdout: "slow_reply": 0, 2024-06-10T00:39:03.918 INFO:teuthology.orchestra.run.smithi135.stdout: "forward": 12640, 2024-06-10T00:39:03.918 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_fetch_complete": 209, 2024-06-10T00:39:03.918 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_fetch_keys": 0, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_commit": 539, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_split": 54, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_merge": 0, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes": 41551, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_top": 2600, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_bottom": 6468, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_pin_tail": 32483, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_pinned": 37835, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_expired": 66947, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "inodes_with_caps": 2, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "caps": 2, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "subtrees": 3, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse": 284367, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_hit": 224111, 2024-06-10T00:39:03.919 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_forward": 12358, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_discover": 1237, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_dir_fetch": 101, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_remote_ino": 0, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "traverse_lock": 947, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "load_cent": 0, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "q": 0, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "exported": 194, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "exported_inodes": 57574, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "imported": 357, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "imported_inodes": 69170, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "openino_dir_fetch": 0, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "openino_backtrace_fetch": 65304, 2024-06-10T00:39:03.920 INFO:teuthology.orchestra.run.smithi135.stdout: "openino_peer_discover": 0, 2024-06-10T00:39:03.921 INFO:teuthology.orchestra.run.smithi135.stdout: "root_rfiles": 11510, 2024-06-10T00:39:03.921 INFO:teuthology.orchestra.run.smithi135.stdout: "root_rbytes": 254329998, 2024-06-10T00:39:03.921 INFO:teuthology.orchestra.run.smithi135.stdout: "root_rsnaps": 6, 2024-06-10T00:39:03.921 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_backtrace_fetch": 65304, 2024-06-10T00:39:03.921 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_set_tag": 0, 2024-06-10T00:39:03.921 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_backtrace_repaired": 0, 2024-06-10T00:39:03.921 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_inotable_repaired": 0, 2024-06-10T00:39:03.921 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_dir_inodes": 443, 2024-06-10T00:39:03.921 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_dir_base_inodes": 0, 2024-06-10T00:39:03.922 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_dirfrag_rstats": 443, 2024-06-10T00:39:03.922 INFO:teuthology.orchestra.run.smithi135.stdout: "scrub_file_inodes": 64861, 2024-06-10T00:39:03.922 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_inode_file_caps": 254, 2024-06-10T00:39:03.922 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_revoke": 59857, 2024-06-10T00:39:03.922 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_grant": 49953, 2024-06-10T00:39:03.922 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_trunc": 0, 2024-06-10T00:39:03.922 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-10T00:39:03.923 INFO:teuthology.orchestra.run.smithi135.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-10T00:39:03.923 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_caps": 76323, 2024-06-10T00:39:03.923 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_caps_dirty": 16447, 2024-06-10T00:39:03.923 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_cap_release": 3160, 2024-06-10T00:39:03.923 INFO:teuthology.orchestra.run.smithi135.stdout: "process_request_cap_release": 31012 2024-06-10T00:39:03.923 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.923 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_cache": { 2024-06-10T00:39:03.923 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_update": 2186, 2024-06-10T00:39:03.923 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_update_receipt": 5998, 2024-06-10T00:39:03.924 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_try_discover": 3425, 2024-06-10T00:39:03.924 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_send_discover": 4737, 2024-06-10T00:39:03.924 INFO:teuthology.orchestra.run.smithi135.stdout: "dir_handle_discover": 3121, 2024-06-10T00:39:03.924 INFO:teuthology.orchestra.run.smithi135.stdout: "num_strays": 90, 2024-06-10T00:39:03.924 INFO:teuthology.orchestra.run.smithi135.stdout: "num_strays_delayed": 0, 2024-06-10T00:39:03.924 INFO:teuthology.orchestra.run.smithi135.stdout: "num_strays_enqueuing": 0, 2024-06-10T00:39:03.924 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_created": 15334, 2024-06-10T00:39:03.924 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_enqueued": 15244, 2024-06-10T00:39:03.924 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_reintegrated": 0, 2024-06-10T00:39:03.925 INFO:teuthology.orchestra.run.smithi135.stdout: "strays_migrated": 0, 2024-06-10T00:39:03.925 INFO:teuthology.orchestra.run.smithi135.stdout: "num_recovering_processing": 0, 2024-06-10T00:39:03.925 INFO:teuthology.orchestra.run.smithi135.stdout: "num_recovering_enqueued": 0, 2024-06-10T00:39:03.925 INFO:teuthology.orchestra.run.smithi135.stdout: "num_recovering_prioritized": 0, 2024-06-10T00:39:03.925 INFO:teuthology.orchestra.run.smithi135.stdout: "recovery_started": 0, 2024-06-10T00:39:03.925 INFO:teuthology.orchestra.run.smithi135.stdout: "recovery_completed": 0, 2024-06-10T00:39:03.925 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_quiesce_path": 8, 2024-06-10T00:39:03.925 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_quiesce_inode": 49055, 2024-06-10T00:39:03.926 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_enqueue_scrub": 0, 2024-06-10T00:39:03.926 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_exportdir": 197, 2024-06-10T00:39:03.926 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_flush": 0, 2024-06-10T00:39:03.926 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_fragmentdir": 55, 2024-06-10T00:39:03.926 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_fragstats": 0, 2024-06-10T00:39:03.926 INFO:teuthology.orchestra.run.smithi135.stdout: "ireq_inodestats": 0 2024-06-10T00:39:03.926 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.926 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_log": { 2024-06-10T00:39:03.927 INFO:teuthology.orchestra.run.smithi135.stdout: "evlrg": 6, 2024-06-10T00:39:03.927 INFO:teuthology.orchestra.run.smithi135.stdout: "evadd": 101432, 2024-06-10T00:39:03.927 INFO:teuthology.orchestra.run.smithi135.stdout: "evex": 11947, 2024-06-10T00:39:03.927 INFO:teuthology.orchestra.run.smithi135.stdout: "evtrm": 1, 2024-06-10T00:39:03.927 INFO:teuthology.orchestra.run.smithi135.stdout: "ev": 101431, 2024-06-10T00:39:03.927 INFO:teuthology.orchestra.run.smithi135.stdout: "evexg": 0, 2024-06-10T00:39:03.927 INFO:teuthology.orchestra.run.smithi135.stdout: "evexd": 11946, 2024-06-10T00:39:03.927 INFO:teuthology.orchestra.run.smithi135.stdout: "segadd": 144, 2024-06-10T00:39:03.928 INFO:teuthology.orchestra.run.smithi135.stdout: "segex": 16, 2024-06-10T00:39:03.928 INFO:teuthology.orchestra.run.smithi135.stdout: "segtrm": 1, 2024-06-10T00:39:03.928 INFO:teuthology.orchestra.run.smithi135.stdout: "seg": 143, 2024-06-10T00:39:03.928 INFO:teuthology.orchestra.run.smithi135.stdout: "segmjr": 9, 2024-06-10T00:39:03.928 INFO:teuthology.orchestra.run.smithi135.stdout: "segexg": 0, 2024-06-10T00:39:03.929 INFO:teuthology.orchestra.run.smithi135.stdout: "segexd": 15, 2024-06-10T00:39:03.929 INFO:teuthology.orchestra.run.smithi135.stdout: "expos": 4194352, 2024-06-10T00:39:03.929 INFO:teuthology.orchestra.run.smithi135.stdout: "wrpos": 324096715, 2024-06-10T00:39:03.929 INFO:teuthology.orchestra.run.smithi135.stdout: "rdpos": 0, 2024-06-10T00:39:03.929 INFO:teuthology.orchestra.run.smithi135.stdout: "jlat": { 2024-06-10T00:39:03.929 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 9751, 2024-06-10T00:39:03.930 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 287.558577325, 2024-06-10T00:39:03.930 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.029490162 2024-06-10T00:39:03.930 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.930 INFO:teuthology.orchestra.run.smithi135.stdout: "replayed": 0 2024-06-10T00:39:03.930 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.930 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_mem": { 2024-06-10T00:39:03.930 INFO:teuthology.orchestra.run.smithi135.stdout: "ino": 22624, 2024-06-10T00:39:03.930 INFO:teuthology.orchestra.run.smithi135.stdout: "ino+": 79121, 2024-06-10T00:39:03.931 INFO:teuthology.orchestra.run.smithi135.stdout: "ino-": 56497, 2024-06-10T00:39:03.931 INFO:teuthology.orchestra.run.smithi135.stdout: "dir": 647, 2024-06-10T00:39:03.931 INFO:teuthology.orchestra.run.smithi135.stdout: "dir+": 3672, 2024-06-10T00:39:03.931 INFO:teuthology.orchestra.run.smithi135.stdout: "dir-": 3025, 2024-06-10T00:39:03.931 INFO:teuthology.orchestra.run.smithi135.stdout: "dn": 41551, 2024-06-10T00:39:03.931 INFO:teuthology.orchestra.run.smithi135.stdout: "dn+": 125722, 2024-06-10T00:39:03.931 INFO:teuthology.orchestra.run.smithi135.stdout: "dn-": 84171, 2024-06-10T00:39:03.931 INFO:teuthology.orchestra.run.smithi135.stdout: "cap": 2, 2024-06-10T00:39:03.931 INFO:teuthology.orchestra.run.smithi135.stdout: "cap+": 44568, 2024-06-10T00:39:03.931 INFO:teuthology.orchestra.run.smithi135.stdout: "cap-": 44566, 2024-06-10T00:39:03.932 INFO:teuthology.orchestra.run.smithi135.stdout: "rss": 276720, 2024-06-10T00:39:03.932 INFO:teuthology.orchestra.run.smithi135.stdout: "heap": 223612 2024-06-10T00:39:03.932 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.932 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_server": { 2024-06-10T00:39:03.932 INFO:teuthology.orchestra.run.smithi135.stdout: "dispatch_client_request": 167043, 2024-06-10T00:39:03.932 INFO:teuthology.orchestra.run.smithi135.stdout: "dispatch_server_request": 6016, 2024-06-10T00:39:03.932 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_request": 156888, 2024-06-10T00:39:03.932 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_client_session": 329, 2024-06-10T00:39:03.932 INFO:teuthology.orchestra.run.smithi135.stdout: "handle_peer_request": 14896, 2024-06-10T00:39:03.933 INFO:teuthology.orchestra.run.smithi135.stdout: "req_create_latency": { 2024-06-10T00:39:03.933 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 9150, 2024-06-10T00:39:03.933 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 330.543063643, 2024-06-10T00:39:03.933 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.036124924 2024-06-10T00:39:03.933 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.933 INFO:teuthology.orchestra.run.smithi135.stdout: "req_getattr_latency": { 2024-06-10T00:39:03.933 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 104, 2024-06-10T00:39:03.933 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 5.964303557, 2024-06-10T00:39:03.934 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.057349072 2024-06-10T00:39:03.934 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.934 INFO:teuthology.orchestra.run.smithi135.stdout: "req_getfilelock_latency": { 2024-06-10T00:39:03.934 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.934 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.934 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.934 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.934 INFO:teuthology.orchestra.run.smithi135.stdout: "req_link_latency": { 2024-06-10T00:39:03.935 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.935 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.935 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.935 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.935 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookup_latency": { 2024-06-10T00:39:03.935 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 102518, 2024-06-10T00:39:03.935 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 3945.254267587, 2024-06-10T00:39:03.935 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.038483527 2024-06-10T00:39:03.935 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookuphash_latency": { 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupino_latency": { 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 1, 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000118408, 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000118408 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupname_latency": { 2024-06-10T00:39:03.936 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.937 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.937 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.937 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.937 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupparent_latency": { 2024-06-10T00:39:03.937 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.937 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.937 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.937 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.938 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lookupsnap_latency": { 2024-06-10T00:39:03.938 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.938 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.938 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.938 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.938 INFO:teuthology.orchestra.run.smithi135.stdout: "req_lssnap_latency": { 2024-06-10T00:39:03.938 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.938 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.938 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.938 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.938 INFO:teuthology.orchestra.run.smithi135.stdout: "req_mkdir_latency": { 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 38, 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 35.519748478, 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.934730223 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: "req_mknod_latency": { 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: "req_mksnap_latency": { 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.939 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "req_open_latency": { 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 9512, 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 145.908092213, 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.015339370 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "req_readdir_latency": { 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 610, 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 1.613860032, 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.002645672 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rename_latency": { 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 9156, 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 360.525750228, 2024-06-10T00:39:03.940 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.039375901 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: "req_renamesnap_latency": { 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: "req_snapdiff_latency": { 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rmdir_latency": { 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 91, 2024-06-10T00:39:03.941 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 32.971202929, 2024-06-10T00:39:03.942 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.362320911 2024-06-10T00:39:03.942 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.942 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rmsnap_latency": { 2024-06-10T00:39:03.942 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.942 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.942 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.942 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.942 INFO:teuthology.orchestra.run.smithi135.stdout: "req_rmxattr_latency": { 2024-06-10T00:39:03.942 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.942 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.942 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setattr_latency": { 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setdirlayout_latency": { 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setfilelock_latency": { 2024-06-10T00:39:03.943 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.944 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.944 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.944 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.944 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setlayout_latency": { 2024-06-10T00:39:03.944 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.944 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.944 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.944 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.944 INFO:teuthology.orchestra.run.smithi135.stdout: "req_setxattr_latency": { 2024-06-10T00:39:03.945 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.945 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.945 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.945 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.945 INFO:teuthology.orchestra.run.smithi135.stdout: "req_symlink_latency": { 2024-06-10T00:39:03.945 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.945 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.945 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.945 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.945 INFO:teuthology.orchestra.run.smithi135.stdout: "req_unlink_latency": { 2024-06-10T00:39:03.945 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 13057, 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 47.433070989, 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.003632769 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: "cap_revoke_eviction": 0, 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: "cap_acquisition_throttle": 0, 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: "req_getvxattr_latency": { 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_sessions": { 2024-06-10T00:39:03.946 INFO:teuthology.orchestra.run.smithi135.stdout: "session_count": 2, 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "session_add": 4, 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "session_remove": 2, 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "sessions_open": 2, 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "sessions_stale": 0, 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "total_load": 12534, 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "average_load": 6267, 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "avg_session_uptime": 616, 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "mdthresh_evicted": 0 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "mempool": { 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "bloom_filter_bytes": 0, 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "bloom_filter_items": 0, 2024-06-10T00:39:03.947 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_extent_items": 0, 2024-06-10T00:39:03.948 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_blob_items": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_txc_items": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_items": 0, 2024-06-10T00:39:03.949 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_bytes": 0, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_items": 0, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_anon_bytes": 1797012, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_anon_items": 20674, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_meta_bytes": 176, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_meta_items": 2, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_bytes": 0, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_items": 0, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:39:03.950 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_mapbl_items": 0, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_pglog_items": 0, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_bytes": 9608, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_items": 184, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "pgmap_bytes": 0, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "pgmap_items": 0, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_co_bytes": 83330456, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_co_items": 875373, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_1_bytes": 0, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_1_items": 0, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_2_bytes": 0, 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_2_items": 0 2024-06-10T00:39:03.951 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.952 INFO:teuthology.orchestra.run.smithi135.stdout: "objecter": { 2024-06-10T00:39:03.952 INFO:teuthology.orchestra.run.smithi135.stdout: "op_active": 0, 2024-06-10T00:39:03.952 INFO:teuthology.orchestra.run.smithi135.stdout: "op_laggy": 0, 2024-06-10T00:39:03.952 INFO:teuthology.orchestra.run.smithi135.stdout: "op_send": 107877, 2024-06-10T00:39:03.952 INFO:teuthology.orchestra.run.smithi135.stdout: "op_send_bytes": 336504020, 2024-06-10T00:39:03.952 INFO:teuthology.orchestra.run.smithi135.stdout: "op_resend": 33, 2024-06-10T00:39:03.952 INFO:teuthology.orchestra.run.smithi135.stdout: "op_reply": 107877, 2024-06-10T00:39:03.952 INFO:teuthology.orchestra.run.smithi135.stdout: "op_latency": { 2024-06-10T00:39:03.952 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 107877, 2024-06-10T00:39:03.952 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 513.545809219, 2024-06-10T00:39:03.952 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.004760475 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: "op_inflight": 0, 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: "oplen_avg": { 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 107877, 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 109357 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: "op": 107877, 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: "op_r": 65932, 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: "op_w": 41945, 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: "op_rmw": 0, 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: "op_pg": 0, 2024-06-10T00:39:03.953 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_stat": 42, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_create": 21, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_read": 419, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_write": 10171, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_writefull": 242, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_writesame": 0, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_append": 0, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_zero": 0, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_truncate": 54, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_delete": 30575, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_mapext": 0, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_sparse_read": 0, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_clonerange": 0, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_getxattr": 65304, 2024-06-10T00:39:03.954 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_setxattr": 42, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_cmpxattr": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_rmxattr": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_resetxattrs": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_call": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_watch": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_notify": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_src_cmpxattr": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_pgls": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_pgls_filter": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "osdop_other": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_active": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_send": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_resend": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "linger_ping": 0, 2024-06-10T00:39:03.955 INFO:teuthology.orchestra.run.smithi135.stdout: "poolop_active": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "poolop_send": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "poolop_resend": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "poolstat_active": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "poolstat_send": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "poolstat_resend": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "statfs_active": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "statfs_send": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "statfs_resend": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "command_active": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "command_send": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "command_resend": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "map_epoch": 109, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "map_full": 0, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "map_inc": 11, 2024-06-10T00:39:03.956 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_sessions": 12, 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_session_open": 12, 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_session_close": 0, 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_laggy": 0, 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_wr": 1728, 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_rd": 418, 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_del": 341 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "oft": { 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_objs": 1, 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_kv_pairs": 0, 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_updates": 0, 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "omap_total_removes": 0 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "osd-slow-ops": { 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: "slow_ops_count": 48851 2024-06-10T00:39:03.957 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "purge_queue": { 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing_ops": 0, 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing_ops_high_water": 16, 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing": 0, 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executing_high_water": 8, 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executed_ops": 30487, 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_executed": 15244, 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "pq_item_in_journal": 0 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 104857600, 2024-06-10T00:39:03.958 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 467360, 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 159853014, 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 467360, 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 0, 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 0, 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 467360, 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 159853014, 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:03.959 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-objecter_bytes": { 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 104857600, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 0, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 0, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 107877, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 338601328, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 107877, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 338601328, 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:39:03.960 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.961 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.961 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.961 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:03.961 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.961 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-objecter_ops": { 2024-06-10T00:39:03.961 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:39:03.962 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 1024, 2024-06-10T00:39:03.962 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:39:03.962 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 0, 2024-06-10T00:39:03.962 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 0, 2024-06-10T00:39:03.962 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:03.962 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:03.962 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 107877, 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 107877, 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 107877, 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 107877, 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-write_buf_throttle": { 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 3758096384, 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:39:03.963 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 15244, 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 2100652, 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 15244, 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 0, 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 0, 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 419, 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 2100652, 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.964 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-write_buf_throttle-0x558e1eb4a500": { 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 3758096384, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 101432, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 319902411, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 101432, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 0, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 0, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 9751, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 319902411, 2024-06-10T00:39:03.965 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:39:03.966 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:03.966 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:03.966 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:03.966 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:03.966 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:03.966 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-06-10T00:39:04.837 INFO:tasks.check_counter:Daemon mds.h mds.exported=194 2024-06-10T00:39:04.837 INFO:tasks.check_counter:Daemon mds.h mds.imported=357 2024-06-10T00:39:04.837 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=2186 2024-06-10T00:39:04.837 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=5998 2024-06-10T00:39:04.837 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-06-10T00:39:04.837 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-06-10T00:39:04.837 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-06-10T00:39:04.837 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=54 2024-06-10T00:39:04.837 DEBUG:tasks.check_counter:Getting stats from k 2024-06-10T00:39:04.838 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.k perf dump 2024-06-10T00:39:05.075 INFO:teuthology.orchestra.run.smithi135.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.b/config 2024-06-10T00:39:05.463 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:05 smithi135 ceph-mon[36424]: pgmap v344: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-06-10T00:39:05.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:05 smithi121 ceph-mon[32378]: pgmap v344: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-06-10T00:39:05.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:05 smithi193 ceph-mon[36716]: pgmap v344: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-06-10T00:39:06.825 INFO:teuthology.orchestra.run.smithi135.stdout:{ 2024-06-10T00:39:06.825 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 271, 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 258, 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 1596736, 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 158288, 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 4, 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 0, 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 0.047384277, 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 0.013208391, 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 0.037808454, 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 0.005815569, 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 258, 2024-06-10T00:39:06.826 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.011225503, 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000043509 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 1596736, 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 158288 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 2, 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 196, 2024-06-10T00:39:06.827 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 320, 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 190320, 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 5, 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 3, 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 0.010622575, 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 0.007298962, 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 0.001265148, 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 193, 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.013768316, 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000071338 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:06.828 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 190320 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_messages": 0, 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages": 2, 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_bytes": 0, 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_bytes": 0, 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_created_connections": 2, 2024-06-10T00:39:06.829 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_active_connections": 1, 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_total_time": 0.000283253, 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_send_time": 0.000000000, 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_recv_time": 0.000212461, 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:06.830 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: "msgr_send_encrypted_bytes": 0 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: "cct": { 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: "total_workers": 0, 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: "unhealthy_workers": 0 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: "mempool": { 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: "bloom_filter_bytes": 0, 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: "bloom_filter_items": 0, 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:39:06.831 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:39:06.832 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:39:06.832 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:39:06.832 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:39:06.832 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:39:06.832 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:39:06.832 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:39:06.832 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:39:06.832 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:39:06.832 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_extent_items": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_blob_items": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:39:06.833 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_txc_items": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "bluestore_writing_items": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_bytes": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_items": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_anon_bytes": 148754, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_anon_items": 31, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_meta_bytes": 0, 2024-06-10T00:39:06.834 INFO:teuthology.orchestra.run.smithi135.stdout: "buffer_meta_items": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_bytes": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_items": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_mapbl_items": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "osd_pglog_items": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_bytes": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_items": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "pgmap_bytes": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "pgmap_items": 0, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_co_bytes": 928, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "mds_co_items": 2, 2024-06-10T00:39:06.835 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_1_bytes": 0, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_1_items": 0, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_2_bytes": 0, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "unittest_2_items": 0 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: }, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "val": 0, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "max": 104857600, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "get_started": 0, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "get": 269, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "get_sum": 1556059, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "get_or_fail_success": 269, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "take": 0, 2024-06-10T00:39:06.836 INFO:teuthology.orchestra.run.smithi135.stdout: "take_sum": 0, 2024-06-10T00:39:06.837 INFO:teuthology.orchestra.run.smithi135.stdout: "put": 269, 2024-06-10T00:39:06.837 INFO:teuthology.orchestra.run.smithi135.stdout: "put_sum": 1556059, 2024-06-10T00:39:06.837 INFO:teuthology.orchestra.run.smithi135.stdout: "wait": { 2024-06-10T00:39:06.837 INFO:teuthology.orchestra.run.smithi135.stdout: "avgcount": 0, 2024-06-10T00:39:06.837 INFO:teuthology.orchestra.run.smithi135.stdout: "sum": 0.000000000, 2024-06-10T00:39:06.837 INFO:teuthology.orchestra.run.smithi135.stdout: "avgtime": 0.000000000 2024-06-10T00:39:06.837 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:06.837 INFO:teuthology.orchestra.run.smithi135.stdout: } 2024-06-10T00:39:06.837 INFO:teuthology.orchestra.run.smithi135.stdout:} 2024-06-10T00:39:07.578 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-06-10T00:39:07.578 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-06-10T00:39:07.578 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-06-10T00:39:07.579 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-06-10T00:39:07.579 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-06-10T00:39:07.579 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-06-10T00:39:07.579 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-06-10T00:39:07.579 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-06-10T00:39:07.579 DEBUG:tasks.check_counter:Getting stats from c 2024-06-10T00:39:07.579 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.c perf dump 2024-06-10T00:39:07.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:07 smithi121 ceph-mon[32378]: pgmap v345: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-06-10T00:39:07.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:07 smithi135 ceph-mon[36424]: pgmap v345: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-06-10T00:39:07.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:07 smithi193 ceph-mon[36716]: pgmap v345: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-06-10T00:39:08.017 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:39:08.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:08 smithi121 ceph-mon[32378]: osdmap e110: 12 total, 12 up, 12 in 2024-06-10T00:39:08.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:08 smithi135 ceph-mon[36424]: osdmap e110: 12 total, 12 up, 12 in 2024-06-10T00:39:08.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:08 smithi193 ceph-mon[36716]: osdmap e110: 12 total, 12 up, 12 in 2024-06-10T00:39:09.569 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:09 smithi193 ceph-mon[36716]: pgmap v347: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:09.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:09 smithi121 ceph-mon[32378]: pgmap v347: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:09.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:09 smithi135 ceph-mon[36424]: pgmap v347: 129 pgs: 129 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:10.073 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-06-10T00:39:10.073 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:39:10.073 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 2137, 2024-06-10T00:39:10.073 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 2117, 2024-06-10T00:39:10.073 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 165782583, 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 746056, 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 7, 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 3, 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.293760308, 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.060255141, 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.158836518, 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.070946945, 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2116, 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.072479686, 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000034253 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.074 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1656656, 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 159024 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 508, 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 697, 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 130169120, 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 639990, 2024-06-10T00:39:10.075 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 9, 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 7, 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.106054811, 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.017765687, 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.072289218, 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.011174702, 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 695, 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.031797037, 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000045751 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:10.076 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 480288 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1144, 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1146, 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 189964646, 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 363799, 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 6, 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 5, 2024-06-10T00:39:10.077 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.167014400, 2024-06-10T00:39:10.078 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.027815208, 2024-06-10T00:39:10.078 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.102449463, 2024-06-10T00:39:10.078 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.028952297, 2024-06-10T00:39:10.078 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:10.078 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1144, 2024-06-10T00:39:10.078 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.043522783, 2024-06-10T00:39:10.078 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000038044 2024-06-10T00:39:10.078 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.078 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:10.084 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.084 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.084 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.084 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.084 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-10T00:39:10.084 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-06-10T00:39:10.084 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.084 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-06-10T00:39:10.084 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-06-10T00:39:10.084 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 5447, 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.143858282, 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000026410 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-06-10T00:39:10.085 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.086 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.086 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.086 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:10.086 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.086 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-06-10T00:39:10.086 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-06-10T00:39:10.086 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-06-10T00:39:10.086 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-06-10T00:39:10.086 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 68340, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 5679, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 62661, 2024-06-10T00:39:10.087 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-06-10T00:39:10.088 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 60228, 2024-06-10T00:39:10.088 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-06-10T00:39:10.088 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-06-10T00:39:10.088 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-06-10T00:39:10.088 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 2, 2024-06-10T00:39:10.088 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-06-10T00:39:10.088 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-06-10T00:39:10.088 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-06-10T00:39:10.088 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-06-10T00:39:10.088 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-06-10T00:39:10.089 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-06-10T00:39:10.089 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-06-10T00:39:10.089 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-06-10T00:39:10.089 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-06-10T00:39:10.089 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-06-10T00:39:10.089 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-06-10T00:39:10.089 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-06-10T00:39:10.089 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-06-10T00:39:10.089 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-06-10T00:39:10.089 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-06-10T00:39:10.089 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 4, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 234, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 6, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-06-10T00:39:10.090 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-06-10T00:39:10.091 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-06-10T00:39:10.092 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-06-10T00:39:10.093 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-06-10T00:39:10.093 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-06-10T00:39:10.093 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-06-10T00:39:10.093 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-06-10T00:39:10.093 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-06-10T00:39:10.093 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-06-10T00:39:10.094 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-06-10T00:39:10.094 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-06-10T00:39:10.094 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-06-10T00:39:10.094 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-06-10T00:39:10.094 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-06-10T00:39:10.094 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.094 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-06-10T00:39:10.094 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 144528, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 139, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 13, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-06-10T00:39:10.095 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 134920736, 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 487453959, 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 487453899, 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 144528 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 24504, 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 104720, 2024-06-10T00:39:10.096 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 80216, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 740, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 2808, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 2068, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 68340, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 168588, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 100248, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 201160, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 215420 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.097 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.098 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-06-10T00:39:10.099 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-06-10T00:39:10.100 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.101 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-06-10T00:39:10.102 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.102 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.102 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.103 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.104 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-06-10T00:39:10.104 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.104 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.104 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.104 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-06-10T00:39:10.104 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.104 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.104 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.105 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.106 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.106 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-06-10T00:39:10.106 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.106 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.106 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.106 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-06-10T00:39:10.106 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.106 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.107 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-06-10T00:39:10.108 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.109 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 6, 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 4, 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 4, 2024-06-10T00:39:10.110 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 3570, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:39:10.111 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:39:10.112 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:39:10.113 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 1158764, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 24925, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 9608, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 184, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:39:10.114 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:39:10.115 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-06-10T00:39:10.115 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-06-10T00:39:10.115 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 106313503, 2024-06-10T00:39:10.115 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 1009350, 2024-06-10T00:39:10.115 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-06-10T00:39:10.115 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-06-10T00:39:10.115 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-06-10T00:39:10.115 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-06-10T00:39:10.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.115 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-06-10T00:39:10.116 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-06-10T00:39:10.116 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-06-10T00:39:10.116 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 3492, 2024-06-10T00:39:10.116 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 26, 2024-06-10T00:39:10.116 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 4, 2024-06-10T00:39:10.116 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 3492, 2024-06-10T00:39:10.116 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-06-10T00:39:10.116 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3492, 2024-06-10T00:39:10.116 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 7.940878694, 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.002274020 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3492, 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 3493 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 3492, 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 3492, 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 2181, 2024-06-10T00:39:10.117 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 1308, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 2, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-06-10T00:39:10.118 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-06-10T00:39:10.119 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-06-10T00:39:10.119 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-06-10T00:39:10.119 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-06-10T00:39:10.119 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-06-10T00:39:10.119 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-06-10T00:39:10.119 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-06-10T00:39:10.120 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-06-10T00:39:10.120 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-06-10T00:39:10.120 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-06-10T00:39:10.120 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-06-10T00:39:10.120 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-06-10T00:39:10.120 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-06-10T00:39:10.120 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-06-10T00:39:10.120 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-06-10T00:39:10.120 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-06-10T00:39:10.120 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 110, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 12, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 11, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 11, 2024-06-10T00:39:10.121 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-06-10T00:39:10.122 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:10.123 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 3785, 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 485557793, 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 3785, 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3785, 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 485557793, 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.124 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3492, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 483295277, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3492, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 483295277, 2024-06-10T00:39:10.125 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:10.126 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3492, 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 3492, 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3492, 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 3492, 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-06-10T00:39:10.127 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:10.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:10.129 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x55d44089e6e0": { 2024-06-10T00:39:10.129 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:10.129 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-06-10T00:39:10.129 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:10.129 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-06-10T00:39:10.129 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-06-10T00:39:10.129 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:10.129 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:10.130 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-06-10T00:39:10.130 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-06-10T00:39:10.130 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-06-10T00:39:10.130 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-06-10T00:39:10.130 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:10.130 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:10.130 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:10.130 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:10.130 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:10.131 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:10.131 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-06-10T00:39:10.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:10 smithi121 ceph-mon[32378]: pgmap v348: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:10.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:10 smithi135 ceph-mon[36424]: pgmap v348: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:10.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:10 smithi193 ceph-mon[36716]: pgmap v348: 129 pgs: 15 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 75 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:11.064 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-06-10T00:39:11.064 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-06-10T00:39:11.064 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-06-10T00:39:11.064 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-06-10T00:39:11.064 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-06-10T00:39:11.064 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-06-10T00:39:11.064 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-06-10T00:39:11.064 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-06-10T00:39:11.064 DEBUG:tasks.check_counter:Getting stats from f 2024-06-10T00:39:11.065 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.f perf dump 2024-06-10T00:39:11.500 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:39:12.938 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:12 smithi193 ceph-mon[36716]: pgmap v349: 129 pgs: 21 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 50 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:13.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:12 smithi135 ceph-mon[36424]: pgmap v349: 129 pgs: 21 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 50 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:13.203 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:12 smithi121 ceph-mon[32378]: pgmap v349: 129 pgs: 21 active+clean+snaptrim, 58 active+clean+snaptrim_wait, 50 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:13.458 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-06-10T00:39:13.458 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:39:13.458 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 2217, 2024-06-10T00:39:13.458 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 2219, 2024-06-10T00:39:13.458 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 101124976, 2024-06-10T00:39:13.458 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 705013, 2024-06-10T00:39:13.459 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 8, 2024-06-10T00:39:13.459 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-06-10T00:39:13.459 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.228636146, 2024-06-10T00:39:13.459 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.057507007, 2024-06-10T00:39:13.459 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.093658936, 2024-06-10T00:39:13.459 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.062603403, 2024-06-10T00:39:13.459 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:13.459 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2217, 2024-06-10T00:39:13.459 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.075869289, 2024-06-10T00:39:13.459 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000034221 2024-06-10T00:39:13.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.460 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:13.460 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.460 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.460 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.460 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-06-10T00:39:13.460 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-06-10T00:39:13.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.460 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:39:13.460 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 448, 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 640, 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 79826008, 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 621906, 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 9, 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 7, 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.082014905, 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.015460701, 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.052155321, 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.009772600, 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 637, 2024-06-10T00:39:13.461 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.029101484, 2024-06-10T00:39:13.462 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000045685 2024-06-10T00:39:13.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.462 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:13.462 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.462 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.462 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.462 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-10T00:39:13.462 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 480016 2024-06-10T00:39:13.462 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.462 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:39:13.463 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 626, 2024-06-10T00:39:13.463 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 604, 2024-06-10T00:39:13.463 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 77253219, 2024-06-10T00:39:13.463 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 265210, 2024-06-10T00:39:13.463 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 5, 2024-06-10T00:39:13.463 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-06-10T00:39:13.463 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.113832296, 2024-06-10T00:39:13.463 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.020794036, 2024-06-10T00:39:13.463 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.088048020, 2024-06-10T00:39:13.463 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.013277820, 2024-06-10T00:39:13.463 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:13.464 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 604, 2024-06-10T00:39:13.464 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.024383730, 2024-06-10T00:39:13.464 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000040370 2024-06-10T00:39:13.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.464 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:13.464 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.464 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.464 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.464 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.464 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1659792, 2024-06-10T00:39:13.464 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 159952 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 4908, 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.122331334, 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000024924 2024-06-10T00:39:13.465 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:13.466 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.466 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-06-10T00:39:13.466 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-06-10T00:39:13.466 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-06-10T00:39:13.466 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.466 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.466 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.466 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:13.466 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.466 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-06-10T00:39:13.467 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 43672, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 3100, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 40572, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 39243, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 1, 2024-06-10T00:39:13.468 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-06-10T00:39:13.469 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-06-10T00:39:13.470 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-06-10T00:39:13.471 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-06-10T00:39:13.471 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-06-10T00:39:13.471 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-06-10T00:39:13.471 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-06-10T00:39:13.471 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-06-10T00:39:13.471 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-06-10T00:39:13.471 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-10T00:39:13.471 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-06-10T00:39:13.472 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-06-10T00:39:13.473 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 0, 2024-06-10T00:39:13.474 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 0, 2024-06-10T00:39:13.474 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-06-10T00:39:13.474 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-06-10T00:39:13.474 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-06-10T00:39:13.474 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-06-10T00:39:13.474 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-06-10T00:39:13.474 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-06-10T00:39:13.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.475 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-06-10T00:39:13.475 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 0, 2024-06-10T00:39:13.475 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-06-10T00:39:13.475 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-06-10T00:39:13.475 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-06-10T00:39:13.475 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 91839, 2024-06-10T00:39:13.475 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-06-10T00:39:13.476 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-06-10T00:39:13.476 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-06-10T00:39:13.476 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-06-10T00:39:13.476 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-06-10T00:39:13.476 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 128, 2024-06-10T00:39:13.476 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 9, 2024-06-10T00:39:13.476 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-06-10T00:39:13.476 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-06-10T00:39:13.476 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194352, 2024-06-10T00:39:13.476 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 259863147, 2024-06-10T00:39:13.476 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 259856941, 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 91839 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 14479, 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 48571, 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 34092, 2024-06-10T00:39:13.477 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 447, 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 1147, 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 700, 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 43672, 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 77551, 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 33879, 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 137184, 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 215420 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.478 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-06-10T00:39:13.479 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-06-10T00:39:13.479 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-06-10T00:39:13.479 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-06-10T00:39:13.479 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-06-10T00:39:13.479 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-06-10T00:39:13.479 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-06-10T00:39:13.479 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.479 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.481 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.482 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.482 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-06-10T00:39:13.482 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.482 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.482 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.482 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-06-10T00:39:13.482 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.482 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.482 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.483 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-06-10T00:39:13.484 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.484 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.484 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.484 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-06-10T00:39:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-06-10T00:39:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-06-10T00:39:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.486 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.486 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.486 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-06-10T00:39:13.486 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.486 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.486 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.486 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-06-10T00:39:13.486 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.486 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.487 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.487 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-06-10T00:39:13.487 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.487 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.487 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.487 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-06-10T00:39:13.487 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.487 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.487 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.488 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-06-10T00:39:13.489 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.489 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.489 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.489 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-06-10T00:39:13.489 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.489 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.489 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.489 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-06-10T00:39:13.489 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.490 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.490 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.490 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-06-10T00:39:13.490 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.490 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.490 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-06-10T00:39:13.491 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 4, 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 2, 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 3, 2024-06-10T00:39:13.493 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 2851, 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:39:13.494 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:39:13.495 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-06-10T00:39:13.496 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 782932, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 14748, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:39:13.497 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 9848, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 190, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 65094202, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 647553, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-06-10T00:39:13.498 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 2994, 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 21, 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 3, 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 2994, 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2994, 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 5.022923295, 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001677663 2024-06-10T00:39:13.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2994, 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 2995 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 2994, 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 2994, 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 1822, 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-06-10T00:39:13.500 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 1170, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 1, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-06-10T00:39:13.501 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-06-10T00:39:13.502 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 110, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-06-10T00:39:13.503 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 12, 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 11, 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 11, 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-06-10T00:39:13.504 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 0, 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-06-10T00:39:13.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 3287, 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 257890461, 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 3287, 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3287, 2024-06-10T00:39:13.506 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 257890461, 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:13.507 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 2994, 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 255710405, 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 2994, 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 255710405, 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:13.508 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 2994, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 2994, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 2994, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 2994, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-06-10T00:39:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x560b878206e0": { 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:13.511 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:13.512 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-06-10T00:39:13.512 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-06-10T00:39:13.512 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-06-10T00:39:13.512 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-06-10T00:39:13.512 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:13.512 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:13.512 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:13.512 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:13.512 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:13.512 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:13.512 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-06-10T00:39:14.230 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-06-10T00:39:14.231 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-06-10T00:39:14.231 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-06-10T00:39:14.231 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-06-10T00:39:14.231 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-06-10T00:39:14.231 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-06-10T00:39:14.231 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-06-10T00:39:14.231 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-06-10T00:39:14.231 DEBUG:tasks.check_counter:Getting stats from i 2024-06-10T00:39:14.231 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.i perf dump 2024-06-10T00:39:14.619 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:39:15.008 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:14 smithi135 ceph-mon[36424]: pgmap v350: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:15.042 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:14 smithi193 ceph-mon[36716]: pgmap v350: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:15.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:14 smithi121 ceph-mon[32378]: pgmap v350: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-06-10T00:39:17.068 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:16 smithi193 ceph-mon[36716]: pgmap v351: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-06-10T00:39:17.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:16 smithi121 ceph-mon[32378]: pgmap v351: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-06-10T00:39:17.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:16 smithi135 ceph-mon[36424]: pgmap v351: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2024-06-10T00:39:17.777 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-06-10T00:39:17.777 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:39:17.777 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 70500, 2024-06-10T00:39:17.777 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 80961, 2024-06-10T00:39:17.777 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 26117103, 2024-06-10T00:39:17.777 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 136763199, 2024-06-10T00:39:17.777 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 11, 2024-06-10T00:39:17.777 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 5, 2024-06-10T00:39:17.777 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 4.366605209, 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 1.644055795, 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 1.678422115, 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 1.057772613, 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 80960, 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 3.655500484, 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000045151 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 67104, 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.010198706, 2024-06-10T00:39:17.778 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000151 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1675504, 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 187584 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 287131, 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 378369, 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 103559276, 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 292086107, 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 34, 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 9, 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 14.377636544, 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 6.763225805, 2024-06-10T00:39:17.779 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 11.427441148, 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.796078012, 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 378365, 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 15.072906838, 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000039836 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 514188, 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.075033615, 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000145 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 96054208, 2024-06-10T00:39:17.780 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 209365952 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 40554, 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 45161, 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 32600995, 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 98320336, 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 12, 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 5, 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 2.787301689, 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 1.088968052, 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 1.311411982, 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.545624470, 2024-06-10T00:39:17.781 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 45161, 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 2.734537357, 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000060550 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 54566, 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.008301495, 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000152 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1568, 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 5712 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.782 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 47253, 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 67.253288142, 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001423259 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-06-10T00:39:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 21536, 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 5.112466016, 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000237391 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 148441, 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 133611, 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 133611, 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 2451.319198501, 2024-06-10T00:39:17.784 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.018346687 2024-06-10T00:39:17.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.785 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-06-10T00:39:17.785 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 14830, 2024-06-10T00:39:17.785 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 34, 2024-06-10T00:39:17.785 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-06-10T00:39:17.785 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 515, 2024-06-10T00:39:17.785 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 56, 2024-06-10T00:39:17.785 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-06-10T00:39:17.785 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 25915, 2024-06-10T00:39:17.785 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 87, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 13294, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 12534, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 25788, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 37892, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 1, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 1, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 4, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 268617, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 213112, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 14547, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 614, 2024-06-10T00:39:17.786 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 4, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 1017, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 160, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 32553, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 127, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 33043, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 50150, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 11510, 2024-06-10T00:39:17.787 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 254329998, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 6, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 50150, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 267, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 267, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 49883, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 556, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 59399, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 45885, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-06-10T00:39:17.788 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 76052, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 15307, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 3435, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 30604 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 2294, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 5905, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 3295, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 4141, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 2985, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 65, 2024-06-10T00:39:17.789 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 12359, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 12294, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_path": 8, 2024-06-10T00:39:17.790 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_quiesce_inode": 46467, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 160, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 59, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "evlrg": 4, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 91839, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 1, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 1, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 91838, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-06-10T00:39:17.791 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 1, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 129, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 1, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 1, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 128, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "segmjr": 8, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 1, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194352, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 259863147, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 0, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 8478, 2024-06-10T00:39:17.792 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 230.487998117, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.027186600 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 0 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 14096, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 49057, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 34961, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 472, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 3796, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 3324, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 25915, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 79217, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 53302, 2024-06-10T00:39:17.793 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 1, 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 33497, 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 33496, 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 200620, 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 223612 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 161739, 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 2589, 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 148441, 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 205, 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 10963, 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-06-10T00:39:17.794 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 9624, 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 154.366906325, 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.016039786 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2028, 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 71.994265563, 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.035500130 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-06-10T00:39:17.795 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 92786, 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1797.635840616, 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.019373998 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1, 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000305089, 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000305089 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.797 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.798 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.798 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.798 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-06-10T00:39:17.798 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 38, 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.833129987, 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.021924473 2024-06-10T00:39:17.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 9206, 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 92.321056462, 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.010028357 2024-06-10T00:39:17.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 476, 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.788004298, 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001655471 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 9619, 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 297.788070555, 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.030958319 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 65, 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 17.693979682, 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.272215072 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.802 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.803 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.804 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 9768, 2024-06-10T00:39:17.805 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 17.897639924, 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001832272 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-06-10T00:39:17.806 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 4, 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 2, 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 2, 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 13336, 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 6668, 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 650, 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:39:17.807 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:39:17.808 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-06-10T00:39:17.809 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 1328397, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 13823, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 88, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 1, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-06-10T00:39:17.810 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 9848, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 190, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 52321914, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 558968, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-06-10T00:39:17.811 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 85119, 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 270093102, 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 32, 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 85119, 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 85119, 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 352.950613419, 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.004146554 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-06-10T00:39:17.812 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 85119, 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 86231 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 85119, 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 50533, 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 34586, 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 0, 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 10, 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 349, 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 8827, 2024-06-10T00:39:17.813 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 194, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 56, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 24660, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 50150, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 20, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-06-10T00:39:17.814 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-06-10T00:39:17.815 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 110, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 12, 2024-06-10T00:39:17.816 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 12, 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 12, 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 1627, 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 68, 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 270 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 1, 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.817 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 46427 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 16, 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 8, 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed_ops": 24588, 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 12294, 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:39:17.818 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 398181, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 113438344, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 398181, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 398181, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 113438344, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.819 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.820 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:17.820 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.820 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-06-10T00:39:17.820 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:17.820 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-06-10T00:39:17.820 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:17.820 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-06-10T00:39:17.820 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-06-10T00:39:17.820 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:17.820 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:17.820 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 85119, 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 271784356, 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 85119, 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 271784356, 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-06-10T00:39:17.821 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 85119, 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 85119, 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 85119, 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 85119, 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:17.822 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 12294, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 1691798, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 12294, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 349, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 1691798, 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:17.823 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x55a48b1d8500": { 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 91839, 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 255668843, 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 91839, 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-06-10T00:39:17.824 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-06-10T00:39:17.825 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 8478, 2024-06-10T00:39:17.825 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 255668843, 2024-06-10T00:39:17.825 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:17.825 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:17.825 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:17.825 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:17.825 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:17.825 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:17.825 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-06-10T00:39:18.623 INFO:tasks.check_counter:Daemon mds.i mds.exported=160 2024-06-10T00:39:18.624 INFO:tasks.check_counter:Daemon mds.i mds.imported=127 2024-06-10T00:39:18.624 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=2294 2024-06-10T00:39:18.624 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=5905 2024-06-10T00:39:18.624 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-06-10T00:39:18.624 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-06-10T00:39:18.624 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-06-10T00:39:18.624 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=56 2024-06-10T00:39:18.624 DEBUG:tasks.check_counter:Getting stats from l 2024-06-10T00:39:18.624 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph daemon mds.l perf dump 2024-06-10T00:39:18.881 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:18 smithi193 ceph-mon[36716]: pgmap v352: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 299 B/s wr, 6 op/s 2024-06-10T00:39:19.033 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.c/config 2024-06-10T00:39:19.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:18 smithi121 ceph-mon[32378]: pgmap v352: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 299 B/s wr, 6 op/s 2024-06-10T00:39:19.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:18 smithi135 ceph-mon[36424]: pgmap v352: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 299 B/s wr, 6 op/s 2024-06-10T00:39:20.884 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:20 smithi193 ceph-mon[36716]: pgmap v353: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-06-10T00:39:20.992 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-06-10T00:39:20.993 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-06-10T00:39:20.993 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 4, 2024-06-10T00:39:20.993 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 4, 2024-06-10T00:39:20.993 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 3024, 2024-06-10T00:39:20.993 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 448, 2024-06-10T00:39:20.993 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 4, 2024-06-10T00:39:20.993 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 0, 2024-06-10T00:39:20.993 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.001373062, 2024-06-10T00:39:20.993 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.000034445, 2024-06-10T00:39:20.993 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.000628692, 2024-06-10T00:39:20.993 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:39:20.994 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:20.994 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3, 2024-06-10T00:39:20.994 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000717151, 2024-06-10T00:39:20.994 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000239050 2024-06-10T00:39:20.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:20.994 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:20.994 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:20.994 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:20.994 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:20.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:20.994 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 448 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 271, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 260, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 1594800, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 160400, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 2, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 1, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.047680708, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.014494267, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.036573199, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.006129246, 2024-06-10T00:39:20.995 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 259, 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.011840885, 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000045717 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1594800, 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 160400 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-06-10T00:39:20.996 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 2, 2024-06-10T00:39:20.997 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 199, 2024-06-10T00:39:20.997 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 320, 2024-06-10T00:39:20.997 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 190848, 2024-06-10T00:39:20.997 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 5, 2024-06-10T00:39:20.997 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 3, 2024-06-10T00:39:20.997 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.011599912, 2024-06-10T00:39:20.997 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.007936257, 2024-06-10T00:39:20.997 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.001508170, 2024-06-10T00:39:20.997 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-06-10T00:39:20.997 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-06-10T00:39:20.997 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 196, 2024-06-10T00:39:20.998 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.014450640, 2024-06-10T00:39:20.998 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000073727 2024-06-10T00:39:20.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:20.998 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-06-10T00:39:20.998 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:20.998 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:20.998 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:20.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 320, 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 190848 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 0, 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-06-10T00:39:20.999 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-06-10T00:39:21.000 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-06-10T00:39:21.001 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 148771, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 32, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-06-10T00:39:21.002 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-06-10T00:39:21.003 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 0, 2024-06-10T00:39:21.003 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 0, 2024-06-10T00:39:21.003 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-06-10T00:39:21.003 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-06-10T00:39:21.003 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-06-10T00:39:21.003 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-06-10T00:39:21.003 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 928, 2024-06-10T00:39:21.003 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 2, 2024-06-10T00:39:21.003 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-06-10T00:39:21.003 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 273, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 1556579, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 273, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-06-10T00:39:21.004 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 273, 2024-06-10T00:39:21.005 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 1556579, 2024-06-10T00:39:21.005 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-06-10T00:39:21.005 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-06-10T00:39:21.005 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-06-10T00:39:21.005 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-06-10T00:39:21.005 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:21.005 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-06-10T00:39:21.005 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-06-10T00:39:21.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:20 smithi121 ceph-mon[32378]: pgmap v353: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-06-10T00:39:21.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:20 smithi135 ceph-mon[36424]: pgmap v353: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-06-10T00:39:21.760 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-06-10T00:39:21.761 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-06-10T00:39:21.761 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-06-10T00:39:21.761 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-06-10T00:39:21.761 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-06-10T00:39:21.761 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-06-10T00:39:21.761 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-06-10T00:39:21.761 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-06-10T00:39:21.762 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-06-10T00:39:21.772 INFO:tasks.kclient:Unmounting kernel clients... 2024-06-10T00:39:21.772 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:39:21.772 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-10T00:39:21.817 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:39:21.817 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-10T00:39:21.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd blocklist ls 2024-06-10T00:39:22.421 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:39:22.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:22 smithi121 ceph-mon[32378]: pgmap v354: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:39:23.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:22 smithi193 ceph-mon[36716]: pgmap v354: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:39:23.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:22 smithi135 ceph-mon[36424]: pgmap v354: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:39:24.814 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:24 smithi121 ceph-mon[32378]: pgmap v355: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:39:25.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:24 smithi193 ceph-mon[36716]: pgmap v355: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:39:25.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:24 smithi135 ceph-mon[36424]: pgmap v355: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-06-10T00:39:25.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.423+0000 7f1ace557700 1 -- 172.21.15.121:0/1954975502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ac8101570 msgr2=0x7f1ac8101950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:25.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.423+0000 7f1ace557700 1 --2- 172.21.15.121:0/1954975502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ac8101570 0x7f1ac8101950 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1ac001c0d0 tx=0x7f1ac001c3e0 comp rx=0 tx=0).stop 2024-06-10T00:39:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.428+0000 7f1ace557700 1 -- 172.21.15.121:0/1954975502 shutdown_connections 2024-06-10T00:39:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.428+0000 7f1ace557700 1 --2- 172.21.15.121:0/1954975502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac8106650 0x7f1ac810e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.428+0000 7f1ace557700 1 --2- 172.21.15.121:0/1954975502 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ac8101570 0x7f1ac8101950 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.428+0000 7f1ace557700 1 --2- 172.21.15.121:0/1954975502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ac8101f20 0x7f1ac8105f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:25.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.428+0000 7f1ace557700 1 -- 172.21.15.121:0/1954975502 >> 172.21.15.121:0/1954975502 conn(0x7f1ac80fce20 msgr2=0x7f1ac80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:25.430 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.430+0000 7f1ace557700 1 -- 172.21.15.121:0/1954975502 shutdown_connections 2024-06-10T00:39:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.431+0000 7f1ace557700 1 -- 172.21.15.121:0/1954975502 wait complete. 2024-06-10T00:39:25.432 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.431+0000 7f1ace557700 1 Processor -- start 2024-06-10T00:39:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.432+0000 7f1ace557700 1 -- start start 2024-06-10T00:39:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.432+0000 7f1ace557700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ac8101570 0x7f1ac8108d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.432+0000 7f1ace557700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac8101f20 0x7f1ac8109280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.432+0000 7f1ace557700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ac8106650 0x7f1ac810d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.432+0000 7f1ace557700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac806e840 con 0x7f1ac8101f20 2024-06-10T00:39:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.432+0000 7f1ace557700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac806e6c0 con 0x7f1ac8101570 2024-06-10T00:39:25.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.432+0000 7f1ace557700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac806e9c0 con 0x7f1ac8106650 2024-06-10T00:39:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.433+0000 7f1accd54700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac8101f20 0x7f1ac8109280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:25.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.433+0000 7f1accd54700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac8101f20 0x7f1ac8109280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45368/0 (socket says 172.21.15.121:45368) 2024-06-10T00:39:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.433+0000 7f1accd54700 1 -- 172.21.15.121:0/1204515387 learned_addr learned my addr 172.21.15.121:0/1204515387 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:39:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.434+0000 7f1accd54700 1 -- 172.21.15.121:0/1204515387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ac8106650 msgr2=0x7f1ac810d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-10T00:39:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.434+0000 7f1accd54700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ac8106650 0x7f1ac810d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.434+0000 7f1accd54700 1 -- 172.21.15.121:0/1204515387 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ac8101570 msgr2=0x7f1ac8108d40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-10T00:39:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.434+0000 7f1accd54700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ac8101570 0x7f1ac8108d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.434+0000 7f1accd54700 1 -- 172.21.15.121:0/1204515387 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac810dd00 con 0x7f1ac8101f20 2024-06-10T00:39:25.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.434+0000 7f1accd54700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac8101f20 0x7f1ac8109280 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f1ac400b550 tx=0x7f1ac400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:25.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.438+0000 7f1abe7fc700 1 -- 172.21.15.121:0/1204515387 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac4015020 con 0x7f1ac8101f20 2024-06-10T00:39:25.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.438+0000 7f1abe7fc700 1 -- 172.21.15.121:0/1204515387 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ac4011d10 con 0x7f1ac8101f20 2024-06-10T00:39:25.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.439+0000 7f1abe7fc700 1 -- 172.21.15.121:0/1204515387 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac400f2e0 con 0x7f1ac8101f20 2024-06-10T00:39:25.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.440+0000 7f1ace557700 1 -- 172.21.15.121:0/1204515387 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac810dff0 con 0x7f1ac8101f20 2024-06-10T00:39:25.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.440+0000 7f1ace557700 1 -- 172.21.15.121:0/1204515387 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac819abe0 con 0x7f1ac8101f20 2024-06-10T00:39:25.443 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.441+0000 7f1ace557700 1 -- 172.21.15.121:0/1204515387 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ac8102a50 con 0x7f1ac8101f20 2024-06-10T00:39:25.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.448+0000 7f1abe7fc700 1 -- 172.21.15.121:0/1204515387 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f1ac4018020 con 0x7f1ac8101f20 2024-06-10T00:39:25.449 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.449+0000 7f1abe7fc700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1ab40a2e70 0x7f1ab40a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:25.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.450+0000 7f1abe7fc700 1 -- 172.21.15.121:0/1204515387 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9613+0+0 (secure 0 0 0) 0x7f1ac40cefc0 con 0x7f1ac8101f20 2024-06-10T00:39:25.451 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.451+0000 7f1acd555700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1ab40a2e70 0x7f1ab40a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:25.454 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.454+0000 7f1acd555700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1ab40a2e70 0x7f1ab40a5330 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1ac001ae70 tx=0x7f1ac001a9f0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:25.455 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.455+0000 7f1abe7fc700 1 -- 172.21.15.121:0/1204515387 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1ac401f020 con 0x7f1ac8101f20 2024-06-10T00:39:25.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.868+0000 7f1ace557700 1 -- 172.21.15.121:0/1204515387 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1ac8069180 con 0x7f1ac8101f20 2024-06-10T00:39:25.868 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.869+0000 7f1abe7fc700 1 -- 172.21.15.121:0/1204515387 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v110) ==== 81+0+956 (secure 0 0 0) 0x7f1ac409b8d0 con 0x7f1ac8101f20 2024-06-10T00:39:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/4266501776 2024-06-11T00:27:57.451553+0000 2024-06-10T00:39:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/4078460254 2024-06-11T00:27:57.451553+0000 2024-06-10T00:39:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/3506007074 2024-06-11T00:27:57.451553+0000 2024-06-10T00:39:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/3506007074 2024-06-11T00:27:57.451553+0000 2024-06-10T00:39:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/58990727 2024-06-11T00:27:57.451553+0000 2024-06-10T00:39:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1710032094 2024-06-11T00:16:42.011037+0000 2024-06-10T00:39:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/704666019 2024-06-11T00:16:42.011037+0000 2024-06-10T00:39:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1002791322 2024-06-11T00:27:57.451553+0000 2024-06-10T00:39:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/986214985 2024-06-11T00:16:42.011037+0000 2024-06-10T00:39:25.870 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6801/3588432900 2024-06-11T00:16:15.442852+0000 2024-06-10T00:39:25.871 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/3588432900 2024-06-11T00:16:15.442852+0000 2024-06-10T00:39:25.871 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:6800/704666019 2024-06-11T00:16:42.011037+0000 2024-06-10T00:39:25.871 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/743919928 2024-06-11T00:16:42.011037+0000 2024-06-10T00:39:25.871 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/2834833132 2024-06-11T00:16:15.442852+0000 2024-06-10T00:39:25.871 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1390094598 2024-06-11T00:16:15.442852+0000 2024-06-10T00:39:25.871 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121:0/1651500994 2024-06-11T00:16:15.442852+0000 2024-06-10T00:39:25.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.873+0000 7f1ace557700 1 -- 172.21.15.121:0/1204515387 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1ab40a2e70 msgr2=0x7f1ab40a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:25.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.874+0000 7f1ace557700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1ab40a2e70 0x7f1ab40a5330 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1ac001ae70 tx=0x7f1ac001a9f0 comp rx=0 tx=0).stop 2024-06-10T00:39:25.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.874+0000 7f1ace557700 1 -- 172.21.15.121:0/1204515387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac8101f20 msgr2=0x7f1ac8109280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:25.874 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.874+0000 7f1ace557700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac8101f20 0x7f1ac8109280 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f1ac400b550 tx=0x7f1ac400b910 comp rx=0 tx=0).stop 2024-06-10T00:39:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.875+0000 7f1ace557700 1 -- 172.21.15.121:0/1204515387 shutdown_connections 2024-06-10T00:39:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.875+0000 7f1ace557700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f1ab40a2e70 0x7f1ab40a5330 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.875+0000 7f1ace557700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ac8106650 0x7f1ac810d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.875+0000 7f1ace557700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ac8101570 0x7f1ac8108d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.875+0000 7f1ace557700 1 --2- 172.21.15.121:0/1204515387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac8101f20 0x7f1ac8109280 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.875+0000 7f1ace557700 1 -- 172.21.15.121:0/1204515387 >> 172.21.15.121:0/1204515387 conn(0x7f1ac80fce20 msgr2=0x7f1ac80fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.878+0000 7f1ace557700 1 -- 172.21.15.121:0/1204515387 shutdown_connections 2024-06-10T00:39:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:25.878+0000 7f1ace557700 1 -- 172.21.15.121:0/1204515387 wait complete. 2024-06-10T00:39:25.882 INFO:teuthology.orchestra.run.smithi121.stderr:listed 16 entries 2024-06-10T00:39:27.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:26 smithi193 ceph-mon[36716]: pgmap v356: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-06-10T00:39:27.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:26 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1204515387' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-10T00:39:27.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:26 smithi121 ceph-mon[32378]: pgmap v356: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-06-10T00:39:27.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:26 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1204515387' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-10T00:39:27.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:26 smithi135 ceph-mon[36424]: pgmap v356: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-06-10T00:39:27.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:26 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1204515387' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-10T00:39:27.263 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-06-10T00:39:27.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:39:27.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-06-10T00:39:27.442 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-06-10T00:39:27.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:39:27.443 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-06-10T00:39:27.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:39:27.488 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-06-10T00:39:27.577 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-06-10T00:39:27.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:39:27.577 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-06-10T00:39:27.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:39:27.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2024-06-10T00:39:27.776 INFO:tasks.kclient:Unmounting kernel clients... 2024-06-10T00:39:27.776 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-06-10T00:39:27.776 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-06-10T00:39:27.806 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056291,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497568k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3321 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248780k,nr_inodes=812195,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/622c4f1a93dd4f2b22b329835cd2812f4c1f15f21c9656191ffee99295e6eb26/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/RPERDF5WAUSCEEFH4O2HFWKSSK:/var/lib/containers/storage/overlay/l/5UKB3OBEXLJDLOWIP5WX73ZHS2,upperdir=/var/lib/containers/storage/overlay/622c4f1a93dd4f2b22b329835cd2812f4c1f15f21c9656191ffee99295e6eb26/diff,workdir=/var/lib/containers/storage/overlay/622c4f1a93dd4f2b22b329835cd2812f4c1f15f21c9656191ffee99295e6eb26/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/73cbe159337f7bf270e1771a1749def1a68efc61c27eb4584cf8eed2cd1ab747/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/RPERDF5WAUSCEEFH4O2HFWKSSK:/var/lib/containers/storage/overlay/l/5UKB3OBEXLJDLOWIP5WX73ZHS2,upperdir=/var/lib/containers/storage/overlay/73cbe159337f7bf270e1771a1749def1a68efc61c27eb4584cf8eed2cd1ab747/diff,workdir=/var/lib/containers/storage/overlay/73cbe159337f7bf270e1771a1749def1a68efc61c27eb4584cf8eed2cd1ab747/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248780k,nr_inodes=812195,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/7b329ac753f76c08018cf4f0475ebe8d9f94e8ce61922451957619e948e75ee0/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/RPERDF5WAUSCEEFH4O2HFWKSSK:/var/lib/containers/storage/overlay/l/5UKB3OBEXLJDLOWIP5WX73ZHS2,upperdir=/var/lib/containers/storage/overlay/7b329ac753f76c08018cf4f0475ebe8d9f94e8ce61922451957619e948e75ee0/diff,workdir=/var/lib/containers/storage/overlay/7b329ac753f76c08018cf4f0475ebe8d9f94e8ce61922451957619e948e75ee0/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/274b75b7f792331c5a30be972722840c37b90da2b7b6eeb5eaea68c66ed9df0f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/RPERDF5WAUSCEEFH4O2HFWKSSK:/var/lib/containers/storage/overlay/l/5UKB3OBEXLJDLOWIP5WX73ZHS2,upperdir=/var/lib/containers/storage/overlay/274b75b7f792331c5a30be972722840c37b90da2b7b6eeb5eaea68c66ed9df0f/diff,workdir=/var/lib/containers/storage/overlay/274b75b7f792331c5a30be972722840c37b90da2b7b6eeb5eaea68c66ed9df0f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0944eb3314744e1698af30d3956b82fcb82bdde6f0b96b440b6662297baa2a4c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/RPERDF5WAUSCEEFH4O2HFWKSSK:/var/lib/containers/storage/overlay/l/5UKB3OBEXLJDLOWIP5WX73ZHS2,upperdir=/var/lib/containers/storage/overlay/0944eb3314744e1698af30d3956b82fcb82bdde6f0b96b440b6662297baa2a4c/diff,workdir=/var/lib/containers/storage/overlay/0944eb3314744e1698af30d3956b82fcb82bdde6f0b96b440b6662297baa2a4c/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/cfaac1049b32142faf0e69d9e9cbdbf2195d8e0cbeb7f52028c91e51dfb6ebb3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/RPERDF5WAUSCEEFH4O2HFWKSSK:/var/lib/containers/storage/overlay/l/5UKB3OBEXLJDLOWIP5WX73ZHS2,upperdir=/var/lib/containers/storage/overlay/cfaac1049b32142faf0e69d9e9cbdbf2195d8e0cbeb7f52028c91e51dfb6ebb3/diff,workdir=/var/lib/containers/storage/overlay/cfaac1049b32142faf0e69d9e9cbdbf2195d8e0cbeb7f52028c91e51dfb6ebb3/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/48ecfac32414f44f1bd2b4cbb4094efa4e3e0d007507efd559be7e5af8208355/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/RPERDF5WAUSCEEFH4O2HFWKSSK:/var/lib/containers/storage/overlay/l/5UKB3OBEXLJDLOWIP5WX73ZHS2,upperdir=/var/lib/containers/storage/overlay/48ecfac32414f44f1bd2b4cbb4094efa4e3e0d007507efd559be7e5af8208355/diff,workdir=/var/lib/containers/storage/overlay/48ecfac32414f44f1bd2b4cbb4094efa4e3e0d007507efd559be7e5af8208355/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e4cea2e006a182f0554fe8a3487a8bcc0f4c7bd4730c70908bf0a84220b3d418/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/RPERDF5WAUSCEEFH4O2HFWKSSK:/var/lib/containers/storage/overlay/l/5UKB3OBEXLJDLOWIP5WX73ZHS2,upperdir=/var/lib/containers/storage/overlay/e4cea2e006a182f0554fe8a3487a8bcc0f4c7bd4730c70908bf0a84220b3d418/diff,workdir=/var/lib/containers/storage/overlay/e4cea2e006a182f0554fe8a3487a8bcc0f4c7bd4730c70908bf0a84220b3d418/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4c3daf5b457f24d95e12b2edff0728cb4e8d6f7d7f2f4857f944125a4496ad39/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/RPERDF5WAUSCEEFH4O2HFWKSSK:/var/lib/containers/storage/overlay/l/5UKB3OBEXLJDLOWIP5WX73ZHS2,upperdir=/var/lib/containers/storage/overlay/4c3daf5b457f24d95e12b2edff0728cb4e8d6f7d7f2f4857f944125a4496ad39/diff,workdir=/var/lib/containers/storage/overlay/4c3daf5b457f24d95e12b2edff0728cb4e8d6f7d7f2f4857f944125a4496ad39/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/568b3a748a1ad52793cf29a7530f00ca68cfe0d3cceb2b90c60571f448a18757/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/RPERDF5WAUSCEEFH4O2HFWKSSK:/var/lib/containers/storage/overlay/l/5UKB3OBEXLJDLOWIP5WX73ZHS2,upperdir=/var/lib/containers/storage/overlay/568b3a748a1ad52793cf29a7530f00ca68cfe0d3cceb2b90c60571f448a18757/diff,workdir=/var/lib/containers/storage/overlay/568b3a748a1ad52793cf29a7530f00ca68cfe0d3cceb2b90c60571f448a18757/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497568k,nr_inodes=819200,mode=755,inode64 0 0 2024-06-10T00:39:27.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:39:27.807 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-06-10T00:39:27.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-06-10T00:39:27.884 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2024-06-10T00:39:27.975 INFO:teuthology.orchestra.run.smithi121.stderr:Cannot find device "ceph-brx" 2024-06-10T00:39:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:39:27.977 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-10T00:39:27.991 INFO:tasks.cephadm:Teardown begin 2024-06-10T00:39:27.991 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-06-10T00:39:27.991 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:39:28.461 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:39:28.878 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:28 smithi121 ceph-mon[32378]: pgmap v357: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:39:29.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:28 smithi193 ceph-mon[36716]: pgmap v357: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:39:29.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:28 smithi135 ceph-mon[36424]: pgmap v357: 129 pgs: 20 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 56 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-06-10T00:39:30.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.584+0000 7f6db9915700 1 -- 172.21.15.121:0/4082109105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db4109390 msgr2=0x7f6db4115830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:30.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.584+0000 7f6db9915700 1 --2- 172.21.15.121:0/4082109105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db4109390 0x7f6db4115830 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f6dac009990 tx=0x7f6dac009ca0 comp rx=0 tx=0).stop 2024-06-10T00:39:30.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.584+0000 7f6db9915700 1 -- 172.21.15.121:0/4082109105 shutdown_connections 2024-06-10T00:39:30.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.584+0000 7f6db9915700 1 --2- 172.21.15.121:0/4082109105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db4109390 0x7f6db4115830 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:30.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.584+0000 7f6db9915700 1 --2- 172.21.15.121:0/4082109105 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db410d8a0 0x7f6db410dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:30.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.584+0000 7f6db9915700 1 --2- 172.21.15.121:0/4082109105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db41089d0 0x7f6db4108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:30.585 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.584+0000 7f6db9915700 1 -- 172.21.15.121:0/4082109105 >> 172.21.15.121:0/4082109105 conn(0x7f6db4078220 msgr2=0x7f6db41072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.586+0000 7f6db9915700 1 -- 172.21.15.121:0/4082109105 shutdown_connections 2024-06-10T00:39:30.587 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.586+0000 7f6db9915700 1 -- 172.21.15.121:0/4082109105 wait complete. 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.586+0000 7f6db9915700 1 Processor -- start 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.587+0000 7f6db9915700 1 -- start start 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.587+0000 7f6db9915700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db41089d0 0x7f6db4073c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.587+0000 7f6db9915700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db4109390 0x7f6db406cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.587+0000 7f6db9915700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db410d8a0 0x7f6db406d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.587+0000 7f6db9915700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6db4118d20 con 0x7f6db410d8a0 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.587+0000 7f6db9915700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6db4118ba0 con 0x7f6db4109390 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.587+0000 7f6db9915700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6db4118ea0 con 0x7f6db41089d0 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.587+0000 7f6db2ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db41089d0 0x7f6db4073c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.587+0000 7f6db2ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db41089d0 0x7f6db4073c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:34990/0 (socket says 172.21.15.121:34990) 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.587+0000 7f6db2ffd700 1 -- 172.21.15.121:0/3165307466 learned_addr learned my addr 172.21.15.121:0/3165307466 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:39:30.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.587+0000 7f6db27fc700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db4109390 0x7f6db406cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.590+0000 7f6db2ffd700 1 -- 172.21.15.121:0/3165307466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db4109390 msgr2=0x7f6db406cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:30.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.590+0000 7f6db2ffd700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db4109390 0x7f6db406cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.590+0000 7f6db2ffd700 1 -- 172.21.15.121:0/3165307466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db410d8a0 msgr2=0x7f6db406d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.590+0000 7f6db2ffd700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db410d8a0 0x7f6db406d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.590+0000 7f6db2ffd700 1 -- 172.21.15.121:0/3165307466 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db406d810 con 0x7f6db41089d0 2024-06-10T00:39:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.590+0000 7f6db2ffd700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db41089d0 0x7f6db4073c50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6da400ada0 tx=0x7f6da400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.591+0000 7f6db8913700 1 -- 172.21.15.121:0/3165307466 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4017020 con 0x7f6db41089d0 2024-06-10T00:39:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.591+0000 7f6db8913700 1 -- 172.21.15.121:0/3165307466 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6da4004bb0 con 0x7f6db41089d0 2024-06-10T00:39:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.591+0000 7f6db8913700 1 -- 172.21.15.121:0/3165307466 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4005670 con 0x7f6db41089d0 2024-06-10T00:39:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.592+0000 7f6db9915700 1 -- 172.21.15.121:0/3165307466 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db406db00 con 0x7f6db41089d0 2024-06-10T00:39:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.592+0000 7f6db9915700 1 -- 172.21.15.121:0/3165307466 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db4071a70 con 0x7f6db41089d0 2024-06-10T00:39:30.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.592+0000 7f6db27fc700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db4109390 0x7f6db406cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:39:30.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.596+0000 7f6db9915700 1 -- 172.21.15.121:0/3165307466 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6db410a0c0 con 0x7f6db41089d0 2024-06-10T00:39:30.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.599+0000 7f6db8913700 1 -- 172.21.15.121:0/3165307466 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6da401a070 con 0x7f6db41089d0 2024-06-10T00:39:30.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.599+0000 7f6db8913700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6d9c0a2f90 0x7f6d9c0a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:30.600 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.600+0000 7f6db8913700 1 -- 172.21.15.121:0/3165307466 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9613+0+0 (secure 0 0 0) 0x7f6da40ce990 con 0x7f6db41089d0 2024-06-10T00:39:30.601 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.600+0000 7f6db27fc700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6d9c0a2f90 0x7f6d9c0a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:30.605 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.605+0000 7f6db27fc700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6d9c0a2f90 0x7f6d9c0a5450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6db406e4a0 tx=0x7f6da8009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:30.607 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:30.607+0000 7f6db8913700 1 -- 172.21.15.121:0/3165307466 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6da404e850 con 0x7f6db41089d0 2024-06-10T00:39:31.073 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.072+0000 7f6db9915700 1 -- 172.21.15.121:0/3165307466 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6db406e240 con 0x7f6db41089d0 2024-06-10T00:39:31.073 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:30 smithi121 ceph-mon[32378]: pgmap v358: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-06-10T00:39:31.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.077+0000 7f6db8913700 1 -- 172.21.15.121:0/3165307466 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11895 (secure 0 0 0) 0x7f6db406e240 con 0x7f6db41089d0 2024-06-10T00:39:31.077 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:39:31.077 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:35:59.291154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":189,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:39:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.082+0000 7f6db9915700 1 -- 172.21.15.121:0/3165307466 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6d9c0a2f90 msgr2=0x7f6d9c0a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.082+0000 7f6db9915700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6d9c0a2f90 0x7f6d9c0a5450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6db406e4a0 tx=0x7f6da8009260 comp rx=0 tx=0).stop 2024-06-10T00:39:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.082+0000 7f6db9915700 1 -- 172.21.15.121:0/3165307466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db41089d0 msgr2=0x7f6db4073c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:31.082 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.082+0000 7f6db9915700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db41089d0 0x7f6db4073c50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6da400ada0 tx=0x7f6da400c330 comp rx=0 tx=0).stop 2024-06-10T00:39:31.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.083+0000 7f6db9915700 1 -- 172.21.15.121:0/3165307466 shutdown_connections 2024-06-10T00:39:31.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.083+0000 7f6db9915700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6d9c0a2f90 0x7f6d9c0a5450 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:31.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.083+0000 7f6db9915700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6db410d8a0 0x7f6db406d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:31.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.083+0000 7f6db9915700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6db41089d0 0x7f6db4073c50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:31.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.083+0000 7f6db9915700 1 --2- 172.21.15.121:0/3165307466 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6db4109390 0x7f6db406cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:31.086 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.083+0000 7f6db9915700 1 -- 172.21.15.121:0/3165307466 >> 172.21.15.121:0/3165307466 conn(0x7f6db4078220 msgr2=0x7f6db4078840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:31.087 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.087+0000 7f6db9915700 1 -- 172.21.15.121:0/3165307466 shutdown_connections 2024-06-10T00:39:31.088 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:31.089+0000 7f6db9915700 1 -- 172.21.15.121:0/3165307466 wait complete. 2024-06-10T00:39:31.092 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 33 2024-06-10T00:39:31.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:30 smithi193 ceph-mon[36716]: pgmap v358: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-06-10T00:39:31.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:30 smithi135 ceph-mon[36424]: pgmap v358: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 7 op/s 2024-06-10T00:39:31.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-06-10T00:39:32.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:31 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3165307466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:39:32.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:31 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3165307466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:39:32.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:31 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3165307466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:39:32.302 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:39:33.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:32 smithi193 ceph-mon[36716]: pgmap v359: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:33.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:32 smithi121 ceph-mon[32378]: pgmap v359: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:33.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:32 smithi135 ceph-mon[36424]: pgmap v359: 129 pgs: 19 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 57 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:34.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.485+0000 7f6b59512700 1 -- 172.21.15.121:0/2872728262 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5410ef90 msgr2=0x7f6b5410f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:34.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.485+0000 7f6b59512700 1 --2- 172.21.15.121:0/2872728262 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5410ef90 0x7f6b5410f370 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c0085a0 tx=0x7f6b4c00acb0 comp rx=0 tx=0).stop 2024-06-10T00:39:34.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.485+0000 7f6b59512700 1 -- 172.21.15.121:0/2872728262 shutdown_connections 2024-06-10T00:39:34.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.485+0000 7f6b59512700 1 --2- 172.21.15.121:0/2872728262 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b54109e50 0x7f6b54115d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:34.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.485+0000 7f6b59512700 1 --2- 172.21.15.121:0/2872728262 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b5410ef90 0x7f6b5410f370 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:34.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.485+0000 7f6b59512700 1 --2- 172.21.15.121:0/2872728262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b54109490 0x7f6b54109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:34.486 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.485+0000 7f6b59512700 1 -- 172.21.15.121:0/2872728262 >> 172.21.15.121:0/2872728262 conn(0x7f6b5406d470 msgr2=0x7f6b5406d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:34.488 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.488+0000 7f6b59512700 1 -- 172.21.15.121:0/2872728262 shutdown_connections 2024-06-10T00:39:34.489 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.488+0000 7f6b59512700 1 -- 172.21.15.121:0/2872728262 wait complete. 2024-06-10T00:39:34.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.490+0000 7f6b59512700 1 Processor -- start 2024-06-10T00:39:34.490 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.490+0000 7f6b59512700 1 -- start start 2024-06-10T00:39:34.491 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.491+0000 7f6b59512700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b54109e50 0x7f6b5419f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:34.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.491+0000 7f6b59512700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b5410ef90 0x7f6b5419f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:34.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.491+0000 7f6b52ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b54109e50 0x7f6b5419f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:34.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.491+0000 7f6b52ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b54109e50 0x7f6b5419f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:35010/0 (socket says 172.21.15.121:35010) 2024-06-10T00:39:34.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.491+0000 7f6b52ffd700 1 -- 172.21.15.121:0/1499052195 learned_addr learned my addr 172.21.15.121:0/1499052195 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:39:34.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.491+0000 7f6b59512700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b541a3cd0 0x7f6b541a4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:34.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.491+0000 7f6b59512700 1 -- 172.21.15.121:0/1499052195 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6b54118ef0 con 0x7f6b5410ef90 2024-06-10T00:39:34.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.491+0000 7f6b59512700 1 -- 172.21.15.121:0/1499052195 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f6b54118d70 con 0x7f6b541a3cd0 2024-06-10T00:39:34.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.492+0000 7f6b537fe700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b541a3cd0 0x7f6b541a4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:34.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.492+0000 7f6b59512700 1 -- 172.21.15.121:0/1499052195 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6b54119070 con 0x7f6b54109e50 2024-06-10T00:39:34.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.492+0000 7f6b537fe700 1 -- 172.21.15.121:0/1499052195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b54109e50 msgr2=0x7f6b5419f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:34.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.492+0000 7f6b537fe700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b54109e50 0x7f6b5419f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:34.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.492+0000 7f6b537fe700 1 -- 172.21.15.121:0/1499052195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b5410ef90 msgr2=0x7f6b5419f940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:39:34.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.492+0000 7f6b537fe700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b5410ef90 0x7f6b5419f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:34.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.492+0000 7f6b537fe700 1 -- 172.21.15.121:0/1499052195 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b541a47f0 con 0x7f6b541a3cd0 2024-06-10T00:39:34.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.492+0000 7f6b537fe700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b541a3cd0 0x7f6b541a4150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6b44011180 tx=0x7f6b44011490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:34.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.494+0000 7f6b4bfff700 1 -- 172.21.15.121:0/1499052195 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44004bb0 con 0x7f6b541a3cd0 2024-06-10T00:39:34.494 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.494+0000 7f6b4bfff700 1 -- 172.21.15.121:0/1499052195 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b44004d50 con 0x7f6b541a3cd0 2024-06-10T00:39:34.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.495+0000 7f6b4bfff700 1 -- 172.21.15.121:0/1499052195 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44019810 con 0x7f6b541a3cd0 2024-06-10T00:39:34.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.495+0000 7f6b59512700 1 -- 172.21.15.121:0/1499052195 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b541a4ae0 con 0x7f6b541a3cd0 2024-06-10T00:39:34.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.495+0000 7f6b59512700 1 -- 172.21.15.121:0/1499052195 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b541ac1b0 con 0x7f6b541a3cd0 2024-06-10T00:39:34.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.498+0000 7f6b4bfff700 1 -- 172.21.15.121:0/1499052195 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f6b440199b0 con 0x7f6b541a3cd0 2024-06-10T00:39:34.499 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.499+0000 7f6b4bfff700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6b400a2e70 0x7f6b400a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:34.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.499+0000 7f6b4bfff700 1 -- 172.21.15.121:0/1499052195 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9613+0+0 (secure 0 0 0) 0x7f6b440d34f0 con 0x7f6b541a3cd0 2024-06-10T00:39:34.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.500+0000 7f6b59512700 1 -- 172.21.15.121:0/1499052195 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b5410a6b0 con 0x7f6b541a3cd0 2024-06-10T00:39:34.500 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.500+0000 7f6b52ffd700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6b400a2e70 0x7f6b400a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:34.501 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.501+0000 7f6b52ffd700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6b400a2e70 0x7f6b400a5330 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c008570 tx=0x7f6b4c005fb0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:34.508 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.508+0000 7f6b4bfff700 1 -- 172.21.15.121:0/1499052195 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6b44014070 con 0x7f6b541a3cd0 2024-06-10T00:39:34.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:34 smithi121 ceph-mon[32378]: pgmap v360: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:34.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.971+0000 7f6b59512700 1 -- 172.21.15.121:0/1499052195 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b541a0760 con 0x7f6b541a3cd0 2024-06-10T00:39:34.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.975+0000 7f6b4bfff700 1 -- 172.21.15.121:0/1499052195 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11895 (secure 0 0 0) 0x7f6b4409fd80 con 0x7f6b541a3cd0 2024-06-10T00:39:34.976 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:39:34.976 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14514,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4144871914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4144871914},{"type":"v1","addr":"172.21.15.121:6835","nonce":4144871914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14526,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/64109840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":64109840},{"type":"v1","addr":"172.21.15.121:6837","nonce":64109840}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14571,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/4288408975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":4288408975},{"type":"v1","addr":"172.21.15.121:6839","nonce":4288408975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14583,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6841/691733071","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6840","nonce":691733071},{"type":"v1","addr":"172.21.15.121:6841","nonce":691733071}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/360772282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":360772282},{"type":"v1","addr":"172.21.15.193:6833","nonce":360772282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24413,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.135:6833/1842796932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6832","nonce":1842796932},{"type":"v1","addr":"172.21.15.135:6833","nonce":1842796932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-06-10T00:23:59.392036+0000","modified":"2024-06-10T00:35:59.291154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24479,"mds_1":24472,"mds_2":24455},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24431":{"gid":24431,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.135:6835/284701222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6834","nonce":284701222},{"type":"v1","addr":"172.21.15.135:6835","nonce":284701222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/3023450024","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3023450024},{"type":"v1","addr":"172.21.15.193:6835","nonce":3023450024}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24448":{"gid":24448,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/2863018656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2863018656},{"type":"v1","addr":"172.21.15.193:6837","nonce":2863018656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":189,"addr":"172.21.15.135:6837/1549198529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6836","nonce":1549198529},{"type":"v1","addr":"172.21.15.135:6837","nonce":1549198529}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24472":{"gid":24472,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":24,"addr":"172.21.15.193:6839/2677954983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2677954983},{"type":"v1","addr":"172.21.15.193:6839","nonce":2677954983}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24479":{"gid":24479,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.135:6839/1599631648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6838","nonce":1599631648},{"type":"v1","addr":"172.21.15.135:6839","nonce":1599631648}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24479,"qdb_cluster":[24455,24479,24472]},"id":1}]} 2024-06-10T00:39:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.981+0000 7f6b49ffb700 1 -- 172.21.15.121:0/1499052195 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6b400a2e70 msgr2=0x7f6b400a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.981+0000 7f6b49ffb700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6b400a2e70 0x7f6b400a5330 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c008570 tx=0x7f6b4c005fb0 comp rx=0 tx=0).stop 2024-06-10T00:39:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.982+0000 7f6b49ffb700 1 -- 172.21.15.121:0/1499052195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b541a3cd0 msgr2=0x7f6b541a4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.982+0000 7f6b49ffb700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b541a3cd0 0x7f6b541a4150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6b44011180 tx=0x7f6b44011490 comp rx=0 tx=0).stop 2024-06-10T00:39:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.982+0000 7f6b49ffb700 1 -- 172.21.15.121:0/1499052195 shutdown_connections 2024-06-10T00:39:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.982+0000 7f6b49ffb700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f6b400a2e70 0x7f6b400a5330 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.982+0000 7f6b49ffb700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b541a3cd0 0x7f6b541a4150 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.982+0000 7f6b49ffb700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b54109e50 0x7f6b5419f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:34.983 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.982+0000 7f6b49ffb700 1 --2- 172.21.15.121:0/1499052195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b5410ef90 0x7f6b5419f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:34.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.982+0000 7f6b49ffb700 1 -- 172.21.15.121:0/1499052195 >> 172.21.15.121:0/1499052195 conn(0x7f6b5406d470 msgr2=0x7f6b5410ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:34.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.983+0000 7f6b49ffb700 1 -- 172.21.15.121:0/1499052195 shutdown_connections 2024-06-10T00:39:34.984 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:34.983+0000 7f6b49ffb700 1 -- 172.21.15.121:0/1499052195 wait complete. 2024-06-10T00:39:34.984 INFO:teuthology.orchestra.run.smithi121.stderr:dumped fsmap epoch 33 2024-06-10T00:39:35.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:34 smithi193 ceph-mon[36716]: pgmap v360: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:35.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:34 smithi135 ceph-mon[36424]: pgmap v360: 129 pgs: 17 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 59 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:35.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-10T00:39:35.949 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:35 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1499052195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:39:36.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:35 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1499052195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:39:36.134 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:39:36.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:35 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1499052195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-06-10T00:39:37.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:36 smithi193 ceph-mon[36716]: pgmap v361: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:37.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:36 smithi121 ceph-mon[32378]: pgmap v361: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:37.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:36 smithi135 ceph-mon[36424]: pgmap v361: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:38.424 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.423+0000 7fe5b145e700 1 -- 172.21.15.121:0/263744680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac101ed0 msgr2=0x7fe5ac105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:38.425 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.424+0000 7fe5b145e700 1 --2- 172.21.15.121:0/263744680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac101ed0 0x7fe5ac105f20 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe5940099c0 tx=0x7fe594009cd0 comp rx=0 tx=0).stop 2024-06-10T00:39:38.428 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.428+0000 7fe5b145e700 1 -- 172.21.15.121:0/263744680 shutdown_connections 2024-06-10T00:39:38.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.428+0000 7fe5b145e700 1 --2- 172.21.15.121:0/263744680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5ac106600 0x7fe5ac10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:38.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.428+0000 7fe5b145e700 1 --2- 172.21.15.121:0/263744680 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5ac101520 0x7fe5ac101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:38.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.428+0000 7fe5b145e700 1 --2- 172.21.15.121:0/263744680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac101ed0 0x7fe5ac105f20 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:38.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.428+0000 7fe5b145e700 1 -- 172.21.15.121:0/263744680 >> 172.21.15.121:0/263744680 conn(0x7fe5ac0fcdb0 msgr2=0x7fe5ac0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:38.429 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.429+0000 7fe5b145e700 1 -- 172.21.15.121:0/263744680 shutdown_connections 2024-06-10T00:39:38.434 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.434+0000 7fe5b145e700 1 -- 172.21.15.121:0/263744680 wait complete. 2024-06-10T00:39:38.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.435+0000 7fe5b145e700 1 Processor -- start 2024-06-10T00:39:38.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.435+0000 7fe5b145e700 1 -- start start 2024-06-10T00:39:38.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.435+0000 7fe5b145e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac101520 0x7fe5ac197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:38.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.435+0000 7fe5b145e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5ac101ed0 0x7fe5ac197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:38.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.435+0000 7fe5b145e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5ac106600 0x7fe5ac19c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:38.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.435+0000 7fe5b145e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac111730 con 0x7fe5ac101520 2024-06-10T00:39:38.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.435+0000 7fe5b145e700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac1115b0 con 0x7fe5ac106600 2024-06-10T00:39:38.436 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.435+0000 7fe5b145e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac1118b0 con 0x7fe5ac101ed0 2024-06-10T00:39:38.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.436+0000 7fe5abfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac101520 0x7fe5ac197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:38.438 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.436+0000 7fe5abfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac101520 0x7fe5ac197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:55514/0 (socket says 172.21.15.121:55514) 2024-06-10T00:39:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.436+0000 7fe5abfff700 1 -- 172.21.15.121:0/2856478446 learned_addr learned my addr 172.21.15.121:0/2856478446 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:39:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.436+0000 7fe5b0c5d700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5ac106600 0x7fe5ac19c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.436+0000 7fe5abfff700 1 -- 172.21.15.121:0/2856478446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5ac101ed0 msgr2=0x7fe5ac197ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-06-10T00:39:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.436+0000 7fe5abfff700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5ac101ed0 0x7fe5ac197ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.436+0000 7fe5abfff700 1 -- 172.21.15.121:0/2856478446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5ac106600 msgr2=0x7fe5ac19c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.436+0000 7fe5abfff700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5ac106600 0x7fe5ac19c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.436+0000 7fe5abfff700 1 -- 172.21.15.121:0/2856478446 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ac19cef0 con 0x7fe5ac101520 2024-06-10T00:39:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.436+0000 7fe5abfff700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac101520 0x7fe5ac197960 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00b550 tx=0x7fe59c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.438+0000 7fe5a9ffb700 1 -- 172.21.15.121:0/2856478446 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c015020 con 0x7fe5ac101520 2024-06-10T00:39:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.438+0000 7fe5a9ffb700 1 -- 172.21.15.121:0/2856478446 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe59c011d10 con 0x7fe5ac101520 2024-06-10T00:39:38.439 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.438+0000 7fe5a9ffb700 1 -- 172.21.15.121:0/2856478446 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c00f2e0 con 0x7fe5ac101520 2024-06-10T00:39:38.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.438+0000 7fe5b145e700 1 -- 172.21.15.121:0/2856478446 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ac19d1e0 con 0x7fe5ac101520 2024-06-10T00:39:38.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.438+0000 7fe5b145e700 1 -- 172.21.15.121:0/2856478446 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ac19d670 con 0x7fe5ac101520 2024-06-10T00:39:38.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.440+0000 7fe5b145e700 1 -- 172.21.15.121:0/2856478446 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe58c005330 con 0x7fe5ac101520 2024-06-10T00:39:38.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.442+0000 7fe5a9ffb700 1 -- 172.21.15.121:0/2856478446 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fe59c018020 con 0x7fe5ac101520 2024-06-10T00:39:38.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.445+0000 7fe5a9ffb700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe5980b3f70 0x7fe5980b6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:38.446 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.446+0000 7fe5a9ffb700 1 -- 172.21.15.121:0/2856478446 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9613+0+0 (secure 0 0 0) 0x7fe59c0cee90 con 0x7fe5ac101520 2024-06-10T00:39:38.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.446+0000 7fe5a37fe700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe5980b3f70 0x7fe5980b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:38.447 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.447+0000 7fe5a37fe700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe5980b3f70 0x7fe5980b6430 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe59400b580 tx=0x7fe594005fb0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:38.452 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.452+0000 7fe5a9ffb700 1 -- 172.21.15.121:0/2856478446 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe59c01f020 con 0x7fe5ac101520 2024-06-10T00:39:38.829 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.829+0000 7fe5b145e700 1 -- 172.21.15.121:0/2856478446 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe58c0058f0 con 0x7fe5ac101520 2024-06-10T00:39:38.832 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.832+0000 7fe5a9ffb700 1 -- 172.21.15.121:0/2856478446 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+25621 (secure 0 0 0) 0x7fe59c09b720 con 0x7fe5ac101520 2024-06-10T00:39:38.832 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-06-10T00:39:38.832 INFO:teuthology.orchestra.run.smithi121.stdout:{"epoch":110,"fsid":"6c432244-26be-11ef-bc9b-c7b262605968","created":"2024-06-10T00:15:42.548845+0000","modified":"2024-06-10T00:39:07.400248+0000","last_up_change":"2024-06-10T00:22:50.148031+0000","last_in_change":"2024-06-10T00:22:29.966293+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-10T00:19:04.337527+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-10T00:23:53.357210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-10T00:23:56.483170+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-10T00:24:07.390280+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d80f9176-d911-4798-98cd-02587aa53bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6803","nonce":1987481040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6805","nonce":1987481040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6809","nonce":1987481040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":1987481040},{"type":"v1","addr":"172.21.15.121:6807","nonce":1987481040}]},"public_addr":"172.21.15.121:6803/1987481040","cluster_addr":"172.21.15.121:6805/1987481040","heartbeat_back_addr":"172.21.15.121:6809/1987481040","heartbeat_front_addr":"172.21.15.121:6807/1987481040","state":["exists","up"]},{"osd":1,"uuid":"3887c2a6-d2b6-43e9-9632-c73154511128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6801","nonce":1153602323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6803","nonce":1153602323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6807","nonce":1153602323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":1153602323},{"type":"v1","addr":"172.21.15.135:6805","nonce":1153602323}]},"public_addr":"172.21.15.135:6801/1153602323","cluster_addr":"172.21.15.135:6803/1153602323","heartbeat_back_addr":"172.21.15.135:6807/1153602323","heartbeat_front_addr":"172.21.15.135:6805/1153602323","state":["exists","up"]},{"osd":2,"uuid":"d37465a0-81c9-499f-a0f3-fe5cdc1538ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6801","nonce":2430648346}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6803","nonce":2430648346}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6807","nonce":2430648346}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":2430648346},{"type":"v1","addr":"172.21.15.193:6805","nonce":2430648346}]},"public_addr":"172.21.15.193:6801/2430648346","cluster_addr":"172.21.15.193:6803/2430648346","heartbeat_back_addr":"172.21.15.193:6807/2430648346","heartbeat_front_addr":"172.21.15.193:6805/2430648346","state":["exists","up"]},{"osd":3,"uuid":"1e4414c2-8c5d-4b4a-895e-b3d0157ab36a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6811","nonce":1510255067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6813","nonce":1510255067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6817","nonce":1510255067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1510255067},{"type":"v1","addr":"172.21.15.121:6815","nonce":1510255067}]},"public_addr":"172.21.15.121:6811/1510255067","cluster_addr":"172.21.15.121:6813/1510255067","heartbeat_back_addr":"172.21.15.121:6817/1510255067","heartbeat_front_addr":"172.21.15.121:6815/1510255067","state":["exists","up"]},{"osd":4,"uuid":"5a6e3f25-d136-4698-8657-ca62ff8d4ffe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6809","nonce":759537459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6811","nonce":759537459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6815","nonce":759537459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":759537459},{"type":"v1","addr":"172.21.15.135:6813","nonce":759537459}]},"public_addr":"172.21.15.135:6809/759537459","cluster_addr":"172.21.15.135:6811/759537459","heartbeat_back_addr":"172.21.15.135:6815/759537459","heartbeat_front_addr":"172.21.15.135:6813/759537459","state":["exists","up"]},{"osd":5,"uuid":"62361748-4968-4095-9e32-756c6cfdfaf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6809","nonce":3919206206}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6811","nonce":3919206206}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6815","nonce":3919206206}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":3919206206},{"type":"v1","addr":"172.21.15.193:6813","nonce":3919206206}]},"public_addr":"172.21.15.193:6809/3919206206","cluster_addr":"172.21.15.193:6811/3919206206","heartbeat_back_addr":"172.21.15.193:6815/3919206206","heartbeat_front_addr":"172.21.15.193:6813/3919206206","state":["exists","up"]},{"osd":6,"uuid":"3541db4e-839e-4ad2-bd35-2566a649d87f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6819","nonce":2905970681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6821","nonce":2905970681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6825","nonce":2905970681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2905970681},{"type":"v1","addr":"172.21.15.121:6823","nonce":2905970681}]},"public_addr":"172.21.15.121:6819/2905970681","cluster_addr":"172.21.15.121:6821/2905970681","heartbeat_back_addr":"172.21.15.121:6825/2905970681","heartbeat_front_addr":"172.21.15.121:6823/2905970681","state":["exists","up"]},{"osd":7,"uuid":"3979b209-73a0-4891-85dc-b7aad76f0ff0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6817","nonce":3426631559}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6818","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6819","nonce":3426631559}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6822","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6823","nonce":3426631559}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6820","nonce":3426631559},{"type":"v1","addr":"172.21.15.135:6821","nonce":3426631559}]},"public_addr":"172.21.15.135:6817/3426631559","cluster_addr":"172.21.15.135:6819/3426631559","heartbeat_back_addr":"172.21.15.135:6823/3426631559","heartbeat_front_addr":"172.21.15.135:6821/3426631559","state":["exists","up"]},{"osd":8,"uuid":"6a099d63-2664-4c39-a992-2095f55cdfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6817","nonce":3981402186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6819","nonce":3981402186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6823","nonce":3981402186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3981402186},{"type":"v1","addr":"172.21.15.193:6821","nonce":3981402186}]},"public_addr":"172.21.15.193:6817/3981402186","cluster_addr":"172.21.15.193:6819/3981402186","heartbeat_back_addr":"172.21.15.193:6823/3981402186","heartbeat_front_addr":"172.21.15.193:6821/3981402186","state":["exists","up"]},{"osd":9,"uuid":"109bd90d-d878-46b2-99c3-af0792345523","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6827","nonce":3736625801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6829","nonce":3736625801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6833","nonce":3736625801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":3736625801},{"type":"v1","addr":"172.21.15.121:6831","nonce":3736625801}]},"public_addr":"172.21.15.121:6827/3736625801","cluster_addr":"172.21.15.121:6829/3736625801","heartbeat_back_addr":"172.21.15.121:6833/3736625801","heartbeat_front_addr":"172.21.15.121:6831/3736625801","state":["exists","up"]},{"osd":10,"uuid":"66e56ceb-7b01-4bec-9d2d-bff37873aa75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6824","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6825","nonce":937452287}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6826","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6827","nonce":937452287}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6830","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6831","nonce":937452287}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6828","nonce":937452287},{"type":"v1","addr":"172.21.15.135:6829","nonce":937452287}]},"public_addr":"172.21.15.135:6825/937452287","cluster_addr":"172.21.15.135:6827/937452287","heartbeat_back_addr":"172.21.15.135:6831/937452287","heartbeat_front_addr":"172.21.15.135:6829/937452287","state":["exists","up"]},{"osd":11,"uuid":"3b8b392a-5b98-4b28-8301-8ceecd572573","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6825","nonce":3765048609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6827","nonce":3765048609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6831","nonce":3765048609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3765048609},{"type":"v1","addr":"172.21.15.193:6829","nonce":3765048609}]},"public_addr":"172.21.15.193:6825/3765048609","cluster_addr":"172.21.15.193:6827/3765048609","heartbeat_back_addr":"172.21.15.193:6831/3765048609","heartbeat_front_addr":"172.21.15.193:6829/3765048609","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:13.972705+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:18:37.557160+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:01.533298+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:25.632131+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:19:50.041505+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:14.736724+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:20:40.105443+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:05.540074+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:30.787144+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:21:56.564613+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:22.031932+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-10T00:22:48.205196+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:0/4266501776":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/4078460254":"2024-06-11T00:27:57.451553+0000","172.21.15.121:6801/3506007074":"2024-06-11T00:27:57.451553+0000","172.21.15.121:6800/3506007074":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/58990727":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/1710032094":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/1002791322":"2024-06-11T00:27:57.451553+0000","172.21.15.121:0/986214985":"2024-06-11T00:16:42.011037+0000","172.21.15.121:6801/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/3588432900":"2024-06-11T00:16:15.442852+0000","172.21.15.121:6800/704666019":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/743919928":"2024-06-11T00:16:42.011037+0000","172.21.15.121:0/2834833132":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1390094598":"2024-06-11T00:16:15.442852+0000","172.21.15.121:0/1651500994":"2024-06-11T00:16:15.442852+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-10T00:39:38.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.840+0000 7fe5b145e700 1 -- 172.21.15.121:0/2856478446 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe5980b3f70 msgr2=0x7fe5980b6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:38.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.840+0000 7fe5b145e700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe5980b3f70 0x7fe5980b6430 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe59400b580 tx=0x7fe594005fb0 comp rx=0 tx=0).stop 2024-06-10T00:39:38.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.840+0000 7fe5b145e700 1 -- 172.21.15.121:0/2856478446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac101520 msgr2=0x7fe5ac197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:38.840 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.840+0000 7fe5b145e700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac101520 0x7fe5ac197960 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00b550 tx=0x7fe59c00b910 comp rx=0 tx=0).stop 2024-06-10T00:39:38.844 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.844+0000 7fe5b145e700 1 -- 172.21.15.121:0/2856478446 shutdown_connections 2024-06-10T00:39:38.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.844+0000 7fe5b145e700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fe5980b3f70 0x7fe5980b6430 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:38.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.844+0000 7fe5b145e700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe5ac106600 0x7fe5ac19c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:38.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.844+0000 7fe5b145e700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac101520 0x7fe5ac197960 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:38.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.844+0000 7fe5b145e700 1 --2- 172.21.15.121:0/2856478446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe5ac101ed0 0x7fe5ac197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:38.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.844+0000 7fe5b145e700 1 -- 172.21.15.121:0/2856478446 >> 172.21.15.121:0/2856478446 conn(0x7fe5ac0fcdb0 msgr2=0x7fe5ac0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:38.845 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.844+0000 7fe5b145e700 1 -- 172.21.15.121:0/2856478446 shutdown_connections 2024-06-10T00:39:38.846 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:38.844+0000 7fe5b145e700 1 -- 172.21.15.121:0/2856478446 wait complete. 2024-06-10T00:39:39.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:38 smithi193 ceph-mon[36716]: pgmap v362: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 6 op/s 2024-06-10T00:39:39.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:38 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2856478446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:39:39.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:38 smithi121 ceph-mon[32378]: pgmap v362: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 6 op/s 2024-06-10T00:39:39.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:38 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2856478446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:39:39.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:38 smithi135 ceph-mon[36424]: pgmap v362: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 6 op/s 2024-06-10T00:39:39.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:38 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2856478446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-10T00:39:40.211 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs fail cephfs 2024-06-10T00:39:40.656 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:39:41.121 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:40 smithi121 ceph-mon[32378]: pgmap v363: 129 pgs: 9 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 102 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:41.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:40 smithi193 ceph-mon[36716]: pgmap v363: 129 pgs: 9 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 102 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:41.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:40 smithi135 ceph-mon[36424]: pgmap v363: 129 pgs: 9 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 102 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 767 B/s wr, 7 op/s 2024-06-10T00:39:42.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.708+0000 7f213000e700 1 -- 172.21.15.121:0/2433553270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2128110c70 msgr2=0x7f2128109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:42.709 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.708+0000 7f213000e700 1 --2- 172.21.15.121:0/2433553270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2128110c70 0x7f2128109490 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f21240098f0 tx=0x7f2124009c00 comp rx=0 tx=0).stop 2024-06-10T00:39:42.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.710+0000 7f213000e700 1 -- 172.21.15.121:0/2433553270 shutdown_connections 2024-06-10T00:39:42.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.710+0000 7f213000e700 1 --2- 172.21.15.121:0/2433553270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f212811bc70 0x7f212811e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:42.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.710+0000 7f213000e700 1 --2- 172.21.15.121:0/2433553270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2128110c70 0x7f2128109490 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:42.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.710+0000 7f213000e700 1 --2- 172.21.15.121:0/2433553270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21281099d0 0x7f2128109e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:42.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.710+0000 7f213000e700 1 -- 172.21.15.121:0/2433553270 >> 172.21.15.121:0/2433553270 conn(0x7f212806d600 msgr2=0x7f212806da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:42.715 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.711+0000 7f213000e700 1 -- 172.21.15.121:0/2433553270 shutdown_connections 2024-06-10T00:39:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.711+0000 7f213000e700 1 -- 172.21.15.121:0/2433553270 wait complete. 2024-06-10T00:39:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.712+0000 7f213000e700 1 Processor -- start 2024-06-10T00:39:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.712+0000 7f213000e700 1 -- start start 2024-06-10T00:39:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.713+0000 7f213000e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21281099d0 0x7f212811b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.713+0000 7f213000e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2128110c70 0x7f212811b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.713+0000 7f213000e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f212811bc70 0x7f2128114420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.713+0000 7f213000e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f21281216d0 con 0x7f212811bc70 2024-06-10T00:39:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.713+0000 7f213000e700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f2128121550 con 0x7f21281099d0 2024-06-10T00:39:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.713+0000 7f213000e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2128121850 con 0x7f2128110c70 2024-06-10T00:39:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.713+0000 7f212ddaa700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21281099d0 0x7f212811b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:42.716 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.713+0000 7f212ddaa700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21281099d0 0x7f212811b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.121:40832/0 (socket says 172.21.15.121:40832) 2024-06-10T00:39:42.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.713+0000 7f212ddaa700 1 -- 172.21.15.121:0/2889370838 learned_addr learned my addr 172.21.15.121:0/2889370838 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:39:42.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.714+0000 7f212ddaa700 1 -- 172.21.15.121:0/2889370838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2128110c70 msgr2=0x7f212811b6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-10T00:39:42.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.714+0000 7f212ddaa700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2128110c70 0x7f212811b6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:42.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.714+0000 7f212ddaa700 1 -- 172.21.15.121:0/2889370838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f212811bc70 msgr2=0x7f2128114420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:42.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.714+0000 7f212ddaa700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f212811bc70 0x7f2128114420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:42.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.714+0000 7f212ddaa700 1 -- 172.21.15.121:0/2889370838 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2128114d20 con 0x7f21281099d0 2024-06-10T00:39:42.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.714+0000 7f212ddaa700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21281099d0 0x7f212811b170 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2124005d60 tx=0x7f2124004b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:42.718 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.718+0000 7f211effd700 1 -- 172.21.15.121:0/2889370838 <== mon.2 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2124004f00 con 0x7f21281099d0 2024-06-10T00:39:42.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.718+0000 7f211effd700 1 -- 172.21.15.121:0/2889370838 <== mon.2 v2:172.21.15.135:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f212400fd00 con 0x7f21281099d0 2024-06-10T00:39:42.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.718+0000 7f211effd700 1 -- 172.21.15.121:0/2889370838 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2124003dd0 con 0x7f21281099d0 2024-06-10T00:39:42.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.718+0000 7f213000e700 1 -- 172.21.15.121:0/2889370838 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2128114fb0 con 0x7f21281099d0 2024-06-10T00:39:42.719 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.718+0000 7f213000e700 1 -- 172.21.15.121:0/2889370838 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2128118e70 con 0x7f21281099d0 2024-06-10T00:39:42.720 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.720+0000 7f213000e700 1 -- 172.21.15.121:0/2889370838 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f212806eb90 con 0x7f21281099d0 2024-06-10T00:39:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.722+0000 7f211effd700 1 -- 172.21.15.121:0/2889370838 <== mon.2 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f212400f820 con 0x7f21281099d0 2024-06-10T00:39:42.723 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.722+0000 7f211effd700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f21140a2f60 0x7f21140a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.723+0000 7f211effd700 1 -- 172.21.15.121:0/2889370838 <== mon.2 v2:172.21.15.135:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9613+0+0 (secure 0 0 0) 0x7f21240d07a0 con 0x7f21281099d0 2024-06-10T00:39:42.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.724+0000 7f212d5a9700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f21140a2f60 0x7f21140a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:42.726 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.725+0000 7f212d5a9700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f21140a2f60 0x7f21140a5420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2118005e50 tx=0x7f2118005de0 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:42.729 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:42.729+0000 7f211effd700 1 -- 172.21.15.121:0/2889370838 <== mon.2 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2124013030 con 0x7f21281099d0 2024-06-10T00:39:43.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:42 smithi193 ceph-mon[36716]: pgmap v364: 129 pgs: 9 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 102 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-06-10T00:39:43.184 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.183+0000 7f213000e700 1 -- 172.21.15.121:0/2889370838 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f2128119190 con 0x7f21281099d0 2024-06-10T00:39:43.184 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:42 smithi121 ceph-mon[32378]: pgmap v364: 129 pgs: 9 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 102 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-06-10T00:39:43.195 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.194+0000 7f211effd700 1 -- 172.21.15.121:0/2889370838 <== mon.2 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) ==== 161+0+0 (secure 0 0 0) 0x7f212409cfb0 con 0x7f21281099d0 2024-06-10T00:39:43.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.198+0000 7f213000e700 1 -- 172.21.15.121:0/2889370838 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f21140a2f60 msgr2=0x7f21140a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:43.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.198+0000 7f213000e700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f21140a2f60 0x7f21140a5420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2118005e50 tx=0x7f2118005de0 comp rx=0 tx=0).stop 2024-06-10T00:39:43.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.198+0000 7f213000e700 1 -- 172.21.15.121:0/2889370838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21281099d0 msgr2=0x7f212811b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:43.199 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.198+0000 7f213000e700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21281099d0 0x7f212811b170 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2124005d60 tx=0x7f2124004b70 comp rx=0 tx=0).stop 2024-06-10T00:39:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.201+0000 7f213000e700 1 -- 172.21.15.121:0/2889370838 shutdown_connections 2024-06-10T00:39:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.201+0000 7f213000e700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f21140a2f60 0x7f21140a5420 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2118005e50 tx=0x7f2118005de0 comp rx=0 tx=0).stop 2024-06-10T00:39:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.201+0000 7f213000e700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f212811bc70 0x7f2128114420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.201+0000 7f213000e700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21281099d0 0x7f212811b170 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2124005d60 tx=0x7f2124004b70 comp rx=0 tx=0).stop 2024-06-10T00:39:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.201+0000 7f213000e700 1 --2- 172.21.15.121:0/2889370838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2128110c70 0x7f212811b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:43.201 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.201+0000 7f213000e700 1 -- 172.21.15.121:0/2889370838 >> 172.21.15.121:0/2889370838 conn(0x7f212806d600 msgr2=0x7f212811d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.202+0000 7f213000e700 1 -- 172.21.15.121:0/2889370838 shutdown_connections 2024-06-10T00:39:43.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:43.202+0000 7f213000e700 1 -- 172.21.15.121:0/2889370838 wait complete. 2024-06-10T00:39:43.206 INFO:teuthology.orchestra.run.smithi121.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-06-10T00:39:43.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:42 smithi135 ceph-mon[36424]: pgmap v364: 129 pgs: 9 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 102 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-06-10T00:39:43.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:43 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a[32374]: 2024-06-10T00:39:43.185+0000 7f460a042700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-10T00:39:43.631 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f[66607]: ignoring --setuser ceph since I am not root 2024-06-10T00:39:43.631 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f[66607]: ignoring --setgroup ceph since I am not root 2024-06-10T00:39:43.631 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f[66607]: starting mds.f at 2024-06-10T00:39:43.632 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i[67214]: ignoring --setuser ceph since I am not root 2024-06-10T00:39:43.632 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i[67214]: ignoring --setgroup ceph since I am not root 2024-06-10T00:39:43.632 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-mds[67218]: -- 172.21.15.193:0/3328332366 <== mon.2 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af6225fa40 con 0x55af62233000 2024-06-10T00:39:43.632 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i[67214]: starting mds.i at 2024-06-10T00:39:43.632 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c[66910]: ignoring --setuser ceph since I am not root 2024-06-10T00:39:43.632 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c[66910]: ignoring --setgroup ceph since I am not root 2024-06-10T00:39:43.632 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c[66910]: starting mds.c at 2024-06-10T00:39:43.758 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e[65561]: ignoring --setuser ceph since I am not root 2024-06-10T00:39:43.758 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e[65561]: ignoring --setgroup ceph since I am not root 2024-06-10T00:39:43.758 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e[65561]: starting mds.e at 2024-06-10T00:39:43.759 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: ignoring --setuser ceph since I am not root 2024-06-10T00:39:43.759 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: ignoring --setgroup ceph since I am not root 2024-06-10T00:39:43.759 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-mds[65882]: -- 172.21.15.135:0/2167174689 <== mon.2 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a67fe05a40 con 0x55a67fdd8c00 2024-06-10T00:39:43.759 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: starting mds.h at 2024-06-10T00:39:43.759 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b[66180]: ignoring --setuser ceph since I am not root 2024-06-10T00:39:43.759 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b[66180]: ignoring --setgroup ceph since I am not root 2024-06-10T00:39:43.759 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-mds[66184]: -- 172.21.15.135:0/2537471130 <== mon.2 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55cf441f44e0 con 0x55cf441ca800 2024-06-10T00:39:43.759 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-mds[66184]: -- 172.21.15.135:0/2537471130 <== mon.2 v2:172.21.15.135:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cf441f7a40 con 0x55cf441ca800 2024-06-10T00:39:43.759 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b[66180]: starting mds.b at 2024-06-10T00:39:44.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2889370838' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-10T00:39:44.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-mon[36716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-10T00:39:44.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-mon[36716]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-06-10T00:39:44.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-mon[36716]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-10T00:39:44.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-mon[36716]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-06-10T00:39:44.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-mon[36716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-06-10T00:39:44.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-mon[36716]: osdmap e111: 12 total, 12 up, 12 in 2024-06-10T00:39:44.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:43 smithi193 ceph-mon[36716]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-06-10T00:39:44.199 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:43 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2889370838' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-10T00:39:44.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:43 smithi121 ceph-mon[32378]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-10T00:39:44.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:43 smithi121 ceph-mon[32378]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-06-10T00:39:44.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:43 smithi121 ceph-mon[32378]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-10T00:39:44.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:43 smithi121 ceph-mon[32378]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-06-10T00:39:44.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:43 smithi121 ceph-mon[32378]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-06-10T00:39:44.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:43 smithi121 ceph-mon[32378]: osdmap e111: 12 total, 12 up, 12 in 2024-06-10T00:39:44.200 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:43 smithi121 ceph-mon[32378]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-06-10T00:39:44.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2889370838' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-10T00:39:44.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-mon[36424]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-06-10T00:39:44.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-mon[36424]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-06-10T00:39:44.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-mon[36424]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-06-10T00:39:44.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-mon[36424]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-06-10T00:39:44.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-mon[36424]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-06-10T00:39:44.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-mon[36424]: osdmap e111: 12 total, 12 up, 12 in 2024-06-10T00:39:44.259 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:43 smithi135 ceph-mon[36424]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-06-10T00:39:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: pgmap v366: 129 pgs: 9 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 102 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-06-10T00:39:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.193:6834/2055776371,v1:172.21.15.193:6835/2055776371] up:boot 2024-06-10T00:39:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.135:6834/3187926557,v1:172.21.15.135:6835/3187926557] up:boot 2024-06-10T00:39:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.193:6838/3658785861,v1:172.21.15.193:6839/3658785861] up:boot 2024-06-10T00:39:45.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.135:6836/3077057489,v1:172.21.15.135:6837/3077057489] up:boot 2024-06-10T00:39:45.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.193:6836/1096193501,v1:172.21.15.193:6837/1096193501] up:boot 2024-06-10T00:39:45.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: mds.? [v2:172.21.15.135:6838/3390259734,v1:172.21.15.135:6839/3390259734] up:boot 2024-06-10T00:39:45.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-06-10T00:39:45.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-10T00:39:45.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-10T00:39:45.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-10T00:39:45.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-10T00:39:45.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-10T00:39:45.509 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:45 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-10T00:39:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: pgmap v366: 129 pgs: 9 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 102 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-06-10T00:39:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.193:6834/2055776371,v1:172.21.15.193:6835/2055776371] up:boot 2024-06-10T00:39:45.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.135:6834/3187926557,v1:172.21.15.135:6835/3187926557] up:boot 2024-06-10T00:39:45.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.193:6838/3658785861,v1:172.21.15.193:6839/3658785861] up:boot 2024-06-10T00:39:45.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.135:6836/3077057489,v1:172.21.15.135:6837/3077057489] up:boot 2024-06-10T00:39:45.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.193:6836/1096193501,v1:172.21.15.193:6837/1096193501] up:boot 2024-06-10T00:39:45.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: mds.? [v2:172.21.15.135:6838/3390259734,v1:172.21.15.135:6839/3390259734] up:boot 2024-06-10T00:39:45.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-06-10T00:39:45.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-10T00:39:45.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-10T00:39:45.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-10T00:39:45.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-10T00:39:45.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-10T00:39:45.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:45 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-10T00:39:45.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: pgmap v366: 129 pgs: 9 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 102 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-06-10T00:39:45.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.193:6834/2055776371,v1:172.21.15.193:6835/2055776371] up:boot 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.135:6834/3187926557,v1:172.21.15.135:6835/3187926557] up:boot 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.193:6838/3658785861,v1:172.21.15.193:6839/3658785861] up:boot 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.135:6836/3077057489,v1:172.21.15.135:6837/3077057489] up:boot 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.193:6836/1096193501,v1:172.21.15.193:6837/1096193501] up:boot 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: mds.? [v2:172.21.15.135:6838/3390259734,v1:172.21.15.135:6839/3390259734] up:boot 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-06-10T00:39:45.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:45 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-06-10T00:39:46.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-06-10T00:39:46.901 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:39:47.277 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:47 smithi121 ceph-mon[32378]: pgmap v367: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-06-10T00:39:47.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:47 smithi193 ceph-mon[36716]: pgmap v367: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-06-10T00:39:47.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:47 smithi135 ceph-mon[36424]: pgmap v367: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-06-10T00:39:48.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.691+0000 7f02edef3700 1 -- 172.21.15.121:0/1368631652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e8101390 msgr2=0x7f02e8101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:48.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.691+0000 7f02edef3700 1 --2- 172.21.15.121:0/1368631652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e8101390 0x7f02e8101810 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009990 tx=0x7f02dc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:39:48.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.691+0000 7f02edef3700 1 -- 172.21.15.121:0/1368631652 shutdown_connections 2024-06-10T00:39:48.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.691+0000 7f02edef3700 1 --2- 172.21.15.121:0/1368631652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02e8101d50 0x7f02e810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:48.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.691+0000 7f02edef3700 1 --2- 172.21.15.121:0/1368631652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02e8107400 0x7f02e81077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:48.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.691+0000 7f02edef3700 1 --2- 172.21.15.121:0/1368631652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e8101390 0x7f02e8101810 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:48.694 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.691+0000 7f02edef3700 1 -- 172.21.15.121:0/1368631652 >> 172.21.15.121:0/1368631652 conn(0x7f02e8078340 msgr2=0x7f02e8078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.691+0000 7f02edef3700 1 -- 172.21.15.121:0/1368631652 shutdown_connections 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.691+0000 7f02edef3700 1 -- 172.21.15.121:0/1368631652 wait complete. 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02edef3700 1 Processor -- start 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02edef3700 1 -- start start 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02edef3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02e8101390 0x7f02e8103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02edef3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e8101d50 0x7f02e81035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02edef3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02e8107400 0x7f02e8103af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02edef3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f02e8111670 con 0x7f02e8101d50 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02edef3700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81114f0 con 0x7f02e8101390 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02edef3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f02e81117f0 con 0x7f02e8107400 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02e6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e8101d50 0x7f02e81035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:48.695 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02e7fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02e8107400 0x7f02e8103af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02e6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e8101d50 0x7f02e81035b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48586/0 (socket says 172.21.15.121:48586) 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02e6ffd700 1 -- 172.21.15.121:0/1187368200 learned_addr learned my addr 172.21.15.121:0/1187368200 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02e7fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02e8107400 0x7f02e8103af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.121:43620/0 (socket says 172.21.15.121:43620) 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.692+0000 7f02e77fe700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02e8101390 0x7f02e8103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.693+0000 7f02e6ffd700 1 -- 172.21.15.121:0/1187368200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02e8107400 msgr2=0x7f02e8103af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.693+0000 7f02e6ffd700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02e8107400 0x7f02e8103af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.693+0000 7f02e6ffd700 1 -- 172.21.15.121:0/1187368200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02e8101390 msgr2=0x7f02e8103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.693+0000 7f02e6ffd700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02e8101390 0x7f02e8103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.693+0000 7f02e6ffd700 1 -- 172.21.15.121:0/1187368200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e81a7120 con 0x7f02e8101d50 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.693+0000 7f02e6ffd700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e8101d50 0x7f02e81035b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f02dc000c00 tx=0x7f02dc00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.693+0000 7f02e4ff9700 1 -- 172.21.15.121:0/1187368200 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc027070 con 0x7f02e8101d50 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.693+0000 7f02e4ff9700 1 -- 172.21.15.121:0/1187368200 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02dc003d70 con 0x7f02e8101d50 2024-06-10T00:39:48.696 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.693+0000 7f02edef3700 1 -- 172.21.15.121:0/1187368200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e81a73b0 con 0x7f02e8101d50 2024-06-10T00:39:48.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.693+0000 7f02e4ff9700 1 -- 172.21.15.121:0/1187368200 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc00f460 con 0x7f02e8101d50 2024-06-10T00:39:48.697 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.693+0000 7f02edef3700 1 -- 172.21.15.121:0/1187368200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e81a7860 con 0x7f02e8101d50 2024-06-10T00:39:48.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.695+0000 7f02edef3700 1 -- 172.21.15.121:0/1187368200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02e8069180 con 0x7f02e8101d50 2024-06-10T00:39:48.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.696+0000 7f02e4ff9700 1 -- 172.21.15.121:0/1187368200 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f02dc02a070 con 0x7f02e8101d50 2024-06-10T00:39:48.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.697+0000 7f02e4ff9700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f02d00a2e10 0x7f02d00a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:48.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.697+0000 7f02e77fe700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f02d00a2e10 0x7f02d00a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:48.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.697+0000 7f02e4ff9700 1 -- 172.21.15.121:0/1187368200 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9783+0+0 (secure 0 0 0) 0x7f02dc0d05c0 con 0x7f02e8101d50 2024-06-10T00:39:48.698 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.698+0000 7f02e77fe700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f02d00a2e10 0x7f02d00a52d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009be0 tx=0x7f02d8009260 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:48.703 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.703+0000 7f02e4ff9700 1 -- 172.21.15.121:0/1187368200 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f02dc013070 con 0x7f02e8101d50 2024-06-10T00:39:48.982 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:48.982+0000 7f02edef3700 1 -- 172.21.15.121:0/1187368200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f02e81a7b80 con 0x7f02e8101d50 2024-06-10T00:39:49.261 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.260+0000 7f02e4ff9700 1 -- 172.21.15.121:0/1187368200 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) ==== 104+0+0 (secure 0 0 0) 0x7f02dc09ce20 con 0x7f02e8101d50 2024-06-10T00:39:49.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 -- 172.21.15.121:0/1187368200 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f02d00a2e10 msgr2=0x7f02d00a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:49.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f02d00a2e10 0x7f02d00a52d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009be0 tx=0x7f02d8009260 comp rx=0 tx=0).stop 2024-06-10T00:39:49.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 -- 172.21.15.121:0/1187368200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e8101d50 msgr2=0x7f02e81035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:49.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e8101d50 0x7f02e81035b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f02dc000c00 tx=0x7f02dc00bab0 comp rx=0 tx=0).stop 2024-06-10T00:39:49.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 -- 172.21.15.121:0/1187368200 shutdown_connections 2024-06-10T00:39:49.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f02d00a2e10 0x7f02d00a52d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:49.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02e8107400 0x7f02e8103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:49.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f02e8101390 0x7f02e8103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:49.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 --2- 172.21.15.121:0/1187368200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e8101d50 0x7f02e81035b0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:49.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 -- 172.21.15.121:0/1187368200 >> 172.21.15.121:0/1187368200 conn(0x7f02e8078340 msgr2=0x7f02e81005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:49.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 -- 172.21.15.121:0/1187368200 shutdown_connections 2024-06-10T00:39:49.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:49.264+0000 7f02edef3700 1 -- 172.21.15.121:0/1187368200 wait complete. 2024-06-10T00:39:49.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:49 smithi193 ceph-mon[36716]: pgmap v368: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-06-10T00:39:49.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:49 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1187368200' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-06-10T00:39:49.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:49 smithi121 ceph-mon[32378]: pgmap v368: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-06-10T00:39:49.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:49 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1187368200' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-06-10T00:39:49.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:49 smithi135 ceph-mon[36424]: pgmap v368: 129 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 118 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-06-10T00:39:49.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:49 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1187368200' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-06-10T00:39:50.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:50 smithi193 ceph-mon[36716]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-06-10T00:39:50.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:50 smithi193 ceph-mon[36716]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-10T00:39:50.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:50 smithi193 ceph-mon[36716]: Cluster is now healthy 2024-06-10T00:39:50.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:50 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1187368200' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-06-10T00:39:50.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:50 smithi193 ceph-mon[36716]: fsmap 12 up:standby 2024-06-10T00:39:50.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:50 smithi121 ceph-mon[32378]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-06-10T00:39:50.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:50 smithi121 ceph-mon[32378]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-10T00:39:50.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:50 smithi121 ceph-mon[32378]: Cluster is now healthy 2024-06-10T00:39:50.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:50 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1187368200' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-06-10T00:39:50.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:50 smithi121 ceph-mon[32378]: fsmap 12 up:standby 2024-06-10T00:39:50.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:50 smithi135 ceph-mon[36424]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-06-10T00:39:50.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:50 smithi135 ceph-mon[36424]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-06-10T00:39:50.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:50 smithi135 ceph-mon[36424]: Cluster is now healthy 2024-06-10T00:39:50.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:50 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1187368200' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-06-10T00:39:50.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:50 smithi135 ceph-mon[36424]: fsmap 12 up:standby 2024-06-10T00:39:51.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-06-10T00:39:51.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:51 smithi193 ceph-mon[36716]: pgmap v369: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-06-10T00:39:51.664 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:39:51.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:51 smithi121 ceph-mon[32378]: pgmap v369: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-06-10T00:39:51.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:51 smithi135 ceph-mon[36424]: pgmap v369: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3 op/s 2024-06-10T00:39:53.404 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:53 smithi121 ceph-mon[32378]: pgmap v370: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-06-10T00:39:53.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.510+0000 7fd6d1812700 1 -- 172.21.15.121:0/2770933644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6cc107500 msgr2=0x7fd6cc1078e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:53.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.510+0000 7fd6d1812700 1 --2- 172.21.15.121:0/2770933644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6cc107500 0x7fd6cc1078e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009990 tx=0x7fd6bc009ca0 comp rx=0 tx=0).stop 2024-06-10T00:39:53.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.510+0000 7fd6d1812700 1 -- 172.21.15.121:0/2770933644 shutdown_connections 2024-06-10T00:39:53.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.510+0000 7fd6d1812700 1 --2- 172.21.15.121:0/2770933644 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6cc102070 0x7fd6cc10e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:53.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.510+0000 7fd6d1812700 1 --2- 172.21.15.121:0/2770933644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6cc107500 0x7fd6cc1078e0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:53.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.510+0000 7fd6d1812700 1 --2- 172.21.15.121:0/2770933644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6cc1016b0 0x7fd6cc101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:53.513 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.510+0000 7fd6d1812700 1 -- 172.21.15.121:0/2770933644 >> 172.21.15.121:0/2770933644 conn(0x7fd6cc0780f0 msgr2=0x7fd6cc078500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:53.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.510+0000 7fd6d1812700 1 -- 172.21.15.121:0/2770933644 shutdown_connections 2024-06-10T00:39:53.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.510+0000 7fd6d1812700 1 -- 172.21.15.121:0/2770933644 wait complete. 2024-06-10T00:39:53.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6d1812700 1 Processor -- start 2024-06-10T00:39:53.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6d1812700 1 -- start start 2024-06-10T00:39:53.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6d1812700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6cc1016b0 0x7fd6cc199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:53.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6d1812700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6cc102070 0x7fd6cc19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:53.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6d1812700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6cc107500 0x7fd6cc1a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:53.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6d1812700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc111720 con 0x7fd6cc1016b0 2024-06-10T00:39:53.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6d1812700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1115a0 con 0x7fd6cc107500 2024-06-10T00:39:53.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6d1812700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc1118a0 con 0x7fd6cc102070 2024-06-10T00:39:53.514 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6caffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6cc1016b0 0x7fd6cc199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6caffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6cc1016b0 0x7fd6cc199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50202/0 (socket says 172.21.15.121:50202) 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6caffd700 1 -- 172.21.15.121:0/3363904821 learned_addr learned my addr 172.21.15.121:0/3363904821 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6cb7fe700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6cc107500 0x7fd6cc1a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6c3fff700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6cc102070 0x7fd6cc19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6caffd700 1 -- 172.21.15.121:0/3363904821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6cc102070 msgr2=0x7fd6cc19a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6caffd700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6cc102070 0x7fd6cc19a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6caffd700 1 -- 172.21.15.121:0/3363904821 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6cc107500 msgr2=0x7fd6cc1a4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6caffd700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6cc107500 0x7fd6cc1a4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.511+0000 7fd6caffd700 1 -- 172.21.15.121:0/3363904821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc1a5100 con 0x7fd6cc1016b0 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.512+0000 7fd6cb7fe700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6cc107500 0x7fd6cc1a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.512+0000 7fd6c3fff700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6cc102070 0x7fd6cc19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-10T00:39:53.515 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.512+0000 7fd6caffd700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6cc1016b0 0x7fd6cc199b40 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc000c00 tx=0x7fd6bc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:53.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.512+0000 7fd6d0810700 1 -- 172.21.15.121:0/3363904821 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc027070 con 0x7fd6cc1016b0 2024-06-10T00:39:53.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.512+0000 7fd6d0810700 1 -- 172.21.15.121:0/3363904821 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6bc004050 con 0x7fd6cc1016b0 2024-06-10T00:39:53.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.512+0000 7fd6d1812700 1 -- 172.21.15.121:0/3363904821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6cc1a52d0 con 0x7fd6cc1016b0 2024-06-10T00:39:53.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.512+0000 7fd6d0810700 1 -- 172.21.15.121:0/3363904821 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc00f460 con 0x7fd6cc1016b0 2024-06-10T00:39:53.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.512+0000 7fd6d1812700 1 -- 172.21.15.121:0/3363904821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6cc1a56c0 con 0x7fd6cc1016b0 2024-06-10T00:39:53.516 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.513+0000 7fd6d1812700 1 -- 172.21.15.121:0/3363904821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6cc102b20 con 0x7fd6cc1016b0 2024-06-10T00:39:53.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.514+0000 7fd6d0810700 1 -- 172.21.15.121:0/3363904821 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7fd6bc02a070 con 0x7fd6cc1016b0 2024-06-10T00:39:53.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.515+0000 7fd6d0810700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6b80a2d90 0x7fd6b80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:53.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.515+0000 7fd6c3fff700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6b80a2d90 0x7fd6b80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:53.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.515+0000 7fd6d0810700 1 -- 172.21.15.121:0/3363904821 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9783+0+0 (secure 0 0 0) 0x7fd6bc0d08b0 con 0x7fd6cc1016b0 2024-06-10T00:39:53.517 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.515+0000 7fd6c3fff700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6b80a2d90 0x7fd6b80a5250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc078750 tx=0x7fd6b4008040 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:53.519 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.519+0000 7fd6d0810700 1 -- 172.21.15.121:0/3363904821 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd6bc05e850 con 0x7fd6cc1016b0 2024-06-10T00:39:53.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:53 smithi193 ceph-mon[36716]: pgmap v370: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-06-10T00:39:53.717 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:53.717+0000 7fd6d1812700 1 -- 172.21.15.121:0/3363904821 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd6cc02dca0 con 0x7fd6cc1016b0 2024-06-10T00:39:53.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:53 smithi135 ceph-mon[36424]: pgmap v370: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-06-10T00:39:54.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.282+0000 7fd6d0810700 1 -- 172.21.15.121:0/3363904821 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v112) ==== 181+0+0 (secure 0 0 0) 0x7fd6bc013070 con 0x7fd6cc1016b0 2024-06-10T00:39:54.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.285+0000 7fd6d1812700 1 -- 172.21.15.121:0/3363904821 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6b80a2d90 msgr2=0x7fd6b80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:54.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.285+0000 7fd6d1812700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6b80a2d90 0x7fd6b80a5250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc078750 tx=0x7fd6b4008040 comp rx=0 tx=0).stop 2024-06-10T00:39:54.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.285+0000 7fd6d1812700 1 -- 172.21.15.121:0/3363904821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6cc1016b0 msgr2=0x7fd6cc199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:54.286 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.285+0000 7fd6d1812700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6cc1016b0 0x7fd6cc199b40 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc000c00 tx=0x7fd6bc0047b0 comp rx=0 tx=0).stop 2024-06-10T00:39:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.286+0000 7fd6d1812700 1 -- 172.21.15.121:0/3363904821 shutdown_connections 2024-06-10T00:39:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.286+0000 7fd6d1812700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7fd6b80a2d90 0x7fd6b80a5250 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.286+0000 7fd6d1812700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6cc107500 0x7fd6cc1a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.286+0000 7fd6d1812700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6cc1016b0 0x7fd6cc199b40 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.286+0000 7fd6d1812700 1 --2- 172.21.15.121:0/3363904821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6cc102070 0x7fd6cc19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.286+0000 7fd6d1812700 1 -- 172.21.15.121:0/3363904821 >> 172.21.15.121:0/3363904821 conn(0x7fd6cc0780f0 msgr2=0x7fd6cc100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:54.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.286+0000 7fd6d1812700 1 -- 172.21.15.121:0/3363904821 shutdown_connections 2024-06-10T00:39:54.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:54.286+0000 7fd6d1812700 1 -- 172.21.15.121:0/3363904821 wait complete. 2024-06-10T00:39:54.288 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'cephfs_metadata' removed 2024-06-10T00:39:54.615 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:54 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3363904821' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-10T00:39:54.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:54 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3363904821' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-10T00:39:54.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:54 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3363904821' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-10T00:39:55.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-06-10T00:39:55.315 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:39:55.441 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:55 smithi121 ceph-mon[32378]: pgmap v371: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-06-10T00:39:55.441 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:55 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/3363904821' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-06-10T00:39:55.441 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:55 smithi121 ceph-mon[32378]: osdmap e112: 12 total, 12 up, 12 in 2024-06-10T00:39:55.441 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:55 smithi121 ceph-mon[32378]: osdmap e113: 12 total, 12 up, 12 in 2024-06-10T00:39:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:55 smithi193 ceph-mon[36716]: pgmap v371: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-06-10T00:39:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:55 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/3363904821' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-06-10T00:39:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:55 smithi193 ceph-mon[36716]: osdmap e112: 12 total, 12 up, 12 in 2024-06-10T00:39:55.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:55 smithi193 ceph-mon[36716]: osdmap e113: 12 total, 12 up, 12 in 2024-06-10T00:39:55.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:55 smithi135 ceph-mon[36424]: pgmap v371: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-06-10T00:39:55.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:55 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/3363904821' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-06-10T00:39:55.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:55 smithi135 ceph-mon[36424]: osdmap e112: 12 total, 12 up, 12 in 2024-06-10T00:39:55.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:55 smithi135 ceph-mon[36424]: osdmap e113: 12 total, 12 up, 12 in 2024-06-10T00:39:56.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:56 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:39:56.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:56 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:39:56.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:56 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:39:56.586 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:56 smithi121 ceph-mon[32378]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:39:56.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:56 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:39:56.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:56 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:39:56.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:56 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:39:56.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:56 smithi193 ceph-mon[36716]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:39:56.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:56 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-10T00:39:56.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:56 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-10T00:39:56.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:56 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-10T00:39:56.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:56 smithi135 ceph-mon[36424]: from='mgr.15216 172.21.15.121:0/3553643506' entity='mgr.x' 2024-06-10T00:39:57.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.397+0000 7ff703915700 1 -- 172.21.15.121:0/1256540479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6fc101df0 msgr2=0x7ff6fc1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:57.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.397+0000 7ff703915700 1 --2- 172.21.15.121:0/1256540479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6fc101df0 0x7ff6fc1021d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80099c0 tx=0x7ff6f8009cd0 comp rx=0 tx=0).stop 2024-06-10T00:39:57.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.397+0000 7ff703915700 1 -- 172.21.15.121:0/1256540479 shutdown_connections 2024-06-10T00:39:57.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.397+0000 7ff703915700 1 --2- 172.21.15.121:0/1256540479 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6fc106750 0x7ff6fc10e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:57.399 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.397+0000 7ff703915700 1 --2- 172.21.15.121:0/1256540479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6fc101df0 0x7ff6fc1021d0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.397+0000 7ff703915700 1 --2- 172.21.15.121:0/1256540479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc1027a0 0x7ff6fc106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.397+0000 7ff703915700 1 -- 172.21.15.121:0/1256540479 >> 172.21.15.121:0/1256540479 conn(0x7ff6fc077a00 msgr2=0x7ff6fc0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.397+0000 7ff703915700 1 -- 172.21.15.121:0/1256540479 shutdown_connections 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.397+0000 7ff703915700 1 -- 172.21.15.121:0/1256540479 wait complete. 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff703915700 1 Processor -- start 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff703915700 1 -- start start 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff703915700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6fc101df0 0x7ff6fc199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff703915700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6fc1027a0 0x7ff6fc19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff703915700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc106750 0x7ff6fc19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff703915700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc1116e0 con 0x7ff6fc101df0 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff703915700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc111560 con 0x7ff6fc1027a0 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff703915700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff6fc111860 con 0x7ff6fc106750 2024-06-10T00:39:57.400 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff7016b1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6fc101df0 0x7ff6fc199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff7016b1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6fc101df0 0x7ff6fc199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:50214/0 (socket says 172.21.15.121:50214) 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff7016b1700 1 -- 172.21.15.121:0/1066849291 learned_addr learned my addr 172.21.15.121:0/1066849291 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff701eb2700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc106750 0x7ff6fc19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff700eb0700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6fc1027a0 0x7ff6fc19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff7016b1700 1 -- 172.21.15.121:0/1066849291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc106750 msgr2=0x7ff6fc19a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff7016b1700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc106750 0x7ff6fc19a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff7016b1700 1 -- 172.21.15.121:0/1066849291 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6fc1027a0 msgr2=0x7ff6fc19a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff7016b1700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6fc1027a0 0x7ff6fc19a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.398+0000 7ff7016b1700 1 -- 172.21.15.121:0/1066849291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6fc1a4f60 con 0x7ff6fc101df0 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.399+0000 7ff7016b1700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6fc101df0 0x7ff6fc199b20 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8006010 tx=0x7ff6f8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.399+0000 7ff6f27fc700 1 -- 172.21.15.121:0/1066849291 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8026070 con 0x7ff6fc101df0 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.399+0000 7ff6f27fc700 1 -- 172.21.15.121:0/1066849291 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6f8004cd0 con 0x7ff6fc101df0 2024-06-10T00:39:57.401 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.399+0000 7ff703915700 1 -- 172.21.15.121:0/1066849291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6fc1a51f0 con 0x7ff6fc101df0 2024-06-10T00:39:57.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.399+0000 7ff6f27fc700 1 -- 172.21.15.121:0/1066849291 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8003ba0 con 0x7ff6fc101df0 2024-06-10T00:39:57.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.399+0000 7ff703915700 1 -- 172.21.15.121:0/1066849291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6fc1a5640 con 0x7ff6fc101df0 2024-06-10T00:39:57.402 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.400+0000 7ff703915700 1 -- 172.21.15.121:0/1066849291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6fc109990 con 0x7ff6fc101df0 2024-06-10T00:39:57.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.401+0000 7ff6f27fc700 1 -- 172.21.15.121:0/1066849291 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7ff6f8029070 con 0x7ff6fc101df0 2024-06-10T00:39:57.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.401+0000 7ff6f27fc700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff6e80a2d90 0x7ff6e80a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:39:57.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.402+0000 7ff700eb0700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff6e80a2d90 0x7ff6e80a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:39:57.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.402+0000 7ff6f27fc700 1 -- 172.21.15.121:0/1066849291 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 9287+0+0 (secure 0 0 0) 0x7ff6f80d0080 con 0x7ff6fc101df0 2024-06-10T00:39:57.403 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.402+0000 7ff700eb0700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff6e80a2d90 0x7ff6e80a5250 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc103f30 tx=0x7ff6ec009190 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:39:57.406 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.406+0000 7ff6f27fc700 1 -- 172.21.15.121:0/1066849291 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff6f805d850 con 0x7ff6fc101df0 2024-06-10T00:39:57.562 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:57 smithi121 ceph-mon[32378]: pgmap v374: 97 pgs: 97 active+clean; 768 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:39:57.614 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:57.613+0000 7ff703915700 1 -- 172.21.15.121:0/1066849291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff6fc02dca0 con 0x7ff6fc101df0 2024-06-10T00:39:57.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:57 smithi193 ceph-mon[36716]: pgmap v374: 97 pgs: 97 active+clean; 768 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:39:57.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:57 smithi135 ceph-mon[36424]: pgmap v374: 97 pgs: 97 active+clean; 768 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:39:58.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.300+0000 7ff6f27fc700 1 -- 172.21.15.121:0/1066849291 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v114) ==== 169+0+0 (secure 0 0 0) 0x7ff6f8013070 con 0x7ff6fc101df0 2024-06-10T00:39:58.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.303+0000 7ff703915700 1 -- 172.21.15.121:0/1066849291 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff6e80a2d90 msgr2=0x7ff6e80a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:58.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.303+0000 7ff703915700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff6e80a2d90 0x7ff6e80a5250 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc103f30 tx=0x7ff6ec009190 comp rx=0 tx=0).stop 2024-06-10T00:39:58.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.303+0000 7ff703915700 1 -- 172.21.15.121:0/1066849291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6fc101df0 msgr2=0x7ff6fc199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:39:58.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.304+0000 7ff703915700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6fc101df0 0x7ff6fc199b20 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8006010 tx=0x7ff6f8009f80 comp rx=0 tx=0).stop 2024-06-10T00:39:58.304 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.304+0000 7ff703915700 1 -- 172.21.15.121:0/1066849291 shutdown_connections 2024-06-10T00:39:58.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.304+0000 7ff703915700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7ff6e80a2d90 0x7ff6e80a5250 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:58.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.304+0000 7ff703915700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6fc106750 0x7ff6fc19a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:58.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.304+0000 7ff703915700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6fc101df0 0x7ff6fc199b20 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:58.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.304+0000 7ff703915700 1 --2- 172.21.15.121:0/1066849291 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff6fc1027a0 0x7ff6fc19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:39:58.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.304+0000 7ff703915700 1 -- 172.21.15.121:0/1066849291 >> 172.21.15.121:0/1066849291 conn(0x7ff6fc077a00 msgr2=0x7ff6fc0fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:39:58.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.304+0000 7ff703915700 1 -- 172.21.15.121:0/1066849291 shutdown_connections 2024-06-10T00:39:58.305 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:39:58.304+0000 7ff703915700 1 -- 172.21.15.121:0/1066849291 wait complete. 2024-06-10T00:39:58.305 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'cephfs_data' removed 2024-06-10T00:39:58.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:58 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1066849291' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-10T00:39:58.636 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:58 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1066849291' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-10T00:39:58.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:58 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1066849291' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-10T00:39:59.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba shell --fsid 6c432244-26be-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-06-10T00:39:59.293 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/mon.a/config 2024-06-10T00:39:59.557 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:59 smithi121 ceph-mon[32378]: pgmap v375: 97 pgs: 97 active+clean; 768 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:39:59.557 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:59 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/1066849291' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-06-10T00:39:59.557 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:39:59 smithi121 ceph-mon[32378]: osdmap e114: 12 total, 12 up, 12 in 2024-06-10T00:39:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:59 smithi193 ceph-mon[36716]: pgmap v375: 97 pgs: 97 active+clean; 768 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:39:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:59 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/1066849291' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-06-10T00:39:59.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:39:59 smithi193 ceph-mon[36716]: osdmap e114: 12 total, 12 up, 12 in 2024-06-10T00:39:59.659 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:59 smithi135 ceph-mon[36424]: pgmap v375: 97 pgs: 97 active+clean; 768 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:39:59.659 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:59 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/1066849291' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-06-10T00:39:59.660 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:39:59 smithi135 ceph-mon[36424]: osdmap e114: 12 total, 12 up, 12 in 2024-06-10T00:40:00.450 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:00 smithi121 ceph-mon[32378]: overall HEALTH_OK 2024-06-10T00:40:00.451 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:00 smithi121 ceph-mon[32378]: osdmap e115: 12 total, 12 up, 12 in 2024-06-10T00:40:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:00 smithi193 ceph-mon[36716]: overall HEALTH_OK 2024-06-10T00:40:00.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:00 smithi193 ceph-mon[36716]: osdmap e115: 12 total, 12 up, 12 in 2024-06-10T00:40:00.678 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:00 smithi135 ceph-mon[36424]: overall HEALTH_OK 2024-06-10T00:40:00.679 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:00 smithi135 ceph-mon[36424]: osdmap e115: 12 total, 12 up, 12 in 2024-06-10T00:40:01.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.165+0000 7f0a1b869700 1 -- 172.21.15.121:0/4007528109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a14101440 msgr2=0x7f0a1410f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.165+0000 7f0a1b869700 1 --2- 172.21.15.121:0/4007528109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a14101440 0x7f0a1410f530 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c009990 tx=0x7f0a0c009ca0 comp rx=0 tx=0).stop 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.165+0000 7f0a1b869700 1 -- 172.21.15.121:0/4007528109 shutdown_connections 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.165+0000 7f0a1b869700 1 --2- 172.21.15.121:0/4007528109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a14101440 0x7f0a1410f530 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.165+0000 7f0a1b869700 1 --2- 172.21.15.121:0/4007528109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a141086f0 0x7f0a14108ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.165+0000 7f0a1b869700 1 --2- 172.21.15.121:0/4007528109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a14100a80 0x7f0a14100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.165+0000 7f0a1b869700 1 -- 172.21.15.121:0/4007528109 >> 172.21.15.121:0/4007528109 conn(0x7f0a14077a00 msgr2=0x7f0a140fedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.165+0000 7f0a1b869700 1 -- 172.21.15.121:0/4007528109 shutdown_connections 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.165+0000 7f0a1b869700 1 -- 172.21.15.121:0/4007528109 wait complete. 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a1b869700 1 Processor -- start 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a1b869700 1 -- start start 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a1b869700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a14100a80 0x7f0a14198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:40:01.168 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a1b869700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a14101440 0x7f0a141992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a1b869700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a141086f0 0x7f0a1419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a1b869700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0a14112990 con 0x7f0a14100a80 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a1b869700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 -- 0x7f0a14112810 con 0x7f0a141086f0 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a1b869700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0a14112b10 con 0x7f0a14101440 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a19605700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a14100a80 0x7f0a14198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a19605700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a14100a80 0x7f0a14198d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:59500/0 (socket says 172.21.15.121:59500) 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a19605700 1 -- 172.21.15.121:0/2017203069 learned_addr learned my addr 172.21.15.121:0/2017203069 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a19e06700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a141086f0 0x7f0a1419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.166+0000 7f0a18e04700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a14101440 0x7f0a141992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.167+0000 7f0a19605700 1 -- 172.21.15.121:0/2017203069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a14101440 msgr2=0x7f0a141992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.167+0000 7f0a19605700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a14101440 0x7f0a141992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.167+0000 7f0a19605700 1 -- 172.21.15.121:0/2017203069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a141086f0 msgr2=0x7f0a1419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.167+0000 7f0a19605700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a141086f0 0x7f0a1419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:40:01.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.167+0000 7f0a19605700 1 -- 172.21.15.121:0/2017203069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1419dd40 con 0x7f0a14100a80 2024-06-10T00:40:01.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.167+0000 7f0a19605700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a14100a80 0x7f0a14198d80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0a10007ee0 tx=0x7f0a1000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:40:01.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.167+0000 7f0a0a7fc700 1 -- 172.21.15.121:0/2017203069 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1001a070 con 0x7f0a14100a80 2024-06-10T00:40:01.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.167+0000 7f0a1b869700 1 -- 172.21.15.121:0/2017203069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1419e030 con 0x7f0a14100a80 2024-06-10T00:40:01.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.167+0000 7f0a0a7fc700 1 -- 172.21.15.121:0/2017203069 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a1000dd20 con 0x7f0a14100a80 2024-06-10T00:40:01.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.167+0000 7f0a0a7fc700 1 -- 172.21.15.121:0/2017203069 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10014710 con 0x7f0a14100a80 2024-06-10T00:40:01.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.167+0000 7f0a1b869700 1 -- 172.21.15.121:0/2017203069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a141a58b0 con 0x7f0a14100a80 2024-06-10T00:40:01.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.168+0000 7f0a1b869700 1 -- 172.21.15.121:0/2017203069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a14069180 con 0x7f0a14100a80 2024-06-10T00:40:01.171 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:40:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:40:00.946+0000 7faf4d77c700 -1 client.15267: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-06-10T00:40:01.171 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:40:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:40:00.947+0000 7faf4d77c700 -1 client.15267: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-06-10T00:40:01.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.169+0000 7f0a0a7fc700 1 -- 172.21.15.121:0/2017203069 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) ==== 137965+0+0 (secure 0 0 0) 0x7f0a1001d070 con 0x7f0a14100a80 2024-06-10T00:40:01.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.170+0000 7f0a0a7fc700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0a000a2d90 0x7f0a000a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-10T00:40:01.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.170+0000 7f0a18e04700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0a000a2d90 0x7f0a000a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-10T00:40:01.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.170+0000 7f0a0a7fc700 1 -- 172.21.15.121:0/2017203069 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 8626+0+0 (secure 0 0 0) 0x7f0a100cf2d0 con 0x7f0a14100a80 2024-06-10T00:40:01.172 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.171+0000 7f0a18e04700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0a000a2d90 0x7f0a000a5250 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0a140fe610 tx=0x7f0a0400b410 comp rx=0 tx=0).ready entity=mgr.15216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-10T00:40:01.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.174+0000 7f0a0a7fc700 1 -- 172.21.15.121:0/2017203069 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0a10010070 con 0x7f0a14100a80 2024-06-10T00:40:01.377 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:01.375+0000 7f0a1b869700 1 -- 172.21.15.121:0/2017203069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0a1419a0c0 con 0x7f0a14100a80 2024-06-10T00:40:01.449 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:01 smithi121 ceph-mon[32378]: pgmap v377: 33 pgs: 33 active+clean; 768 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:40:01.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:01 smithi135 ceph-mon[36424]: pgmap v377: 33 pgs: 33 active+clean; 768 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:40:01.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:01 smithi193 ceph-mon[36716]: pgmap v377: 33 pgs: 33 active+clean; 768 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:40:02.315 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.315+0000 7f0a0a7fc700 1 -- 172.21.15.121:0/2017203069 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v116) ==== 178+0+0 (secure 0 0 0) 0x7f0a1419a0c0 con 0x7f0a14100a80 2024-06-10T00:40:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 -- 172.21.15.121:0/2017203069 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0a000a2d90 msgr2=0x7f0a000a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:40:02.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0a000a2d90 0x7f0a000a5250 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0a140fe610 tx=0x7f0a0400b410 comp rx=0 tx=0).stop 2024-06-10T00:40:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 -- 172.21.15.121:0/2017203069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a14100a80 msgr2=0x7f0a14198d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-10T00:40:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a14100a80 0x7f0a14198d80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0a10007ee0 tx=0x7f0a1000d3b0 comp rx=0 tx=0).stop 2024-06-10T00:40:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 -- 172.21.15.121:0/2017203069 shutdown_connections 2024-06-10T00:40:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.121:6800/2387911981,v1:172.21.15.121:6801/2387911981] conn(0x7f0a000a2d90 0x7f0a000a5250 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:40:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0a141086f0 0x7f0a1419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:40:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a14100a80 0x7f0a14198d80 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:40:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 --2- 172.21.15.121:0/2017203069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a14101440 0x7f0a141992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-10T00:40:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 -- 172.21.15.121:0/2017203069 >> 172.21.15.121:0/2017203069 conn(0x7f0a14077a00 msgr2=0x7f0a140fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-10T00:40:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 -- 172.21.15.121:0/2017203069 shutdown_connections 2024-06-10T00:40:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-06-10T00:40:02.319+0000 7f0a1b869700 1 -- 172.21.15.121:0/2017203069 wait complete. 2024-06-10T00:40:02.323 INFO:teuthology.orchestra.run.smithi121.stderr:pool 'cephfs_data_ec' removed 2024-06-10T00:40:02.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:02 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2017203069' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-10T00:40:02.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:02 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2017203069' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-10T00:40:02.758 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:02 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2017203069' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-06-10T00:40:03.157 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-10T00:40:03.193 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-10T00:40:03.230 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-10T00:40:03.275 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-10T00:40:03.275 DEBUG:teuthology.orchestra.run.smithi121:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-10T00:40:03.295 DEBUG:teuthology.orchestra.run.smithi135:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-10T00:40:03.310 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-10T00:40:03.329 INFO:tasks.cephadm:Stopping all daemons... 2024-06-10T00:40:03.329 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-06-10T00:40:03.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.a 2024-06-10T00:40:03.368 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:03 smithi121 ceph-mon[32378]: pgmap v379: 33 pgs: 33 active+clean; 768 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:40:03.368 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:03 smithi121 ceph-mon[32378]: from='client.? 172.21.15.121:0/2017203069' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-06-10T00:40:03.368 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:03 smithi121 ceph-mon[32378]: osdmap e116: 12 total, 12 up, 12 in 2024-06-10T00:40:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:03 smithi135 ceph-mon[36424]: pgmap v379: 33 pgs: 33 active+clean; 768 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:40:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:03 smithi135 ceph-mon[36424]: from='client.? 172.21.15.121:0/2017203069' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-06-10T00:40:03.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:03 smithi135 ceph-mon[36424]: osdmap e116: 12 total, 12 up, 12 in 2024-06-10T00:40:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:03 smithi193 ceph-mon[36716]: pgmap v379: 33 pgs: 33 active+clean; 768 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-06-10T00:40:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:03 smithi193 ceph-mon[36716]: from='client.? 172.21.15.121:0/2017203069' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-06-10T00:40:03.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:03 smithi193 ceph-mon[36716]: osdmap e116: 12 total, 12 up, 12 in 2024-06-10T00:40:03.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:03 smithi121 systemd[1]: Stopping Ceph mon.a for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:04.014 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:03 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a[32374]: 2024-06-10T00:40:03.754+0000 7f460f84d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:40:04.014 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:03 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a[32374]: 2024-06-10T00:40:03.754+0000 7f460f84d700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-06-10T00:40:04.014 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:03 smithi121 podman[121094]: 2024-06-10 00:40:03.797763603 +0000 UTC m=+0.164780358 container died abafb9418971b452a9073e897aa81dc2000522fb57ae1e29f9c0b38e8a76495a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-06-10T00:40:04.699 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:04 smithi121 podman[121094]: 2024-06-10 00:40:04.321300813 +0000 UTC m=+0.688317559 container cleanup abafb9418971b452a9073e897aa81dc2000522fb57ae1e29f9c0b38e8a76495a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux ) 2024-06-10T00:40:04.700 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:04 smithi121 bash[121094]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a 2024-06-10T00:40:05.092 INFO:journalctl@ceph.mon.a.smithi121.stdout:Jun 10 00:40:04 smithi121 podman[121106]: 2024-06-10 00:40:04.840012009 +0000 UTC m=+1.035413665 container remove abafb9418971b452a9073e897aa81dc2000522fb57ae1e29f9c0b38e8a76495a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-a, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-10T00:40:05.112 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.a.service' 2024-06-10T00:40:05.201 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:40:05.201 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-06-10T00:40:05.202 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-06-10T00:40:05.202 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.b 2024-06-10T00:40:05.508 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:05 smithi135 systemd[1]: Stopping Ceph mon.b for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:05.956 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:05 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-b[36420]: 2024-06-10T00:40:05.588+0000 7fb76b1bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:40:05.956 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:05 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-b[36420]: 2024-06-10T00:40:05.588+0000 7fb76b1bb700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-06-10T00:40:05.956 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:05 smithi135 podman[72694]: 2024-06-10 00:40:05.671193398 +0000 UTC m=+0.192840748 container died 37fc9b55c2e6556e87d6494e2489de8ccaec4b01e63e193a5d13496a1258ec74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-b, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-10T00:40:06.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:06 smithi135 podman[72694]: 2024-06-10 00:40:06.056350405 +0000 UTC m=+0.577997744 container cleanup 37fc9b55c2e6556e87d6494e2489de8ccaec4b01e63e193a5d13496a1258ec74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-b, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-06-10T00:40:06.258 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:06 smithi135 bash[72694]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-b 2024-06-10T00:40:06.580 INFO:journalctl@ceph.mon.b.smithi135.stdout:Jun 10 00:40:06 smithi135 podman[72707]: 2024-06-10 00:40:06.325444762 +0000 UTC m=+0.653189611 container remove 37fc9b55c2e6556e87d6494e2489de8ccaec4b01e63e193a5d13496a1258ec74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:40:06.585 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.b.service' 2024-06-10T00:40:06.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:40:06.627 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-06-10T00:40:06.627 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-06-10T00:40:06.628 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.c 2024-06-10T00:40:07.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:06 smithi193 systemd[1]: Stopping Ceph mon.c for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:07 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-c[36712]: 2024-06-10T00:40:07.278+0000 7ff5ce6dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:40:07.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:07 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-c[36712]: 2024-06-10T00:40:07.278+0000 7ff5ce6dc700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-06-10T00:40:07.632 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:07 smithi193 podman[74119]: 2024-06-10 00:40:07.421929929 +0000 UTC m=+0.558455876 container died 55317b68c71622ce383a508d371e0305cf07b7233ce1ecba54af4530d8980e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:40:08.131 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:07 smithi193 podman[74119]: 2024-06-10 00:40:07.854236233 +0000 UTC m=+0.990762163 container cleanup 55317b68c71622ce383a508d371e0305cf07b7233ce1ecba54af4530d8980e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-10T00:40:08.132 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:07 smithi193 bash[74119]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-c 2024-06-10T00:40:08.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jun 10 00:40:08 smithi193 podman[74131]: 2024-06-10 00:40:08.381594585 +0000 UTC m=+0.956475052 container remove 55317b68c71622ce383a508d371e0305cf07b7233ce1ecba54af4530d8980e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-10T00:40:08.636 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mon.c.service' 2024-06-10T00:40:08.694 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:40:08.694 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-06-10T00:40:08.694 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-06-10T00:40:08.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.x 2024-06-10T00:40:09.001 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:40:08 smithi121 systemd[1]: Stopping Ceph mgr.x for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:09.319 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:40:09 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:40:09.000+0000 7fb00a622700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:40:09.319 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:40:09 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x[32618]: 2024-06-10T00:40:09.000+0000 7fb00a622700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-06-10T00:40:09.320 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:40:09 smithi121 podman[121212]: 2024-06-10 00:40:09.033393893 +0000 UTC m=+0.143394768 container died a57b33200d7e35e61755f9a2fb650e53d6eef3559516af6db996e36dcce0e23c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528) 2024-06-10T00:40:09.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:40:09 smithi121 podman[121212]: 2024-06-10 00:40:09.519538964 +0000 UTC m=+0.629539837 container cleanup a57b33200d7e35e61755f9a2fb650e53d6eef3559516af6db996e36dcce0e23c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-06-10T00:40:09.699 INFO:journalctl@ceph.mgr.x.smithi121.stdout:Jun 10 00:40:09 smithi121 bash[121212]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-x 2024-06-10T00:40:09.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.x.service' 2024-06-10T00:40:09.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:40:09.926 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-06-10T00:40:09.926 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-06-10T00:40:09.926 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.y 2024-06-10T00:40:10.258 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:40:09 smithi135 systemd[1]: Stopping Ceph mgr.y for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:10.621 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:40:10 smithi135 podman[72814]: 2024-06-10 00:40:10.273267853 +0000 UTC m=+0.152308223 container died a4bbc28e66c799635b9e3120da4decd0f74a4ff1f148a805ccbbcbf33b0ec96d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-10T00:40:10.929 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:40:10 smithi135 podman[72814]: 2024-06-10 00:40:10.712144602 +0000 UTC m=+0.591184971 container cleanup a4bbc28e66c799635b9e3120da4decd0f74a4ff1f148a805ccbbcbf33b0ec96d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:40:10.930 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:40:10 smithi135 bash[72814]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y 2024-06-10T00:40:11.184 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:40:10 smithi135 podman[72826]: 2024-06-10 00:40:10.930135911 +0000 UTC m=+0.651818484 container remove a4bbc28e66c799635b9e3120da4decd0f74a4ff1f148a805ccbbcbf33b0ec96d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, ceph=True) 2024-06-10T00:40:11.184 INFO:journalctl@ceph.mgr.y.smithi135.stdout:Jun 10 00:40:10 smithi135 systemd[1]: ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-06-10T00:40:11.188 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.y.service' 2024-06-10T00:40:11.221 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:40:11.222 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-06-10T00:40:11.222 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-06-10T00:40:11.222 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.z 2024-06-10T00:40:11.631 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:40:11 smithi193 systemd[1]: Stopping Ceph mgr.z for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:12.131 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:40:11 smithi193 podman[74239]: 2024-06-10 00:40:11.813731751 +0000 UTC m=+0.398659505 container died 22997f2ae24b565936a7e4b776a4e9306fe9dd0029bd2e606753dcf5da967c9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:40:12.631 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:40:12 smithi193 podman[74239]: 2024-06-10 00:40:12.475161036 +0000 UTC m=+1.060088787 container cleanup 22997f2ae24b565936a7e4b776a4e9306fe9dd0029bd2e606753dcf5da967c9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:40:12.631 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jun 10 00:40:12 smithi193 bash[74239]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mgr-z 2024-06-10T00:40:14.763 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mgr.z.service' 2024-06-10T00:40:14.797 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:40:14.797 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-06-10T00:40:14.797 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-10T00:40:14.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.0 2024-06-10T00:40:15.107 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:14 smithi121 systemd[1]: Stopping Ceph osd.0 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:15.449 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:15 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0[46997]: 2024-06-10T00:40:15.106+0000 7efc7a12a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:40:15.450 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:15 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0[46997]: 2024-06-10T00:40:15.106+0000 7efc7a12a700 -1 osd.0 116 *** Got signal Terminated *** 2024-06-10T00:40:15.450 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:15 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0[46997]: 2024-06-10T00:40:15.106+0000 7efc7a12a700 -1 osd.0 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:40:20.766 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:20 smithi121 podman[121336]: 2024-06-10 00:40:20.501572492 +0000 UTC m=+5.508079009 container died c6630c87df7f152410546496348cea581cb03f4dafef5b0f84168f4f5099803d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux ) 2024-06-10T00:40:21.171 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:20 smithi121 podman[121336]: 2024-06-10 00:40:20.866259614 +0000 UTC m=+5.872766130 container cleanup c6630c87df7f152410546496348cea581cb03f4dafef5b0f84168f4f5099803d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-10T00:40:21.171 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:20 smithi121 bash[121336]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0 2024-06-10T00:40:21.449 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:21 smithi121 podman[121355]: 2024-06-10 00:40:21.167788742 +0000 UTC m=+0.663004233 container remove c6630c87df7f152410546496348cea581cb03f4dafef5b0f84168f4f5099803d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-06-10T00:40:21.763 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:21 smithi121 podman[121446]: 2024-06-10 00:40:21.388340432 +0000 UTC m=+0.017444841 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:40:21.763 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:21 smithi121 podman[121446]: 2024-06-10 00:40:21.510366804 +0000 UTC m=+0.139471178 container create 2254400a436bf1bdb984556d18348071ed10a8fedc3580c8885d31d5899082d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:40:22.067 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:21 smithi121 podman[121446]: 2024-06-10 00:40:21.763149251 +0000 UTC m=+0.392270055 container init 2254400a436bf1bdb984556d18348071ed10a8fedc3580c8885d31d5899082d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:40:22.067 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:21 smithi121 podman[121446]: 2024-06-10 00:40:21.766549176 +0000 UTC m=+0.395653563 container start 2254400a436bf1bdb984556d18348071ed10a8fedc3580c8885d31d5899082d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS) 2024-06-10T00:40:22.449 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:22 smithi121 podman[121446]: 2024-06-10 00:40:22.138678583 +0000 UTC m=+0.767782967 container attach 2254400a436bf1bdb984556d18348071ed10a8fedc3580c8885d31d5899082d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0-deactivate, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528) 2024-06-10T00:40:22.449 INFO:journalctl@ceph.osd.0.smithi121.stdout:Jun 10 00:40:22 smithi121 podman[121446]: 2024-06-10 00:40:22.13916106 +0000 UTC m=+0.768265435 container died 2254400a436bf1bdb984556d18348071ed10a8fedc3580c8885d31d5899082d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-06-10T00:40:22.887 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.0.service' 2024-06-10T00:40:22.924 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:40:22.925 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-10T00:40:22.925 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-10T00:40:22.925 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.1 2024-06-10T00:40:23.233 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:22 smithi135 systemd[1]: Stopping Ceph osd.1 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:23.508 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:23 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1[40983]: 2024-06-10T00:40:23.232+0000 7fb4afe90700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:40:23.508 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:23 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1[40983]: 2024-06-10T00:40:23.232+0000 7fb4afe90700 -1 osd.1 116 *** Got signal Terminated *** 2024-06-10T00:40:23.508 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:23 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1[40983]: 2024-06-10T00:40:23.232+0000 7fb4afe90700 -1 osd.1 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:40:29.001 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:28 smithi135 podman[72943]: 2024-06-10 00:40:28.664200644 +0000 UTC m=+5.550025678 container died cd0a3fb1734dc7d4ac9d1eb96708cd706d52f315deec5e6542b021fca5f27e51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-3d89acc, ceph=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:40:29.258 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:29 smithi135 podman[72943]: 2024-06-10 00:40:29.091610995 +0000 UTC m=+5.977436087 container cleanup cd0a3fb1734dc7d4ac9d1eb96708cd706d52f315deec5e6542b021fca5f27e51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-06-10T00:40:29.258 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:29 smithi135 bash[72943]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1 2024-06-10T00:40:29.758 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:29 smithi135 podman[72962]: 2024-06-10 00:40:29.384845397 +0000 UTC m=+0.718364924 container remove cd0a3fb1734dc7d4ac9d1eb96708cd706d52f315deec5e6542b021fca5f27e51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:40:30.182 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:29 smithi135 podman[73054]: 2024-06-10 00:40:29.774331431 +0000 UTC m=+0.017556046 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:40:30.182 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:29 smithi135 podman[73054]: 2024-06-10 00:40:29.911047262 +0000 UTC m=+0.154271858 container create bc91cb05cdeac98af47ef817f379472b2197fd6661b5a2128dbb20a050af12bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-10T00:40:30.182 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:30 smithi135 podman[73054]: 2024-06-10 00:40:30.18032301 +0000 UTC m=+0.423547606 container init bc91cb05cdeac98af47ef817f379472b2197fd6661b5a2128dbb20a050af12bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1-deactivate, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-10T00:40:30.475 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:30 smithi135 podman[73054]: 2024-06-10 00:40:30.183623475 +0000 UTC m=+0.426848088 container start bc91cb05cdeac98af47ef817f379472b2197fd6661b5a2128dbb20a050af12bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-06-10T00:40:30.475 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:30 smithi135 podman[73054]: 2024-06-10 00:40:30.313767338 +0000 UTC m=+0.556991947 container attach bc91cb05cdeac98af47ef817f379472b2197fd6661b5a2128dbb20a050af12bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:40:31.075 INFO:journalctl@ceph.osd.1.smithi135.stdout:Jun 10 00:40:30 smithi135 podman[73054]: 2024-06-10 00:40:30.773932752 +0000 UTC m=+1.017157359 container died bc91cb05cdeac98af47ef817f379472b2197fd6661b5a2128dbb20a050af12bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-10T00:40:31.435 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.1.service' 2024-06-10T00:40:31.474 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:40:31.474 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-10T00:40:31.475 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-10T00:40:31.475 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.2 2024-06-10T00:40:31.772 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:31 smithi193 systemd[1]: Stopping Ceph osd.2 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:32.132 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:31 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2[41840]: 2024-06-10T00:40:31.771+0000 7f354e286700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:40:32.132 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:31 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2[41840]: 2024-06-10T00:40:31.771+0000 7f354e286700 -1 osd.2 116 *** Got signal Terminated *** 2024-06-10T00:40:32.132 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:31 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2[41840]: 2024-06-10T00:40:31.771+0000 7f354e286700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:40:37.557 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:37 smithi193 podman[74365]: 2024-06-10 00:40:37.259864524 +0000 UTC m=+5.591940216 container died 0448d3a1d3356c69d63edf96e9b28136564ca7dc4fba0e1739c00cf572b934c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-06-10T00:40:37.881 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:37 smithi193 podman[74365]: 2024-06-10 00:40:37.639068835 +0000 UTC m=+5.971144531 container cleanup 0448d3a1d3356c69d63edf96e9b28136564ca7dc4fba0e1739c00cf572b934c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:40:37.881 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:37 smithi193 bash[74365]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2 2024-06-10T00:40:38.318 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:37 smithi193 podman[74392]: 2024-06-10 00:40:37.88238757 +0000 UTC m=+0.611898899 container remove 0448d3a1d3356c69d63edf96e9b28136564ca7dc4fba0e1739c00cf572b934c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-3d89acc) 2024-06-10T00:40:38.611 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:38 smithi193 podman[74483]: 2024-06-10 00:40:38.221835892 +0000 UTC m=+0.017536076 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:40:38.612 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:38 smithi193 podman[74483]: 2024-06-10 00:40:38.350211856 +0000 UTC m=+0.145912039 container create 0146d6c93429305df519eb9906a1bb4aa6f4d13b8c12fa9bec168501ebc7d5ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-06-10T00:40:38.881 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:38 smithi193 podman[74483]: 2024-06-10 00:40:38.611938509 +0000 UTC m=+0.407638699 container init 0146d6c93429305df519eb9906a1bb4aa6f4d13b8c12fa9bec168501ebc7d5ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:40:38.882 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:38 smithi193 podman[74483]: 2024-06-10 00:40:38.615431679 +0000 UTC m=+0.411131860 container start 0146d6c93429305df519eb9906a1bb4aa6f4d13b8c12fa9bec168501ebc7d5ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:40:38.882 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:38 smithi193 podman[74483]: 2024-06-10 00:40:38.72000247 +0000 UTC m=+0.515702669 container attach 0146d6c93429305df519eb9906a1bb4aa6f4d13b8c12fa9bec168501ebc7d5ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-10T00:40:39.272 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jun 10 00:40:38 smithi193 podman[74483]: 2024-06-10 00:40:38.917988452 +0000 UTC m=+0.713688635 container died 0146d6c93429305df519eb9906a1bb4aa6f4d13b8c12fa9bec168501ebc7d5ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-06-10T00:40:39.603 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.2.service' 2024-06-10T00:40:39.640 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:40:39.640 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-10T00:40:39.641 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-10T00:40:39.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.3 2024-06-10T00:40:39.949 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:39 smithi121 systemd[1]: Stopping Ceph osd.3 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:40.211 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:39 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3[53872]: 2024-06-10T00:40:39.968+0000 7fc43e05d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:40:40.211 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:39 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3[53872]: 2024-06-10T00:40:39.968+0000 7fc43e05d700 -1 osd.3 116 *** Got signal Terminated *** 2024-06-10T00:40:40.211 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:39 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3[53872]: 2024-06-10T00:40:39.968+0000 7fc43e05d700 -1 osd.3 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:40:42.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:42 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:42.852+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:44.131 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:43.849+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:45.131 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:44 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:44.809+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:45.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:45 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:45.079+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:45.877 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:45 smithi121 podman[121554]: 2024-06-10 00:40:45.598943151 +0000 UTC m=+5.757070855 container died 0e5f92768ccc01fc1108e49828e8cd239215b9e1ab25c0a8fba3f4c604469c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc) 2024-06-10T00:40:46.131 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:45 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:45.809+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:46.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:45 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:45.789+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:46.199 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:46 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:46.097+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:46.200 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:46 smithi121 podman[121554]: 2024-06-10 00:40:46.0774761 +0000 UTC m=+6.235603825 container cleanup 0e5f92768ccc01fc1108e49828e8cd239215b9e1ab25c0a8fba3f4c604469c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-10T00:40:46.200 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:46 smithi121 bash[121554]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3 2024-06-10T00:40:46.863 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:46 smithi121 podman[121582]: 2024-06-10 00:40:46.471390691 +0000 UTC m=+0.866425612 container remove 0e5f92768ccc01fc1108e49828e8cd239215b9e1ab25c0a8fba3f4c604469c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-06-10T00:40:47.131 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:46 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:46.782+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:47.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:46 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:46.753+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:47.148 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:47 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:47.049+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:47.149 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:46 smithi121 podman[121668]: 2024-06-10 00:40:46.76620008 +0000 UTC m=+0.016956369 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:40:47.149 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:46 smithi121 podman[121668]: 2024-06-10 00:40:46.897180878 +0000 UTC m=+0.147937151 container create 2862f8577c53546e8f9191e316636a244c920569bfdcff3fdb529492c78550cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-06-10T00:40:47.449 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:47 smithi121 podman[121668]: 2024-06-10 00:40:47.14958006 +0000 UTC m=+0.400336330 container init 2862f8577c53546e8f9191e316636a244c920569bfdcff3fdb529492c78550cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:40:47.449 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:47 smithi121 podman[121668]: 2024-06-10 00:40:47.152810672 +0000 UTC m=+0.403566945 container start 2862f8577c53546e8f9191e316636a244c920569bfdcff3fdb529492c78550cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-06-10T00:40:47.449 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:47 smithi121 podman[121668]: 2024-06-10 00:40:47.249754669 +0000 UTC m=+0.500510955 container attach 2862f8577c53546e8f9191e316636a244c920569bfdcff3fdb529492c78550cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-10T00:40:47.899 INFO:journalctl@ceph.osd.3.smithi121.stdout:Jun 10 00:40:47 smithi121 podman[121668]: 2024-06-10 00:40:47.450070312 +0000 UTC m=+0.700826589 container died 2862f8577c53546e8f9191e316636a244c920569bfdcff3fdb529492c78550cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-10T00:40:48.076 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.3.service' 2024-06-10T00:40:48.115 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:40:48.115 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-10T00:40:48.116 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-10T00:40:48.116 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.4 2024-06-10T00:40:48.131 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:47 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:47.756+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:48.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:47 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:47.796+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:48.199 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:48 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:48.040+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:48.437 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:48 smithi135 systemd[1]: Stopping Ceph osd.4 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:48.758 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:48 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4[47587]: 2024-06-10T00:40:48.437+0000 7f5c0c173700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:40:48.758 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:48 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4[47587]: 2024-06-10T00:40:48.437+0000 7f5c0c173700 -1 osd.4 116 *** Got signal Terminated *** 2024-06-10T00:40:48.758 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:48 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4[47587]: 2024-06-10T00:40:48.437+0000 7f5c0c173700 -1 osd.4 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:40:49.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:48 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:48.779+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:49.131 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:48 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:48.721+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:49.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:49 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:49.086+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:50.131 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:49.684+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:50.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:49.741+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:50.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:50 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:50.093+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:51.131 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:50 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:50.690+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:51.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:50 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:50.787+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:51.402 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:51 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:51.066+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:51.699 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:40:51 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:40:51.402+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:40:52.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:51 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:51.776+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:52.131 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:51 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:51.645+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:52.131 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:51 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:51.645+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.749236+0000 front 2024-06-10T00:40:26.749215+0000 (oldest deadline 2024-06-10T00:40:51.449366+0000) 2024-06-10T00:40:52.424 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:52 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:52.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:52.424 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:52 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:52.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:40:52.699 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:40:52 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:40:52.423+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:40:52.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:40:52 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:40:52.381+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:40:52.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:52 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:52.598+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:52.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:52 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:52.598+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.749236+0000 front 2024-06-10T00:40:26.749215+0000 (oldest deadline 2024-06-10T00:40:51.449366+0000) 2024-06-10T00:40:52.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:52 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:52.794+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:52.882 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:40:52 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:40:52.845+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:40:53.258 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:52 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4[47587]: 2024-06-10T00:40:52.828+0000 7f5c08767700 -1 osd.4 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.279563+0000 front 2024-06-10T00:40:27.279588+0000 (oldest deadline 2024-06-10T00:40:51.979809+0000) 2024-06-10T00:40:53.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:53 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:53.095+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:53.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:53 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:53.095+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:40:53.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:40:53 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:40:53.349+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:40:53.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:53.802+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:53.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:53.579+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:53.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:53.579+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.749236+0000 front 2024-06-10T00:40:26.749215+0000 (oldest deadline 2024-06-10T00:40:51.449366+0000) 2024-06-10T00:40:53.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:40:53 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:40:53.473+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:40:54.154 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:53 smithi135 podman[73173]: 2024-06-10 00:40:53.843215993 +0000 UTC m=+5.531201494 container died dfb4c7ebd91fbce95a81acbed2410a5d81619ae95695e06a5de394f582b265e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-10T00:40:54.381 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:40:53 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:40:53.894+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:40:54.411 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:40:54 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:40:54.374+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:40:54.411 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:54 smithi135 podman[73173]: 2024-06-10 00:40:54.253957019 +0000 UTC m=+5.941942509 container cleanup dfb4c7ebd91fbce95a81acbed2410a5d81619ae95695e06a5de394f582b265e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-3d89acc, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-06-10T00:40:54.411 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:54 smithi135 bash[73173]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4 2024-06-10T00:40:54.446 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:54 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:54.064+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:54.446 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:54 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:54.064+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:40:54.699 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:40:54 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:40:54.446+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:40:54.758 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:54 smithi135 podman[73192]: 2024-06-10 00:40:54.522152997 +0000 UTC m=+0.673610154 container remove dfb4c7ebd91fbce95a81acbed2410a5d81619ae95695e06a5de394f582b265e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-10T00:40:54.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:40:54 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:40:54.410+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:40:54.858 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:54 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:54.583+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:54.859 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:54 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:54.583+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.749236+0000 front 2024-06-10T00:40:26.749215+0000 (oldest deadline 2024-06-10T00:40:51.449366+0000) 2024-06-10T00:40:54.859 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:54 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:54.777+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:55.125 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:54 smithi135 podman[73281]: 2024-06-10 00:40:54.739870564 +0000 UTC m=+0.016743953 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:40:55.125 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:54 smithi135 podman[73281]: 2024-06-10 00:40:54.864516924 +0000 UTC m=+0.141390285 container create 6fcf544f38b2c4dfe17161685b5cf0847d0705fea917472882fdb6163b96724b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0) 2024-06-10T00:40:55.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:40:54 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:40:54.858+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:40:55.389 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:55 smithi135 podman[73281]: 2024-06-10 00:40:55.12556277 +0000 UTC m=+0.402436137 container init 6fcf544f38b2c4dfe17161685b5cf0847d0705fea917472882fdb6163b96724b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_CLEAN=True) 2024-06-10T00:40:55.389 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:55 smithi135 podman[73281]: 2024-06-10 00:40:55.129537191 +0000 UTC m=+0.406410555 container start 6fcf544f38b2c4dfe17161685b5cf0847d0705fea917472882fdb6163b96724b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-06-10T00:40:55.390 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:55 smithi135 podman[73281]: 2024-06-10 00:40:55.217115472 +0000 UTC m=+0.493988841 container attach 6fcf544f38b2c4dfe17161685b5cf0847d0705fea917472882fdb6163b96724b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:40:55.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:55 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:55.112+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:55.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:55 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:55.112+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:40:55.728 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:40:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:40:55.388+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:40:55.728 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:40:55 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:40:55.450+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:40:55.729 INFO:journalctl@ceph.osd.4.smithi135.stdout:Jun 10 00:40:55 smithi135 podman[73281]: 2024-06-10 00:40:55.419028172 +0000 UTC m=+0.695901535 container died 6fcf544f38b2c4dfe17161685b5cf0847d0705fea917472882fdb6163b96724b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_CLEAN=True) 2024-06-10T00:40:55.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:55.610+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:55.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:55.610+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.749236+0000 front 2024-06-10T00:40:26.749215+0000 (oldest deadline 2024-06-10T00:40:51.449366+0000) 2024-06-10T00:40:55.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:55.741+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:55.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:40:55 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:40:55.473+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:40:56.039 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.4.service' 2024-06-10T00:40:56.075 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:40:56.075 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-10T00:40:56.075 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-10T00:40:56.075 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.5 2024-06-10T00:40:56.381 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:56 smithi193 systemd[1]: Stopping Ceph osd.5 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:40:56.381 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:40:55 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:40:55.906+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:40:56.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:56 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:56.094+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:56.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:56 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:56.094+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:40:56.754 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:56.393+0000 7f8d4fab5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:40:56.754 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:56.393+0000 7f8d4fab5700 -1 osd.5 116 *** Got signal Terminated *** 2024-06-10T00:40:56.754 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:56.393+0000 7f8d4fab5700 -1 osd.5 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:40:56.754 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:56.604+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:56.754 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:56.604+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.749236+0000 front 2024-06-10T00:40:26.749215+0000 (oldest deadline 2024-06-10T00:40:51.449366+0000) 2024-06-10T00:40:56.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:40:56 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:40:56.418+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:40:56.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:40:56 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:40:56.456+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:40:56.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:40:56 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:40:56.520+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:40:57.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:40:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:40:56.902+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:40:57.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:56 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:56.754+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:57.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:57.076+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:57.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:57.076+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:40:57.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:40:57 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:40:57.422+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:40:57.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:40:57 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:40:57.464+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:40:57.864 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:57 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:57.576+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:57.864 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:57 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:57.576+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.749236+0000 front 2024-06-10T00:40:26.749215+0000 (oldest deadline 2024-06-10T00:40:51.449366+0000) 2024-06-10T00:40:57.864 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:57 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:57.729+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:57.864 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:57 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:57.729+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:40:57.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:40:57 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:40:57.495+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:40:58.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:40:57 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:40:57.863+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:40:58.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:58 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:58.099+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:58.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:58 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:58.099+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:40:58.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:40:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:40:58.486+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:40:58.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:40:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:40:58.452+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:40:58.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:40:58 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:40:58.452+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:40:58.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:58 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:58.584+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:58.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:58 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:58.584+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.749236+0000 front 2024-06-10T00:40:26.749215+0000 (oldest deadline 2024-06-10T00:40:51.449366+0000) 2024-06-10T00:40:58.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:58 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:58.736+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:58.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:58 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:58.736+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:40:58.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:40:58 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:40:58.466+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:40:59.381 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:40:58 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:40:58.897+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:40:59.445 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:59.084+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:40:59.445 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:59.084+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:40:59.445 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:40:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:40:59.084+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:40:59.699 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:40:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:40:59.445+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:40:59.699 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:40:59 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:40:59.445+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.452994+0000 front 2024-06-10T00:40:36.452910+0000 (oldest deadline 2024-06-10T00:40:59.352999+0000) 2024-06-10T00:40:59.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:40:59 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:40:59.496+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:40:59.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:40:59 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:40:59.496+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:40:59.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:40:59 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:40:59.435+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:40:59.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:40:59 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:40:59.435+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:40:59.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:59 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:59.542+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:40:59.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:59 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:59.542+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.749236+0000 front 2024-06-10T00:40:26.749215+0000 (oldest deadline 2024-06-10T00:40:51.449366+0000) 2024-06-10T00:40:59.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:40:59 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:40:59.542+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.351304+0000 front 2024-06-10T00:40:35.351282+0000 (oldest deadline 2024-06-10T00:40:58.851407+0000) 2024-06-10T00:40:59.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:59 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:59.716+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:40:59.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:40:59 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:40:59.716+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:00.381 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:40:59 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:40:59.887+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:00.444 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:00.058+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:00.444 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:00.058+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:00.444 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:00.058+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:00.699 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:00.444+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:41:00.699 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:00 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:00.444+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.452994+0000 front 2024-06-10T00:40:36.452910+0000 (oldest deadline 2024-06-10T00:40:59.352999+0000) 2024-06-10T00:41:00.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:00 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:00.440+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:00.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:00 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:00.440+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:00.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:00 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:00.461+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:00.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:00 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:00.461+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:00.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:00 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:41:00.558+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:16.748232+0000 front 2024-06-10T00:40:16.748314+0000 (oldest deadline 2024-06-10T00:40:42.048123+0000) 2024-06-10T00:41:00.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:00 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:41:00.558+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.749236+0000 front 2024-06-10T00:40:26.749215+0000 (oldest deadline 2024-06-10T00:40:51.449366+0000) 2024-06-10T00:41:00.881 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:00 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5[48316]: 2024-06-10T00:41:00.558+0000 7f8d480a1700 -1 osd.5 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.351304+0000 front 2024-06-10T00:40:35.351282+0000 (oldest deadline 2024-06-10T00:40:58.851407+0000) 2024-06-10T00:41:00.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:00 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:00.709+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:00.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:00 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:00.709+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:01.381 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:00 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:00.928+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:01.381 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:00 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:00.928+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:01.433 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:01 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:01.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:01.433 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:01 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:01.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:01.433 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:01 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:01.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:01.699 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:01 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:01.433+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:41:01.700 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:01 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:01.433+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.452994+0000 front 2024-06-10T00:40:36.452910+0000 (oldest deadline 2024-06-10T00:40:59.352999+0000) 2024-06-10T00:41:01.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:01 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:01.499+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:01.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:01 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:01.499+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:01.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:01 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:01.408+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:01.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:01 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:01.408+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:02.119 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:01 smithi193 podman[74587]: 2024-06-10 00:41:01.859056898 +0000 UTC m=+5.593024581 container died 0ca640f8d67e9b0c5ba482c4fa90b4245b581c7a7c5d71cd7017c31a75d8d03b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:41:02.120 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:01 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:01.881+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:02.120 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:01 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:01.881+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:02.120 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:01 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:01.699+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:02.120 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:01 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:01.699+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:02.381 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:02 smithi193 podman[74587]: 2024-06-10 00:41:02.211266768 +0000 UTC m=+5.945234437 container cleanup 0ca640f8d67e9b0c5ba482c4fa90b4245b581c7a7c5d71cd7017c31a75d8d03b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528) 2024-06-10T00:41:02.381 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:02 smithi193 bash[74587]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5 2024-06-10T00:41:02.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:02 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:02.098+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:02.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:02 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:02.098+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:02.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:02 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:02.098+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:02.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:02 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:02.520+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:02.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:02 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:02.520+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:02.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:02 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:02.380+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:02.759 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:02 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:02.380+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:02.875 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:02 smithi193 podman[74611]: 2024-06-10 00:41:02.521201645 +0000 UTC m=+0.656632980 container remove 0ca640f8d67e9b0c5ba482c4fa90b4245b581c7a7c5d71cd7017c31a75d8d03b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-06-10T00:41:02.876 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:02 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:02.694+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:02.876 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:02 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:02.694+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:02.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:02 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:02.464+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:41:02.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:02 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:02.464+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.452994+0000 front 2024-06-10T00:40:36.452910+0000 (oldest deadline 2024-06-10T00:40:59.352999+0000) 2024-06-10T00:41:03.131 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:02 smithi193 podman[74704]: 2024-06-10 00:41:02.794802989 +0000 UTC m=+0.017657460 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:41:03.132 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:02 smithi193 podman[74704]: 2024-06-10 00:41:02.914163105 +0000 UTC m=+0.137017552 container create 79d472561e4c893d8a441ee100dd137f56f2a0139c27647d12080c55cc5ed1e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5-deactivate, ceph=True, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:41:03.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:02 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:02.875+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:03.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:02 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:02.875+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:03.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:03 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:03.052+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:03.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:03 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:03.052+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:03.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:03 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:03.052+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:03.475 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:03 smithi193 podman[74704]: 2024-06-10 00:41:03.17522609 +0000 UTC m=+0.398080546 container init 79d472561e4c893d8a441ee100dd137f56f2a0139c27647d12080c55cc5ed1e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc) 2024-06-10T00:41:03.475 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:03 smithi193 podman[74704]: 2024-06-10 00:41:03.178270599 +0000 UTC m=+0.401125057 container start 79d472561e4c893d8a441ee100dd137f56f2a0139c27647d12080c55cc5ed1e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-10T00:41:03.475 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:03 smithi193 podman[74704]: 2024-06-10 00:41:03.258539529 +0000 UTC m=+0.481393989 container attach 79d472561e4c893d8a441ee100dd137f56f2a0139c27647d12080c55cc5ed1e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, ceph=True, org.label-schema.vendor=CentOS) 2024-06-10T00:41:03.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:03 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:03.427+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:03.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:03 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:03.427+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:03.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:03 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:03.553+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:03.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:03 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:03.553+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:03.832 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jun 10 00:41:03 smithi193 podman[74704]: 2024-06-10 00:41:03.475997579 +0000 UTC m=+0.698852034 container died 79d472561e4c893d8a441ee100dd137f56f2a0139c27647d12080c55cc5ed1e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-06-10T00:41:03.832 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:03 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:03.657+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:03.833 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:03 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:03.657+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:03.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:03 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:03.491+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:41:03.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:03 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:03.491+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.452994+0000 front 2024-06-10T00:40:36.452910+0000 (oldest deadline 2024-06-10T00:40:59.352999+0000) 2024-06-10T00:41:04.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:03 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:03.832+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:04.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:03 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:03.832+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:04.169 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.5.service' 2024-06-10T00:41:04.207 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:41:04.207 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-10T00:41:04.208 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-06-10T00:41:04.208 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.6 2024-06-10T00:41:04.449 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:04 smithi121 systemd[1]: Stopping Ceph osd.6 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:41:04.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:04 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:04.059+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:04.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:04 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:04.059+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:04.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:04 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:04.059+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:04.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:04 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:04.471+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:04.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:04 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:04.471+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:04.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:04 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:04.537+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:04.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:04 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:04.537+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:04.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:04 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:04.496+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:41:04.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:04 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:04.496+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.452994+0000 front 2024-06-10T00:40:36.452910+0000 (oldest deadline 2024-06-10T00:40:59.352999+0000) 2024-06-10T00:41:04.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:04 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:04.530+0000 7f8bdbd17700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:41:04.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:04 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:04.530+0000 7f8bdbd17700 -1 osd.6 116 *** Got signal Terminated *** 2024-06-10T00:41:04.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:04 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:04.530+0000 7f8bdbd17700 -1 osd.6 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:41:05.133 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:04 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:04.805+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:05.134 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:04 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:04.805+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:05.134 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:04 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:04.686+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:05.134 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:04 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:04.686+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:05.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:05 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:05.013+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:05.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:05 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:05.013+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:05.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:05 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:05.013+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:05.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:05 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:05.436+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:05.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:05 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:05.436+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:05.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:05 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:05.519+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:05.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:05 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:05.519+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:05.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:05 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:05.533+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:41:05.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:05 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:05.533+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.452994+0000 front 2024-06-10T00:40:36.452910+0000 (oldest deadline 2024-06-10T00:40:59.352999+0000) 2024-06-10T00:41:06.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:05 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:05.829+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:06.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:05 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:05.829+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:06.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:05 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:05.641+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:06.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:05 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:05.641+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:06.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:06 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:06.056+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:06.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:06 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:06.056+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:06.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:06 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:06.056+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:06.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:06 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:06.400+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:06.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:06 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:06.400+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:06.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:06 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:06.565+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:06.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:06 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:06.565+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:06.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:06 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:06.486+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:41:06.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:06 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:06.486+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.452994+0000 front 2024-06-10T00:40:36.452910+0000 (oldest deadline 2024-06-10T00:40:59.352999+0000) 2024-06-10T00:41:06.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:06 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:06.486+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:41.653718+0000 front 2024-06-10T00:40:41.653746+0000 (oldest deadline 2024-06-10T00:41:06.353802+0000) 2024-06-10T00:41:07.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:06 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:06.852+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:07.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:06 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:06.852+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:07.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:06 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:06.852+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:07.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:06 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:06.654+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:07.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:06 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:06.654+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:07.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:06 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:06.654+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:07.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:07 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:07.085+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:07.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:07 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:07.085+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:07.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:07 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:07.085+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:07.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:07 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:07.418+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:07.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:07 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:07.418+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:07.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:07 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:07.556+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:07.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:07 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:07.556+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:07.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:07 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:07.857+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:07.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:07 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:07.857+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:07.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:07 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:07.857+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:07.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:07 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:07.619+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:07.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:07 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:07.619+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:07.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:07 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:07.619+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:07.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:07 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:07.492+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:41:07.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:07 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:07.492+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.452994+0000 front 2024-06-10T00:40:36.452910+0000 (oldest deadline 2024-06-10T00:40:59.352999+0000) 2024-06-10T00:41:07.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:07 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:07.492+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:41.653718+0000 front 2024-06-10T00:40:41.653746+0000 (oldest deadline 2024-06-10T00:41:06.353802+0000) 2024-06-10T00:41:08.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:08 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:08.075+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:08.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:08 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:08.075+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:08.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:08 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:08.075+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:08.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:08 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:08.436+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:08.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:08 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:08.436+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:08.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:08 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:08.553+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:08.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:08 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:08.553+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:08.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:08 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:08.553+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:08.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:08 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:08.825+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:08.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:08 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:08.825+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:08.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:08 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:08.825+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:08.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:08 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:08.626+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:08.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:08 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:08.626+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:08.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:08 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:08.626+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:08.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:08 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:08.524+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:26.050638+0000 front 2024-06-10T00:40:26.050489+0000 (oldest deadline 2024-06-10T00:40:51.350584+0000) 2024-06-10T00:41:08.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:08 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:08.524+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.452994+0000 front 2024-06-10T00:40:36.452910+0000 (oldest deadline 2024-06-10T00:40:59.352999+0000) 2024-06-10T00:41:08.950 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:08 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6[60787]: 2024-06-10T00:41:08.524+0000 7f8bd4303700 -1 osd.6 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:41.653718+0000 front 2024-06-10T00:40:41.653746+0000 (oldest deadline 2024-06-10T00:41:06.353802+0000) 2024-06-10T00:41:09.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:09 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:09.051+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:09.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:09 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:09.051+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:09.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:09 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:09.051+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:09.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:09 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:09.051+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:09.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:09 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:09.437+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:09.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:09 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:09.437+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:09.759 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:09 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:09.437+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.681641+0000 front 2024-06-10T00:40:39.681535+0000 (oldest deadline 2024-06-10T00:41:09.082100+0000) 2024-06-10T00:41:09.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:09 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:09.524+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:09.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:09 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:09.524+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:09.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:09 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:09.524+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:10.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:09 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:09.824+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:10.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:09 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:09.824+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:10.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:09 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:09.824+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:10.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:09 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:09.672+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:10.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:09 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:09.672+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:10.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:09 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:09.672+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:10.197 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:09 smithi121 podman[121772]: 2024-06-10 00:41:09.926542721 +0000 UTC m=+5.522297739 container died c6851f316f4a77ce92ecde64aef77060da40998d1e45973308f0513c686e8693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528) 2024-06-10T00:41:10.254 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:10 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:10.091+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:10.254 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:10 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:10.091+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:10.254 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:10 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:10.091+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:10.254 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:10 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:10.091+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:10.449 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:10 smithi121 podman[121772]: 2024-06-10 00:41:10.388491125 +0000 UTC m=+5.984246132 container cleanup c6851f316f4a77ce92ecde64aef77060da40998d1e45973308f0513c686e8693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=) 2024-06-10T00:41:10.449 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:10 smithi121 bash[121772]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6 2024-06-10T00:41:10.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:10 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:10.424+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:10.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:10 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:10.424+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:10.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:10 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:10.424+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.681641+0000 front 2024-06-10T00:40:39.681535+0000 (oldest deadline 2024-06-10T00:41:09.082100+0000) 2024-06-10T00:41:10.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:10 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:10.491+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:10.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:10 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:10.491+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:10.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:10 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:10.491+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:10.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:10 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:10.835+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:10.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:10 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:10.835+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:10.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:10 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:10.835+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:10.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:10 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:10.624+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:10.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:10 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:10.624+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:10.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:10 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:10.624+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:10.949 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:10 smithi121 podman[121787]: 2024-06-10 00:41:10.682000389 +0000 UTC m=+0.749877643 container remove c6851f316f4a77ce92ecde64aef77060da40998d1e45973308f0513c686e8693 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-10T00:41:11.301 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:11 smithi121 podman[121889]: 2024-06-10 00:41:10.945273292 +0000 UTC m=+0.016931856 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:41:11.301 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:11 smithi121 podman[121889]: 2024-06-10 00:41:11.049432269 +0000 UTC m=+0.121090831 container create 9191731cddc97dd24f7335135226cb59077625077ae4c204fee6943876a66ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:41:11.301 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:11.083+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:11.301 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:11.083+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:11.302 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:11.083+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:11.302 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:11 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:11.083+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:11.603 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:11 smithi121 podman[121889]: 2024-06-10 00:41:11.301748286 +0000 UTC m=+0.373406829 container init 9191731cddc97dd24f7335135226cb59077625077ae4c204fee6943876a66ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS) 2024-06-10T00:41:11.604 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:11 smithi121 podman[121889]: 2024-06-10 00:41:11.305113668 +0000 UTC m=+0.376772229 container start 9191731cddc97dd24f7335135226cb59077625077ae4c204fee6943876a66ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-10T00:41:11.604 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:11 smithi121 podman[121889]: 2024-06-10 00:41:11.410273095 +0000 UTC m=+0.481931658 container attach 9191731cddc97dd24f7335135226cb59077625077ae4c204fee6943876a66ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-06-10T00:41:11.604 INFO:journalctl@ceph.osd.6.smithi121.stdout:Jun 10 00:41:11 smithi121 podman[121889]: 2024-06-10 00:41:11.604039604 +0000 UTC m=+0.675698159 container died 9191731cddc97dd24f7335135226cb59077625077ae4c204fee6943876a66ba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:41:11.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:11 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:11.427+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:11.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:11 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:11.427+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:11.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:11 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:11.427+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.681641+0000 front 2024-06-10T00:40:39.681535+0000 (oldest deadline 2024-06-10T00:41:09.082100+0000) 2024-06-10T00:41:11.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:11 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:11.511+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:11.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:11 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:11.511+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:11.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:11 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:11.511+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:12.100 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:11 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:11.655+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:12.101 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:11 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:11.655+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:12.101 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:11 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:11.655+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:12.101 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:11 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:11.808+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:12.101 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:11 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:11.808+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:12.101 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:11 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:11.808+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:12.147 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:12 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:12.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:12.147 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:12 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:12.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:12.147 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:12 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:12.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:12.147 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:12 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:12.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:12.177 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.6.service' 2024-06-10T00:41:12.217 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:41:12.217 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-06-10T00:41:12.217 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-06-10T00:41:12.217 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.7 2024-06-10T00:41:12.508 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:12 smithi135 systemd[1]: Stopping Ceph osd.7 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:41:12.508 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:12 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:12.475+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:12.508 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:12 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:12.475+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:12.508 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:12 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:12.475+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.681641+0000 front 2024-06-10T00:40:39.681535+0000 (oldest deadline 2024-06-10T00:41:09.082100+0000) 2024-06-10T00:41:13.008 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:12 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:12.551+0000 7f4dd491b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:41:13.008 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:12 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:12.551+0000 7f4dd491b700 -1 osd.7 116 *** Got signal Terminated *** 2024-06-10T00:41:13.008 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:12 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:12.551+0000 7f4dd491b700 -1 osd.7 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:41:13.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:12 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:12.529+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:13.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:12 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:12.529+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:13.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:12 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:12.529+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:13.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:12 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:12.852+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:13.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:12 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:12.852+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:13.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:12 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:12.852+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:13.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:12 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:12.654+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:13.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:12 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:12.654+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:13.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:12 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:12.654+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:13.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:13 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:13.057+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:13.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:13 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:13.057+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:13.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:13 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:13.057+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:13.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:13 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:13.057+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:13.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:13 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:13.452+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:13.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:13 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:13.452+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:13.759 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:13 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:13.452+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.681641+0000 front 2024-06-10T00:40:39.681535+0000 (oldest deadline 2024-06-10T00:41:09.082100+0000) 2024-06-10T00:41:13.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:13 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:13.509+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:13.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:13 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:13.509+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:13.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:13 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:13.509+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:13.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:13 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:13.804+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:13.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:13 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:13.804+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:13.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:13 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:13.804+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:13.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:13 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:13.629+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:13.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:13 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:13.630+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:13.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:13 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:13.630+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:14.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:14 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:14.089+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:14.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:14 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:14.089+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:14.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:14 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:14.089+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:14.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:14 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:14.089+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:14.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:14 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:14.500+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:14.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:14 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:14.500+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:14.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:14 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:14.500+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.681641+0000 front 2024-06-10T00:40:39.681535+0000 (oldest deadline 2024-06-10T00:41:09.082100+0000) 2024-06-10T00:41:14.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:14 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:14.512+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:14.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:14 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:14.512+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:14.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:14 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:14.512+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:14.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:14 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:14.512+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:14.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:14 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:14.784+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:14.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:14 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:14.784+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:14.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:14 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:14.784+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:14.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:14 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:14.602+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:14.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:14 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:14.602+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:14.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:14 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:14.602+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:15.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:15 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:15.053+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:15.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:15 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:15.053+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:15.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:15 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:15.053+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:15.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:15 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:15.053+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:15.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:15 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:15.739+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:15.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:15 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:15.739+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:15.882 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:15 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:15.739+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:15.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:15 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:15.562+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:15.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:15 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:15.562+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:15.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:15 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:15.562+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:16.008 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:15 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:15.520+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:16.008 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:15 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:15.520+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:16.008 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:15 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:15.520+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.681641+0000 front 2024-06-10T00:40:39.681535+0000 (oldest deadline 2024-06-10T00:41:09.082100+0000) 2024-06-10T00:41:16.008 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:15 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:15.520+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.483495+0000 front 2024-06-10T00:40:52.483534+0000 (oldest deadline 2024-06-10T00:41:15.383582+0000) 2024-06-10T00:41:16.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:15 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:15.513+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:16.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:15 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:15.513+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:16.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:15 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:15.513+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:16.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:15 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:15.513+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:16.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:16 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:16.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:16.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:16 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:16.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:16.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:16 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:16.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:16.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:16 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:16.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:16.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:16 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:16.699+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:16.882 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:16 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:16.699+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:16.882 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:16 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:16.699+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:16.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:16 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:16.597+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:16.883 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:16 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:16.597+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:16.883 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:16 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:16.597+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:17.008 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:16 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:16.543+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:17.008 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:16 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:16.543+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:17.008 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:16 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:16.543+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.681641+0000 front 2024-06-10T00:40:39.681535+0000 (oldest deadline 2024-06-10T00:41:09.082100+0000) 2024-06-10T00:41:17.009 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:16 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:16.543+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.483495+0000 front 2024-06-10T00:40:52.483534+0000 (oldest deadline 2024-06-10T00:41:15.383582+0000) 2024-06-10T00:41:17.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:16 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:16.555+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:17.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:16 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:16.555+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:17.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:16 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:16.555+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:17.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:16 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:16.555+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:17.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:17 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:17.100+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:17.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:17 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:17.100+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:17.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:17 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:17.100+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:17.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:17 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:17.100+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:17.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:17 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:17.100+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:17.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:17 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:17.496+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.578844+0000 front 2024-06-10T00:40:27.578961+0000 (oldest deadline 2024-06-10T00:40:51.678883+0000) 2024-06-10T00:41:17.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:17 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:17.496+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:32.179654+0000 front 2024-06-10T00:40:32.179688+0000 (oldest deadline 2024-06-10T00:40:58.079662+0000) 2024-06-10T00:41:17.758 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:17 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:17.496+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.681641+0000 front 2024-06-10T00:40:39.681535+0000 (oldest deadline 2024-06-10T00:41:09.082100+0000) 2024-06-10T00:41:17.759 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:17 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7[54254]: 2024-06-10T00:41:17.496+0000 7f4dd0f0f700 -1 osd.7 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.483495+0000 front 2024-06-10T00:40:52.483534+0000 (oldest deadline 2024-06-10T00:41:15.383582+0000) 2024-06-10T00:41:17.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:17 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:17.529+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:17.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:17 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:17.529+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:17.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:17 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:17.529+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:17.759 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:17 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:17.529+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:17.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:17 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:17.727+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:17.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:17 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:17.727+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:17.882 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:17 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:17.727+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:17.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:17 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:17.616+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:17.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:17 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:17.616+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:17.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:17 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:17.616+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:17.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:17 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:17.616+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:18.417 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:18 smithi135 podman[73397]: 2024-06-10 00:41:18.080501377 +0000 UTC m=+5.666875129 container died a957e4720bb26822b44475f02390fc08b7687facabc636202ea6aa2665fd9838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-06-10T00:41:18.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:18 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:18.118+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:18.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:18 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:18.118+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:18.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:18 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:18.118+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:18.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:18 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:18.118+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:18.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:18 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:18.118+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:18.725 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:18 smithi135 podman[73397]: 2024-06-10 00:41:18.515605328 +0000 UTC m=+6.101979084 container cleanup a957e4720bb26822b44475f02390fc08b7687facabc636202ea6aa2665fd9838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-06-10T00:41:18.725 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:18 smithi135 bash[73397]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7 2024-06-10T00:41:18.725 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:18 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:18.530+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:18.725 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:18 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:18.530+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:18.725 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:18 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:18.530+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:18.725 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:18 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:18.530+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:19.008 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:18 smithi135 podman[73413]: 2024-06-10 00:41:18.725379037 +0000 UTC m=+0.643257727 container remove a957e4720bb26822b44475f02390fc08b7687facabc636202ea6aa2665fd9838 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:41:19.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:18 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:18.687+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:19.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:18 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:18.687+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:19.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:18 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:18.687+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:19.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:18 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:18.687+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:53.433618+0000 front 2024-06-10T00:40:53.433752+0000 (oldest deadline 2024-06-10T00:41:18.133532+0000) 2024-06-10T00:41:19.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:18 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:18.661+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:19.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:18 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:18.661+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:19.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:18 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:18.661+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:19.133 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:18 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:18.661+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:19.386 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:19 smithi135 podman[73501]: 2024-06-10 00:41:19.032350854 +0000 UTC m=+0.017167205 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:41:19.387 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:19 smithi135 podman[73501]: 2024-06-10 00:41:19.142921257 +0000 UTC m=+0.127737600 container create ace0fd333e07ca8229487a08e09eae9de5b66704e8a5efae5704c4ca2a504f92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7-deactivate, RELEASE=squid-3d89acc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-06-10T00:41:19.387 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:19 smithi135 podman[73501]: 2024-06-10 00:41:19.387241577 +0000 UTC m=+0.372057928 container init ace0fd333e07ca8229487a08e09eae9de5b66704e8a5efae5704c4ca2a504f92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7-deactivate, RELEASE=squid-3d89acc, ceph=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-10T00:41:19.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:19 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:19.104+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:19.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:19 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:19.104+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:19.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:19 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:19.104+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:19.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:19 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:19.104+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:19.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:19 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:19.104+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:19.679 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:19 smithi135 podman[73501]: 2024-06-10 00:41:19.390469478 +0000 UTC m=+0.375285830 container start ace0fd333e07ca8229487a08e09eae9de5b66704e8a5efae5704c4ca2a504f92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-06-10T00:41:19.679 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:19 smithi135 podman[73501]: 2024-06-10 00:41:19.478751296 +0000 UTC m=+0.463567673 container attach ace0fd333e07ca8229487a08e09eae9de5b66704e8a5efae5704c4ca2a504f92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-10T00:41:19.680 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:19 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:19.502+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:19.680 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:19 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:19.502+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:19.680 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:19 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:19.502+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:19.680 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:19 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:19.502+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:19.973 INFO:journalctl@ceph.osd.7.smithi135.stdout:Jun 10 00:41:19 smithi135 podman[73501]: 2024-06-10 00:41:19.680322852 +0000 UTC m=+0.665139204 container died ace0fd333e07ca8229487a08e09eae9de5b66704e8a5efae5704c4ca2a504f92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-06-10T00:41:20.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:19 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:19.716+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:20.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:19 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:19.716+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:20.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:19 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:19.716+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:20.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:19 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:19.716+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:53.433618+0000 front 2024-06-10T00:40:53.433752+0000 (oldest deadline 2024-06-10T00:41:18.133532+0000) 2024-06-10T00:41:20.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:19 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:19.673+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:20.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:19 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:19.673+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:20.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:19 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:19.673+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:20.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:19 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:19.673+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:20.330 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.7.service' 2024-06-10T00:41:20.369 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:41:20.370 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-06-10T00:41:20.370 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-06-10T00:41:20.370 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.8 2024-06-10T00:41:20.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:20.115+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:20.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:20.115+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:20.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:20.115+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:20.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:20.115+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:20.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:20 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:20.115+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:20.582 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:20 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:20.527+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:20.582 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:20 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:20.527+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:20.582 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:20 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:20.527+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:20.582 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:20 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:20.527+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:20.690 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:20 smithi193 systemd[1]: Stopping Ceph osd.8 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:41:20.690 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:20 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:20.639+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:20.690 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:20 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:20.639+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:20.691 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:20 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:20.639+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:20.691 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:20 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:20.639+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:21.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:20 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:20.689+0000 7fa51a4ab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:41:21.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:20 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:20.689+0000 7fa51a4ab700 -1 osd.8 116 *** Got signal Terminated *** 2024-06-10T00:41:21.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:20 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:20.689+0000 7fa51a4ab700 -1 osd.8 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:41:21.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:20 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:20.762+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:21.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:20 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:20.762+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:21.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:20 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:20.762+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:21.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:20 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:20.762+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:53.433618+0000 front 2024-06-10T00:40:53.433752+0000 (oldest deadline 2024-06-10T00:41:18.133532+0000) 2024-06-10T00:41:21.199 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:21 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:21.090+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:21.199 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:21 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:21.090+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:21.200 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:21 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:21.090+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:21.200 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:21 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:21.090+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:21.200 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:21 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:21.090+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:21.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:21 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:21.549+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:21.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:21 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:21.549+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:21.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:21 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:21.549+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:21.758 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:21 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:21.549+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:21.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:21 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:21.713+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:21.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:21 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:21.713+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:21.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:21 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:21.713+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:21.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:21 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:21.713+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:53.433618+0000 front 2024-06-10T00:40:53.433752+0000 (oldest deadline 2024-06-10T00:41:18.133532+0000) 2024-06-10T00:41:21.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:21 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:21.602+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:21.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:21 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:21.602+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:21.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:21 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:21.602+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:21.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:21 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:21.602+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:22.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:22 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:22.110+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:22.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:22 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:22.110+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:22.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:22 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:22.110+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:22.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:22 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:22.110+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:22.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:22 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:22.110+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:22.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:22 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:22.611+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:22.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:22 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:22.611+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:22.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:22 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:22.611+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:22.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:22 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:22.611+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:22.882 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:22 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:22.678+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:22.882 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:22 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:22.678+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:22.882 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:22 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:22.678+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:22.882 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:22 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:22.678+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:53.433618+0000 front 2024-06-10T00:40:53.433752+0000 (oldest deadline 2024-06-10T00:41:18.133532+0000) 2024-06-10T00:41:22.882 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:22 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:22.678+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.134003+0000 front 2024-06-10T00:40:58.133927+0000 (oldest deadline 2024-06-10T00:41:22.234065+0000) 2024-06-10T00:41:23.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:22 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:22.597+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:23.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:22 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:22.597+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:23.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:22 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:22.597+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:23.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:22 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:22.597+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:23.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:22 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:22.597+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:23.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:23 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:23.062+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:23.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:23 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:23.062+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:23.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:23 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:23.062+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:23.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:23 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:23.062+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:23.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:23 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:23.063+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:23.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:23 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:23.063+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:56.465481+0000 front 2024-06-10T00:40:56.465531+0000 (oldest deadline 2024-06-10T00:41:22.365449+0000) 2024-06-10T00:41:23.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:23 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:23.725+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:23.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:23 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:23.725+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:23.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:23 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:23.725+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:23.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:23 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:23.725+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:53.433618+0000 front 2024-06-10T00:40:53.433752+0000 (oldest deadline 2024-06-10T00:41:18.133532+0000) 2024-06-10T00:41:23.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:23 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:23.725+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.134003+0000 front 2024-06-10T00:40:58.133927+0000 (oldest deadline 2024-06-10T00:41:22.234065+0000) 2024-06-10T00:41:23.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:23 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:23.589+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:23.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:23 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:23.589+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:23.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:23 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:23.589+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:23.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:23 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:23.589+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:24.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:23 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:23.646+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:24.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:23 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:23.646+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:24.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:23 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:23.646+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:24.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:23 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:23.646+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:24.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:23 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:23.646+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:24.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:24 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:24.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:24.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:24 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:24.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:24.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:24 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:24.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:24.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:24 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:24.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:24.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:24 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:24.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:24.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:24 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:24.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:56.465481+0000 front 2024-06-10T00:40:56.465531+0000 (oldest deadline 2024-06-10T00:41:22.365449+0000) 2024-06-10T00:41:25.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:24 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:24.685+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:25.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:24 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:24.685+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:25.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:24 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:24.685+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:25.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:24 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:24.685+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:25.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:24 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:24.685+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:25.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:24 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:24.631+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:25.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:24 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:24.631+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:25.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:24 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:24.631+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:25.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:24 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:24.631+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:25.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:24 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:24.758+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:27.829175+0000 front 2024-06-10T00:40:27.829373+0000 (oldest deadline 2024-06-10T00:40:51.929293+0000) 2024-06-10T00:41:25.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:24 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:24.758+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.330811+0000 front 2024-06-10T00:40:35.330674+0000 (oldest deadline 2024-06-10T00:41:00.630985+0000) 2024-06-10T00:41:25.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:24 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:24.758+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:40.631439+0000 front 2024-06-10T00:40:40.631462+0000 (oldest deadline 2024-06-10T00:41:05.931505+0000) 2024-06-10T00:41:25.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:24 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:24.758+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:53.433618+0000 front 2024-06-10T00:40:53.433752+0000 (oldest deadline 2024-06-10T00:41:18.133532+0000) 2024-06-10T00:41:25.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:24 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8[55028]: 2024-06-10T00:41:24.758+0000 7fa516a9f700 -1 osd.8 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.134003+0000 front 2024-06-10T00:40:58.133927+0000 (oldest deadline 2024-06-10T00:41:22.234065+0000) 2024-06-10T00:41:25.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:25 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:25.094+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:25.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:25 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:25.095+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:25.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:25 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:25.095+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:25.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:25 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:25.095+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:25.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:25 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:25.095+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:25.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:25 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:25.095+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:56.465481+0000 front 2024-06-10T00:40:56.465531+0000 (oldest deadline 2024-06-10T00:41:22.365449+0000) 2024-06-10T00:41:26.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:25 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:25.648+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:26.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:25 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:25.648+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:26.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:25 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:25.648+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:26.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:25 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:25.648+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:26.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:25 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:25.648+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:26.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:25 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:25.668+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:26.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:25 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:25.668+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:26.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:25 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:25.668+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:26.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:25 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:25.668+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:26.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:25 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:25.668+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:26.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:26 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:26.085+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:26.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:26 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:26.085+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:26.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:26 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:26.085+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:26.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:26 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:26.085+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:26.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:26 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:26.085+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:26.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:26 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:26.085+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:56.465481+0000 front 2024-06-10T00:40:56.465531+0000 (oldest deadline 2024-06-10T00:41:22.365449+0000) 2024-06-10T00:41:26.532 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:26 smithi193 podman[74813]: 2024-06-10 00:41:26.271384194 +0000 UTC m=+5.710941078 container died c197cbbedb7124fd884134a3189cc971687834e2e3c1dc7b5dde32e7840f1cc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:41:26.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:26 smithi193 podman[74813]: 2024-06-10 00:41:26.640070838 +0000 UTC m=+6.079627714 container cleanup c197cbbedb7124fd884134a3189cc971687834e2e3c1dc7b5dde32e7840f1cc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-10T00:41:26.881 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:26 smithi193 bash[74813]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8 2024-06-10T00:41:26.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:26 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:26.688+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:26.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:26 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:26.688+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:26.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:26 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:26.688+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:26.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:26 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:26.688+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:26.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:26 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:26.688+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:27.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:26 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:26.631+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:27.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:26 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:26.631+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:27.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:26 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:26.631+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:27.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:26 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:26.631+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:27.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:26 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:26.631+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:27.256 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:26 smithi193 podman[74828]: 2024-06-10 00:41:26.933653849 +0000 UTC m=+0.660882648 container remove c197cbbedb7124fd884134a3189cc971687834e2e3c1dc7b5dde32e7840f1cc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS) 2024-06-10T00:41:27.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:27 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:27.122+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:27.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:27 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:27.122+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:27.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:27 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:27.122+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:27.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:27 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:27.122+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:27.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:27 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:27.122+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:27.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:27 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:27.122+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:56.465481+0000 front 2024-06-10T00:40:56.465531+0000 (oldest deadline 2024-06-10T00:41:22.365449+0000) 2024-06-10T00:41:27.545 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:27 smithi193 podman[74915]: 2024-06-10 00:41:27.159858497 +0000 UTC m=+0.017116143 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:41:27.545 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:27 smithi193 podman[74915]: 2024-06-10 00:41:27.276062913 +0000 UTC m=+0.133320549 container create 0794df637ec0063b44a60f3b85fb380fc911b4be32cbae2c5b9b6f8790a1b74e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:41:27.848 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:27 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:27.673+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:27.849 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:27 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:27.673+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:27.849 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:27 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:27.673+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:27.849 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:27 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:27.673+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:27.849 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:27 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:27.673+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:27.849 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:27 smithi193 podman[74915]: 2024-06-10 00:41:27.545741798 +0000 UTC m=+0.402999441 container init 0794df637ec0063b44a60f3b85fb380fc911b4be32cbae2c5b9b6f8790a1b74e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-06-10T00:41:27.849 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:27 smithi193 podman[74915]: 2024-06-10 00:41:27.549378151 +0000 UTC m=+0.406635790 container start 0794df637ec0063b44a60f3b85fb380fc911b4be32cbae2c5b9b6f8790a1b74e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:41:27.849 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:27 smithi193 podman[74915]: 2024-06-10 00:41:27.645497925 +0000 UTC m=+0.502755577 container attach 0794df637ec0063b44a60f3b85fb380fc911b4be32cbae2c5b9b6f8790a1b74e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:41:28.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:27 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:27.662+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:28.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:27 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:27.662+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:28.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:27 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:27.662+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:28.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:27 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:27.662+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:28.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:27 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:27.662+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:28.131 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:27 smithi193 podman[74915]: 2024-06-10 00:41:27.849129888 +0000 UTC m=+0.706387525 container died 0794df637ec0063b44a60f3b85fb380fc911b4be32cbae2c5b9b6f8790a1b74e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:41:28.434 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jun 10 00:41:28 smithi193 podman[74942]: 2024-06-10 00:41:28.41600604 +0000 UTC m=+0.560921013 container remove 0794df637ec0063b44a60f3b85fb380fc911b4be32cbae2c5b9b6f8790a1b74e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0) 2024-06-10T00:41:28.441 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.8.service' 2024-06-10T00:41:28.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:28 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:28.089+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:28.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:28 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:28.089+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:28.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:28 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:28.089+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:28.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:28 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:28.089+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:28.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:28 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:28.089+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:28.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:28 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:28.089+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:56.465481+0000 front 2024-06-10T00:40:56.465531+0000 (oldest deadline 2024-06-10T00:41:22.365449+0000) 2024-06-10T00:41:28.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:41:28.480 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-06-10T00:41:28.480 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-06-10T00:41:28.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.9 2024-06-10T00:41:28.692 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:28 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:28.675+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:28.692 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:28 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:28.675+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:28.692 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:28 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:28.675+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:28.692 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:28 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:28.675+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:28.692 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:28 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:28.675+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:28.790 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:28 smithi121 systemd[1]: Stopping Ceph osd.9 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:41:29.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:28 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:28.703+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:29.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:28 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:28.703+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:29.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:28 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:28.703+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:29.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:28 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:28.703+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:29.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:28 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:28.703+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:29.069 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:28 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:28.789+0000 7f7747381700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:41:29.070 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:28 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:28.789+0000 7f7747381700 -1 osd.9 116 *** Got signal Terminated *** 2024-06-10T00:41:29.070 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:28 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:28.789+0000 7f7747381700 -1 osd.9 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:41:29.070 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:29 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:29.070+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:29.070 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:29 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:29.070+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:29.070 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:29 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:29.070+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:29.070 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:29 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:29.070+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:29.070 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:29 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:29.070+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:29.070 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:29 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:29.070+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:56.465481+0000 front 2024-06-10T00:40:56.465531+0000 (oldest deadline 2024-06-10T00:41:22.365449+0000) 2024-06-10T00:41:29.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:29 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:29.683+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:29.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:29 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:29.683+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:29.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:29 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:29.683+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:29.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:29 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:29.683+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:29.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:29 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:29.683+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:30.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:29 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:29.688+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:30.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:29 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:29.688+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:30.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:29 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:29.688+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:30.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:29 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:29.688+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:30.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:29 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:29.688+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:30.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:30 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:30.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:30.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:30 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:30.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:30.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:30 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:30.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:30.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:30 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:30.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:30.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:30 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:30.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:30.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:30 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:30.088+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:56.465481+0000 front 2024-06-10T00:40:56.465531+0000 (oldest deadline 2024-06-10T00:41:22.365449+0000) 2024-06-10T00:41:31.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:30 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:30.714+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:31.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:30 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:30.714+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:31.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:30 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:30.714+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:31.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:30 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:30.714+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:31.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:30 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:30.714+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:31.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:30 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:30.727+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:31.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:30 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:30.727+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:31.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:30 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:30.727+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:31.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:30 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:30.727+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:31.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:30 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:30.727+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:31.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:31 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:31.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:31.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:31 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:31.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:31.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:31 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:31.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:31.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:31 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:31.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:31.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:31 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:31.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:31.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:31 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:31.102+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:56.465481+0000 front 2024-06-10T00:40:56.465531+0000 (oldest deadline 2024-06-10T00:41:22.365449+0000) 2024-06-10T00:41:32.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:31 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:31.679+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:32.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:31 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:31.679+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:32.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:31 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:31.679+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:32.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:31 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:31.679+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:32.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:31 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:31.679+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:32.032 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:31 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:31.743+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:32.032 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:31 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:31.743+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:32.033 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:31 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:31.743+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:32.033 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:31 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:31.743+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:32.033 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:31 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:31.743+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:32.449 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:32 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:32.056+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:32.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:32 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:32.056+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:32.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:32 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:32.056+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:32.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:32 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:32.056+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:32.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:32 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:32.056+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:32.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:32 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:32.056+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:56.465481+0000 front 2024-06-10T00:40:56.465531+0000 (oldest deadline 2024-06-10T00:41:22.365449+0000) 2024-06-10T00:41:33.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:32 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:32.645+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:33.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:32 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:32.645+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:33.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:32 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:32.645+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:33.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:32 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:32.645+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:33.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:32 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:32.645+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:33.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:32 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:32.767+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:33.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:32 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:32.767+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:33.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:32 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:32.767+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:33.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:32 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:32.767+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:33.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:32 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:32.767+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:33.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:32 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:32.767+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:33.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:33 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:33.051+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.158816+0000 front 2024-06-10T00:40:19.158857+0000 (oldest deadline 2024-06-10T00:40:44.458978+0000) 2024-06-10T00:41:33.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:33 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:33.051+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:25.560088+0000 front 2024-06-10T00:40:25.560201+0000 (oldest deadline 2024-06-10T00:40:51.460252+0000) 2024-06-10T00:41:33.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:33 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:33.051+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:36.062257+0000 front 2024-06-10T00:40:36.062333+0000 (oldest deadline 2024-06-10T00:40:58.962209+0000) 2024-06-10T00:41:33.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:33 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:33.051+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:43.563720+0000 front 2024-06-10T00:40:43.563740+0000 (oldest deadline 2024-06-10T00:41:08.863988+0000) 2024-06-10T00:41:33.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:33 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:33.051+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:48.864632+0000 front 2024-06-10T00:40:48.864706+0000 (oldest deadline 2024-06-10T00:41:16.465016+0000) 2024-06-10T00:41:33.450 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:33 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9[67311]: 2024-06-10T00:41:33.051+0000 7f7743975700 -1 osd.9 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:56.465481+0000 front 2024-06-10T00:40:56.465531+0000 (oldest deadline 2024-06-10T00:41:22.365449+0000) 2024-06-10T00:41:34.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:33 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:33.608+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:34.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:33 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:33.608+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:34.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:33 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:33.608+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:34.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:33 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:33.608+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:34.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:33 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:33.608+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:34.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:33 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:33.786+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:34.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:33 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:33.786+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:34.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:33 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:33.786+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:34.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:33 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:33.786+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:34.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:33 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:33.786+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:34.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:33 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:33.786+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:34.614 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:34 smithi121 podman[121996]: 2024-06-10 00:41:34.32803953 +0000 UTC m=+5.639483122 container died ecc4e366ea0eff912b03c054386b4fee89e950e04149fb15a25dc1599d722d6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528) 2024-06-10T00:41:34.923 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:34 smithi121 podman[121996]: 2024-06-10 00:41:34.756144158 +0000 UTC m=+6.067587741 container cleanup ecc4e366ea0eff912b03c054386b4fee89e950e04149fb15a25dc1599d722d6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528) 2024-06-10T00:41:34.924 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:34 smithi121 bash[121996]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9 2024-06-10T00:41:35.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:34 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:34.577+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:35.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:34 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:34.577+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:35.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:34 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:34.577+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:35.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:34 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:34.577+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:35.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:34 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:34.577+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:35.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:34 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:34.577+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:09.398716+0000 front 2024-06-10T00:41:09.398732+0000 (oldest deadline 2024-06-10T00:41:34.098646+0000) 2024-06-10T00:41:35.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:34 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:34.792+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:35.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:34 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:34.792+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:35.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:34 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:34.792+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:35.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:34 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:34.792+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:35.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:34 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:34.792+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:35.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:34 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:34.792+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:35.199 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:34 smithi121 podman[122011]: 2024-06-10 00:41:34.924126957 +0000 UTC m=+0.598634949 container remove ecc4e366ea0eff912b03c054386b4fee89e950e04149fb15a25dc1599d722d6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=) 2024-06-10T00:41:35.585 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:35 smithi121 podman[122101]: 2024-06-10 00:41:35.229592172 +0000 UTC m=+0.017378714 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:41:35.585 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:35 smithi121 podman[122101]: 2024-06-10 00:41:35.333180828 +0000 UTC m=+0.120967352 container create a72017450b6344399da388b9e8f50ae7b77f19f25e23a344b2db8b185a4b7a38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:41:35.585 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:35 smithi121 podman[122101]: 2024-06-10 00:41:35.527441946 +0000 UTC m=+0.315228470 container init a72017450b6344399da388b9e8f50ae7b77f19f25e23a344b2db8b185a4b7a38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, ceph=True, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-06-10T00:41:35.585 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:35 smithi121 podman[122101]: 2024-06-10 00:41:35.530742555 +0000 UTC m=+0.318529083 container start a72017450b6344399da388b9e8f50ae7b77f19f25e23a344b2db8b185a4b7a38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:41:35.585 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:35 smithi121 podman[122101]: 2024-06-10 00:41:35.585585873 +0000 UTC m=+0.373372401 container attach a72017450b6344399da388b9e8f50ae7b77f19f25e23a344b2db8b185a4b7a38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9-deactivate, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:41:35.949 INFO:journalctl@ceph.osd.9.smithi121.stdout:Jun 10 00:41:35 smithi121 podman[122101]: 2024-06-10 00:41:35.828083438 +0000 UTC m=+0.615869972 container died a72017450b6344399da388b9e8f50ae7b77f19f25e23a344b2db8b185a4b7a38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-10T00:41:36.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:35 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:35.612+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:36.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:35 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:35.612+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:36.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:35 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:35.612+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:36.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:35 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:35.612+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:36.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:35 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:35.612+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:36.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:35 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:35.612+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:09.398716+0000 front 2024-06-10T00:41:09.398732+0000 (oldest deadline 2024-06-10T00:41:34.098646+0000) 2024-06-10T00:41:36.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:35 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:35.785+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:36.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:35 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:35.785+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:36.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:35 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:35.785+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:36.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:35 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:35.785+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:36.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:35 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:35.785+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:36.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:35 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:35.785+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:36.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.9.service' 2024-06-10T00:41:36.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:41:36.579 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-06-10T00:41:36.579 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-06-10T00:41:36.580 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.10 2024-06-10T00:41:36.885 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:36 smithi135 systemd[1]: Stopping Ceph osd.10 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:41:36.885 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:36 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:36.626+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:36.885 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:36 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:36.627+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:36.885 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:36 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:36.627+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:36.885 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:36 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:36.627+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:36.885 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:36 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:36.627+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:36.886 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:36 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:36.627+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:09.398716+0000 front 2024-06-10T00:41:09.398732+0000 (oldest deadline 2024-06-10T00:41:34.098646+0000) 2024-06-10T00:41:36.886 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:36 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:36.884+0000 7ff5501b0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:41:36.886 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:36 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:36.884+0000 7ff5501b0700 -1 osd.10 116 *** Got signal Terminated *** 2024-06-10T00:41:36.886 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:36 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:36.884+0000 7ff5501b0700 -1 osd.10 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:41:37.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:36 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:36.739+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:37.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:36 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:36.739+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:37.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:36 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:36.739+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:37.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:36 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:36.739+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:37.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:36 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:36.739+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:37.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:36 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:36.739+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:38.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:37 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:37.595+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:38.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:37 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:37.595+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:38.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:37 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:37.595+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:38.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:37 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:37.595+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:38.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:37 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:37.595+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:38.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:37 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:37.595+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:09.398716+0000 front 2024-06-10T00:41:09.398732+0000 (oldest deadline 2024-06-10T00:41:34.098646+0000) 2024-06-10T00:41:38.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:37 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:37.740+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:38.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:37 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:37.740+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:38.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:37 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:37.740+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:38.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:37 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:37.740+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:38.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:37 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:37.740+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:38.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:37 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:37.740+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:39.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:38 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:38.601+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:39.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:38 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:38.601+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:39.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:38 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:38.601+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:39.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:38 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:38.601+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:39.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:38 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:38.601+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:39.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:38 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:38.601+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:09.398716+0000 front 2024-06-10T00:41:09.398732+0000 (oldest deadline 2024-06-10T00:41:34.098646+0000) 2024-06-10T00:41:39.118 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:38 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:38.697+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:39.118 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:38 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:38.697+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:39.118 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:38 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:38.697+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:39.118 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:38 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:38.697+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:39.119 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:38 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:38.697+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:39.119 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:38 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:38.697+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:39.119 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:38 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:38.697+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:39.972 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:39 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:39.568+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:39.972 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:39 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:39.568+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:39.972 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:39 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:39.568+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:39.972 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:39 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:39.568+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:39.972 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:39 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:39.568+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:39.972 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:39 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:39.568+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:09.398716+0000 front 2024-06-10T00:41:09.398732+0000 (oldest deadline 2024-06-10T00:41:34.098646+0000) 2024-06-10T00:41:39.972 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:39 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:39.568+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:14.099153+0000 front 2024-06-10T00:41:14.099184+0000 (oldest deadline 2024-06-10T00:41:38.799242+0000) 2024-06-10T00:41:40.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:39 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:39.722+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:40.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:39 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:39.722+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:40.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:39 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:39.722+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:40.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:39 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:39.722+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:40.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:39 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:39.722+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:40.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:39 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:39.722+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:40.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:39 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:39.722+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:41.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:40 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:40.562+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:41.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:40 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:40.562+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:41.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:40 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:40.562+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:41.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:40 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:40.562+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:41.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:40 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:40.562+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:41.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:40 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:40.562+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:09.398716+0000 front 2024-06-10T00:41:09.398732+0000 (oldest deadline 2024-06-10T00:41:34.098646+0000) 2024-06-10T00:41:41.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:40 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:40.562+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:14.099153+0000 front 2024-06-10T00:41:14.099184+0000 (oldest deadline 2024-06-10T00:41:38.799242+0000) 2024-06-10T00:41:41.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:40 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:40.708+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:41.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:40 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:40.708+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:41.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:40 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:40.708+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:41.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:40 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:40.708+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:41.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:40 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:40.708+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:41.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:40 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:40.708+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:41.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:40 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:40.708+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:42.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:41 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:41.565+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6804 osd.1 since back 2024-06-10T00:40:28.191039+0000 front 2024-06-10T00:40:28.191007+0000 (oldest deadline 2024-06-10T00:40:53.491259+0000) 2024-06-10T00:41:42.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:41 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:41.565+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:35.792419+0000 front 2024-06-10T00:40:35.792431+0000 (oldest deadline 2024-06-10T00:40:59.292331+0000) 2024-06-10T00:41:42.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:41 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:41.565+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:39.292829+0000 front 2024-06-10T00:40:39.292842+0000 (oldest deadline 2024-06-10T00:41:08.093434+0000) 2024-06-10T00:41:42.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:41 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:41.565+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:52.595551+0000 front 2024-06-10T00:40:52.595411+0000 (oldest deadline 2024-06-10T00:41:14.295607+0000) 2024-06-10T00:41:42.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:41 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:41.565+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:58.396567+0000 front 2024-06-10T00:40:58.396476+0000 (oldest deadline 2024-06-10T00:41:22.496545+0000) 2024-06-10T00:41:42.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:41 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:41.565+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:09.398716+0000 front 2024-06-10T00:41:09.398732+0000 (oldest deadline 2024-06-10T00:41:34.098646+0000) 2024-06-10T00:41:42.009 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:41 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10[60819]: 2024-06-10T00:41:41.565+0000 7ff54879c700 -1 osd.10 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:14.099153+0000 front 2024-06-10T00:41:14.099184+0000 (oldest deadline 2024-06-10T00:41:38.799242+0000) 2024-06-10T00:41:42.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:41 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:41.671+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:42.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:41 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:41.671+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:42.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:41 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:41.671+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:42.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:41 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:41.671+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:42.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:41 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:41.671+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:42.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:41 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:41.671+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:42.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:41 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:41.671+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:42.678 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:42 smithi135 podman[73622]: 2024-06-10 00:41:42.367080298 +0000 UTC m=+5.592200165 container died 9a00ffc841715b27fcd04380c60a00aad11963de26c91fd8219295e3ae970fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-10T00:41:43.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:42 smithi135 podman[73622]: 2024-06-10 00:41:42.784858355 +0000 UTC m=+6.009978226 container cleanup 9a00ffc841715b27fcd04380c60a00aad11963de26c91fd8219295e3ae970fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-10T00:41:43.008 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:42 smithi135 bash[73622]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10 2024-06-10T00:41:43.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:42 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:42.651+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:43.131 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:42 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:42.651+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:43.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:42 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:42.651+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:43.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:42 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:42.651+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:43.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:42 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:42.651+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:43.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:42 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:42.651+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:43.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:42 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:42.651+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:43.464 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:43 smithi135 podman[73639]: 2024-06-10 00:41:43.027742686 +0000 UTC m=+0.642895672 container remove 9a00ffc841715b27fcd04380c60a00aad11963de26c91fd8219295e3ae970fd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:41:43.714 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:43 smithi135 podman[73725]: 2024-06-10 00:41:43.366870117 +0000 UTC m=+0.017216619 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:41:43.715 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:43 smithi135 podman[73725]: 2024-06-10 00:41:43.486737662 +0000 UTC m=+0.137084145 container create 007873b365957493f74bf3032aff2467d3ca18d4e6809863755ecaf58ff3c87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0) 2024-06-10T00:41:43.715 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:43 smithi135 podman[73725]: 2024-06-10 00:41:43.711154951 +0000 UTC m=+0.361501433 container init 007873b365957493f74bf3032aff2467d3ca18d4e6809863755ecaf58ff3c87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc) 2024-06-10T00:41:43.715 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:43 smithi135 podman[73725]: 2024-06-10 00:41:43.715009602 +0000 UTC m=+0.365356100 container start 007873b365957493f74bf3032aff2467d3ca18d4e6809863755ecaf58ff3c87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux ) 2024-06-10T00:41:43.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:43.628+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:43.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:43.628+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:43.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:43.628+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:43.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:43.628+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:43.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:43.628+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:43.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:43.628+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:43.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:43 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:43.628+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:44.007 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:43 smithi135 podman[73725]: 2024-06-10 00:41:43.806022948 +0000 UTC m=+0.456369434 container attach 007873b365957493f74bf3032aff2467d3ca18d4e6809863755ecaf58ff3c87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528) 2024-06-10T00:41:44.258 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:44 smithi135 podman[73725]: 2024-06-10 00:41:44.007993054 +0000 UTC m=+0.658339549 container died 007873b365957493f74bf3032aff2467d3ca18d4e6809863755ecaf58ff3c87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:41:44.542 INFO:journalctl@ceph.osd.10.smithi135.stdout:Jun 10 00:41:44 smithi135 podman[73751]: 2024-06-10 00:41:44.525784079 +0000 UTC m=+0.512676100 container remove 007873b365957493f74bf3032aff2467d3ca18d4e6809863755ecaf58ff3c87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-06-10T00:41:44.547 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.10.service' 2024-06-10T00:41:44.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:41:44.586 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-06-10T00:41:44.586 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-06-10T00:41:44.586 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.11 2024-06-10T00:41:44.876 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:44 smithi193 systemd[1]: Stopping Ceph osd.11 for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:41:44.876 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:44 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:44.666+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:44.876 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:44 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:44.666+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:44.876 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:44 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:44.666+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:44.877 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:44 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:44.666+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:44.877 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:44 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:44.666+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:44.877 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:44 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:44.666+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:44.877 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:44 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:44.666+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:44.877 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:44 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:44.875+0000 7fb438785700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:41:44.877 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:44 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:44.875+0000 7fb438785700 -1 osd.11 116 *** Got signal Terminated *** 2024-06-10T00:41:44.877 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:44 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:44.875+0000 7fb438785700 -1 osd.11 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-10T00:41:46.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:45 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:45.641+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:46.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:45 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:45.641+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:46.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:45 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:45.641+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:46.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:45 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:45.641+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:46.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:45 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:45.641+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:46.132 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:45 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:45.641+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:46.133 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:45 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:45.641+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:46.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:46 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:46.623+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:46.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:46 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:46.623+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:46.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:46 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:46.623+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:46.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:46 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:46.623+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:46.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:46 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:46.623+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:46.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:46 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:46.623+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:46.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:46 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:46.623+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:47.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:47 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:47.610+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:47.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:47 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:47.610+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:47.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:47 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:47.610+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:47.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:47 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:47.610+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:47.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:47 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:47.610+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:47.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:47 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:47.610+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:47.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:47 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:47.610+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:47.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:47 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:47.610+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6820 osd.8 since back 2024-06-10T00:41:21.967623+0000 front 2024-06-10T00:41:21.967651+0000 (oldest deadline 2024-06-10T00:41:47.267640+0000) 2024-06-10T00:41:48.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:48 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:48.605+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:48.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:48 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:48.605+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:48.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:48 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:48.605+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:48.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:48 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:48.605+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:48.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:48 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:48.605+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:48.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:48 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:48.605+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:48.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:48 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:48.605+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:48.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:48 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:48.605+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6820 osd.8 since back 2024-06-10T00:41:21.967623+0000 front 2024-06-10T00:41:21.967651+0000 (oldest deadline 2024-06-10T00:41:47.267640+0000) 2024-06-10T00:41:49.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:49.576+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6806 osd.0 since back 2024-06-10T00:40:19.458238+0000 front 2024-06-10T00:40:19.458271+0000 (oldest deadline 2024-06-10T00:40:45.358189+0000) 2024-06-10T00:41:49.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:49.576+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-06-10T00:40:31.759844+0000 front 2024-06-10T00:40:31.759873+0000 (oldest deadline 2024-06-10T00:40:57.660038+0000) 2024-06-10T00:41:49.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:49.576+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6814 osd.3 since back 2024-06-10T00:40:42.162328+0000 front 2024-06-10T00:40:42.162427+0000 (oldest deadline 2024-06-10T00:41:05.662348+0000) 2024-06-10T00:41:49.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:49.576+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6812 osd.4 since back 2024-06-10T00:40:51.563577+0000 front 2024-06-10T00:40:51.563551+0000 (oldest deadline 2024-06-10T00:41:17.463325+0000) 2024-06-10T00:41:49.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:49.576+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-06-10T00:40:57.463916+0000 front 2024-06-10T00:40:57.463733+0000 (oldest deadline 2024-06-10T00:41:25.064427+0000) 2024-06-10T00:41:49.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:49.576+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.121:6822 osd.6 since back 2024-06-10T00:41:06.765754+0000 front 2024-06-10T00:41:06.765610+0000 (oldest deadline 2024-06-10T00:41:32.665518+0000) 2024-06-10T00:41:49.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:49.576+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.135:6820 osd.7 since back 2024-06-10T00:41:12.666288+0000 front 2024-06-10T00:41:12.666343+0000 (oldest deadline 2024-06-10T00:41:38.466515+0000) 2024-06-10T00:41:49.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:49 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11[62043]: 2024-06-10T00:41:49.576+0000 7fb434d79700 -1 osd.11 116 heartbeat_check: no reply from 172.21.15.193:6820 osd.8 since back 2024-06-10T00:41:21.967623+0000 front 2024-06-10T00:41:21.967651+0000 (oldest deadline 2024-06-10T00:41:47.267640+0000) 2024-06-10T00:41:50.536 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:50 smithi193 podman[75034]: 2024-06-10 00:41:50.270998462 +0000 UTC m=+5.489895541 container died ef2a891f69112719d35c8ec39cce5f30fce10b75054cebe182bacb4572d7838b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:41:50.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:50 smithi193 podman[75034]: 2024-06-10 00:41:50.63443151 +0000 UTC m=+5.853328597 container cleanup ef2a891f69112719d35c8ec39cce5f30fce10b75054cebe182bacb4572d7838b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:41:50.882 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:50 smithi193 bash[75034]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11 2024-06-10T00:41:51.185 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:50 smithi193 podman[75050]: 2024-06-10 00:41:50.885376795 +0000 UTC m=+0.612045970 container remove ef2a891f69112719d35c8ec39cce5f30fce10b75054cebe182bacb4572d7838b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-06-10T00:41:51.480 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:51 smithi193 podman[75140]: 2024-06-10 00:41:51.186184806 +0000 UTC m=+0.101185466 container create 58edaa8b0ff555fb160c0dcdffae2e9701116355da07d365ff9f4306f55aee5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11-deactivate, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:41:51.480 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:51 smithi193 podman[75140]: 2024-06-10 00:41:51.101963215 +0000 UTC m=+0.016963894 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9 2024-06-10T00:41:51.481 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:51 smithi193 podman[75140]: 2024-06-10 00:41:51.38898887 +0000 UTC m=+0.303989530 container init 58edaa8b0ff555fb160c0dcdffae2e9701116355da07d365ff9f4306f55aee5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528) 2024-06-10T00:41:51.481 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:51 smithi193 podman[75140]: 2024-06-10 00:41:51.392205671 +0000 UTC m=+0.307206334 container start 58edaa8b0ff555fb160c0dcdffae2e9701116355da07d365ff9f4306f55aee5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:41:51.481 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:51 smithi193 podman[75140]: 2024-06-10 00:41:51.480607511 +0000 UTC m=+0.395608172 container attach 58edaa8b0ff555fb160c0dcdffae2e9701116355da07d365ff9f4306f55aee5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc) 2024-06-10T00:41:51.881 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jun 10 00:41:51 smithi193 podman[75140]: 2024-06-10 00:41:51.682396967 +0000 UTC m=+0.597397628 container died 58edaa8b0ff555fb160c0dcdffae2e9701116355da07d365ff9f4306f55aee5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:41:52.249 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@osd.11.service' 2024-06-10T00:41:52.290 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:41:52.290 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-06-10T00:41:52.290 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-06-10T00:41:52.290 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.a 2024-06-10T00:41:52.590 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:41:52 smithi121 systemd[1]: Stopping Ceph mds.a for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:41:52.949 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:41:52 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-a[74275]: 2024-06-10T00:41:52.588+0000 7fb6d06f9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:41:52.950 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:41:52 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-a[74275]: 2024-06-10T00:41:52.588+0000 7fb6d06f9700 -1 mds.a *** got signal Terminated *** 2024-06-10T00:41:54.767 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:41:54 smithi121 podman[122218]: 2024-06-10 00:41:54.515144115 +0000 UTC m=+2.026740554 container died ab8ef27468e7502dfd515b707a0d165218b5587d540de7df7674fc067626acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-a, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-06-10T00:41:55.132 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:41:54 smithi121 podman[122218]: 2024-06-10 00:41:54.939586719 +0000 UTC m=+2.451183216 container cleanup ab8ef27468e7502dfd515b707a0d165218b5587d540de7df7674fc067626acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.build-date=20240528) 2024-06-10T00:41:55.132 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:41:54 smithi121 bash[122218]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-a 2024-06-10T00:41:55.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.a.service' 2024-06-10T00:41:55.389 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:41:55 smithi121 podman[122232]: 2024-06-10 00:41:55.132586982 +0000 UTC m=+0.609470211 container remove ab8ef27468e7502dfd515b707a0d165218b5587d540de7df7674fc067626acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-a, ceph=True, RELEASE=squid-3d89acc, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-06-10T00:41:55.389 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:41:55 smithi121 systemd[1]: ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.a.service: Deactivated successfully. 2024-06-10T00:41:55.389 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:41:55 smithi121 systemd[1]: Stopped Ceph mds.a for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:41:55.389 INFO:journalctl@ceph.mds.a.smithi121.stdout:Jun 10 00:41:55 smithi121 systemd[1]: ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.a.service: Consumed 1.066s CPU time. 2024-06-10T00:41:55.402 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:41:55.402 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-06-10T00:41:55.402 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-06-10T00:41:55.403 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.d 2024-06-10T00:41:55.699 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:41:55 smithi121 systemd[1]: Stopping Ceph mds.d for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:41:56.199 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:41:55 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-d[73020]: 2024-06-10T00:41:55.736+0000 7f4cc54ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:41:56.200 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:41:55 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-d[73020]: 2024-06-10T00:41:55.736+0000 7f4cc54ed700 -1 mds.d *** got signal Terminated *** 2024-06-10T00:41:57.401 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:41:56 smithi121 podman[122338]: 2024-06-10 00:41:56.986041719 +0000 UTC m=+1.366417101 container died cbb96db319935fb5de29038bb707c8ae8dcd1a8cae9f746e2c8b7ba3e50cae46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2) 2024-06-10T00:41:57.401 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:41:57 smithi121 podman[122338]: 2024-06-10 00:41:57.401720094 +0000 UTC m=+1.782095471 container cleanup cbb96db319935fb5de29038bb707c8ae8dcd1a8cae9f746e2c8b7ba3e50cae46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, ceph=True, GIT_BRANCH=HEAD) 2024-06-10T00:41:57.699 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:41:57 smithi121 bash[122338]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-d 2024-06-10T00:41:57.979 INFO:journalctl@ceph.mds.d.smithi121.stdout:Jun 10 00:41:57 smithi121 podman[122350]: 2024-06-10 00:41:57.719679126 +0000 UTC m=+0.733245238 container remove cbb96db319935fb5de29038bb707c8ae8dcd1a8cae9f746e2c8b7ba3e50cae46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-10T00:41:57.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.d.service' 2024-06-10T00:41:58.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:41:58.016 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-06-10T00:41:58.016 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-06-10T00:41:58.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.g 2024-06-10T00:41:58.339 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:41:58 smithi121 systemd[1]: Stopping Ceph mds.g for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:41:58.699 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:41:58 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-g[73818]: 2024-06-10T00:41:58.338+0000 7f5b87c3f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:41:58.699 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:41:58 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-g[73818]: 2024-06-10T00:41:58.338+0000 7f5b87c3f700 -1 mds.g *** got signal Terminated *** 2024-06-10T00:42:02.449 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:42:02 smithi121 podman[122459]: 2024-06-10 00:42:02.177196241 +0000 UTC m=+3.944365856 container died 6b9283c82a030c1b568a063f5c4ddb16956ca84ff5431a68d0dec3a5423af40a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:42:02.949 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:42:02 smithi121 podman[122459]: 2024-06-10 00:42:02.655651683 +0000 UTC m=+4.422821293 container cleanup 6b9283c82a030c1b568a063f5c4ddb16956ca84ff5431a68d0dec3a5423af40a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-g, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True) 2024-06-10T00:42:02.949 INFO:journalctl@ceph.mds.g.smithi121.stdout:Jun 10 00:42:02 smithi121 bash[122459]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-g 2024-06-10T00:42:03.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.g.service' 2024-06-10T00:42:03.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:42:03.163 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-06-10T00:42:03.164 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-06-10T00:42:03.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.j 2024-06-10T00:42:03.699 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:42:03 smithi121 systemd[1]: Stopping Ceph mds.j for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:42:03.700 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:42:03 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-j[72577]: 2024-06-10T00:42:03.484+0000 7f17c64c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:42:03.700 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:42:03 smithi121 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-j[72577]: 2024-06-10T00:42:03.484+0000 7f17c64c3700 -1 mds.j *** got signal Terminated *** 2024-06-10T00:42:04.942 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:42:04 smithi121 podman[122581]: 2024-06-10 00:42:04.621441829 +0000 UTC m=+1.239620662 container died 4bd66b89868871b209b2e5c6f3dbee7f01c1084bb6948d94128600777975e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-j, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:42:04.942 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:42:04 smithi121 podman[122581]: 2024-06-10 00:42:04.942954544 +0000 UTC m=+1.561133376 container cleanup 4bd66b89868871b209b2e5c6f3dbee7f01c1084bb6948d94128600777975e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-j, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True) 2024-06-10T00:42:05.199 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:42:04 smithi121 bash[122581]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-j 2024-06-10T00:42:05.199 INFO:journalctl@ceph.mds.j.smithi121.stdout:Jun 10 00:42:05 smithi121 podman[122594]: 2024-06-10 00:42:05.093879318 +0000 UTC m=+0.468888719 container remove 4bd66b89868871b209b2e5c6f3dbee7f01c1084bb6948d94128600777975e232 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240528, ceph=True, RELEASE=squid-3d89acc) 2024-06-10T00:42:05.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.j.service' 2024-06-10T00:42:05.358 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:42:05.358 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-06-10T00:42:05.358 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-06-10T00:42:05.358 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.b 2024-06-10T00:42:05.659 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:42:05 smithi135 systemd[1]: Stopping Ceph mds.b for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:42:06.008 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:42:05 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b[66180]: 2024-06-10T00:42:05.659+0000 7f2db6ab0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:42:06.008 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:42:05 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b[66180]: 2024-06-10T00:42:05.659+0000 7f2db6ab0700 -1 mds.b *** got signal Terminated *** 2024-06-10T00:42:07.585 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:42:07 smithi135 podman[73831]: 2024-06-10 00:42:07.314195543 +0000 UTC m=+1.763690360 container died 13bb6b83c0c3a1f53087af06776b31458ef1da2d5c0ad9740e3b4e890536c4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528) 2024-06-10T00:42:07.882 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:42:07 smithi135 podman[73831]: 2024-06-10 00:42:07.693443374 +0000 UTC m=+2.142938181 container cleanup 13bb6b83c0c3a1f53087af06776b31458ef1da2d5c0ad9740e3b4e890536c4a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.build-date=20240528, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-10T00:42:07.882 INFO:journalctl@ceph.mds.b.smithi135.stdout:Jun 10 00:42:07 smithi135 bash[73831]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-b 2024-06-10T00:42:08.073 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.b.service' 2024-06-10T00:42:08.108 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:42:08.108 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-06-10T00:42:08.108 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-06-10T00:42:08.109 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.e 2024-06-10T00:42:08.439 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:42:08 smithi135 systemd[1]: Stopping Ceph mds.e for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:42:08.758 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:42:08 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e[65561]: 2024-06-10T00:42:08.438+0000 7f9b8a45c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:42:08.758 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:42:08 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e[65561]: 2024-06-10T00:42:08.438+0000 7f9b8a45c700 -1 mds.e *** got signal Terminated *** 2024-06-10T00:42:11.581 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:42:11 smithi135 podman[73964]: 2024-06-10 00:42:11.316312755 +0000 UTC m=+2.989446013 container died d4a9ad535a9d0b1a0c343a6cb19dd9232a6130e382e401e2d4c8a097535bd45f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:42:11.956 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:42:11 smithi135 podman[73964]: 2024-06-10 00:42:11.663717514 +0000 UTC m=+3.336850776 container cleanup d4a9ad535a9d0b1a0c343a6cb19dd9232a6130e382e401e2d4c8a097535bd45f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-06-10T00:42:11.956 INFO:journalctl@ceph.mds.e.smithi135.stdout:Jun 10 00:42:11 smithi135 bash[73964]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-e 2024-06-10T00:42:12.095 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.e.service' 2024-06-10T00:42:12.126 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:42:12.127 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-06-10T00:42:12.127 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-06-10T00:42:12.127 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.h 2024-06-10T00:42:12.500 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:42:12 smithi135 systemd[1]: Stopping Ceph mds.h for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:42:12.758 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:42:12 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:42:12.500+0000 7fe96b590700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:42:12.758 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:42:12 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h[65878]: 2024-06-10T00:42:12.500+0000 7fe96b590700 -1 mds.h *** got signal Terminated *** 2024-06-10T00:42:15.660 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:42:15 smithi135 podman[74085]: 2024-06-10 00:42:15.31546635 +0000 UTC m=+2.969621068 container died 6707dedba949b1781422c1782f7f480c30d9b47b48e2439a5eb9f658a17a4c86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-06-10T00:42:15.968 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:42:15 smithi135 podman[74085]: 2024-06-10 00:42:15.767510952 +0000 UTC m=+3.421665670 container cleanup 6707dedba949b1781422c1782f7f480c30d9b47b48e2439a5eb9f658a17a4c86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=) 2024-06-10T00:42:15.968 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:42:15 smithi135 bash[74085]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h 2024-06-10T00:42:16.202 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.h.service' 2024-06-10T00:42:16.222 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:42:15 smithi135 podman[74100]: 2024-06-10 00:42:15.96870062 +0000 UTC m=+0.605890026 container remove 6707dedba949b1781422c1782f7f480c30d9b47b48e2439a5eb9f658a17a4c86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-h, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:42:16.222 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:42:16 smithi135 systemd[1]: ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.h.service: Deactivated successfully. 2024-06-10T00:42:16.222 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:42:16 smithi135 systemd[1]: Stopped Ceph mds.h for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:42:16.222 INFO:journalctl@ceph.mds.h.smithi135.stdout:Jun 10 00:42:16 smithi135 systemd[1]: ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.h.service: Consumed 6min 48.357s CPU time. 2024-06-10T00:42:16.232 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:42:16.233 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-06-10T00:42:16.233 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-06-10T00:42:16.233 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.k 2024-06-10T00:42:16.508 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:42:16 smithi135 systemd[1]: Stopping Ceph mds.k for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:42:17.008 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:42:16 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-k[65248]: 2024-06-10T00:42:16.553+0000 7f19fc11e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:42:17.008 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:42:16 smithi135 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-k[65248]: 2024-06-10T00:42:16.553+0000 7f19fc11e700 -1 mds.k *** got signal Terminated *** 2024-06-10T00:42:20.463 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:42:20 smithi135 podman[74208]: 2024-06-10 00:42:20.209485679 +0000 UTC m=+3.755019288 container died e4647ca801fdac7fc2d6c2be74158c6254d6781ddc004cd2d925350676838a02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-k, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-06-10T00:42:20.758 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:42:20 smithi135 podman[74208]: 2024-06-10 00:42:20.546487662 +0000 UTC m=+4.092021261 container cleanup e4647ca801fdac7fc2d6c2be74158c6254d6781ddc004cd2d925350676838a02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-k, ceph=True, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:42:20.758 INFO:journalctl@ceph.mds.k.smithi135.stdout:Jun 10 00:42:20 smithi135 bash[74208]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-k 2024-06-10T00:42:20.993 DEBUG:teuthology.orchestra.run.smithi135:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.k.service' 2024-06-10T00:42:21.023 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:42:21.023 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-06-10T00:42:21.023 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-06-10T00:42:21.023 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.c 2024-06-10T00:42:21.308 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:42:21 smithi193 systemd[1]: Stopping Ceph mds.c for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:42:21.566 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:42:21 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c[66910]: 2024-06-10T00:42:21.307+0000 7fcd18d53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:42:21.566 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:42:21 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c[66910]: 2024-06-10T00:42:21.307+0000 7fcd18d53700 -1 mds.c *** got signal Terminated *** 2024-06-10T00:42:23.585 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:42:23 smithi193 podman[75245]: 2024-06-10 00:42:23.317315365 +0000 UTC m=+2.099527742 container died 94260ed35f301a1e9c4aa79ed600c76667cb1e4b85adbd27a16868e35a9fddf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-06-10T00:42:23.881 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:42:23 smithi193 podman[75245]: 2024-06-10 00:42:23.696148813 +0000 UTC m=+2.478361188 container cleanup 94260ed35f301a1e9c4aa79ed600c76667cb1e4b85adbd27a16868e35a9fddf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-10T00:42:23.881 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jun 10 00:42:23 smithi193 bash[75245]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-c 2024-06-10T00:42:24.102 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.c.service' 2024-06-10T00:42:24.136 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:42:24.136 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-06-10T00:42:24.137 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-06-10T00:42:24.137 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.f 2024-06-10T00:42:24.381 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:42:24 smithi193 systemd[1]: Stopping Ceph mds.f for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:42:24.881 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:42:24 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f[66607]: 2024-06-10T00:42:24.487+0000 7fddbb338700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:42:24.881 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:42:24 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f[66607]: 2024-06-10T00:42:24.487+0000 7fddbb338700 -1 mds.f *** got signal Terminated *** 2024-06-10T00:42:27.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:42:27 smithi193 podman[75378]: 2024-06-10 00:42:27.327106919 +0000 UTC m=+2.972105378 container died e76d9836ad77514ef787f398c68e405757b2a2c72ba321d742c97927f0c4edca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240528) 2024-06-10T00:42:27.855 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:42:27 smithi193 podman[75378]: 2024-06-10 00:42:27.662958635 +0000 UTC m=+3.307957090 container cleanup e76d9836ad77514ef787f398c68e405757b2a2c72ba321d742c97927f0c4edca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, ceph=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-06-10T00:42:27.855 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:42:27 smithi193 bash[75378]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f 2024-06-10T00:42:28.090 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.f.service' 2024-06-10T00:42:28.110 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:42:27 smithi193 podman[75392]: 2024-06-10 00:42:27.855941852 +0000 UTC m=+0.529035854 container remove e76d9836ad77514ef787f398c68e405757b2a2c72ba321d742c97927f0c4edca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240528, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-10T00:42:28.110 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:42:28 smithi193 systemd[1]: ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.f.service: Deactivated successfully. 2024-06-10T00:42:28.110 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:42:28 smithi193 systemd[1]: Stopped Ceph mds.f for 6c432244-26be-11ef-bc9b-c7b262605968. 2024-06-10T00:42:28.111 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jun 10 00:42:28 smithi193 systemd[1]: ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.f.service: Consumed 21.300s CPU time. 2024-06-10T00:42:28.122 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:42:28.122 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-06-10T00:42:28.122 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-06-10T00:42:28.123 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.i 2024-06-10T00:42:28.381 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:42:28 smithi193 systemd[1]: Stopping Ceph mds.i for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:42:28.881 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:42:28 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i[67214]: 2024-06-10T00:42:28.450+0000 7f4264506700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:42:28.881 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:42:28 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i[67214]: 2024-06-10T00:42:28.450+0000 7f4264506700 -1 mds.i *** got signal Terminated *** 2024-06-10T00:42:31.631 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:42:31 smithi193 podman[75497]: 2024-06-10 00:42:31.318145416 +0000 UTC m=+2.979084449 container died e78cfb13592ac06d1be9ba102ce6b8819c52114f029cb15b6314d664494cbd0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-10T00:42:31.948 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:42:31 smithi193 podman[75497]: 2024-06-10 00:42:31.633462124 +0000 UTC m=+3.294401157 container cleanup e78cfb13592ac06d1be9ba102ce6b8819c52114f029cb15b6314d664494cbd0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-10T00:42:31.949 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:42:31 smithi193 bash[75497]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i 2024-06-10T00:42:31.949 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jun 10 00:42:31 smithi193 podman[75510]: 2024-06-10 00:42:31.843097213 +0000 UTC m=+0.523319941 container remove e78cfb13592ac06d1be9ba102ce6b8819c52114f029cb15b6314d664494cbd0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-06-10T00:42:31.971 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.i.service' 2024-06-10T00:42:32.002 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:42:32.002 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-06-10T00:42:32.002 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-06-10T00:42:32.003 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.l 2024-06-10T00:42:32.329 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:42:32 smithi193 systemd[1]: Stopping Ceph mds.l for 6c432244-26be-11ef-bc9b-c7b262605968... 2024-06-10T00:42:32.631 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:42:32 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-l[66293]: 2024-06-10T00:42:32.328+0000 7f478f0a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-10T00:42:32.631 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:42:32 smithi193 ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-l[66293]: 2024-06-10T00:42:32.328+0000 7f478f0a2700 -1 mds.l *** got signal Terminated *** 2024-06-10T00:42:33.678 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:42:33 smithi193 podman[75618]: 2024-06-10 00:42:33.423721867 +0000 UTC m=+1.198427021 container died 3fe4e86aa09ffa3484cdf78deb1cdd4274de00adfb28e1b4671a19f893a87578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-06-10T00:42:33.955 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:42:33 smithi193 podman[75618]: 2024-06-10 00:42:33.754401132 +0000 UTC m=+1.529106203 container cleanup 3fe4e86aa09ffa3484cdf78deb1cdd4274de00adfb28e1b4671a19f893a87578 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9, name=ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-l, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-10T00:42:33.955 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jun 10 00:42:33 smithi193 bash[75618]: ceph-6c432244-26be-11ef-bc9b-c7b262605968-mds-l 2024-06-10T00:42:34.163 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6c432244-26be-11ef-bc9b-c7b262605968@mds.l.service' 2024-06-10T00:42:34.193 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-10T00:42:34.193 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-06-10T00:42:34.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6c432244-26be-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-10T00:42:34.385 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:42:36.794 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6c432244-26be-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-10T00:42:36.984 INFO:teuthology.orchestra.run.smithi135.stdout:Deleting cluster with fsid: 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:42:39.514 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6c432244-26be-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-10T00:42:39.703 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:42:42.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-10T00:42:42.101 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-10T00:42:42.133 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-10T00:42:42.159 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-10T00:42:42.160 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320/remote/smithi121/crash 2024-06-10T00:42:42.161 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/crash -- . 2024-06-10T00:42:42.188 INFO:teuthology.orchestra.run.smithi121.stderr:tar: /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-10T00:42:42.188 INFO:teuthology.orchestra.run.smithi121.stderr:tar: Error is not recoverable: exiting now 2024-06-10T00:42:42.190 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320/remote/smithi135/crash 2024-06-10T00:42:42.191 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/crash -- . 2024-06-10T00:42:42.228 INFO:teuthology.orchestra.run.smithi135.stderr:tar: /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-10T00:42:42.228 INFO:teuthology.orchestra.run.smithi135.stderr:tar: Error is not recoverable: exiting now 2024-06-10T00:42:42.230 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320/remote/smithi193/crash 2024-06-10T00:42:42.231 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/crash -- . 2024-06-10T00:42:42.280 INFO:teuthology.orchestra.run.smithi193.stderr:tar: /var/lib/ceph/6c432244-26be-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-10T00:42:42.280 INFO:teuthology.orchestra.run.smithi193.stderr:tar: Error is not recoverable: exiting now 2024-06-10T00:42:42.282 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-10T00:42:42.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-06-10T00:42:42.326 INFO:teuthology.orchestra.run.smithi121.stdout:2024-06-10T00:35:57.433800+0000 mds.h (mds.2) 29 : cluster [WRN] Scrub error on inode 0x10000001d4a (/volumes/qa/sv_1/3fc9b7f7-a431-4fce-8235-be0cc96de6ca/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-42) see mds.h log and `damage ls` output for details 2024-06-10T00:42:42.327 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-10T00:42:42.327 INFO:tasks.cephadm:Compressing logs... 2024-06-10T00:42:42.327 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-10T00:42:42.370 DEBUG:teuthology.orchestra.run.smithi135:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-10T00:42:42.373 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-10T00:42:42.394 INFO:teuthology.orchestra.run.smithi121.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.log 2024-06-10T00:42:42.394 INFO:teuthology.orchestra.run.smithi121.stderr:: No such file or directory 2024-06-10T00:42:42.395 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-10T00:42:42.395 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.g.log 2024-06-10T00:42:42.397 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.j.log 2024-06-10T00:42:42.397 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-06-10T00:42:42.398 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-06-10T00:42:42.398 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mgr.x.log 2024-06-10T00:42:42.398 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mon.a.log 2024-06-10T00:42:42.398 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.6.log 2024-06-10T00:42:42.402 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mon.a.log: 90.3% 88.3% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-10T00:42:42.403 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-10T00:42:42.403 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-10T00:42:42.403 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.a.log 2024-06-10T00:42:42.403 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-10T00:42:42.403 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-10T00:42:42.403 INFO:teuthology.orchestra.run.smithi121.stderr: 94.6% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.g.log.gz 2024-06-10T00:42:42.403 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.j.log.gz 2024-06-10T00:42:42.403 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.d.log 2024-06-10T00:42:42.404 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.9.log 2024-06-10T00:42:42.404 INFO:teuthology.orchestra.run.smithi193.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-10T00:42:42.405 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.log 2024-06-10T00:42:42.405 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-06-10T00:42:42.410 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.f.log 2024-06-10T00:42:42.410 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.log: /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-10T00:42:42.410 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-06-10T00:42:42.410 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.c.log 2024-06-10T00:42:42.410 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mon.c.log 2024-06-10T00:42:42.410 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mgr.z.log 2024-06-10T00:42:42.411 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.i.log 2024-06-10T00:42:42.411 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-10T00:42:42.411 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.11.log 2024-06-10T00:42:42.411 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.i.log: /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.8.log 2024-06-10T00:42:42.411 INFO:teuthology.orchestra.run.smithi193.stderr: 83.3% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-10T00:42:42.411 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-10T00:42:42.411 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.l.log 2024-06-10T00:42:42.411 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.8.log: 90.5% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-10T00:42:42.412 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-10T00:42:42.412 INFO:teuthology.orchestra.run.smithi193.stderr: 88.3% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-10T00:42:42.413 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 94.5% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.l.log.gz 2024-06-10T00:42:42.419 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.a.log.gz 2024-06-10T00:42:42.419 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-10T00:42:42.419 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.9.log: 94.5% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.d.log.gz 2024-06-10T00:42:42.419 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-06-10T00:42:42.419 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-10T00:42:42.422 INFO:teuthology.orchestra.run.smithi121.stderr: 94.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-10T00:42:42.422 INFO:teuthology.orchestra.run.smithi135.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.log 2024-06-10T00:42:42.422 INFO:teuthology.orchestra.run.smithi135.stderr:: No such file or directory 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.7.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.b.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mgr.y.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.h.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.10.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.e.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mon.b.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.k.log 2024-06-10T00:42:42.423 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-10T00:42:42.424 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-06-10T00:42:42.486 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 90.3% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-10T00:42:42.507 INFO:teuthology.orchestra.run.smithi193.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-10T00:42:42.525 INFO:teuthology.orchestra.run.smithi193.stderr: 92.1% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mgr.z.log.gz 2024-06-10T00:42:42.530 INFO:teuthology.orchestra.run.smithi135.stderr: 83.3% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-10T00:42:42.568 INFO:teuthology.orchestra.run.smithi135.stderr: 88.3% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-10T00:42:42.598 INFO:teuthology.orchestra.run.smithi135.stderr: 92.1% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mgr.y.log.gz 2024-06-10T00:42:42.601 INFO:teuthology.orchestra.run.smithi135.stderr: 94.5% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.k.log.gz 2024-06-10T00:42:42.614 INFO:teuthology.orchestra.run.smithi193.stderr: 91.1% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-10T00:42:42.629 INFO:teuthology.orchestra.run.smithi135.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-10T00:42:42.743 INFO:teuthology.orchestra.run.smithi193.stderr: 91.9% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mon.c.log.gz 2024-06-10T00:42:42.797 INFO:teuthology.orchestra.run.smithi121.stderr: 90.2% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mgr.x.log.gz 2024-06-10T00:42:43.004 INFO:teuthology.orchestra.run.smithi135.stderr: 91.0% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-10T00:42:43.765 INFO:teuthology.orchestra.run.smithi121.stderr: 90.4% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mon.a.log.gz 2024-06-10T00:42:50.119 INFO:teuthology.orchestra.run.smithi135.stderr: 91.8% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mon.b.log.gz 2024-06-10T00:43:05.686 INFO:teuthology.orchestra.run.smithi193.stderr: 91.7% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.8.log.gz 2024-06-10T00:43:09.208 INFO:teuthology.orchestra.run.smithi121.stderr: 91.6% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-06-10T00:43:13.218 INFO:teuthology.orchestra.run.smithi121.stderr: 91.5% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-06-10T00:43:15.408 INFO:teuthology.orchestra.run.smithi121.stderr: 91.5% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.6.log.gz 2024-06-10T00:43:19.062 INFO:teuthology.orchestra.run.smithi121.stderr: 91.5% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.9.log.gz 2024-06-10T00:43:19.064 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-06-10T00:43:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m36.680s 2024-06-10T00:43:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:user 2m4.888s 2024-06-10T00:43:19.064 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m4.022s 2024-06-10T00:43:26.637 INFO:teuthology.orchestra.run.smithi193.stderr: 95.2% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.f.log.gz 2024-06-10T00:43:29.217 INFO:teuthology.orchestra.run.smithi193.stderr: 95.2% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.c.log.gz 2024-06-10T00:43:45.891 INFO:teuthology.orchestra.run.smithi193.stderr: 91.5% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-06-10T00:43:54.412 INFO:teuthology.orchestra.run.smithi193.stderr: 91.8% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-06-10T00:43:55.141 INFO:teuthology.orchestra.run.smithi193.stderr: 91.7% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.11.log.gz 2024-06-10T00:44:52.716 INFO:teuthology.orchestra.run.smithi193.stderr: 94.3% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.i.log.gz 2024-06-10T00:44:52.717 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-06-10T00:44:52.717 INFO:teuthology.orchestra.run.smithi193.stderr:real 2m10.330s 2024-06-10T00:44:52.717 INFO:teuthology.orchestra.run.smithi193.stderr:user 3m58.721s 2024-06-10T00:44:52.717 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m8.413s 2024-06-10T00:45:24.405 INFO:teuthology.orchestra.run.smithi135.stderr: 95.1% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.e.log.gz 2024-06-10T00:46:19.245 INFO:teuthology.orchestra.run.smithi135.stderr: 91.7% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.10.log.gz 2024-06-10T00:46:34.824 INFO:teuthology.orchestra.run.smithi135.stderr: 91.6% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-06-10T00:48:17.888 INFO:teuthology.orchestra.run.smithi135.stderr: 91.5% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.7.log.gz 2024-06-10T00:49:49.216 INFO:teuthology.orchestra.run.smithi135.stderr: 93.8% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.b.log.gz 2024-06-10T00:50:27.471 INFO:teuthology.orchestra.run.smithi135.stderr: 91.8% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-06-10T00:52:00.702 INFO:teuthology.orchestra.run.smithi135.stderr: 94.2% -- replaced with /var/log/ceph/6c432244-26be-11ef-bc9b-c7b262605968/ceph-mds.h.log.gz 2024-06-10T00:52:00.704 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-06-10T00:52:00.704 INFO:teuthology.orchestra.run.smithi135.stderr:real 9m18.316s 2024-06-10T00:52:00.704 INFO:teuthology.orchestra.run.smithi135.stderr:user 6m21.791s 2024-06-10T00:52:00.704 INFO:teuthology.orchestra.run.smithi135.stderr:sys 0m16.087s 2024-06-10T00:52:00.705 INFO:tasks.cephadm:Archiving logs... 2024-06-10T00:52:00.706 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320/remote/smithi121/log 2024-06-10T00:52:00.707 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-10T00:52:21.466 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320/remote/smithi135/log 2024-06-10T00:52:21.467 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-10T00:53:17.691 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320/remote/smithi193/log 2024-06-10T00:53:17.692 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-10T00:53:52.887 INFO:tasks.cephadm:Removing cluster... 2024-06-10T00:53:52.888 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6c432244-26be-11ef-bc9b-c7b262605968 --force 2024-06-10T00:53:53.078 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:53:53.544 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6c432244-26be-11ef-bc9b-c7b262605968 --force 2024-06-10T00:53:53.744 INFO:teuthology.orchestra.run.smithi135.stdout:Deleting cluster with fsid: 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:53:54.418 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6c432244-26be-11ef-bc9b-c7b262605968 --force 2024-06-10T00:53:54.613 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: 6c432244-26be-11ef-bc9b-c7b262605968 2024-06-10T00:53:55.160 INFO:tasks.cephadm:Removing cephadm ... 2024-06-10T00:53:55.161 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-10T00:53:55.178 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-10T00:53:55.196 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-10T00:53:55.212 INFO:tasks.cephadm:Teardown complete 2024-06-10T00:53:55.212 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-10T00:53:55.229 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-10T00:53:55.230 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-10T00:53:55.232 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-10T00:53:55.240 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-10T00:53:55.351 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-10T00:53:55.364 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-10T00:53:55.364 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-10T00:53:55.367 DEBUG:teuthology.orchestra.run.smithi135:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-10T00:53:55.369 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-10T00:53:55.379 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-06-10T00:53:55.382 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-06-10T00:53:55.383 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-06-10T00:53:55.384 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-10T00:53:55.384 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-06-10T00:53:55.384 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv01.front.sepia.ceph.com 3 6 377 9 -129us[ -125us] +/- 54ms 2024-06-10T00:53:55.384 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 11 -983us[ -979us] +/- 77ms 2024-06-10T00:53:55.384 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +1150us[+1154us] +/- 74ms 2024-06-10T00:53:55.384 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:53:55.399 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-10T00:53:55.399 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-06-10T00:53:55.399 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv01.front.sepia.ceph.com 3 6 377 11 -136us[ -112us] +/- 54ms 2024-06-10T00:53:55.399 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 15 -1005us[ -981us] +/- 77ms 2024-06-10T00:53:55.399 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +1139us[+1139us] +/- 74ms 2024-06-10T00:53:55.399 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:53:55.411 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-10T00:53:55.411 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-06-10T00:53:55.411 INFO:teuthology.orchestra.run.smithi135.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 -163us[ -165us] +/- 55ms 2024-06-10T00:53:55.411 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 -953us[ -955us] +/- 78ms 2024-06-10T00:53:55.411 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +1114us[+1114us] +/- 74ms 2024-06-10T00:53:55.411 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-10T00:53:55.412 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-10T00:53:55.426 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-10T00:53:55.426 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-10T00:53:55.442 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-10T00:53:55.525 DEBUG:teuthology.orchestra.run.smithi135:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-10T00:53:55.613 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-10T00:53:55.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-10T00:53:55.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:53:55.722 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-10T00:53:55.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:53:55.752 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-10T00:53:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:53:55.779 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-10T00:53:55.790 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-10T00:53:55.804 INFO:teuthology.task.internal:Duration was 2755.878395 seconds 2024-06-10T00:53:55.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-10T00:53:55.928 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-10T00:53:55.928 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-10T00:53:55.931 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-10T00:53:55.934 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-10T00:53:55.969 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-10T00:53:55.995 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-10T00:53:56.011 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-10T00:53:56.427 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-10T00:53:56.427 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-06-10T00:53:56.427 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-10T00:53:56.448 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi135.front.sepia.ceph.com 2024-06-10T00:53:56.448 DEBUG:teuthology.orchestra.run.smithi135:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-10T00:53:56.487 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-06-10T00:53:56.487 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-10T00:53:56.522 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-10T00:53:56.522 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-10T00:53:56.525 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-10T00:53:56.530 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-10T00:53:56.587 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-10T00:53:56.587 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-10T00:53:56.592 DEBUG:teuthology.orchestra.run.smithi135:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-10T00:53:56.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-10T00:53:57.645 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-10T00:53:57.660 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-10T00:53:57.661 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-10T00:53:57.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-10T00:53:57.716 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-10T00:53:57.744 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-10T00:53:57.755 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-10T00:53:57.758 DEBUG:teuthology.orchestra.run.smithi135:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-10T00:53:57.760 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-10T00:53:57.778 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-06-10T00:53:57.792 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = core 2024-06-10T00:53:57.808 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-06-10T00:53:57.835 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-10T00:53:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:53:57.849 DEBUG:teuthology.orchestra.run.smithi135:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-10T00:53:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:53:57.892 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-10T00:53:57.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-10T00:53:57.907 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-10T00:53:57.918 INFO:teuthology.task.internal:Transferring archived files... 2024-06-10T00:53:57.919 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320/remote/smithi121 2024-06-10T00:53:57.919 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-10T00:53:57.980 DEBUG:teuthology.misc:Transferring archived files from smithi135:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320/remote/smithi135 2024-06-10T00:53:57.980 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-10T00:53:58.018 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747320/remote/smithi193 2024-06-10T00:53:58.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-10T00:53:58.053 INFO:teuthology.task.internal:Removing archive directory... 2024-06-10T00:53:58.053 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-10T00:53:58.056 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-10T00:53:58.058 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-10T00:53:58.102 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-10T00:53:58.115 INFO:teuthology.task.internal:Not uploading archives. 2024-06-10T00:53:58.115 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-10T00:53:58.129 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-10T00:53:58.129 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-10T00:53:58.131 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-10T00:53:58.134 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-10T00:53:58.144 INFO:teuthology.orchestra.run.smithi121.stdout: 265316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 10 00:53 /home/ubuntu/cephtest 2024-06-10T00:53:58.148 INFO:teuthology.orchestra.run.smithi135.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 10 00:53 /home/ubuntu/cephtest 2024-06-10T00:53:58.157 INFO:teuthology.orchestra.run.smithi193.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 10 00:53 /home/ubuntu/cephtest 2024-06-10T00:53:58.158 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-10T00:53:58.172 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-10T00:53:58.208 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/blogbench}} duration: 2755.878395318985 failure_reason: 'error during scrub thrashing: rank damage found: {''backtrace''}' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-06-10T00:53:58.209 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-10T00:53:58.296 INFO:teuthology.run:FAIL